Image processing, model training, live room background switching method

Abstract

The embodiment of the application provides an image processing method, a model training method and a live room background switching method. In the image processing method, the input data of the matting model in the feature extraction stage is a target image and a background image thereof, and the features on the target image and the background image can be mined. In the decoding stage, the matting model can decode the feature map output in the feature extraction stage with the assistance of the background image and / or the target image, so that the decoding process can be further focused on the difference between the target image and the background image. In the output stage, the matting model uses a depth guided filtering network to perform depth guided filtering on the decoding result image, and the decoding result can be corrected to make the matting result more accurate. Based on this implementation of guiding the separation of foreground and background according to known background information, a more accurate matting result can be obtained with the assistance of the target image and / or the background image without a large amount of calculation, and the matting efficiency is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to an image processing method, a model training method and a live room background switching method. BACKGROUND

[0002] In many scenarios, image matting processing is required to separate the foreground from the background in an image. In some scenarios, real-time image matting processing is required to meet the background or scene replacement requirements. For example, in a live streaming scenario, it is required to separate the foreground objects such as people and objects from the real-time image of a live streaming video and replace the live streaming environment in real time.

[0003] However, the existing matting algorithm has a large amount of calculation and low matting efficiency, and cannot meet the real-time matting requirements. Therefore, a new solution is needed. SUMMARY

[0004] The present application provides an image processing method, a model training method and a live room background switching method to reduce the calculation amount of matting operation and improve the matting efficiency.

[0005] The present application provides an image processing method, comprising: obtaining a target image to be matting and a background image of the target image; using a feature extraction network in a matting model to perform feature extraction on the target image and the background image to obtain a feature map; using a decoder in the matting model to perform feature decoding on the feature map in combination with the background image and / or the target image to obtain a first decoding result image; and using a depth-guided filtering network in the matting model to perform depth-guided filtering on the first decoding result image under the guidance of the target image to obtain a target matting mask.

[0006] The present application also provides a training method of a matting model, comprising: obtaining a target image to be matting and a background image of the target image; using a feature extraction network in a matting model to perform feature extraction on the target image and the background image to obtain a feature map; using a decoder in the matting model to perform feature decoding on the feature map in combination with the background image and / or the target image to obtain a first decoding result image; using a depth-guided filtering network in the matting model to perform depth-guided filtering on the first decoding result image under the guidance of the target image to obtain a target matting mask; and performing parameter optimization on the matting model according to the difference between the target matting mask and an expected matting mask of the target image.

[0007] The embodiment of the present application further provides a live room background switching method, comprising: acquiring a background image of a live room and acquiring a live video stream generated in real time by the live room; for any frame of live image in the live video stream, using a feature extraction network in a matting model to perform feature extraction on the live image and the background image to obtain a feature map; using a decoder in the matting model to perform feature decoding on the feature map in combination with the background image and / or the live image to obtain a first decoding result image; using a depth guided filtering network in the matting model to perform depth guided filtering on the first decoding result image under the guidance of the live image to obtain a target matting mask; performing matting processing on the live image according to the target matting mask to obtain a foreground image; and fusing the foreground image with a preset image to obtain a new live image.

[0008] The embodiment of the present application further provides an electronic device, comprising: a memory and a processor; the memory is used to store one or more computer instructions; and the processor is used to execute the one or more computer instructions to perform the steps in the method provided by the embodiment of the present application.

[0009] The embodiment of the present application further provides a computer readable storage medium storing a computer program, which can implement the steps in the method provided by the embodiment of the present application when executed by a processor.

[0010] In the image processing method provided by the embodiment of the present application, the input data of the matting model in the feature extraction stage is a target image and a background image thereof, and the features on the target image and the background image can be mined. In the decoding stage, the matting model can decode the feature map output in the feature extraction stage under the assistance of the background image and / or the target image, and can make the decoding process further focus on the difference between the target image and the background image. In the output stage, the matting model uses a depth guided filtering network to perform depth guided filtering on the decoding result image, and can correct the decoding result to make the matting result more accurate. Based on this implementation manner of guiding the separation of the foreground and the background according to the known background information, the target image and / or the background image can be used to obtain a more accurate matting result without a large amount of calculation, and the matting efficiency is greatly improved. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application and illustrate the exemplary embodiments of the present application and their descriptions serve to explain the present application and do not constitute improper limitations on the present application. In the drawings:

[0012] Figure 1 A method flowchart of image processing provided by an exemplary embodiment of the present application is shown in the figure;

[0013] Figure 2 A structural diagram of a matting model provided for an example embodiment of the present application is shown in FIG. 1.

[0014] Figure 3 A structural diagram of a matting model provided for another example embodiment of the present application is shown in FIG. 2.

[0015] Figure 4a A schematic diagram of a live streaming system provided with a matting model for an example embodiment of the present application is shown in FIG. 3.

[0016] Figure 4b A flowchart of a live streaming room background switching method provided for an example embodiment of the present application is shown in FIG. 4.

[0017] Figure 5 A flowchart of a matting model training method provided for an example embodiment of the present application is shown in FIG. 5.

[0018] Figure 6 A structural diagram of an electronic device provided for an example embodiment of the present application is shown in FIG. 6. DETAILED DESCRIPTION

[0019] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below with reference to the embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0020] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms “a”, “an” and “the” used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. “Plural” generally includes at least two, but does not exclude the case of including at least one.

[0021] It should be understood that the term “and / or” used herein is only to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character “ / ” herein generally represents an “or” relationship between the front and rear associated objects.

[0022] It is also to be noted that the terms "comprising", "comprises" or any other variation thereof are intended to cover a non-exclusive inclusion, such that a composition or method that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such composition or method. An element proceeded by "comprises a" does not, without further constraints, exclude the presence of additional identical elements in the composition or method that contains the referred element.

[0023] In many scenarios, image matting is needed to separate foreground from background in an image. In some scenarios, real-time image matting is needed to meet the requirement of background or scene replacement. For example, in a live broadcast scenario, it is needed to separate foreground objects such as people and objects from real-time pictures of a live broadcast video and replace the live broadcast environment in real time. However, existing image matting algorithms have a large amount of calculation and low matting efficiency, and cannot meet the requirement of real-time image matting.

[0024] To solve the above technical problems, in some embodiments of the present application, a solution is provided, which will be described in detail below in combination with the accompanying drawings.

[0025] Figure 1 A flowchart of an image processing method provided by an exemplary embodiment of the present application is shown in FIG. 1, which includes the following steps. Figure 1

[0026] In step 101, a target image to be matting and a background image of the target image are obtained.

[0027] In step 102, a feature extraction network in a matting model is used to perform feature extraction on the target image and the background image, to obtain a feature map.

[0028] In step 103, a decoder in the matting model is used to perform feature decoding on the feature map in combination with the background image and / or the target image, to obtain a first decoding result image.

[0029] In step 104, a depth-guided filtering network in the matting model is used to perform depth-guided filtering on the first decoding result image under the guidance of the target image, to obtain a target matting mask.

[0030] ​The execution subject of the embodiment can be an electronic device running the matting model. In different application scenarios, the electronic device has different implementation forms. For example, in some scenarios, the electronic device can be implemented as a terminal device such as a mobile phone, a tablet computer, or a computer, and the matting model runs on the terminal device. A user can implement offline matting based on the matting model. In other scenarios, the electronic device can be implemented as a server, for example, a conventional server or a cloud server, and the matting model runs on the server and can provide remote matting services to terminal devices.

[0031] The matting model can be obtained by training a neural network model (NN) according to image samples in advance. The specific training process will be described in subsequent embodiments, and will not be described here in detail.

[0032] The target image refers to any image to be matting. The target image can include a foreground part and a background part. The foreground part refers to a subject object in the image, such as a person, a commodity, or a prop. The subject object is located in a target environment, and the subject object and the environment are photographed to obtain the target image. The background image of the target image does not include the subject object. The background image can be obtained by photographing the target environment when the subject object does not enter the target environment. The target image and the background image contain different information, and the matting model can segment the foreground and the background in the target image based on the learned ability to distinguish the foreground and the background.

[0033] In this embodiment, the structure of the matting model is as shown in Figure 2 The matting model includes at least a feature extraction network, a decoder, and a refiner. The refiner can be implemented based on a deep guided filter network (DGF).

[0034] The feature extraction network is configured to extract features from the input image to obtain a matrix or tensor for fully expressing the features of the input image. The feature extraction network can include a plurality of feature calculation layers, which form a deep feature extraction network from low to high, and can perform deep feature extraction on the input image to obtain low-level features and high-level semantic features. The high-level semantic features are abstract feature information obtained after several times of convolution (feature extraction), have a large receptive field, and are conducive to classifying objects on the image. The high-level semantic features are expressed by a feature map with a specific resolution, which is much smaller than the resolution of the target image and the background image, thereby increasing the receptive field of the pixel points on the feature map. The receptive field of the pixel points on the feature map refers to the size of the corresponding image region on the target image and the background image.

[0035] After the feature extraction network in the matting model extracts the feature map, the feature map can be input to a decoder of the matting model. The decoder is configured to map the input feature map to the size of the original image. In this embodiment, the input data of the decoder can also include the target image and / or the background image of the target image. Thus, in the process of decoding the feature map, the decoder can make full use of the existing information on the background image and / or the target image to decode the pixel points on the feature map. The background image contains known background information, and the target image contains known edge information between the background and the foreground. In this embodiment, the above information on the background image and / or the target image can assist and guide the decoding process of the decoder, so that the decoder can achieve more accurate decoding operation without a large amount of calculation, thereby improving the quality of the final matting result. For ease of description, the decoding result output by the decoder is described as a first decoding result image in this embodiment.

[0036] The first decoding result image is the result of decoding the feature map with a low resolution, and the decoding result in the first decoding result image is relatively rough, which provides approximate region positioning, but some pixels have inaccurate decoding results. Therefore, in this embodiment, a refining link can be further used to correct the relatively rough decoding result. The first decoding result image output by the decoder can be input to a deep guided filtering network of the matting model, so as to correct the first decoding result image based on the deep guided filtering network, thereby obtaining a more accurate decoding result.

[0037] The input data of the deep guided filtering network includes the target image. The target image contains complete image information, and the deep guided filtering can perform deep guided filtering on the first decoding result image under the guidance of the target image to obtain a target matting mask.

[0038] In this cutout mask, the pixel values ​​are used to represent the foreground transparency. Typically, the foreground transparency value of a pixel is between 0 and 1. If the foreground transparency of a pixel is 0, then the pixel is determined to belong to the foreground; if the foreground transparency of a pixel is 1, then the pixel is determined to belong to the background.

[0039] In this embodiment, the input data for the matting model in the feature extraction stage is the target image and its background image, allowing for feature extraction from both. In the decoding stage, the matting model, aided by the background image and / or the target image, decodes the feature map output from the feature extraction stage, further focusing the decoding process on the differences between the target and background images. In the output stage, the matting model uses a depth-guided filtering network to perform depth-guided filtering on the decoded result image, correcting the decoding result for greater accuracy. Based on this implementation method that guides foreground and background separation using known background information, relatively accurate matting results can be obtained with the aid of the target image and / or background image without requiring extensive computation, significantly improving matting efficiency.

[0040] Meanwhile, modeling the background image as pre-defined known information reduces the number of parameters in the matting model. With reduced computational load and fewer parameters, the model's demand for computing resources also decreases, allowing it to be used in real-time on a CPU (central processing unit). Before extracting features from the target and background images using the feature extraction network within the matting model, the model can be loaded and run on the CPU.

[0041] In addition, during the refinement process, the above-mentioned matting model uses a deep guided filter network to refine the low-resolution matting results, achieving end-to-end model processing. During the training phase and the prediction phase after training, there is no need to make differentiated adjustments to the structure of the matting model, which is beneficial to the optimization and deployment of the model.

[0042] In some exemplary embodiments, such as Figure 3 As shown, the feature extraction network in the image matting model includes an encoder and a feature computation network.

[0043] Accordingly, when using the feature extraction network in the matting model to extract features from the target image and the background image to obtain feature maps, an encoder can first be used to downsample and encode the target image and the background image to obtain encoded feature maps.

[0044] The encoder is a backbone network of the matting model, and can be implemented based on a Mobilenetv3 (large) model, a VGG model, a ResNet model, or other optional neural network model, which is not limited in the present application.

[0045] Optionally, the encoder comprises a plurality of encoding layers with different resolutions, and the resolutions of the plurality of encoding layers decrease successively. That is, the deeper the encoding layer is, the greater the down-sampling rate is, and the smaller the size of the output encoding feature map is. For any encoding layer in the plurality of encoding layers, the input data of the encoding layer can be down-sampled and encoded according to the preset convolution kernel and the down-sampling rate of the encoding layer, to obtain an encoding feature map with a set resolution.

[0046] As shown in Figure 3 , the input data of the first encoding layer in the plurality of encoding layers is the target image and the background image of the target image, and the input data of the non-first encoding layer is the encoding feature map output by the previous layer. Based on this implementation, the input target image and the background image thereof can be deeply encoded in a multi-layer down-sampling and encoding manner, to obtain an encoding feature map with a lower resolution. The encoding feature map with a lower resolution has a smaller size, which is beneficial to reducing the subsequent calculation amount.

[0047] The encoding feature map output by the encoder can be input into a feature calculation network. The feature calculation network can be implemented based on an improved depthwise-atrous spatial pyramid pooling (DW-ASPP). In the feature calculation network, a depthwise-atrous convolution calculation can be performed on the encoding feature map by using an atrous convolution kernel, to obtain a multi-channel semantic feature; and a point-by-point convolution can be performed on the multi-channel semantic feature, to obtain the feature map.

[0048] The atrous convolution kernel is obtained by injecting holes into a standard convolution kernel. The atrous convolution kernel has a smaller calculation amount and a larger receptive field. The depthwise-atrous convolution is a convolution calculation manner with fewer parameters. In the depthwise-atrous convolution calculation process, each channel can be calculated by using the same convolution kernel, and no addition calculation between channels is performed. In the present embodiment, the matting model performs depthwise-atrous convolution calculation by using the atrous convolution kernel, which can greatly reduce the calculation amount of multiplication and addition in the convolution calculation, improve the calculation efficiency, and improve the real-time performance of the matting processing operation.

[0049] In some exemplary embodiments, as shown in Figure 3 , the decoder in the matting model comprises a plurality of decoding layers with different resolutions, and the resolutions of the plurality of decoding layers increase successively.

[0050] Correspondingly, when decoding the feature map in combination with the background image and / or the target image, any decoding layer in the plurality of decoding layers can perform up-sampling decoding on input data of the decoding layer according to a preset convolution kernel and an up-sampling rate of the decoding layer to obtain a second decoding result map with a set resolution.

[0051] In the plurality of decoding layers, input data of a first decoding layer includes the feature map output by the feature calculation network, and input data of a non-first decoding layer includes a decoding result map output by a previous layer.

[0052] In the plurality of decoding layers, input data of at least one decoding layer includes the background image and / or the target image down-sampled to a corresponding resolution. For example, in the plurality of decoding layers, input data of a part of the decoding layers can include the background image and the target image, and input data of another part of the decoding layers can include the target image. The above will be described below in combination with Figure 3 for an exemplary description.

[0053] As shown in Figure 3 , the decoder includes decoding layers D1, D2, D3, and D4. Input data of the decoding layers D1 and D2 includes the target image and the background image thereof, and input data of the decoding layers D3 and D4 includes the target image. Thus, the decoding layers D1 and D2 can be decoded according to the difference between the target image and the background image thereof, to improve the decoding speed and the accuracy of the decoding result. The decoding layers D3 and D4 can be decoded with the assistance of the target image, to further improve the decoding speed and reduce the decoding calculation amount.

[0054] It should be noted that before inputting into any decoding layer, the target image or the background image can be down-sampled according to the resolution corresponding to the decoding layer, so that the image and the feature map input into the decoding layer have the same resolution for calculation. As shown in Figure 3 , the target image or the background image can be down-sampled based on an average pooling layer (Avg pool) to obtain a target image with different resolutions or a background image with different resolutions.

[0055] Optionally, in the plurality of decoding layers of the decoder, input data of any decoding layer further includes an encoded feature map output by an encoding layer with the same resolution. As shown in Figure 3 , input data of the decoding layer D1 includes an encoded feature map output by the encoding layer E4 with the same resolution, input data of the decoding layer D2 includes an encoded feature map output by the encoding layer E3 with the same resolution, input data of the decoding layer D3 includes an encoded feature map output by the encoding layer E2 with the same resolution, and input data of the decoding layer D4 includes an encoded feature map output by the encoding layer E1 with the same resolution.

[0056] Based on this implementation method, the decoding layer can make full use of the low-level features extracted by the encoding layer during the decoding process. The low-level features have relatively obvious texture and shape features, so they can help the decoding layer achieve more accurate decoding results without a lot of computation.

[0057] In the above implementation, as the resolution of the decoding layer increases, the computational workload of each decoding layer will also increase. To reduce the computational workload, dilated convolution kernels can be used to perform convolution calculations on the input data in higher-level decoding layers, as illustrated below.

[0058] Optionally, for any decoding layer, if the resolution corresponding to that decoding layer is greater than a preset resolution threshold, then based on the preset dilated convolution kernel and upsampling rate of that decoding layer, depthwise separation convolution is performed on the input data of that decoding layer to obtain the second decoding result image with a set resolution corresponding to that decoding layer. The preset resolution threshold can be half, one-third, or one-quarter of the resolution of the target image or the inference image of the input matting model, etc., and this embodiment does not impose any limitations. The inference image can be obtained by downsampling the target image.

[0059] Continue to combine Figure 3 The illustration is used for explanation.

[0060] like Figure 3 As shown, the resolutions of decoding layers D1, D2, D3, and D4 are 1 / 16, 1 / 8, 1 / 4, and 1 / 2 of the inference image, respectively. If the preset resolution threshold is 1 / 4, then the resolution of decoding layer D4 is greater than the preset resolution threshold. Decoding layer D4 can use dilated convolution kernels to perform depthwise separation convolution on the input data, thereby reducing the computational cost required for high-resolution feature maps.

[0061] After obtaining the first decoded result image based on the above embodiments, the depth-guided filtering network in the matting model can be used to perform depth-guided filtering on the first decoded result image under the guidance of the target image to obtain the target matting mask.

[0062] The Deep Guided Filter (DBF) network is primarily used for upsampling based on the guide map. This network has learnable parameters and can learn the ability to upsample input data based on the guide map through continuous training under the supervision of a monitoring signal. DBF performs joint upsampling on the low-resolution output and high-resolution guide map based on the learned parameters, generating a high-resolution output. The image output by the DBF typically fits the edges of the guide map, thus correcting the input data. An example will be provided below.

[0063] In this embodiment, the matting mask is represented by the transparency of the pixel points, and generally, the transparency of the pixel points is a value between 0 and 1. In the pixel points on the second decoding result image output by the decoder, there can be some pixel points whose feature values are not in the range of [0, 1], and therefore, the transparency of the pixel points on the first decoding result image can be normalized according to the feature values of the pixel points on the first decoding result image and the preset transparency operator in the depth-guided filtering network, to obtain a low-resolution matting mask. In the normalization process, for any pixel point, if the feature value of the pixel point is less than 0, the feature value of the pixel point is marked as 0; if the feature value of the pixel point is greater than 1, the feature value of the pixel point is marked as 1.

[0064] After obtaining the low-resolution matting mask based on the above normalization, the low-resolution matting mask and the first decoding result image can be merged in the channel to obtain a channel merging result. The channel merging result can be up-sampled to obtain a target channel merging result with the same resolution as the target image. Based on this step, it can be ensured that the guide image and the image to be filtered have the same resolution. Next, under the guidance of the target image, the target channel merging result can be jointly up-sampled to obtain a high-resolution matting mask as a target matting mask.

[0065] In this implementation, the low-resolution matting result can be corrected by the filtering operation of the DGF, to improve the accuracy of the matting result.

[0066] The above image processing method provided by the embodiments of the present application can be applied in various scenes requiring matting, such as a commodity poster automatic drawing scene, a background replacement scene in live broadcast or video call, a portrait photo editing scene, and the like. In the above scenes, the demand for matting the target image is usually not only portrait matting. For example, in the live broadcast scene, the objects requiring matting include not only the host but also the item arrangement in the live broadcast room, such as a table, an ornament, a stool, an item held by the host, and the like. The above objects requiring matting are “non-background” objects, and simple portrait matting cannot obtain an accurate matting result. Based on the matting model provided by the embodiments, the background image of the target image is taken as preset known information, so that the matting model perceives the difference between the actual background information and the foreground information, thereby accurately segmenting the “non-background” objects in the target image, to meet the accuracy requirement and real-time requirement in the live broadcast scene.

[0067] In the above scenes, the image processing can be implemented based on a system architecture composed of a terminal device and a server, which will be exemplarily described in combination with specific scenes.

[0068] The terminal device refers to a device capable of providing image processing operation to a user and having a communication function. In different application scenarios, the implementation form of the terminal device will also be different. For example, in some scenarios, the terminal device can be a mobile phone, a tablet computer, a computer device, etc. on the user side, and the user can initiate a processing operation on an image or a video through a plug-in, an application program or a browser provided by the terminal device.

[0069] The server can be implemented as a general server, a cloud server, a cloud host, a virtual center, etc. The embodiments of the present application do not limit the server. The server device mainly includes a processor, a hard disk, a memory, a system bus, etc., which is similar to the general computer architecture and will not be described here. The server can run the matting model described in the foregoing embodiments. The terminal device can send an image to be cut out to the server and can provide a background image corresponding to the image or the video. After receiving the image and the background image, the server can input the image and the background image into the matting model, and output a matting result from the matting model.

[0070] In some cases, the terminal device can send video data (such as stream video data generated by a live streaming terminal) and a background image in a video picture to a live streaming server. After receiving the video data, the live streaming server can extract each frame of image in the video and input each frame of image and the background image into the matting model to obtain a matting result of each frame of image.

[0071] In some embodiments, after obtaining the matting result, the server can return the matting result to the terminal device for use by the terminal device. For example, in a portrait beautifying picture scenario, after obtaining the segmentation result of the foreground and the background in the portrait picture, the server can return the segmentation result to the terminal device to enable the terminal device to perform a background replacement operation.

[0072] In some embodiments, after obtaining the matting result, the server can fuse the matting result with a new background provided by the user to obtain a new image, and send the new image to a data receiving party for viewing. For example, as shown in Figure 4a In a live streaming scenario, a live streaming terminal can send a live streaming video stream generated in real time to a live streaming server, and can specify a background image for replacement. After replacing the background image, the live streaming server can send the newly generated live streaming video stream to a live streaming terminal for viewing.

[0073] The following will be further illustrated by taking a live streaming scenario as an example. Figure 4b

[0074] Figure 4b A flowchart of a live streaming room background switching method provided by an exemplary embodiment of the present application is shown in Figure 4b ​As shown, the method comprises:

[0075] Step 401: acquiring a background image of a live broadcast room and acquiring a live broadcast video stream generated in real time in the live broadcast room.

[0076] Step 402: for any live broadcast image in the live broadcast video stream, using a feature extraction network in a matting model to perform feature extraction on the live broadcast image and the background image, to obtain a feature map.

[0077] Step 403: using a decoder in the matting model to perform feature decoding on the feature map in combination with the background image and / or the live broadcast image, to obtain a first decoding result image.

[0078] Step 404: using a depth-guided filtering network in the matting model to perform depth-guided filtering on the first decoding result image under the guidance of the live broadcast image, to obtain a target matting mask.

[0079] Step 405: performing matting processing on the live broadcast image according to the target matting mask, to obtain a foreground image.

[0080] Step 406: fusing the foreground image with a preset image, to obtain a new live broadcast image.

[0081] The execution subject of the embodiment can be a live broadcast server. The live broadcast server can receive a live broadcast video stream sent by a stream pushing end (for example, a mobile phone or a computer of a host) and send the live broadcast video stream to a stream pulling end (for example, a mobile phone or a computer of a live broadcast audience). In the embodiment, the live broadcast server can also acquire a background image of a live broadcast room. The background image can be provided by the host. Alternatively, before the live broadcast starts, the live broadcast server can prompt the host to remove an item in the live broadcast room that needs to be subjected to matting, and acquire the background image of the live broadcast room by capturing the live broadcast environment by the stream pushing end.

[0082] For a live broadcast video stream generated in real time in a live broadcast room, the live broadcast server can perform frame processing on the live broadcast video stream, to obtain each live broadcast image, and perform frame-by-frame matting based on a matting model. For each live broadcast image, the live broadcast server can perform steps 402 to 404 to perform matting processing, to obtain a matting mask of each live broadcast image. The specific implementation of steps 402 to 404 can refer to the description of the foregoing embodiments, which will not be described here.

[0083] In the target matting mask, a transparency value of each pixel point indicates a probability that the pixel point belongs to the foreground. For any pixel point on the live image, if a corresponding transparency of the pixel point in the target matting mask is greater than a set transparency threshold, it can be considered that the pixel point belongs to the foreground region. The transparency threshold can be 0.6, 0.8, 0.9, etc., and the embodiment is not limited. After determining the region to which each pixel point on the live image belongs based on the target matting mask, the foreground image can be segmented from the live image.

[0084] After the live server performs matting on each frame of live image in the live video stream, the segmentation result of the foreground (such as people and objects) and the background in the live scene can be obtained. The foreground can be fused with a preset image to obtain a new live image after replacing the background, and a new live video stream is generated. The preset image can be provided by the host or the live server. In some embodiments, the live server can provide background templates in multiple different scenes for the host to select, such as conference room background templates, scenic area background templates, theme park background templates, studio background templates, etc., and the embodiment is not limited.

[0085] After obtaining the new live video stream, the live server can directly send the new live video stream to the live streaming client for viewing. In this implementation, the live server can efficiently perform matting processing based on the matting model, meeting the high real-time requirement of the live scene.

[0086] In addition to the image processing method described in the foregoing embodiments, the embodiment of the present application also provides a training method of a matting model, which will be exemplarily described below. As shown in Figure 5 The training method of the matting model includes:

[0087] Step 501, obtaining a target image to be cut out and a background image of the target image.

[0088] Step 502, using a feature extraction network in the matting model to perform feature extraction on the target image and the background image to obtain a feature map.

[0089] Step 503, using a decoder in the matting model to combine the background image and / or the target image to perform feature decoding on the feature map to obtain a first decoding result image.

[0090] Step 504, using a depth-guided filtering network in the matting model to perform depth-guided filtering on the first decoding result image under the guidance of the target image to obtain a target matting mask.

[0091] Step 505, performing parameter optimization on the matting model according to a difference between the target matting mask and an expected matting mask of the target image.

[0092] The optional implementation of steps 501-504 can refer to the description of the foregoing embodiments, and will not be described here.

[0093] In step 505, the expected matting mask of the target image is a supervision signal of the training process, which is used to supervise the training process of the matting model. The expected matting mask can be generated in advance by labeling. The parameter optimization operation can be iteratively performed, and the parameters of the matting model are continuously optimized in multiple rounds of iterative training processes, so that the matting error of the matting model continuously converges and finally converges to a preset value or a preset range.

[0094] Optionally, in each training round, a large number of image samples can be used to train the matting model, and the training process of each image sample can be as shown in the foregoing steps 501-505. To improve the robustness of the matting model and enable the matting model to perceive the slight difference between the target image and the background image, the training data can be generated in the following manner:

[0095] The foreground sample set and the background sample set are obtained, and the foreground samples in the foreground sample set are synthesized with the background samples in the background sample set to obtain an image sample set.

[0096] For any foreground sample in the foreground sample set, the operation of synthesizing the foreground sample with any background sample in the background sample set can be implemented based on the following formula:

[0097] I = aF + (1-a)B

[0098] Where I represents the synthesized image sample, a represents the transparency, a ranges from 0 to 1, F represents the foreground sample, and B represents the background sample. Based on the formula, each foreground sample in the foreground sample set can be synthesized with each background sample in the background sample set to obtain a large amount of training data. In each training round, any one image in the image sample set can be selected as a target image, and the target image and the background image of the target image are input into the matting model for training.

[0099] To enhance the adaptability of the model to subtle background differences (such as light, jitter, and noise), the background image input into the matting model together with the target image can not be completely one-to-one corresponding to the background sample of the synthesized target image, that is, the background information on the background image input into the matting model together with the target image can not be completely consistent with the background information on the target image.

[0100] To meet the above requirements, after the target image is synthesized, the background sample corresponding to the target image in the background sample set can be enhanced to obtain an enhanced background sample set of the target image. In each training round, the background image input into the matting model together with the target image can be any image in the enhanced background sample set. The following will be described in conjunction with specific examples.

[0101] Suppose that image A is synthesized by foreground sample a and background sample B. After the background sample B is enhanced, background images B1, B2, B3, and B4 are obtained. When training is performed based on image A, image A and background image B1 can be input into the matting model, or image A and background image B2 can be input into the matting model, or image A and background image B3 can be input into the matting model, or image A and background image B4 can be input into the matting model. Further, the matting model can learn the ability to distinguish foreground and background when the background has a slight change in the continuous iterative training process, thereby improving the robustness of the model.

[0102] Optionally, the enhancement of the background sample includes at least one of affine transformation processing, noise addition processing, brightness adjustment processing, contrast adjustment processing, and foreground local region pasting processing. The foreground local region pasting processing refers to pasting part of the region of the foreground sample on the background sample according to a certain probability to improve the ability of the matting model to distinguish images in which the foreground and the background have similar or consistent colors.

[0103] In some optional embodiments, to improve training efficiency, the training method of the matting model provided in the embodiments of the present application can be executed in stages, for example, can be divided into at least two training stages. The following will be described by way of example.

[0104] In the first training stage, low-resolution image samples are used to train the feature extraction network and the decoder in the matting model.

[0105] Optionally, after the image sample set is obtained based on the method provided in the foregoing embodiments, the image samples in the image sample set and the background images of the image samples can be down-sampled to obtain down-sampled image samples and down-sampled background images. Next, the down-sampled image samples and the down-sampled background images can be used to train the feature extraction network and the decoder in the matting model until the error of the feature extraction network and the decoder converges to a specified range.

[0106] In the training process, after the above reduced resolution image sample and the reduced resolution background image are input into the matting model, the feature extraction network in the matting model can perform feature extraction on the reduced resolution image sample and the reduced resolution background image to obtain a feature map. The decoder in the matting model can combine the reduced resolution background image and / or the reduced resolution image sample to perform feature decoding on the feature map to obtain a decoding result image. The difference between the decoding result image and the expected decoding result image corresponding to the image sample can be obtained, that is, the training loss of the feature extraction network and the decoder. Based on the training loss, the parameters of the feature extraction network and the decoder can be optimized until the training loss meets the convergence condition.

[0107] In this training stage, by reducing the resolution of the input image, the computational amount required for training can be reduced, and the training efficiency can be improved.

[0108] When the error of the feature extraction network and the decoder converges to a specified range, the next training stage can be performed:

[0109] The second training stage is to train the entire matting model.

[0110] In this training stage, each iteration can be performed based on steps 501-505. It should be noted that in this training stage, the main training object is the deep guided filter network, and therefore, when optimizing the parameters of the matting model, the learning rate of the deep guided filter network can be set to be greater than the learning rate of the feature extraction network and the decoder. Based on this implementation, the learning rate of the deep guided filter network can be accelerated, thereby improving the training efficiency.

[0111] The training loss function of the matting model can be expressed by the following formula:

[0112] L = md (||pre- a||1 + ||sobel (pre) - sobel (a)||1)

[0113] Wherein, md represents the weight given by the static matting algorithm (TRIMAP), wherein the weight of the uncertain area calculated by the static matting algorithm can be set to 5, and the weight of the remaining area is 1; pre represents the target matting mask output by the matting model, a represents the expected matting mask of the target image; || ||1 represents the L1 norm function; sobel() represents a function for extracting edge information. Wherein, the sobel() can use a 3*3 sobel convolution kernel for edge extraction. For example, the sobel convolution kernel can be implemented as [[-1,-2,-1],[0,0,0],[1,2,1]], or can be implemented as [[-1,0,1],[-2,0,2],[-1,0,1]], which is not limited in the embodiment.

[0114] It is also worth mentioning that when the matting model as a whole is trained, a training method of interleaving low-resolution training and high-resolution training can also be used to enhance the processing performance of the matting model. In the low-resolution training link, the input image of the matting model can be a low-resolution image sample and a low-resolution background image; in the high-resolution training link, the input image of the matting model can be a high-resolution image sample and a high-resolution background image. Through this way of changing resolution, the model can learn to cope with complex input data, thereby improving the robustness and generalization ability of the model.

[0115] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can also be executed by different devices as the execution subject. For example, the execution subject of steps 501 to 504 can be device A; for another example, the execution subject of steps 501 and 502 can be device A, and the execution subject of step 503 can be device B; and so on.

[0116] In addition, in some of the processes described in the above embodiments and the accompanying drawings, a plurality of operations appearing in a certain order are included, but it should be clearly understood that these operations can be executed or executed in parallel without the order in which they appear in this text. The serial numbers of the operations, such as 501, 502, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and these operations can be executed in sequence or in parallel.

[0117] It should be noted that the "first", "second", and the like in this text are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence. "First" and "second" are not of different types.

[0118] Figure 6 is a structural schematic diagram of an electronic device provided by an exemplary embodiment of the present application, like Figure 6As shown, the electronic device includes a memory 601 and a processor 602.

[0119] The memory 601 is configured to store computer programs and can be configured to store other various data to support operations on the electronic device. Examples of these data include instructions for any application or method operating on the electronic device, contact data, phonebook data, messages, pictures, videos, and the like.

[0120] The memory 601 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic storage, a flash memory, a magnetic disk, or an optical disk.

[0121] The processor 602 is coupled to the memory 601 and is configured to execute the computer programs in the memory 601 for performing the image processing method and / or the training method of the matting model as described in the foregoing embodiments.

[0122] In some embodiments, Figure 6 When the electronic device performs the image processing method, the processor 602 is mainly configured to: obtain a target image to be matting and a background image of the target image; perform feature extraction on the target image and the background image by using a feature extraction network in a matting model to obtain a feature map; perform feature decoding on the feature map by using a decoder in the matting model in combination with the background image and / or the target image to obtain a first decoding result image; and perform depth-guided filtering on the first decoding result image by using a depth-guided filtering network in the matting model under the guidance of the target image to obtain a target matting mask.

[0123] Further optionally, the feature extraction network includes an encoder and a feature calculation network; when the processor 602 performs feature extraction on the target image and the background image by using the feature extraction network in the matting model to obtain the feature map, the processor 602 is specifically configured to: perform down-sampling encoding on the target image and the background image by using the encoder to obtain an encoded feature map; perform depth separation convolution calculation on the encoded feature map by using a hole convolution kernel to obtain multi-channel semantic features by using the feature calculation network; and perform channel integration on the multi-channel semantic features by using point-wise convolution to obtain the feature map.

[0124] Further optionally, the encoder comprises a plurality of encoding layers with different resolutions; when the processor 602 adopts the encoder to perform down-sampling encoding on the target image and the background image to obtain the encoded features, the processor 602 is specifically configured to: for any encoding layer in the plurality of encoding layers, perform down-sampling encoding on input data of the encoding layer according to a preset convolution kernel and a down-sampling rate of the encoding layer to obtain an encoded feature map with a set resolution; resolutions of the plurality of encoding layers decrease in turn; wherein the input data of a first encoding layer in the plurality of encoding layers is the target image and the background image, and the input data of a non-first encoding layer is an encoded feature map output by a previous layer.

[0125] Further optionally, the decoder comprises a plurality of decoding layers with different resolutions; when the processor 602 uses the decoder in the matting model to perform feature decoding on the feature map in combination with the background image and / or the target image to obtain the first decoding result map, the processor 602 is specifically configured to: for any decoding layer in the plurality of decoding layers, perform up-sampling decoding on input data of the decoding layer according to a preset convolution kernel and an up-sampling rate of the decoding layer to obtain a second decoding result map with a set resolution; resolutions of the plurality of decoding layers increase in turn; wherein the input data of a first decoding layer in the plurality of decoding layers comprises the feature map, and the input data of a non-first decoding layer comprises a decoding result map output by a previous layer; the input data of at least one decoding layer in the plurality of decoding layers comprises a background image and / or a target image down-sampled to a corresponding resolution.

[0126] Further optionally, the input data of any decoding layer in the plurality of decoding layers further comprises an encoded feature map output by an encoding layer with the same resolution.

[0127] Further optionally, when the processor 602 performs up-sampling decoding on the input data of the decoding layer according to the preset convolution kernel and the up-sampling rate of the decoding layer to obtain the second decoding result map with the set resolution, the processor 602 is specifically configured to: if a resolution corresponding to the decoding layer is greater than a preset resolution threshold, perform depthwise separable convolution on the input data of the decoding layer according to a preset dilated convolution kernel and the up-sampling rate of the decoding layer to obtain the second decoding result map with the set resolution corresponding to the decoding layer.

[0128] Further optionally, when the processor 602 performs depth guided filtering on the first decoding result image under the guidance of the target image by using a depth guided filtering network in the matting model, the processor 602 is specifically configured to: in the depth guided filtering network, normalize the transparency of the pixel points on the first decoding result image according to the feature values of the pixel points on the first decoding result image and a preset transparency operator, to obtain a low-resolution matting mask; perform channel merging on the low-resolution matting mask and the first decoding result image to obtain a channel merging result; perform up-sampling on the channel merging result to obtain a target channel merging result having the same resolution as the target image; and perform joint up-sampling on the target channel merging result under the guidance of the target image to obtain a high-resolution matting mask as the target matting mask.

[0129] In some embodiments, Figure 6 In the process of executing the image processing method, the processor 602 is mainly configured to: obtain a target image to be matting and a background image of the target image; perform feature extraction on the target image and the background image by using a feature extraction network in a matting model to obtain a feature map; perform feature decoding on the feature map by using a decoder in the matting model in combination with the background image and / or the target image to obtain a first decoding result image; perform depth guided filtering on the first decoding result image under the guidance of the target image by using a depth guided filtering network in the matting model to obtain a target matting mask; and perform parameter optimization on the matting model according to the difference between the target matting mask and an expected matting mask of the target image.

[0130] Further optionally, before obtaining the target image to be matting, the processor 602 is further configured to: obtain a foreground sample set and a background sample set; synthesize the foreground samples in the foreground sample set with the background samples in the background sample set respectively to obtain an image sample set; the target image is any image in the image sample set; and perform enhancement processing on the background sample corresponding to the target image in the background sample set to obtain an enhanced background sample set of the target image, wherein the background image of the target image is any image in the enhanced background sample set.

[0131] Further optionally, the enhancement processing includes at least one of affine transformation processing, noise adding processing, brightness adjustment processing, contrast adjustment processing, and foreground local region fitting processing.

[0132] Further optionally, the processor 602 is further configured to, before performing the parameter optimization on the matting model according to the difference between the target matting mask and the expected matting mask of the target image, perform resolution reduction on the image samples in the image sample set and the background images of the image samples to obtain reduced-resolution image samples and reduced-resolution background images, and train the feature extraction network and the decoder in the matting model using the reduced-resolution image samples and the reduced-resolution background images until the error of the feature extraction network and the decoder converges to a specified range.

[0133] Further optionally, the processor 602 is further configured to, when performing the parameter optimization on the matting model, set the learning rate of the depth-guided filtering network to be greater than the learning rate of the feature extraction network and the decoder.

[0134] Further, as shown in Figure 6 the electronic device further includes a communication component 603, a display 604, a power component 605, an audio component 606, and other components. Figure 6 only some of the components are shown in the electronic device, and the electronic device does not necessarily include Figure 6 only the components shown in the figure.

[0135] The communication component in the above Figure 6 is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, or 5G, or a combination thereof. In an example embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component can be implemented based on near field communication (NFC) technology, radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0136] The display in the above Figure 6 includes a screen, which can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touch or a slide action, but also detect a duration and a pressure related to the touch or slide operation.

[0137] The display in the above Figure 6The power component in the electronic device provides power to various components of the device in which the power component is located. The power component can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which the power component is located.

[0138] In this embodiment, the input data of the matting model running in the electronic device in the feature extraction stage is the target image and its background image, and the features on the target image and the background image can be mined. In the decoding stage, the matting model can decode the feature map output in the feature extraction stage with the assistance of the background image and / or the target image, so that the decoding process can be further focused on the difference between the target image and the background image. In the output stage, the matting model uses a depth-guided filtering network to perform depth-guided filtering on the decoding result image, and the decoding result can be corrected to make the matting result more accurate. Based on this implementation of guiding the separation of foreground and background according to known background information, a more accurate matting result can be obtained with the assistance of the target image and / or the background image without a large amount of calculation, greatly improving the matting efficiency.

[0139] Correspondingly, the embodiment of the present application also provides a computer readable storage medium storing a computer program, which can implement each step that can be executed by the electronic device in the above method embodiment when the computer program is executed.

[0140] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0141] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The functions specified in a flow or multiple flows and / or blocks

[0142] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0143] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the Figure 1 function specified in the flow or flows and / or blocks Figure 1 Figure 1 of the block or blocks.

[0144] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0145] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) about which the computer stores information about an operating system, application software, and / or the like. Memory is an example of computer readable media.

[0146] Computer readable media includes permanent and non-permanent, moveable and non- moveable media that can be implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile discs (DVDs) or other optical storage, magnetic cassettes, magnetic disks storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that is accessible to a computing device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0147] It should also be noted that the terms "comprising," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0148] The above description is merely illustrative of the application, and not restrictive. Various modifications and changes can become apparent to those skilled in the art. Incorporating any modification, equivalent substitution, improvement, etc. within the spirit and principle of the application, shall be included in the scope of the claims of the application.

Claims

1. An image processing method, characterized by, The method comprises the following steps: obtaining a target image to be cut out and a background image of the target image; using a feature extraction network in a matting model to perform feature extraction on the target image and the background image to obtain a feature map; using a decoder in the matting model to perform feature decoding on the feature map in combination with the background image and / or the target image to obtain a first decoding result image; using a depth-guided filtering network in the matting model to perform depth-guided filtering on the first decoding result image under the guidance of the target image to obtain a target matting mask, including: in the depth-guided filtering network, normalizing the transparency of a pixel point on the first decoding result image according to the feature value of the pixel point and a preset transparency operator to obtain a low-resolution matting mask; performing channel merging on the low-resolution matting mask and the first decoding result image to obtain a channel merging result; performing up-sampling on the channel merging result to obtain a target channel merging result having the same resolution as the target image; performing joint up-sampling on the target channel merging result under the guidance of the target image to obtain a high-resolution matting mask as the target matting mask; and the depth-guided filtering network comprises learnable parameters for learning the ability of up-sampling input data based on a guided image under the supervision of a supervision signal.

2. The method of claim 1, wherein, The feature extraction network comprises an encoder and a feature calculation network; using a feature extraction network in a matting model to perform feature extraction on the target image and the background image to obtain a feature map, comprising: using the encoder to perform down-sampling encoding on the target image and the background image to obtain an encoded feature map; using the feature calculation network to perform depth separation convolution calculation on the encoded feature map using a hole convolution kernel to obtain multi-channel semantic features, and using point-by-point convolution to integrate the channels of the multi-channel semantic features to obtain the feature map.

3. The method of claim 2, wherein, The encoder comprises a plurality of encoding layers of different resolutions; using the encoder to perform down-sampling encoding on the target image and the background image to obtain an encoded feature, comprising: for any one of the plurality of encoding layers, performing down-sampling encoding on input data of the encoding layer according to a preset convolution kernel and a down-sampling rate of the encoding layer to obtain an encoded feature map having a set resolution; the resolutions of the plurality of encoding layers decrease in turn; wherein, the input data of the first encoding layer in the plurality of encoding layers is the target image and the background image, and the input data of the non-first encoding layer is the encoded feature map output by the previous layer.

4. The method of claim 3, wherein, The decoder comprises a plurality of decoding layers of different resolutions; using a decoder in the matting model to perform feature decoding on the feature map in combination with the background image and / or the target image to obtain a first decoding result image, comprising: According to the preset convolution kernel and the upsampling rate of the decoding layer, the input data of the decoding layer is upsampled and decoded to obtain a second decoding result map with a set resolution; the resolutions of the plurality of decoding layers are sequentially increased; The input data of the first decoding layer in the plurality of decoding layers includes the feature map, and the input data of the non-first decoding layer includes the decoding result map output by the previous layer; the input data of at least one decoding layer in the plurality of decoding layers includes a background image and / or a target image downsampled to a corresponding resolution.

5. The method of claim 4, wherein, The input data of any decoding layer in the plurality of decoding layers further includes an encoded feature map output by an encoding layer with the same resolution.

6. The method of claim 4, wherein, According to the preset convolution kernel and the upsampling rate of the decoding layer, the input data of the decoding layer is upsampled and decoded to obtain a second decoding result map with a set resolution, including: If the resolution corresponding to the decoding layer is greater than a preset resolution threshold, then according to the preset dilated convolution kernel and the upsampling rate of the decoding layer, the input data of the decoding layer is depthwise separable convolution to obtain the second decoding result map with a set resolution corresponding to the decoding layer.

7. The method according to any one of claims 1 to 6, characterized in that, Before the feature extraction network in the matting model is used to perform feature extraction on the target image and the background image, further comprising: loading and running the matting model by CPU.

8. A method for training a matting model, the method comprising: Including: Obtaining a target image to be cut out and a background image of the target image; Using a feature extraction network in a matting model to perform feature extraction on the target image and the background image to obtain a feature map; Using a decoder in the matting model, combining the background image and / or the target image, to perform feature decoding on the feature map to obtain a first decoding result map; Using a depth-guided filtering network in the matting model, under the guidance of the target image, to perform depth-guided filtering on the first decoding result map to obtain a target matting mask, including: in the depth-guided filtering network, according to the feature value of the pixel point on the first decoding result map and the preset transparency operator, the transparency of the pixel point on the first decoding result map is normalized to obtain a low-resolution matting mask; the low-resolution matting mask and the first decoding result map are merged to obtain a channel merging result; the channel merging result is upsampled to obtain a target channel merging result with the same resolution as the target image; under the guidance of the target image, the target channel merging result is jointly upsampled to obtain a high-resolution matting mask as the target matting mask; the depth-guided filtering network includes learnable parameters for learning the ability of up-sampling input data based on a guided image under the supervision of a supervision signal; According to the difference between the target matting mask and the expected matting mask of the target image, the parameters of the matting model are optimized.

9. The method of claim 8, wherein, Before obtaining the target image to be cut out, further comprising: Obtaining a foreground sample set and a background sample set; Synthesizing foreground samples in the foreground sample set with background samples in the background sample set respectively to obtain an image sample set; the target image is any image in the image sample set; For the target image, performing enhancement processing on the corresponding background sample in the background sample set to obtain an enhanced background sample set of the target image; the background image of the target image is any image in the enhanced background sample set.

10. The method of claim 9, wherein, The enhancement processing includes at least one of affine transformation processing, noise addition processing, brightness adjustment processing, contrast adjustment processing and foreground local region fitting processing.

11. The method of claim 9, wherein, Before the parameter optimization of the matting model according to the difference between the target matting mask and the expected matting mask of the target image, further comprising: Performing resolution reduction processing on the image samples in the image sample set and the background images of the image samples to obtain reduced resolution image samples and reduced resolution background images; Using the reduced resolution image samples and the reduced resolution background images, training the feature extraction network and the decoder in the matting model until the error of the feature extraction network and the decoder converges to a specified range.

12. The method of claim 11, wherein, Further comprising: When the parameter optimization of the matting model is performed, the learning rate of the depth guided filtering network is set to be greater than the learning rate of the feature extraction network and the decoder.

13. A live room background switching method, characterized in that, Including: Obtaining a background image of a live broadcast room and obtaining a live broadcast video stream generated in real time in the live broadcast room; For any frame of live broadcast image in the live broadcast video stream, using a feature extraction network in a matting model to perform feature extraction on the live broadcast image and the background image to obtain a feature map; Using a decoder in the matting model, combining the background image and / or the live broadcast image, performing feature decoding on the feature map to obtain a first decoding result image; Using a depth guided filtering network in the matting model, under the guidance of the live broadcast image, performing depth guided filtering on the first decoding result image to obtain a target matting mask, including: in the depth guided filtering network, according to the feature value of the pixel point on the first decoding result image and the preset transparency operator, performing normalization processing on the transparency of the pixel point on the first decoding result image to obtain a low-resolution matting mask; merging the channels of the low-resolution matting mask and the first decoding result image to obtain a channel merging result; upsampling the channel merging result to obtain a target channel merging result with the same resolution as the live broadcast image; under the guidance of the live broadcast image, jointly upsampling the target channel merging result to obtain a high-resolution matting mask as the target matting mask; the depth guided filtering network includes learnable parameters for learning the ability of upsampling input data based on a guided image under the supervision of a supervision signal; According to the target matting mask, performing matting processing on the live broadcast image to obtain a foreground image; Fusing the foreground image with a preset image to obtain a new live broadcast image.

14. An electronic device, comprising: Including: a memory and a processor; the memory for storing one or more computer instructions; the processor for executing the one or more computer instructions for: performing the steps in the method of any one of claims 1-13.

15. A computer readable storage medium storing a computer program, characterized in that, a computer program for performing the steps in the method of any one of claims 1-13 by a processor.

Images