Image processing method, device, equipment and storage medium

Abstract

The present invention discloses an image processing method, device, equipment and storage medium. The image processing method includes: obtaining a current video stream, the current video stream includes at least one frame of a video frame to be processed containing a target object image; inputting the video frame to be processed into a mask processing model to obtain a foreground mask, the foreground mask corresponds to the target object image, the mask processing model is used to output the foreground feature map and the background feature map of the video frame to be processed respectively through a background decoding subunit and a foreground decoding subunit, and determining the foreground mask according to the foreground feature map and the background feature map; performing background replacement on the video frame to be processed according to the foreground mask to update the video frame to be processed in the current video stream; sending the updated current video stream to a server, so as to be sent from the server to a target user terminal. The image processing method provided by the embodiment of the present disclosure solves the problem of low quality of the foreground portrait determined by the foreground portrait determination method of the prior art.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to computer data processing technology, and more particularly to an image processing method, apparatus, device and storage medium. Background Art

[0002] With the popularity of live streaming, the demand for portrait background replacement is increasing. Portrait background replacement refers to accurately determining the foreground portrait from natural images or videos containing portraits and integrating it with a preset background image. Taking the live streaming of fresh goods as an example, portrait background replacement can be used to integrate the anchor with the video showing the production environment of fresh goods. This can not only increase the diversity and authenticity of the live broadcast, but also allow the audience to see the anchor's portrait, thereby increasing the audience's desire to buy related goods. Therefore, to perform portrait background replacement, it is necessary to first accurately determine the foreground portrait from natural images or videos containing portraits.

[0003] Existing foreground portrait determination methods mainly include semantic segmentation and portrait cutout. The former has the disadvantage of serious edge loss and coarse granularity; the latter retains better edge details of the portrait, but has poor integrity. It can be seen that the quality of the foreground portrait determined by the foreground portrait determination method in the existing technology is low and usually cannot meet people's aesthetic requirements. Summary of the invention

[0004] The embodiments of the present disclosure provide an image processing method, apparatus, device and storage medium to solve the problem of low quality of foreground portraits determined by foreground portrait determination methods in the prior art.

[0005] In a first aspect, an embodiment of the present disclosure provides an image processing method, including:

[0006] Acquire a current video stream, wherein the current video stream includes at least one frame of video frame to be processed that includes an image of a target object;

[0007] Inputting the video frame to be processed into a mask processing model to obtain a foreground mask, wherein the foreground mask corresponds to the target object image, and the mask processing model is used to output a foreground feature map and a background feature map of the video frame to be processed respectively through a background decoding subunit and a foreground decoding subunit, and determine the foreground mask according to the foreground feature map and the background feature map;

[0008] Performing background replacement on the to-be-processed video frame according to the foreground mask to update the to-be-processed video frame in the current video stream;

[0009] The updated current video stream is sent to the server, so that it can be sent to the target user end.

[0010] In a second aspect, the present disclosure also provides an image processing device, including:

[0011] A video acquisition module, used to acquire a current video stream, wherein the current video stream includes at least one frame of video frame to be processed that contains an image of a target object;

[0012] A mask module, used for inputting the video frame to be processed into a mask processing model to obtain a foreground mask, wherein the foreground mask corresponds to the target object image, and the mask processing model is used for outputting a foreground feature map and a background feature map of the video frame to be processed respectively through a background decoding subunit and a foreground decoding subunit, and determining the foreground mask according to the foreground feature map and the background feature map;

[0013] A background replacement module, used for performing background replacement on the to-be-processed video frame according to the foreground mask to update the to-be-processed video frame in the current video stream;

[0014] The output module is used to send the updated current video stream to the server, so as to be sent from the server to the target user.

[0015] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0016] one or more processors;

[0017] A memory for storing one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any embodiment of the present disclosure.

[0019] In a fourth aspect, the embodiments of the present disclosure further provide a computer storage medium on which a computer program is stored, and when the program is executed by a processor, the image processing method as described in any of the embodiments of the present disclosure is implemented.

[0020] The technical solution of the image processing method provided by the embodiment of the present disclosure is that since the mask processing model outputs the foreground feature map and the background feature map of the video frame to be processed respectively through the background decoding subunit and the foreground decoding subunit, and determines the foreground mask of the video frame to be processed based on the foreground feature map and the background feature map, it is possible to reduce the impact of the error of the foreground feature map or the background feature map on the quality of the foreground mask, which helps to improve the quality of the foreground mask. Therefore, the foreground mask of the video frame to be processed determined by the mask processing model has a higher mask quality. Then, by replacing the background of the video frame to be processed in the current video stream based on the foreground mask, the updated current video stream can have a higher quality target object image, thereby satisfying the user's aesthetic needs for the target object image. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0022] Figure 1 A flow chart of an image processing method provided by Embodiment 1 of the present disclosure is shown;

[0023] Figure 2 A schematic diagram of a foreground mask provided in the first embodiment of the present disclosure is shown;

[0024] Figure 3 A schematic diagram of a trained mask processing model provided in Embodiment 1 of the present disclosure is shown;

[0025] Figure 4 A schematic diagram of processing a live video stream provided by Embodiment 1 of the present disclosure is shown;

[0026] Figure 5 A block diagram of an image processing device provided by Embodiment 2 of the present disclosure is shown;

[0027] Figure 6 A flowchart of a mask processing model training method provided in Embodiment 3 of the present disclosure is shown;

[0028] Fig. 7A A schematic diagram of a foreground training template provided in Embodiment 3 of the present disclosure is shown;

[0029] Figure 7B A schematic diagram of an edge extraction template provided in Embodiment 3 of the present disclosure is shown;

[0030] Figure 8 A flowchart of another mask processing model training method provided by the fourth embodiment of the present disclosure is shown;

[0031] Fig. 9 A schematic block diagram of a mask processing model training device provided in Embodiment 5 of the present disclosure is shown;

[0032] Fig.10 A schematic diagram of the structure of an electronic device provided in Embodiment 6 of the present disclosure is shown. DETAILED DESCRIPTION

[0033] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0034] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0035] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0036] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0037] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0038] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0039] Embodiment 1

[0040] Figure 1 The flowchart of the image processing method provided by the first embodiment of the present disclosure is shown. The embodiment of the present disclosure can be applied to the case where the foreground mask of the video frame to be processed in the current live video stream is determined by the foreground mask processing model, and the background replacement of the video frame to be processed is completed according to the foreground mask. The method is usually implemented by the electronic device of the live client, and can be implemented by software and / or hardware configured in the electronic device. The electronic device is a computer, a mobile phone, a handheld computer, and other devices that can be used for image processing.

[0041] like Figure 1As shown, the image processing method provided in the embodiment of the present disclosure includes the following steps:

[0042] S101. Obtain a current video stream, where the current video stream includes at least one to-be-processed video frame containing an image of a target object.

[0043] The video frame to be processed is preferably a live video frame image. In addition, the embodiment of the present disclosure does not limit the format of the video frame to be processed, as long as it is an existing image format, such as RGB format.

[0044] The target object refers to the object whose background needs to be replaced, which may be a portrait, an animal image, an object image, etc. In actual use, the user may make a selection according to the specific situation.

[0045] S102. Input the video frame to be processed into the mask processing model to obtain a foreground mask, which corresponds to the target object image. The mask processing model is used to output the foreground feature map and the background feature map of the video frame to be processed through the background decoding subunit and the foreground decoding subunit respectively, and determine the foreground mask according to the foreground feature map and the background feature map.

[0046] The foreground mask refers to the transparency of the video frame to be processed, usually represented by alpha, which is a matrix composed of floating-point values ​​between 0 and 1 and the same size as the image. It is used to limit the position range of the target object in the video frame to be processed, such as Figure 2 shown.

[0047] Among them, the mask processing model includes an encoding unit, a decoding unit and an output unit; the encoding unit is used to extract encoding features from the video frame to be processed; the decoding unit includes a parallel foreground decoding subunit and a background decoding subunit, the foreground decoding subunit is used to reconstruct the foreground feature map of the video frame to be processed according to the encoding features, and the background decoding subunit is used to reconstruct the background feature map of the video frame to be processed according to the encoding features; the output unit is used to determine the foreground mask according to the foreground feature map and the background feature map.

[0048] Among them, the encoding unit preferably extracts image features based on network structures such as convolutional neural network (CNN), residual network (ResNet), dense convolutional network (DenseNeT) or MobileNet.

[0049] The foreground decoding subunit and the background decoding subunit have the same structure, and their weights are independent and not shared. Moreover, whether it is the foreground decoding subunit or the background decoding subunit, the number of the first reverse feature extraction layers used for semantic segmentation and the number of the second reverse feature extraction layers used for cutout are preferably the same.

[0050] Among them, semantic segmentation is classification at the pixel level. Pixels belonging to the same category are classified into one category. Therefore, semantic segmentation understands the image at the pixel level.

[0051] Among them, cutout is used to predict the transparency of the image, so as to separate the foreground and background of the image. It is a regression problem. Any image can be expressed by the following formula:

[0052] image=fg×alpha+bg×(1-alpha)

[0053] Among them, image is the target image, fg is the foreground of the target image, bg is the background of the target image, and alpha is the foreground mask of the target image.

[0054] In a disclosed embodiment, Figure 3 As shown, the foreground decoding subunit and the background decoding subunit each include two first reverse feature extraction layers for semantic segmentation to extract the contour of the target object from the corresponding feature map, and two second reverse feature extraction layers for cutout connected behind the two first reverse feature extraction layers for semantic segmentation. The second reverse feature extraction layer is used to cutout the feature map carrying the contour information of the target object to obtain the detail information of the target object, so that the feature map output by the second reverse feature extraction layer retains the overall information of the target object image while also containing the edge detail information of the target object.

[0055] For the foreground decoding subunit, Figure 3As shown, the reverse feature extraction layers identified as F8 and F4 are both the first reverse feature extraction layers for semantic segmentation; the reverse feature extraction layers identified as F2 and F are both the second reverse feature extraction layers for cutout; specifically, the first reverse feature extraction layer identified as F8 is used to upsample the received feature map to 1 / 8 of the original image size, and perform semantic segmentation on it to obtain a first segmentation feature map, and then pass through a 1×1 convolution layer to convert the number of channels of the first segmentation feature map to 1, and the format becomes (1, H / 8, W / 8); correspondingly, the first reverse feature extraction layer identified as F4 is used to upsample the received first segmentation feature map to 1 / 4 of the original image size, and perform semantic segmentation on it to obtain a second segmentation feature map, and then pass through A 1×1 convolution layer converts the number of channels of the second segmentation feature map to 1, and the format becomes (1, H / 4, W / 4); the second reverse feature extraction layer marked as F2 is used to upsample the received second segmentation feature map to 1 / 2 of the original image size, and perform portrait cutout to obtain a first cutout feature map, and then pass through a 1×1 convolution layer to convert the number of channels of the first cutout feature map to 1, and the format becomes (1, H / 2, W / 2); the second reverse feature extraction layer marked as F is used to upsample the received first cutout feature map to the original image size, and perform portrait cutout to obtain a foreground feature map, and then pass through a 1×1 convolution layer to convert the number of channels of the foreground feature map to 1, and the format becomes (1, H, W).

[0056] It should be noted that, since the foreground decoding subunit and the background decoding subunit have the same structure, the embodiment of the present disclosure does not describe the composition of the background decoding subunit in detail.

[0057] Among them, the output unit is used to splice the foreground feature map output by the foreground decoding subunit and the background feature map output by the background decoding subunit through the splicing subunit to obtain the foreground weight; then determine the foreground mask according to the foreground weight, the foreground feature map and the background feature map. Compared with determining the foreground mask only based on the foreground feature map or the background feature map, the influence of the error of the foreground feature map or the background feature map on the quality of the foreground mask is reduced, which helps to improve the quality of the foreground mask. In the embodiment of the present disclosure, the formula for determining the foreground mask based on the foreground weight, the foreground feature map and the background feature map is as follows:

[0058] alpha=beta×F+(1-beta)×(1-B)

[0059] Among them, F is the foreground feature map, B is the background feature map, and beta is the foreground weight.

[0060] In some disclosed embodiments, after the splicing subunit completes the splicing of the foreground feature map output by the splicing decoding subunit and the background feature map output by the background decoding subunit, the splicing subunit sequentially inputs the splicing result into at least two convolutional layers, such as three convolutional layers, and the last convolutional layer is a 1×1 convolutional layer, so that the output foreground mask format is (1, H, W).

[0061] In some disclosed embodiments, the mask processing model also includes an atrous spatial pyramid pooling (ASPP) unit disposed between the encoding unit and the decoding unit. The ASPP unit is used to perform parallel sampling with atrous convolutions at different sampling rates according to image features extracted by the encoding unit to capture the context of the video frame to be processed.

[0062] S103: Replace the background of the video frame to be processed according to the foreground mask to update the video frame to be processed in the current video stream.

[0063] After the foreground mask of the video frame to be processed is determined, the background of the video frame to be processed is replaced according to the foreground mask to update the video frame to be processed, thereby updating the current video stream.

[0064] In some disclosed embodiments, a background replacement template selected by a user in a background replacement template library is determined, or a background replacement template uploaded by a user is received, and then the background of the video frame to be processed is replaced according to the background replacement template and the foreground mask to update the video frame to be processed, thereby updating the current video stream.

[0065] S104: Send the updated current video stream to the server, so as to be sent from the server to the target user.

[0066] The updated current video stream is sent to the server, so that the server can send the current video stream to the target user. It can be understood that the video processing object of the server includes not only the current video stream, but also the video streams uploaded by other clients. Therefore, the server will merge the video streams uploaded by each client to generate video merge data (see Figure 4 ), and then sends the corresponding video stream in the video merging data to the corresponding user terminal according to the playback request of the user terminal.

[0067] The technical solution of the image processing method provided by the embodiment of the present disclosure is that since the mask processing model outputs the foreground feature map and the background feature map of the video frame to be processed respectively through the background decoding subunit and the foreground decoding subunit, and determines the foreground mask of the video frame to be processed based on the foreground feature map and the background feature map, the foreground mask of the video frame to be processed is determined by the foreground feature map and the background feature map, which can reduce the impact of the error of the foreground feature map or the background feature map on the quality of the foreground mask, and help to improve the quality of the foreground mask. Therefore, the foreground mask of the video frame to be processed determined by the mask processing model has a higher mask quality. Then, according to the foreground mask, the background of the video frame to be processed in the current video stream is replaced, so that the updated current video stream can have a higher quality target object image, thereby meeting the user's aesthetic needs for the target object image.

[0068] Embodiment 2

[0069] Figure 5 A block diagram of an image processing device provided by an embodiment of the present disclosure is shown. The embodiment of the present disclosure is usually configured in an electronic device and implemented through software or hardware, and can implement the image processing method provided by the above-mentioned disclosed embodiment.

[0070] like Figure 5 As shown, the image processing device provided by the embodiment of the present disclosure includes:

[0071] The video acquisition module 21 is used to acquire a current video stream, wherein the current video stream includes at least one frame of a video frame to be processed that includes an image of a target object;

[0072] A mask module 22, used for inputting the video frame to be processed into a mask processing model to obtain a foreground mask, wherein the foreground mask corresponds to the target object image, and the mask processing model is used for outputting a foreground feature map and a background feature map respectively through a parallel background decoding subunit and a foreground decoding subunit, and determining the foreground mask according to the foreground feature map and the background feature map;

[0073] A background replacement module 23, configured to perform background replacement on the to-be-processed video frame according to the foreground mask to update the to-be-processed video frame in the current video stream;

[0074] The output module 24 is used to send the updated current video stream to the server, so as to be sent from the server to the target user.

[0075] Optionally, the background replacement module is used to receive a background replacement template uploaded by a user or determine a background replacement template selected by a user in a background replacement template library; perform background replacement on the video frame to be processed according to the background replacement template and the foreground mask to update the video frame to be processed in the current video stream.

[0076] Optionally, the mask processing model includes an encoding unit, a decoding unit and an output unit; the encoding unit is used to extract encoding features from the video frame to be processed; the decoding unit includes a parallel foreground decoding subunit and a background decoding subunit, the foreground decoding subunit is used to reconstruct the foreground feature map of the video frame to be processed according to the encoding features, and the background decoding subunit is used to reconstruct the background feature map of the video frame to be processed according to the encoding features; the output unit is used to determine the foreground mask based on the foreground feature map and the background feature map.

[0077] Optionally, the foreground decoding subunit and the background decoding subunit respectively include at least one first reverse feature extraction layer for semantic segmentation and at least one second reverse feature extraction layer for cutout, which are sequentially connected.

[0078] Optionally, in a decoding subunit, the number of first reverse feature extraction layers is the same as the number of second reverse feature extraction layers.

[0079] Optionally, the output unit is used to concatenate the foreground feature map and the background feature map to obtain a foreground weight, and to determine a foreground mask according to the foreground feature map, the background feature map and the foreground weight.

[0080] Optionally, the output unit is specifically configured to determine the foreground mask according to the following formula:

[0081] alpha=beta×F+(1-beta)×(1-B)

[0082] Among them, alpha is the foreground mask, beta is the foreground weight, F is the foreground feature map, and B is the background feature map.

[0083] The technical solution of the image processing device provided by the embodiment of the present disclosure is that since the mask processing model outputs the foreground feature map and the background feature map of the video frame to be processed respectively through the background decoding subunit and the foreground decoding subunit, and determines the foreground mask of the video frame to be processed based on the foreground feature map and the background feature map, it is possible to reduce the impact of the error of the foreground feature map or the background feature map on the quality of the foreground mask, which helps to improve the quality of the foreground mask. Therefore, the foreground mask of the video frame to be processed determined by the mask processing model has a higher mask quality. Then, by replacing the background of the video frame to be processed in the current video stream based on the foreground mask, the updated current video stream can have a higher quality target object image, thereby satisfying the user's aesthetic needs for the target object image.

[0084] The image processing device provided in the embodiment of the present disclosure belongs to the same inventive concept as the image processing method provided in the above-mentioned embodiment of the present disclosure. The technical details not fully described in the embodiment of the present disclosure can be referred to the above-mentioned embodiment of the present disclosure, and the embodiment of the present disclosure has the same beneficial effects as the above-mentioned embodiment of the present disclosure.

[0085] Embodiment 3

[0086] Figure 6 The flowchart of the model training method provided by the embodiment of the present disclosure is shown. The mask processing model described in the aforementioned embodiment is trained based on the model training method described in the embodiment of the present disclosure, and the method is usually implemented by an electronic device, and can be implemented by software and / or hardware configured in the electronic device. Among them, the electronic device in the embodiment of the present disclosure is preferably a computer device.

[0087] like Figure 6 As shown, the mask processing model training method provided in the embodiment of the present disclosure includes the following steps:

[0088] S301, obtaining a training sample set, wherein the training samples in the training sample set include a template image and a to-be-processed video frame including a target object image, and the template image includes a foreground training template corresponding to a foreground mask and a background training template corresponding to the foreground training template.

[0089] In a disclosed embodiment, a method for acquiring a training sample set includes: acquiring a sample set of a foreground training template of a video frame to be processed including a target object image and a corresponding foreground mask; performing inversion processing on the foreground training template to generate a background training template corresponding to the foreground training template; and adding the background training template to the training sample set to update the training sample set. Exemplarily, when a foreground training template corresponding to the foreground mask is acquired, the foreground training template is inverted to generate a background training template corresponding to the foreground training template, the background training template is added to the training sample set, and the non-zero pixel values ​​of the foreground training template and the background training template are both greater than a preset threshold, such as 0.05, to update the training sample set.

[0090] The format of the foreground training template is (1, H, W), which is a foreground mask corresponding to the video frame to be processed, or an image mask corresponding to the video frame to be processed. The pixel value of the image mask is either 0 or 1.

[0091] S302: input the training samples in the training sample set into the mask processing model to be trained for training, and optimize the model parameters based on the set loss function to generate the mask processing model.

[0092] Among them, the mask processing model includes an encoding unit, a decoding unit and an output unit; the encoding unit is used to extract encoding features from the video frame to be processed; the decoding unit includes a parallel foreground decoding subunit and a background decoding subunit, the foreground decoding subunit is used to reconstruct the foreground feature map of the video frame to be processed according to the encoding features, and the background decoding subunit is used to reconstruct the background feature map of the video frame to be processed according to the encoding features; the output unit is used to determine the foreground mask according to the foreground feature map and the background feature map; the foreground training template is the target result image of the training foreground decoding subunit, and the background training template is the target result image of the training background decoding subunit.

[0093] In order to improve the reconstruction effect of the foreground feature map and the background feature map, the foreground decoding subunit and the background decoding subunit of the embodiment of the present disclosure respectively include at least one first reverse feature extraction layer for semantic segmentation and at least one second reverse feature extraction layer for cutout, which are sequentially connected. It can be understood that the contour information of the target object image can be extracted from the corresponding feature map through the first directional feature extraction layer for semantic segmentation, and the detail information of the target object image can be extracted from the corresponding feature map carrying the contour information of the target object image through the second reverse feature extraction layer for cutout, thereby improving the reconstruction effect of the corresponding feature map, so that it contains the edge detail information of the target object while containing the overall information of the target object image, and the improvement of the reconstruction effect of the corresponding feature map can significantly improve the accuracy of the foreground mask. Further, first determining the contour information of the target object and then determining the detail information of the target object can reduce the image processing operation amount of the second reverse feature extraction layer, and the reduction of the image processing operation amount can improve the image processing speed of the second reverse feature extraction layer, so the embodiment of the present disclosure can improve the image quality of the foreground mask by improving the reconstruction effect of the foreground feature map and the background feature map while ensuring a high image processing speed.

[0094] In the disclosed embodiment, the foreground decoding subunit preferably includes the same number of first directional feature extraction layers for semantic segmentation as the number of second directional feature extraction layers for matting.

[0095] In a disclosed embodiment, the foreground decoding subunit and the background decoding subunit each include two first reverse feature extraction layers for semantic segmentation to extract the contour of the target object from the corresponding feature map, and two second reverse feature extraction layers for cutout connected to the two first reverse feature extraction layers for semantic segmentation, and the reverse feature extraction layers are used to cutout the feature map carrying the contour information of the target object to obtain the detail information of the target object.

[0096] For the foreground decoding subunit, Figure 3As shown, the first reverse feature extraction layers identified as F8 and F4 are both first reverse feature extraction layers for semantic segmentation; the reverse feature extraction layers identified as F2 and F are second reverse feature extraction layers for cutout; specifically, the first reverse feature extraction layer identified as F8 is used to upsample the received feature map to 1 / 8 of the original image size, and perform semantic segmentation on it to obtain a first segmentation feature map, and then pass through a 1×1 convolution layer to convert the number of channels of the first segmentation feature map to 1, and the format becomes (1, H / 8, W / 8); correspondingly, the first reverse feature extraction layer identified as F4 is used to upsample the received first segmentation feature map to 1 / 4 of the original image size, and perform semantic segmentation on it to obtain a second segmentation feature map , and then pass through a 1×1 convolution layer to convert the number of channels of the second segmentation feature map to 1, and the format becomes (1, H / 4, W / 4); the second reverse feature extraction layer identified as F2 is used to upsample the received second segmentation feature map to 1 / 2 of the original image size, and perform portrait cutout to obtain the first cutout feature map, and then pass through a 1×1 convolution layer to convert the number of channels of the first cutout feature map to 1, and the format becomes (1, H / 2, W / 2); the second reverse feature extraction layer identified as F is used to upsample the received feature map to the original image size, perform portrait cutout to obtain a foreground feature map, and then pass through a 1×1 convolution layer to convert the number of channels of the foreground feature map to 1, and the format becomes (1, H, W).

[0097] It should be noted that, since the foreground decoding subunit and the background decoding subunit have the same structure, the embodiment of the present disclosure does not provide a detailed description of the background decoding subunit.

[0098] Among them, the set loss function used by the first reverse feature extraction layer for semantic segmentation is a binary cross entropy loss function; the set loss function used by the second reverse feature extraction layer for cutout is the minimum absolute value deviation.

[0099] During the training process of the mask processing model, because we hope that the output of the mask processing model is as close as possible to the target value required by the user, the current output value of the network and the target value will be compared, and then the weight vector of each layer of the model network will be updated according to the difference between the two (of course, the relevant parameters of each layer of the model network will be pre-configured). For example, if the predicted value of the model is too high, the weight vector of the corresponding network part of the model is adjusted to make the predicted value of the model lower; if the predicted value of the model is too low, the weight vector of the corresponding network part of the model is also adjusted to make the predicted value of the model higher. This adjustment is continued until the difference between the predicted value of the model and the target value is within the preset error threshold. Since the comparison method of the predicted value and the target value will vary depending on the purpose of each network part, it is necessary to select a suitable loss function according to the purpose of different network parts to measure the degree of difference between the predicted value and the target value.

[0100] The embodiment of the present disclosure uses a cross entropy loss function to measure the difference between the predicted value and the target value of the first reverse feature extraction layer for semantic segmentation. The cross entropy loss function is used to measure the difference information between the two probability distributions of the target value and the predicted value. The smaller the cross entropy, the closer the two are. The cross entropy loss function formula for N samples is as follows:

[0101]

[0102] Among them, y (i) is the target value, is the predicted value.

[0103] The disclosed embodiment uses the minimum absolute value deviation to measure the difference between the predicted value and the target value of the second reverse feature extraction layer used for cutout. The minimum absolute value deviation is also called the L1 norm loss function, or L1 loss function for short. It minimizes the sum of the absolute differences between the target value and the predicted value. The specific formula is as follows:

[0104]

[0105] Among them, y (i) is the target value, is the predicted value.

[0106] In order to improve the detail learning effect of the mask processing model, the training sample set obtained by the embodiment of the present disclosure includes the video frame to be processed, the foreground training template and the background training template, as well as the edge extraction template. Correspondingly, the parameter optimization process of the model also includes: determining the difference image range between the current feature map output by the second reverse feature extraction layer defined by the edge extraction template and the corresponding training template as the target edge image; optimizing the parameters of the current mask processing model according to the calculation result of the minimum absolute value deviation corresponding to the pixel mean of the target edge image and the current feature map.

[0107] The method for obtaining the edge extraction template includes: Fig. 7A ) are respectively subjected to dilation and erosion processing, and the difference image between the dilation processing result and the erosion processing result is used as the edge extraction template (see Figure 7B ), and add the edge extraction template to the training sample set to update the training sample set.

[0108] For example, Figure 3 As shown, the difference image between the current feature map output by the second reverse feature extraction layer identified as F2 and the foreground training template is determined, and the image range defined by the edge extraction template in the difference image is used as the target edge image; the pixel mean of the target edge image is determined, and the parameters of the current mask processing model are optimized according to the calculation result of the minimum absolute value deviation corresponding to the pixel mean and the current feature map, so as to improve the edge extraction effect of the foreground mask output by the mask processing model. The pixel mean can be determined by the following formula:

[0109] wights_l1_M=mean(|M-fg_alpha_gt|×|edge_region)

[0110] Among them, M is the current feature image output by the second reverse feature extraction layer, fg_alpha_gt is the foreground training template, and edge_region is the edge extraction template.

[0111] In order to improve the smoothness and continuity of the foreground mask, the embodiment of the present disclosure, in the process of optimizing the model parameters based on the set loss function, also includes: calculating the gradient difference between the current feature map output by the second reverse feature extraction layer and the corresponding training template in at least one direction; optimizing the parameters of the current mask processing model according to the calculation result of the gradient difference in at least one direction and the corresponding minimum absolute value deviation.

[0112] For example, Figure 3As shown, the gradient difference between the current feature map output by the second directional feature extraction layer identified as B2 and the background training template in the x, y and xy directions is calculated, and the parameters of the current mask processing model are optimized according to the gradient difference in the three directions and the calculation result of the corresponding minimum absolute value deviation, so as to improve the parameter optimization speed and optimization quality of each layer network of the mask processing model, as well as the smoothness and continuity of the corresponding foreground mask. The calculation formula of the gradient difference in the three directions is as follows:

[0113]

[0114] Among them, M is the feature map currently output by the second reverse feature extraction layer, and bg_alpha_gt is the background training template.

[0115] In a disclosed embodiment, during the training of the mask processing model, the parameters of the current mask processing model are optimized based on the calculation result of the minimum absolute value deviation of the current feature map, the pixel mean of the target edge image, and the gradient difference between the current feature map and the corresponding training template in at least one direction.

[0116] Compared with the prior art, the technical solution of the mask processing model training method provided in the embodiment of the present disclosure optimizes model parameters based on a set loss function to improve the ability of the parallel foreground decoding subunit and the background decoding subunit to predict the corresponding foreground feature map and the background feature map, thereby improving the ability of the output unit to predict the foreground mask based on the foreground feature map and the background feature map, so that the mask processing model can accurately extract the foreground mask of the target object image in the video frame to be processed after training.

[0117] Embodiment 4

[0118] Based on the above disclosed embodiments, the present disclosed embodiments optimize the model parameter steps based on setting the loss function, and belong to the same inventive concept as the mask processing model training method proposed in the above disclosed embodiments. For technical details not fully described in the present disclosed embodiments, please refer to the above disclosed embodiments.

[0119] Correspondingly, such as Figure 8 As shown, the steps of the mask processing model training method include:

[0120] S401, obtaining a training sample set, wherein the training samples in the training sample set include a template image and a to-be-processed video frame including a target object image, and the template image includes a foreground training template corresponding to a foreground mask and a background training template corresponding to the foreground training template.

[0121] S401, inputting the training samples in the training sample set into the mask processing model to be trained for training, and optimizing the model parameters based on the set loss function to generate the mask processing model, wherein the model parameters include the network parameters of the output unit.

[0122] Among them, the output unit is used to splice the foreground feature map and the background feature map to obtain the foreground weight, and determine the foreground mask according to the foreground feature map, the background feature map and the foreground weight. The specific calculation formula is as follows:

[0123] alpha=beta×F+(1-beta)×(1-B)

[0124] Among them, F is the foreground feature map, B is the background feature map, and beta is the foreground weight.

[0125] In the process of optimizing the model parameters based on the set loss function, the disclosed embodiment also includes, when the output unit determines the current foreground mask according to the foreground feature map, the background feature map and the foreground weight, performing minimum absolute deviation processing on the foreground training template and the current foreground mask, and adjusting the model parameters according to the processing result of the minimum absolute deviation. This helps to improve the accuracy of the foreground weight determined by the output unit according to the foreground feature map and the background feature map, and the accuracy of the foreground mask determined according to the foreground feature map, the background feature map and the foreground weight.

[0126] In another disclosed embodiment, the process of optimizing model parameters based on the set loss function includes: determining the difference image range between the current foreground mask and the foreground training template defined by the edge extraction template as the target edge image; and optimizing the parameters of the current mask processing model based on the calculation result of the pixel mean of the target edge image and the current minimum absolute value deviation.

[0127] Exemplarily, the difference image between the current foreground mask and the foreground training template, and the effective image range of the difference image defined by the edge extraction template are determined as the target edge image, and the parameters of the current mask processing model are optimized according to the calculation result of the pixel mean of the target edge image and the current minimum absolute value deviation. This helps to improve the edge detail effect of the foreground mask output by the mask processing model, thereby improving the edge processing effect of the target object of the video frame to be processed. The method for determining the target edge image includes:

[0128] wights_l1_alpha=mean(|alpha-fg_alpha_gt|×|edge_region)

[0129] Among them, alpha is the current foreground mask output by the output unit, and fg_alpha_gt is the foreground training template.

[0130] In another disclosed embodiment, the process of optimizing model parameters based on a set loss function includes: calculating the gradient difference between the current foreground mask and the foreground training template in at least one direction; optimizing the parameters of the current mask processing model according to the gradient difference processing result in at least one direction and the calculation result of the current minimum absolute deviation loss function.

[0131] Exemplarily, the gradient difference between the current foreground mask and the foreground training template in the x, y and xy directions is calculated, and the parameters of the current mask processing model are optimized according to the calculation results of the gradient difference in the three directions, the pixel mean of the target edge image and the corresponding minimum absolute value deviation. The calculation formula of the gradient difference in the three directions is as follows:

[0132]

[0133] Among them, alpha is the current foreground mask and fg_alpha_gt is the foreground training template.

[0134] The mask processing model training method provided by the disclosed embodiment of the present invention adjusts the parameters of the output unit of the mask processing model according to the calculation result of the minimum absolute deviation to improve the ability of the output unit to predict the foreground mask according to the foreground feature map, the background feature map and the foreground weight, thereby improving the quality of the predicted foreground mask.

[0135] Embodiment 5

[0136] Fig. 9 A block diagram of a mask processing model training device provided in a fourth embodiment of the present disclosure is shown. The present embodiment of the present disclosure is usually configured in an electronic device and implemented through software or hardware, and can implement the mask processing model training method provided in the above-mentioned disclosed embodiment.

[0137] like Fig. 9 As shown, the mask processing model training device provided by the embodiment of the present disclosure includes:

[0138] The sample acquisition module 51 is used to acquire a training sample set, wherein the training samples in the training sample set include a template image and a to-be-processed video frame including a target object image, and the template image includes a foreground training template and a background training template corresponding to the foreground training template;

[0139] The training module 52 is used to input the training samples in the training sample set into the mask processing model to be trained for training, and optimize the model parameters based on the set loss function; wherein, the mask processing model includes an encoding unit, a decoding unit and an output unit; the encoding unit is used to extract encoding features from the video frame to be processed; the decoding unit includes a foreground decoding subunit and a background decoding subunit, the foreground decoding subunit is used to reconstruct the foreground feature map of the video frame to be processed according to the encoding features, and the background decoding subunit is used to reconstruct the background feature map of the video frame to be processed according to the encoding features; the output unit is used to determine the foreground mask according to the foreground feature map and the background feature map; the foreground training template is the target result image of the training foreground decoding subunit, and the background training template is the target result image of the training background decoding subunit.

[0140] Preferably, the foreground decoding subunit and the background decoding subunit respectively include at least one first reverse feature extraction layer for semantic segmentation and at least one second reverse feature extraction layer for matting, which are sequentially connected.

[0141] Preferably, in one decoding subunit, the number of first inverse feature extraction layers is the same as the number of second inverse feature extraction layers.

[0142] Preferably, the set loss function used by the first reverse feature extraction layer for semantic segmentation is a binary cross entropy loss function; the set loss function used by the second reverse feature extraction layer for cutout is a minimum absolute deviation loss function.

[0143] Preferably, the sample acquisition module is used to obtain a sample set of a foreground training template containing a video frame to be processed of an image of a target object and a corresponding foreground mask; invert the foreground training template to generate a background training template corresponding to the foreground training template; and add the background training template to the training sample set to update the training sample set.

[0144] Preferably, the difference image between the current feature map output by the second reverse feature extraction layer and the corresponding training template, as well as the target image range of the difference image limited by the edge extraction template, are determined as the target edge image; and the current mask processing model is parameter optimized based on the calculation result of the pixel mean of the target edge image and the minimum absolute value deviation corresponding to the current feature map to generate the mask processing model.

[0145] Preferably, the training module 52 is also used to calculate the gradient difference between the current feature map output by the second reverse feature extraction layer and the corresponding training template in at least one direction; and optimize the parameters of the current mask processing model according to the calculation result of the gradient difference in at least one direction and the corresponding minimum absolute value deviation to generate a mask processing model.

[0146] Optionally, the training module 52 is also used to perform minimum absolute value deviation processing on the foreground training template and the current foreground mask after the current foreground mask is determined based on the foreground feature map, the background feature map and the foreground weight, and to optimize the parameters of the current mask processing model based on the processing result of the minimum absolute value deviation.

[0147] Optionally, the training module 52 is also used to determine the difference image between the current foreground mask and the foreground training template, and the target image range of the difference image defined by the edge extraction template, as the target edge image; and to optimize the parameters of the current mask processing model based on the calculation result of the pixel mean of the target edge image and the current minimum absolute value deviation.

[0148] Optionally, the training module 52 is also used to calculate the gradient difference between the current foreground mask and the foreground training template in at least one direction; and optimize the parameters of the current mask processing model based on the calculation result of the gradient difference in at least one direction and the current minimum absolute value deviation to generate a mask processing model.

[0149] The technical solution of the mask processing model training device provided in the embodiment of the present disclosure obtains a training sample set through a sample acquisition module, and optimizes the model parameters based on a set loss function through the training module to improve the ability of the foreground decoding subunit and the background decoding subunit to predict the corresponding foreground feature map and the background feature map, thereby improving the ability of the output unit to predict the foreground mask according to the foreground feature map and the background feature map, so that the mask processing model can accurately extract the foreground mask corresponding to the target object image from the video frame to be processed after training.

[0150] Embodiment 6

[0151] Fig.10 The structural diagram of the electronic device provided by the sixth embodiment of the present disclosure is shown. The electronic device in the embodiment of the present disclosure may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage device 608 to the random access memory (RAM) 603. In RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, ROM 602 and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0152] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Fig.10 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0153] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined by the method of the embodiment of the present disclosure are executed.

[0154] The electronic device provided in the embodiment of the present disclosure and the image processing method provided in the above-mentioned embodiment of the present disclosure belong to the same inventive concept. The technical details not fully described in the embodiment of the present disclosure can be referred to the above-mentioned embodiment of the present disclosure, and the embodiment of the present disclosure has the same beneficial effects as the above-mentioned embodiment of the present disclosure.

[0155] Embodiment 7

[0156] Embodiment 7 of the present disclosure provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, the image processing method provided by the above-mentioned embodiment of the disclosure is implemented.

[0157] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0158] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0159] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0160] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0161] Acquire a current video stream, wherein the current video stream includes at least one frame of video frame to be processed that includes an image of a target object;

[0162] Inputting the video frame to be processed into a mask processing model to obtain a foreground mask, wherein the foreground mask corresponds to the target object image, and the mask processing model is used to output a foreground feature map and a background feature map of the video frame to be processed respectively through a background decoding subunit and a foreground decoding subunit, and determine the foreground mask according to the foreground feature map and the background feature map;

[0163] Performing background replacement on the to-be-processed video frame according to the foreground mask to update the to-be-processed video frame in the current video stream;

[0164] The updated current video stream is sent to the server, so that it can be sent to the target user end.

[0165] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0166] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0167] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0168] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0169] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0170] According to one or more embodiments of the present disclosure, [Example 1] provides an image processing method, including:

[0171] Acquire a current video stream, wherein the current video stream includes at least one frame of video frame to be processed that includes an image of a target object;

[0172] Inputting the video frame to be processed into a mask processing model to obtain a foreground mask, wherein the foreground mask corresponds to the target object image, and the mask processing model is used to output a foreground feature map and a background feature map of the video frame to be processed respectively through a background decoding subunit and a foreground decoding subunit, and determine the foreground mask according to the foreground feature map and the background feature map;

[0173] Performing background replacement on the to-be-processed video frame according to the foreground mask to update the to-be-processed video frame in the current video stream;

[0174] The updated current video stream is sent to the server, so that it can be sent to the target user end.

[0175] According to one or more embodiments of the present disclosure, [Example 2] provides an image processing method, which, before performing background replacement on the to-be-processed video frame according to the foreground mask to update the to-be-processed video frame in the current video stream, further includes:

[0176] Receiving a background replacement template uploaded by a user or determining a background replacement template selected by a user in a background replacement template library;

[0177] The background of the video frame to be processed is replaced according to the background replacement template and the foreground mask to update the video frame to be processed in the current video stream.

[0178] According to one or more embodiments of the present disclosure, [Example Three] provides an image processing method, wherein the mask processing model includes an encoding unit, a decoding unit and an output unit; the encoding unit is used to extract encoding features from the video frame to be processed; the decoding unit includes a parallel foreground decoding subunit and a background decoding subunit, the foreground decoding subunit is used to reconstruct a foreground feature map of the video frame to be processed according to the encoding features, and the background decoding subunit is used to reconstruct a background feature map of the video frame to be processed according to the encoding features; the output unit is used to determine the foreground mask based on the foreground feature map and the background feature map.

[0179] According to one or more embodiments of the present disclosure, [Example 4] provides an image processing method, which also includes: the foreground decoding subunit and the background decoding subunit respectively include at least one first reverse feature extraction layer for semantic segmentation and at least one second reverse feature extraction layer for cutout, which are sequentially connected.

[0180] According to one or more embodiments of the present disclosure, [Example Five] provides an image processing method, further comprising: in a decoding subunit, the number of first reverse feature extraction layers is the same as the number of second reverse feature extraction layers.

[0181] According to one or more embodiments of the present disclosure, [Example Six] provides an image processing method, which also includes: the output unit is used to splice the foreground feature map and the background feature map to obtain a foreground weight, and determine the foreground mask based on the foreground feature map, the background feature map and the foreground weight.

[0182] According to one or more embodiments of the present disclosure, [Example 7] provides an image processing method, further comprising: the output unit is specifically used to determine the foreground mask according to the following formula:

[0183] alpha=beta×F+(1-beta)×(1-B)

[0184] Among them, alpha is the foreground mask, beta is the foreground weight, F is the foreground feature map, and B is the background feature map.

[0185] According to one or more embodiments of the present disclosure, [Example 8] provides an image processing method, further comprising: the mask processing model is trained by the mask processing model to be trained through the following training method, the training method comprising:

[0186] Acquire a training sample set, wherein the training samples in the training sample set include a template image and an image to be processed and a video frame to be processed containing an image of a target object, and the template image includes a foreground training template corresponding to a foreground mask and a background training template corresponding to the foreground training template;

[0187] The training samples in the training sample set are input into the mask processing model to be trained for training, and the model parameters are optimized based on a set loss function to generate the mask processing model.

[0188] According to one or more embodiments of the present disclosure, [Example 9] provides an image processing method, including:

[0189] In the mask processing model to be trained, the set loss function used by the first reverse feature extraction layer for semantic segmentation is a binary cross entropy loss function; the set loss function used by the second reverse feature extraction layer for cutout is a minimum absolute value deviation.

[0190] According to one or more embodiments of the present disclosure, [Example 10] provides an image processing method, including: a method for obtaining a training sample set includes:

[0191] Obtaining a sample set of a video frame to be processed including a target object image and a foreground training template corresponding to a foreground mask;

[0192] The foreground training template is inverted to generate a background training template corresponding to the foreground training template;

[0193] The background training template is added to the training sample set to update the training sample set.

[0194] According to one or more embodiments of the present disclosure, [Example 11] provides an image processing method, further comprising:

[0195] Performing expansion processing and corrosion processing on the foreground training template respectively, taking the difference image between the expansion processing result and the corrosion processing result as the edge extraction template, and adding the edge extraction template to the training sample set;

[0196] The step of inputting the training samples in the training sample set into the mask processing model to be trained for training, and optimizing the model parameters based on a set loss function to generate the mask processing model includes:

[0197] Determine a difference image between a current feature map output by the second reverse feature extraction layer and a corresponding training template, and a target image range of the difference image defined by the edge extraction template, as a target edge image;

[0198] According to the calculation result of the pixel mean of the target edge image and the minimum absolute value deviation corresponding to the current feature map, the parameters of the current mask processing model are optimized to generate the mask processing model.

[0199] According to one or more embodiments of the present disclosure, [Example 12] provides an image processing method, further comprising:

[0200] The training samples are input into the mask processing model to be trained for training, and model parameters are optimized based on a set loss function to generate the mask processing model, further comprising:

[0201] Calculating the gradient difference between the current feature map output by the second reverse feature extraction layer and the corresponding training template in at least one direction;

[0202] The parameters of the current mask processing model are optimized according to the calculation result of the gradient difference in the at least one direction and the corresponding minimum absolute value deviation to generate a mask processing model.

[0203] According to one or more embodiments of the present disclosure, [Example 13] provides an image processing method, further comprising:

[0204] The step of inputting the training samples in the training sample set into the mask processing model to be trained for training, and optimizing the model parameters based on a set loss function to generate the mask processing model comprises: after the current foreground mask is determined according to the foreground feature map, the background feature map and the foreground weight, performing minimum absolute deviation processing on the foreground training template and the current foreground mask;

[0205] The parameters of the current mask processing model are optimized according to the processing result of the minimum absolute value deviation to generate a mask processing model.

[0206] According to one or more embodiments of the present disclosure, [Example 14] provides an image processing method, further comprising:

[0207] The step of inputting the training samples in the training sample set into the mask processing model to be trained for training, and optimizing the model parameters based on a set loss function to generate the mask processing model, further includes:

[0208] Determine a difference image between a current foreground mask and the foreground training template, and a target image range of the difference image defined by the edge extraction template, as a target edge image;

[0209] The parameters of the current mask processing model are optimized according to the calculation result of the pixel mean value of the target edge image and the current minimum absolute value deviation to generate the mask processing model.

[0210] According to one or more embodiments of the present disclosure, [Example 15] provides an image processing method, further comprising:

[0211] The step of inputting the training samples in the training sample set into the mask processing model to be trained for training, and optimizing the model parameters based on the set loss function to generate the mask processing model, further includes:

[0212] Calculating the gradient difference between the current foreground mask and the foreground training template in at least one direction;

[0213] The parameters of the current mask processing model are optimized according to the calculation result of the gradient difference in the at least one direction and the current minimum absolute value deviation to generate a mask processing model.

[0214] According to one or more embodiments of the present disclosure, [Example 16] provides an image processing device, including:

[0215] A video acquisition module, used to acquire a current video stream, wherein the current video stream includes at least one frame of video frame to be processed that contains an image of a target object;

[0216] A mask module, used for inputting the video frame to be processed into a mask processing model to obtain a foreground mask, wherein the foreground mask corresponds to the target object image, and the mask processing model is used for outputting a foreground feature map and a background feature map of the video frame to be processed respectively through a background decoding subunit and a foreground decoding subunit, and determining the foreground mask according to the foreground feature map and the background feature map;

[0217] A background replacement module, used for performing background replacement on the to-be-processed video frame according to the foreground mask to update the to-be-processed video frame in the current video stream;

[0218] The output module is used to send the updated current video stream to the server, so as to be sent from the server to the target user.

[0219] According to one or more embodiments of the present disclosure, [Example 17] provides an electronic device, the electronic device comprising:

[0220] one or more processors;

[0221] A memory for storing one or more programs;

[0222] When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in the above examples.

[0223] According to one or more embodiments of the present disclosure, [Example 18] provides a computer storage medium on which a computer program is stored, and when the program is executed by a processor, the image processing method described in the above example is implemented.

[0224] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0225] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0226] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. An image processing method, characterized in that: include: Acquire a current video stream, wherein the current video stream includes at least one frame of video frame to be processed that includes an image of a target object; Inputting the video frame to be processed into a mask processing model to obtain a foreground mask, wherein the foreground mask corresponds to the target object image, and the mask processing model is used to output a foreground feature map and a background feature map of the video frame to be processed respectively through a background decoding subunit and a foreground decoding subunit, and determine the foreground mask according to the foreground feature map and the background feature map; Performing background replacement on the to-be-processed video frame according to the foreground mask to update the to-be-processed video frame in the current video stream; Send the updated current video stream to the server, so that the server can send it to the target user end; The foreground decoding subunit and the background decoding subunit respectively include at least one first reverse feature extraction layer for semantic segmentation and at least one second reverse feature extraction layer for cutout, which are sequentially connected; in one decoding subunit, the number of the first reverse feature extraction layers is the same as the number of the second reverse feature extraction layers; The first reverse feature extraction layer of the semantic segmentation is used to extract the contour information of the target object image from the corresponding feature map, and the second reverse feature extraction layer of the cutout is used to extract the detail information of the target object image from the corresponding feature map carrying the contour information of the target object image.

2. The method according to claim 1, characterized in that Before performing background replacement on the to-be-processed video frame according to the foreground mask to update the to-be-processed video frame in the current video stream, the method further includes: Receiving a background replacement template uploaded by a user or determining a background replacement template selected by a user in a background replacement template library; The background of the video frame to be processed is replaced according to the background replacement template and the foreground mask to update the video frame to be processed in the current video stream.

3. The method according to claim 1, characterized in that The mask processing model includes an encoding unit, a decoding unit and an output unit; the encoding unit is used to extract encoding features from the video frame to be processed; the decoding unit includes a parallel foreground decoding subunit and a background decoding subunit, the foreground decoding subunit is used to reconstruct the foreground feature map of the video frame to be processed according to the encoding features, and the background decoding subunit is used to reconstruct the background feature map of the video frame to be processed according to the encoding features; the output unit is used to determine the foreground mask according to the foreground feature map and the background feature map.

4. The method according to claim 1, characterized in that: The output unit is used for splicing the foreground feature map and the background feature map to obtain a foreground weight, and determining the foreground mask according to the foreground feature map, the background feature map and the foreground weight.

5. The method according to claim 4, characterized in that The output unit is specifically used to determine the foreground mask according to the following formula: alpha=beta×F+(1-beta)×(1-B) Among them, alpha is the foreground mask, beta is the foreground weight, F is the foreground feature map, and B is the background feature map.

6. The method according to claim 4, characterized in that The mask processing model is trained by the mask processing model to be trained through the following training method, and the training method includes: Acquire a training sample set, wherein the training samples in the training sample set include a template image and a to-be-processed video frame including a target object image, and the template image includes a foreground training template corresponding to a foreground mask and a background training template corresponding to the foreground training template; The training samples in the training sample set are input into the mask processing model to be trained for training, and the model parameters are optimized based on a set loss function to generate the mask processing model.

7. The method according to claim 6, characterized in that In the mask processing model to be trained, the set loss function used by the first reverse feature extraction layer for semantic segmentation is a binary cross entropy loss function; the set loss function used by the second reverse feature extraction layer for cutout is a minimum absolute value deviation.

8. The method according to claim 6, characterized in that The method for obtaining the training sample set includes: Obtaining a sample set of a video frame to be processed including a target object image and a foreground training template corresponding to a foreground mask; The foreground training template is inverted to generate a background training template corresponding to the foreground training template; The background training template is added to the training sample set to update the training sample set.

9. The method according to claim 7, characterized in that: Also includes: Performing expansion processing and corrosion processing on the foreground training template respectively, taking the difference image between the expansion processing result and the corrosion processing result as the edge extraction template, and adding the edge extraction template to the training sample set; The step of inputting the training samples in the training sample set into the mask processing model to be trained for training, and optimizing the model parameters based on a set loss function to generate the mask processing model includes: Determine a difference image between a current feature map output by the second reverse feature extraction layer and a corresponding training template, and a target image range of the difference image defined by the edge extraction template, as a target edge image; According to the calculation result of the pixel mean of the target edge image and the minimum absolute value deviation corresponding to the current feature map, the parameters of the current mask processing model are optimized to generate the mask processing model.

10. The method according to claim 9, characterized in that The step of inputting the training samples in the training sample set into the mask processing model to be trained for training, and optimizing the model parameters based on a set loss function to generate the mask processing model, further includes: Calculating the gradient difference between the current feature map output by the second reverse feature extraction layer and the corresponding training template in at least one direction; The parameters of the current mask processing model are optimized according to the calculation result of the gradient difference in the at least one direction and the corresponding minimum absolute value deviation to generate a mask processing model.

11. The method according to claim 9, characterized in that The step of inputting the training samples in the training sample set into the mask processing model to be trained for training, and optimizing the model parameters based on a set loss function to generate the mask processing model includes: When the current foreground mask is determined according to the foreground feature map, the background feature map and the foreground weight, performing minimum absolute deviation processing on the foreground training template and the current foreground mask; The parameters of the current mask processing model are optimized according to the processing result of the minimum absolute value deviation to generate a mask processing model.

12. The method according to claim 11, characterized in that The step of inputting the training samples in the training sample set into the mask processing model to be trained for training, and optimizing the model parameters based on a set loss function to generate the mask processing model, further includes: Determine a difference image between a current foreground mask and the foreground training template, and a target image range of the difference image defined by the edge extraction template, as a target edge image; The parameters of the current mask processing model are optimized according to the calculation result of the pixel mean value of the target edge image and the current minimum absolute value deviation to generate the mask processing model.

13. The method according to claim 11 or 12, characterized in that: The step of inputting the training samples in the training sample set into the mask processing model to be trained for training, and optimizing the model parameters based on the set loss function to generate the mask processing model, further includes: Calculating the gradient difference between the current foreground mask and the foreground training template in at least one direction; The parameters of the current mask processing model are optimized according to the calculation result of the gradient difference in the at least one direction and the current minimum absolute value deviation to generate a mask processing model.

14. An image processing device, characterized in that: include: A video acquisition module, used to acquire a current video stream, wherein the current video stream includes at least one frame of video frame to be processed that contains an image of a target object; A mask module, used for inputting the video frame to be processed into a mask processing model to obtain a foreground mask, wherein the foreground mask corresponds to the target object image, and the mask processing model is used for outputting a foreground feature map and a background feature map of the video frame to be processed respectively through a background decoding subunit and a foreground decoding subunit, and determining the foreground mask according to the foreground feature map and the background feature map; wherein the foreground decoding subunit and the background decoding subunit respectively comprise at least one first reverse feature extraction layer for semantic segmentation and at least one second reverse feature extraction layer for cutout, which are sequentially connected; in one decoding subunit, the number of the first reverse feature extraction layers is the same as the number of the second reverse feature extraction layers, the first reverse feature extraction layer for semantic segmentation is used for extracting contour information of the target object image from the corresponding feature map, and the second reverse feature extraction layer for cutout is used for extracting detail information of the target object image from the corresponding feature map carrying the contour information of the target object image; A background replacement module, used for performing background replacement on the to-be-processed video frame according to the foreground mask to update the to-be-processed video frame in the current video stream; The output module is used to send the updated current video stream to the server, so as to be sent from the server to the target user.

15. An electronic device, characterized in that: The electronic device comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any one of claims 1 to 13.

16. A computer storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image processing method as described in any one of claims 1 to 13 is implemented.

Images