E-commerce live broadcast image background replacement method

Abstract

The invention provides an e-commerce live broadcast image background replacement method. The method comprises the following steps: step 1, constructing a semantic segmentation model; 2, training a semantic segmentation model; 3, obtaining a semantic segmentation result of the e-commerce live broadcast image to be subjected to background replacement; and 4, replacing the background of the e-commerce live broadcast image to be subjected to background replacement. The semantic segmentation model constructed by the invention can effectively improve the precision of the semantic segmentation result of the image, reduce the edge jitter rate and improve the processing speed; the semantic segmentation result of the image is obtained by using the constructed semantic segmentation model so as to replace the background of the image, so that the quality of the background replacement result is improved, the real-time processing capability is ensured, and the real-time interaction requirements of commodity display, scene switching and the like in the e-commerce live broadcast are met.

Description

Technical Field

[0001] The present invention relates to a method for replacing the background of an e-commerce live broadcast image, and belongs to the technical field of image processing. Background Art

[0002] In the field of e-commerce live streaming, background replacement technology is of great significance for improving live streaming effects and user experience.

[0003] In the process of background replacement, the image needs to be semantically segmented. Capturing the long-distance dependencies and global context information in the image is crucial to improving segmentation accuracy. Usually, semantic segmentation methods use convolutional neural networks. Although convolutional neural networks perform well in extracting local features, they have limitations in processing dependencies between distant pixels. This leads to inaccurate separation of foreground and background during background replacement. To overcome this problem, the multi-head self-attention mechanism in the Transformer architecture was introduced into the field of computer vision. By calculating the similarity between queries, keys, and values, it can effectively model long-distance dependencies between pixels, thereby improving segmentation accuracy. However, when processing high-resolution images, the multi-head self-attention mechanism faces significant computational complexity challenges. Specifically, the time complexity of the multi-head self-attention mechanism is O(N 2 ), where N is the length of the input sequence. For high-resolution images, this can lead to a large consumption of computing resources. In addition, the multi-head self-attention mechanism may ignore local structural information, resulting in insufficient learning of fine-grained features and affecting the detail effect of background replacement. In semantic segmentation methods, the extracted features are nonlinearly transformed through a feedforward network to generate the final segmentation result. Although the feedforward network based on the convolutional neural network effectively reduces the number of parameters through the convolution operation and preserves the spatial structure of the input data, its computational complexity is still high when processing large-scale images. The conditional random field mechanism can effectively combine the spatial relationship between pixels and the dependency between labels, thereby improving the accuracy and coherence of the segmentation boundary. Typically, conditional random fields are added on the basis of convolutional neural networks to fine-tune the segmentation results. However, its segmentation accuracy is still insufficient when processing images of complex scenes, and it is not suitable for application in high-frame-rate video streams.

[0004] In summary, existing background replacement methods have many problems when processing complex scene images, high-resolution images, and real-time processing of high-frame rate video streams. Therefore, existing background replacement methods are not suitable for replacing e-commerce live broadcast backgrounds. Summary of the Invention

[0005] The present invention provides a method for replacing the background of an e-commerce live broadcast image. The semantic segmentation model constructed by the method can effectively improve the accuracy of the semantic segmentation results of the image, reduce the edge jitter rate, and increase the processing speed. The semantic segmentation results of the image are obtained by using the semantic segmentation model constructed by the method to replace the background of the image. This not only improves the quality of the background replacement results, but also ensures the real-time processing capability, which can meet the real-time interaction needs such as product display and scene switching in e-commerce live broadcast.

[0006] In order to solve the above technical problems, the present invention provides a method for replacing the background of an e-commerce live broadcast image, comprising the following steps:

[0007] Step 1: Build a semantic segmentation model:

[0008] The semantic segmentation model includes an encoder, a decoder and a segmentation head. The encoder includes a preprocessing module and first to fourth defective blocks connected in sequence, and the output end of the preprocessing module is connected to the first defective block; the decoder includes first to third multi-scale Transformer modules connected in sequence through a fusion device A, the output end of the fourth defective block is connected to the first multi-scale Transformer module, the output end of the fusion device A between two adjacent multi-scale Transformer modules is also connected to the segmentation head, the output end of the third defective block is also connected to the fusion device A between the first and second multi-scale Transformer modules, the output end of the second defective block is also connected to the fusion device A between the second and third multi-scale Transformer modules, the third multi-scale Transformer module is connected to the segmentation head through a fusion device B, and the output end of the first defective block is also connected to the fusion device B between the third multi-scale Transformer module and the segmentation head;

[0009] Each of the multi-scale Transformer modules includes a window multi-head self-attention module, a batch normalization layer, a feedforward network based on depthwise separable convolution, and a fusion device C. The output end of the window multi-head self-attention module is connected to the batch normalization layer, the batch normalization layer and the feedforward network based on depthwise separable convolution are connected through a fusion device D. The output end of the window multi-head self-attention module is also connected to the fusion device D, the output end of the feedforward network based on depthwise separable convolution is connected to the fusion device C, the output end of the fusion device C of the first multi-scale Transformer module is connected to the fusion device A between the first and second multi-scale Transformer modules, the output end of the fusion device C of the second multi-scale Transformer module is connected to the fusion device A between the second and third multi-scale Transformer modules, and the output end of the fusion device C of the third multi-scale Transformer module is connected to the fusion device B between the third multi-scale Transformer module and the segmentation head;

[0010] The segmentation head includes a multi-scale attention fusion module based on conditional random fields, a convolutional layer and an upsampling block, the output end of the multi-scale attention fusion module is connected to the convolutional layer, and the output end of the convolutional layer is connected to the upsampling block;

[0011] Step 2: Train the semantic segmentation model: Use the e-commerce live broadcast image set to train the semantic segmentation model;

[0012] Step 3: Obtain the semantic segmentation result of the e-commerce live broadcast image to be replaced with the background: Input the e-commerce live broadcast image to be replaced with the background into the trained semantic segmentation model to obtain the semantic segmentation result;

[0013] Step 4. Replace the background of the e-commerce live broadcast image to be replaced: Based on the semantic segmentation results, the target background map B and the e-commerce live broadcast image to be replaced with the background, replace the background of the e-commerce live broadcast image to be replaced with the background.

[0014] In a specific embodiment, the step 3 includes:

[0015] Step 3.1, the encoder extracts four local feature maps of different resolutions of the e-commerce live broadcast image to be background replaced, wherein the first defective block outputs a local feature map of the first resolution, the second defective block outputs a local feature map of the second resolution, the third defective block outputs a local feature map of the third resolution, and the fourth defective block outputs a local feature map of the fourth resolution;

[0016] Step 3.2: Based on the four local feature maps of different resolutions extracted by the encoder, each multi-scale Transformer module of the decoder sequentially performs a windowed multi-head self-attention mechanism, batch normalization, and depthwise separable convolution to extract three global context feature maps of different scales, wherein the first multi-scale Transformer module outputs a first-scale global context feature map, the second multi-scale Transformer module outputs a second-scale global context feature map, and the third multi-scale Transformer module outputs a third-scale global context feature map;

[0017] Step 3.3: Based on four local feature maps of different resolutions and three global context feature maps of different scales, the segmentation head performs a multi-scale attention mechanism fusion mechanism based on conditional random fields to obtain semantic segmentation results.

[0018] In a specific embodiment, in step 3.2, each multi-scale Transformer module performs the following steps of window multi-head self-attention mechanism, batch normalization and depth-wise separable convolution:

[0019] Step 3.2.1: The window multi-head self-attention module of the multi-scale Transformer module divides the input feature map into multiple windows and solves the self-attention feature map of each window;

[0020] Step 3.2.2, based on the self-attention feature map of each window, a shift window strategy is executed on the divided multiple windows to shift the windows, and the self-attention feature map of each window obtained after the shift is solved;

[0021] Step 3.2.3, perform a depth-wise separable convolution operation on the self-attention feature map of each window obtained after the shift, and obtain a feature map based on depth-wise separable convolution enhancement for each window obtained after the shift;

[0022] Step 3.2.4, splicing and convolving the feature maps based on depthwise separable convolution enhancement of each window obtained after the shift, and fusing the overlapping parts between windows before splicing to obtain a first global context feature map based on the input feature map;

[0023] In step 3.2.5, each of the first global context feature maps is fused with the first global context feature map processed by the batch normalization layer to obtain a fused first global context feature map, and the fused first global context feature map is fused with the fused first global context feature map processed by the feedforward network based on depthwise separable convolution to obtain the global context feature map of the corresponding multi-scale Transformer module.

[0024] In a specific embodiment, in step 3.2, the feature map input to the first multi-scale Transformer module is a local feature map of the fourth resolution, the feature map input to the second multi-scale Transformer module is a feature map obtained by fusion of the local feature map of the third resolution and the first-scale global context feature map; the feature map input to the third multi-scale Transformer module is a feature map obtained by fusion of the local feature map of the second resolution and the second-scale global context feature map.

[0025] In a specific embodiment, relative position encoding is introduced in the process of solving the self-attention feature map of the window.

[0026] In a specific embodiment, the feature map obtained by fusing the third-resolution local feature map with the first-scale global context feature map is input into the multi-scale attention fusion module based on conditional random fields of the segmentation head to generate a first feature map F1, the feature map obtained by fusing the second-resolution local feature map with the second-scale global context feature map is input into the multi-scale attention fusion module based on conditional random fields of the segmentation head to generate a second feature map F2, the feature map obtained by fusing the first-resolution local feature map with the third-scale global context feature map is input into the multi-scale attention fusion module based on conditional random fields of the segmentation head to generate a third feature map F3, the multi-scale attention fusion module based on conditional random fields fuses the first feature map F1, the second feature map F2 and the third feature map F3 to obtain a fourth feature map F4 of the e-commerce live broadcast image to be replaced with the background, the fourth feature map of the e-commerce live broadcast image to be replaced with the background is passed through a convolutional layer and an upsampling block to obtain a feature map of the e-commerce live broadcast image to be replaced with the background, and the feature map of the e-commerce live broadcast image to be replaced with the background is the semantic segmentation result of the e-commerce live broadcast image to be replaced with the background;

[0027] The feature map of the e-commerce live broadcast image to be background replaced is a binary segmentation mask, represented by S∈{0, 1}H0×W0, where 0 represents the background, 1 represents the foreground, and H0×W0 is the size of S.

[0028] In a specific embodiment, the step 4 is specifically as follows: step 4.1, performing morphological operations and edge smoothing on the semantic segmentation result S; step 4.2, based on the semantic segmentation result S, extracting the foreground image F of the e-commerce live broadcast image I to be replaced with the background by element-by-element multiplication; step 4.3, fusing the foreground image F and the target background image B to obtain the e-commerce live broadcast image R under the target background.

[0029] In a specific embodiment, the step 2 is specifically as follows:

[0030] Step 2.1: Set the training parameters of the semantic segmentation model;

[0031] Step 2.2: Obtain a training dataset, perform data augmentation on the training dataset, and divide the augmented training dataset into a training set, a validation set, and a test set.

[0032] Step 2.3: Use the training set to train the semantic segmentation model. When the preset number of training times is reached, the semantic segmentation model training is completed.

[0033] Step 2.4: Use the validation set to validate the trained semantic segmentation model. If the performance of the semantic segmentation model does not meet the preset evaluation index A, repeat steps 2.2 to 2.3 to train the semantic segmentation model. If the performance of the semantic segmentation model meets the preset evaluation index A, the semantic segmentation model validation passes.

[0034] Step 2.5: Use the test set to test the trained semantic segmentation model. If the performance of the semantic segmentation model does not meet the preset evaluation index B, repeat steps 2.2 to 2.4 to train and verify the semantic segmentation model. If the performance of the semantic segmentation model meets the preset evaluation index B, the semantic segmentation model test passes.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. The semantic segmentation model constructed by the present invention includes an encoder, a decoder, and a segmentation head. The encoder obtains four local feature maps of different frequencies, the decoder obtains three global context feature maps of different scales, and the segmentation head obtains the semantic segmentation result.

[0037] In the process of obtaining three global context feature maps of different scales, the decoder of the semantic segmentation model of the present invention first solves the self-attention feature map of the input feature map. In the process of solving the self-attention feature map, window division and window shifting are utilized and relative position coding is introduced. Therefore, in the process of solving the self-attention feature map, it can not only accurately capture local context relationships and long-distance dependencies, but also realize cross-window information interaction and greatly reduce the computational complexity. Then, a feedforward network based on deep separable convolution is used on the self-attention feature map. On the basis of reducing the number of parameters, it strengthens the interaction between channels, improves the local feature expression ability, and further optimizes the three global context feature maps of different scales output by the decoder.

[0038] The multi-scale attention fusion module based on conditional random fields in the segmentation head of the semantic segmentation model of the present invention deeply fuses the spatial details of shallow features with the semantic information of deep features. The conditional random fields can be used to adjust the predicted probability distribution according to the image space and color information, realize accurate feature representation, improve boundary accuracy and smoothness, accurately outline the contours of various target objects in the e-commerce live broadcast screen, accurately capture complex structures, and ensure that the semantic segmentation results are both meticulous and meet the needs of actual scenarios.

[0039] Using the semantic segmentation model constructed by the present invention to perform semantic segmentation of images can effectively improve segmentation accuracy, reduce edge jitter rate, and increase processing speed.

[0040] 2. The present invention performs semantic segmentation on the e-commerce live broadcast image to be replaced with the background through the constructed semantic segmentation model to obtain the foreground of the e-commerce live broadcast image to be replaced with the background, and replaces the background of the e-commerce live broadcast image to be replaced with the background based on the foreground, the target background map B and the e-commerce live broadcast image to be replaced with the background. It not only improves the quality of the background replacement result, but also ensures the real-time processing capability, thereby meeting the real-time interaction needs such as product display and scene switching in e-commerce live broadcast. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of the method for replacing the background of an e-commerce live broadcast image of the present invention.

[0042] Figure 2 Schematic diagram of the semantic segmentation model of the present invention.

[0043] Figure 3 An e-commerce live broadcast image to be background-replaced is provided in one embodiment of the present invention.

[0044] Figure 4 For the present invention Figure 3 Semantic segmentation results.

[0045] Figure 5 For the present invention Figure 3 The target background image to be replaced.

[0046] Figure 6 For the present invention Figure 3 The image after background replacement. DETAILED DESCRIPTION

[0047] The present invention will be described in detail below with reference to the embodiments and accompanying drawings. It should be noted that the embodiments and features of the embodiments of the present invention can be combined with each other without conflict.

[0048] refer to Figure 1 , a method for replacing the background of an e-commerce live broadcast image, comprising the following steps:

[0049] Step 1: Build a semantic segmentation model:

[0050] The semantic segmentation model includes an encoder, a decoder and a segmentation head. The encoder includes a preprocessing module and first to fourth defective blocks connected in sequence. The output end of the preprocessing module is connected to the first defective block. The preprocessing module includes a convolutional layer and a maximum pooling layer. Each defective block contains three convolutional layers connected in sequence. The encoder extracts multiple local feature maps of different resolutions of the e-commerce live broadcast image to be replaced with the background; the decoder includes first to third multi-scale Transformer modules connected in sequence through a fusion device A. The decoder decodes the multiple local feature maps of different resolutions extracted by the encoder to extract multiple global context feature maps F of different scales. n The output end of the fourth defective block is connected to the first multi-scale Transformer module, the output ends of the fusion device A between the two adjacent multi-scale Transformer modules are also connected to the segmentation head, the output end of the third defective block is also connected to the fusion device A between the first and second multi-scale Transformer modules, the output end of the second defective block is also connected to the fusion device A between the second and third multi-scale Transformer modules, the third multi-scale Transformer module is connected to the segmentation head through the fusion device B, and the output end of the first defective block is also connected to the fusion device B between the third multi-scale Transformer module and the segmentation head. The segmentation head obtains the semantic segmentation result of the e-commerce live broadcast image to be background replaced.

[0051] The encoder obtains feature maps of different resolutions of the input e-commerce live broadcast image through four residual blocks. While ensuring the retention of key features, it reduces the computational burden, lays a solid foundation for subsequent processing flow, and ensures that effective information can be stably extracted from complex e-commerce live broadcast images.

[0052] Each of the multi-scale Transformer modules includes a window multi-head self-attention module, a batch normalization layer, a feedforward network based on depthwise separable convolution, and a fusion device C. The output end of the window multi-head self-attention module is connected to the batch normalization layer. The batch normalization layer is connected to the feedforward network based on depthwise separable convolution through a fusion device D. The output end of the window multi-head self-attention module is also connected to the fusion device D. The output end of the feedforward network based on depthwise separable convolution is connected to the fusion device C. The output end of the fusion device C of the first multi-scale Transformer module is connected to the fusion device A between the first and second multi-scale Transformer modules. The output end of the fusion device C of the second multi-scale Transformer module is connected to the fusion device A between the second and third multi-scale Transformer modules. The output end of the fusion device C of the third multi-scale Transformer module is connected to the fusion device B between the third multi-scale Transformer module and the segmentation head.

[0053] The segmentation head includes a multi-scale attention fusion module based on conditional random fields, a convolution layer and an upsampling block. The output end of the multi-scale attention fusion module is connected to the convolution layer, and the output end of the convolution layer is connected to the upsampling block.

[0054] Step 2: Train the semantic segmentation model: Use the e-commerce live broadcast image set to train the semantic segmentation model.

[0055] Specifically, step 2 includes the following steps:

[0056] Step 2.1. Set the training parameters of the semantic segmentation model.

[0057] The Adam optimizer is used to accelerate training by adaptively adjusting the learning rate of each parameter, preventing instability caused by excessively large gradients and slow training caused by excessively small gradients. The initial learning rate is 0.001.

[0058] Step 2.2: Obtain a training dataset, perform data enhancement on the training dataset, and divide the enhanced training dataset into a training set, a validation set, and a test set.

[0059] The training dataset uses the LV-MHP-v2 dataset. The LV-MHP-v2 dataset is a multi-pose, multi-view portrait dataset. It is an upgraded version of the MHP (Multi-Human Parsing) dataset and is mainly used to support research on multi-person parsing. The LV-MHP-v2 dataset has scenes with more people, more complex human poses, and more occlusion situations.

[0060] The data enhancement includes the following methods: color jittering, random horizontal flipping, and random cropping of the input image to increase the diversity of training samples and improve the generalization ability of the semantic segmentation model.

[0061] Step 2.3: Use the training set to train the semantic segmentation model. When the preset number of training times is reached, the semantic segmentation model training is completed.

[0062] Step 2.4: Use the validation set to verify the trained semantic segmentation model. If the performance of the semantic segmentation model meets the preset evaluation index A, the semantic segmentation model verification passes.

[0063] Step 2.5: Use the test set to test the trained semantic segmentation model. If the performance of the semantic segmentation model meets the preset evaluation index B, the semantic segmentation model test passes.

[0064] Evaluation indicators include frames per second (FPS) and mean intersection over union (mIoU).

[0065] Step 3. Obtain the semantic segmentation result of the e-commerce live broadcast image to be replaced with the background: input the e-commerce live broadcast image to be replaced with the background into the trained semantic segmentation model to obtain the semantic segmentation result.

[0066] Specifically, the step 3 includes:

[0067] Step 3.1. The encoder extracts four local feature maps of different resolutions of the e-commerce live broadcast image to be background replaced, wherein the first defective block outputs a local feature map of the first resolution, the second defective block outputs a local feature map of the second resolution, the third defective block outputs a local feature map of the third resolution, and the fourth defective block outputs a local feature map of the fourth resolution.

[0068] Step 3.2. Based on the four local feature maps of different resolutions extracted by the encoder, each multi-scale Transformer module of the decoder sequentially executes a window multi-head self-attention mechanism, batch normalization, and depth-separable convolution to extract three global context feature maps of different scales, wherein the first multi-scale Transformer module outputs a first-scale global context feature map, the second multi-scale Transformer module outputs a second-scale global context feature map, and the third multi-scale Transformer module outputs a third-scale global context feature map.

[0069] Furthermore, in step 3.2, the specific steps of each multi-scale Transformer module performing the window multi-head self-attention mechanism, batch normalization, and depth-wise separable convolution are as follows:

[0070] In step 3.2.1, the window multi-head self-attention module of the multi-scale Transformer module divides the input feature map into multiple windows and solves the self-attention feature map of each window.

[0071] Step 3.2.2: Based on the self-attention feature map of each window, a shift window strategy is executed on the divided multiple windows to shift the windows, and the self-attention feature map of each window obtained after the shift is solved.

[0072] Step 3.2.3, perform a depth-wise separable convolution operation on the self-attention feature map of each window obtained after the shift, and obtain a feature map based on depth-wise separable convolution enhancement for each window obtained after the shift.

[0073] Step 3.2.4, the feature map based on depth-separable convolution enhancement of each window obtained after the shift is spliced ​​and convolved. Before splicing, the overlapping parts between windows are fused to obtain a first global context feature map based on the input feature map.

[0074] In step 3.2.5, each of the first global context feature maps is fused with the first global context feature map processed by the batch normalization layer to obtain a fused first global context feature map, and the fused first global context feature map is fused with the fused first global context feature map processed by the feedforward network based on depthwise separable convolution to obtain the global context feature map of the corresponding multi-scale Transformer module.

[0075] A ReLU activation function is added to each branch of the feedforward network based on depthwise separable convolution to increase the nonlinear expression ability. After the results of all branches are fused, the result is processed by the ReLU activation function to increase the nonlinear expression ability again, which helps to improve the accuracy of the semantic segmentation results.

[0076] The windowed multi-head self-attention module divides and shifts the input feature map into windows, accurately capturing local contextual relationships while enabling cross-window information exchange and significantly reducing computational complexity. Furthermore, by utilizing a feedforward network based on depthwise separable convolutions, while reducing the number of parameters, it strengthens inter-channel interaction, enhances the ability to express local features, and further optimizes the decoding output, allowing the decoder to adapt to the complex interweaving of products and backgrounds in e-commerce live streaming scenarios.

[0077] Specifically, in step 3.2, the feature map input to the first multi-scale Transformer module is the local feature map of the fourth resolution, the feature map input to the second multi-scale Transformer module is the feature map obtained by fusion of the local feature map of the third resolution and the first-scale global context feature map; the feature map input to the third multi-scale Transformer module is the feature map obtained by fusion of the local feature map of the second resolution and the second-scale global context feature map.

[0078] Furthermore, relative position encoding is introduced in the process of solving the self-attention feature map of the window. The relative position encoding encodes the relative position relationship between pixels within the window, which enhances the ability of the window multi-head self-attention module to capture local context and helps the window multi-head self-attention module to handle long-distance dependencies.

[0079] Specifically, the size of the input feature map X is H×W, and the input feature map X is divided into multiple non-overlapping small windows X w , each window size is M×M, and the number of windows N is N=(H / M)×(W / M), that is, the input feature map X is evenly divided into N windows.

[0080] For each window X w The features in the ,are generated through linear transformation to the query tensor Q (Query), key tensor K (Key), and value tensor V (Value).

[0081] Use the Q, K, V generated above to calculate the self-attention feature Attention (X w ), the formula is as follows:

[0082]

[0083] Where d is the key tensor K channel dimension, and Softmax is a normalized exponential function used for multi-classification problems.

[0084] The input feature map X is divided into multiple windows X w The steps of shifting are as follows:

[0085] Determine the shift amount: For a window X of size M×M w , select to move half the window size horizontally or vertically, that is, M / 2.

[0086] Perform shift: For each window X w , according to the shift amount determined above, the shift direction can be horizontal, vertical or both directions at the same time.

[0087] Processing overlapping areas: If adjacent windows produce overlapping areas after gram shift, the features of these overlapping areas are merged through weighted averaging.

[0088] Calculate the self-attention features of each window after the shift.

[0089] Step 3.3: Based on four local feature maps of different resolutions and three global context feature maps of different scales, the segmentation head performs a multi-scale attention mechanism fusion mechanism based on conditional random fields to obtain semantic segmentation results.

[0090] Specifically, in step 3.3, the feature map obtained by fusing the third-resolution local feature map with the first-scale global context feature map is input into the multi-scale attention fusion module based on conditional random fields of the segmentation head to generate a first feature map F1, the feature map obtained by fusing the second-resolution local feature map with the second-scale global context feature map is input into the multi-scale attention fusion module based on conditional random fields of the segmentation head to generate a second feature map F2, the feature map obtained by fusing the first-resolution local feature map with the third-scale global context feature map is input into the multi-scale attention fusion module based on conditional random fields of the segmentation head to generate a third feature map F3, and the multi-scale attention fusion module based on conditional random fields fuses the first feature map F1, the second feature map F2 and the third feature map F3 to obtain a fourth feature map F4 of the e-commerce live broadcast image to be replaced with the background. The fourth feature map of the e-commerce live broadcast image to be replaced with the background is subjected to a convolutional layer, an upsampling block and an Argmax function to obtain a feature map of the e-commerce live broadcast image to be replaced with the background. The feature map of the e-commerce live broadcast image to be replaced with the background is the semantic segmentation result of the e-commerce live broadcast image to be replaced with the background.

[0091] The feature map of the e-commerce live broadcast image to be background replaced is a binary segmentation mask, represented by S∈{0, 1}H0×W0, where 0 represents the background, 1 represents the foreground, and H0×W0 is the size of S.

[0092] In order to make full use of multi-scale information, conditional random fields are used to process feature maps at different levels to generate optimized feature maps F1, F2, and F3 respectively. The feature maps F1, F2, and F3 are fused to output the fourth feature map F4 of the e-commerce live broadcast image to be used for background replacement. F4 is calculated by the following formula:

[0093]

[0094] W i It is a learnable weight parameter used to adaptively adjust the importance of features in each layer. It not only improves the accuracy and smoothness of the segmentation boundary, but also reduces the impact of noise, making the model perform better when processing complex scenes.

[0095] Preferably, a mean field approximation algorithm or fast approximate inference is used to solve the conditional random field.

[0096] The multi-scale attention fusion module based on conditional random fields deeply fuses the spatial details of shallow features with the semantic information of deep features. The conditional random fields can be used to adjust the predicted probability distribution according to the image space and color information, achieve accurate feature representation, improve boundary accuracy and smoothness, accurately outline the contours of various target objects in e-commerce live broadcast images, accurately capture complex structures, and ensure that the semantic segmentation results are both meticulous and meet the needs of actual scenarios.

[0097] Step 4. Replace the background of the e-commerce live broadcast image to be replaced: Based on the semantic segmentation results, the target background map B and the e-commerce live broadcast image to be replaced with the background, replace the background of the e-commerce live broadcast image to be replaced with the background.

[0098] Furthermore, the step 4 is specifically as follows: step 4.1, performing morphological operations and edge smoothing on the semantic segmentation result S; step 4.2, based on the semantic segmentation result S, extracting the foreground image F of the e-commerce live broadcast image I to be replaced with the background by element-by-element multiplication, the formula is as follows: F=I⊙S, where ⊙ represents element-by-element multiplication, if the pixel belongs to the foreground, the color information of the pixel will be retained; if the pixel belongs to the background, the color information of the pixel will be set to zero (ie black); step 4.3, fusing the foreground image F and the target background image B to obtain the e-commerce live broadcast image R under the target background, the formula is as follows: R=F+(1-S)⊙B, where 1-S is the inverse mask used to identify which areas should display the background image B.

[0099] Through step 4, the complete process from semantic segmentation results to live background replacement is realized. It not only improves the quality of background replacement results, but also ensures the ability of real-time processing. It is suitable for various application scenarios such as online meetings and virtual studios.

[0100] The above content is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions and substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A method for replacing the background of an e-commerce live broadcast image, characterized in that: The following steps are involved: Step 1: Build a semantic segmentation model: The semantic segmentation model includes an encoder, a decoder and a segmentation head. The encoder includes a preprocessing module and first to fourth defective blocks connected in sequence, and the output end of the preprocessing module is connected to the first defective block; the decoder includes first to third multi-scale Transformer modules connected in sequence through a fusion device A, the output end of the fourth defective block is connected to the first multi-scale Transformer module, the output end of the fusion device A between two adjacent multi-scale Transformer modules is also connected to the segmentation head, the output end of the third defective block is also connected to the fusion device A between the first and second multi-scale Transformer modules, the output end of the second defective block is also connected to the fusion device A between the second and third multi-scale Transformer modules, the third multi-scale Transformer module is connected to the segmentation head through a fusion device B, and the output end of the first defective block is also connected to the fusion device B between the third multi-scale Transformer module and the segmentation head; Each of the multi-scale Transformer modules includes a window multi-head self-attention module, a batch normalization layer, a feedforward network based on depthwise separable convolution, and a fusion device C. The output end of the window multi-head self-attention module is connected to the batch normalization layer, the batch normalization layer and the feedforward network based on depthwise separable convolution are connected through a fusion device D. The output end of the window multi-head self-attention module is also connected to the fusion device D, the output end of the feedforward network based on depthwise separable convolution is connected to the fusion device C, the output end of the fusion device C of the first multi-scale Transformer module is connected to the fusion device A between the first and second multi-scale Transformer modules, the output end of the fusion device C of the second multi-scale Transformer module is connected to the fusion device A between the second and third multi-scale Transformer modules, and the output end of the fusion device C of the third multi-scale Transformer module is connected to the fusion device B between the third multi-scale Transformer module and the segmentation head; The segmentation head includes a multi-scale attention fusion module based on conditional random fields, a convolutional layer and an upsampling block, the output end of the multi-scale attention fusion module is connected to the convolutional layer, and the output end of the convolutional layer is connected to the upsampling block; Step 2: Train the semantic segmentation model: Use the e-commerce live broadcast image set to train the semantic segmentation model; Step 3: Obtain the semantic segmentation result of the e-commerce live broadcast image to be replaced with the background: Input the e-commerce live broadcast image to be replaced with the background into the trained semantic segmentation model to obtain the semantic segmentation result; Step 4. Replace the background of the e-commerce live broadcast image to be replaced: Based on the semantic segmentation results, the target background map B and the e-commerce live broadcast image to be replaced with the background, replace the background of the e-commerce live broadcast image to be replaced with the background.

2. The method for replacing the background of an e-commerce live broadcast image according to claim 1, characterized in that: The step 3 comprises: Step 3.1, the encoder extracts four local feature maps of different resolutions of the e-commerce live broadcast image to be background replaced, wherein the first defective block outputs a local feature map of the first resolution, the second defective block outputs a local feature map of the second resolution, the third defective block outputs a local feature map of the third resolution, and the fourth defective block outputs a local feature map of the fourth resolution; Step 3.2: Based on the four local feature maps of different resolutions extracted by the encoder, each multi-scale Transformer module of the decoder sequentially performs a windowed multi-head self-attention mechanism, batch normalization, and depthwise separable convolution to extract three global context feature maps of different scales, wherein the first multi-scale Transformer module outputs a first-scale global context feature map, the second multi-scale Transformer module outputs a second-scale global context feature map, and the third multi-scale Transformer module outputs a third-scale global context feature map; Step 3.3: Based on four local feature maps of different resolutions and three global context feature maps of different scales, the segmentation head performs a multi-scale attention mechanism fusion mechanism based on conditional random fields to obtain semantic segmentation results.

3. The method for replacing the background of an e-commerce live broadcast image according to claim 2, characterized in that: In step 3.2, the specific steps for each multi-scale Transformer module to perform the window multi-head self-attention mechanism, batch normalization, and depth-wise separable convolution are as follows: Step 3.2.1: The window multi-head self-attention module of the multi-scale Transformer module divides the input feature map into multiple windows and solves the self-attention feature map of each window; Step 3.2.2, based on the self-attention feature map of each window, a shift window strategy is executed on the divided multiple windows to shift the windows, and the self-attention feature map of each window obtained after the shift is solved; Step 3.2.3, perform a depth-wise separable convolution operation on the self-attention feature map of each window obtained after the shift, and obtain a feature map based on depth-wise separable convolution enhancement for each window obtained after the shift; Step 3.2.4, splicing and convolving the feature maps based on depthwise separable convolution enhancement of each window obtained after the shift, and fusing the overlapping parts between windows before splicing to obtain a first global context feature map based on the input feature map; In step 3.2.5, each of the first global context feature maps is fused with the first global context feature map processed by the batch normalization layer to obtain a fused first global context feature map, and the fused first global context feature map is fused with the fused first global context feature map processed by the feedforward network based on depthwise separable convolution to obtain the global context feature map of the corresponding multi-scale Transformer module.

4. The method for replacing the background of an e-commerce live broadcast image according to claim 3, wherein: In step 3.2, the feature map input to the first multi-scale Transformer module is the local feature map of the fourth resolution, the feature map input to the second multi-scale Transformer module is the feature map obtained by fusion of the local feature map of the third resolution and the first-scale global context feature map; the feature map input to the third multi-scale Transformer module is the feature map obtained by fusion of the local feature map of the second resolution and the second-scale global context feature map.

5. The method for replacing the background of an e-commerce live broadcast image according to claim 4, characterized in that: Relative position encoding is introduced in the process of solving the self-attention feature map of the window.

6. The method for replacing the background of an e-commerce live broadcast image according to claim 5, characterized in that: The feature map formed by fusing the third-resolution local feature map with the first-scale global context feature map is input into the multi-scale attention fusion module based on conditional random fields of the segmentation head to generate a first feature map F1, the feature map formed by fusing the second-resolution local feature map with the second-scale global context feature map is input into the multi-scale attention fusion module based on conditional random fields of the segmentation head to generate a second feature map F2, the feature map formed by fusing the first-resolution local feature map with the third-scale global context feature map is input into the multi-scale attention fusion module based on conditional random fields of the segmentation head to generate a third feature map F3, the multi-scale attention fusion module based on conditional random fields fuses the first feature map F1, the second feature map F2 and the third feature map F3 to obtain a fourth feature map F4 of the e-commerce live broadcast image to be subjected to background replacement, the fourth feature map of the e-commerce live broadcast image to be subjected to background replacement is subjected to a convolutional layer and an upsampling block to obtain a feature map of the e-commerce live broadcast image to be subjected to background replacement, and the feature map of the e-commerce live broadcast image to be subjected to background replacement is the semantic segmentation result of the e-commerce live broadcast image to be subjected to background replacement; The feature map of the e-commerce live broadcast image to be background replaced is a binary segmentation mask, represented by S∈{0, 1}H0×W0, where 0 represents the background, 1 represents the foreground, and H0×W0 is the size of S.

7. The method for replacing the background of an e-commerce live broadcast image according to claim 6, characterized in that: The specific steps of step 4 are: step 4.1, performing morphological operations and edge smoothing on the semantic segmentation result S; step 4.2, based on the semantic segmentation result S, extracting the foreground image F of the e-commerce live broadcast image I to be replaced with the background by element-by-element multiplication; step 4.3, fusing the foreground image F and the target background image B to obtain the e-commerce live broadcast image R under the target background.

8. The method for replacing the background of an e-commerce live broadcast image according to claim 7, wherein: The step 2 is specifically as follows: Step 2.1: Set the training parameters of the semantic segmentation model; Step 2.2: Obtain a training dataset, perform data augmentation on the training dataset, and divide the augmented training dataset into a training set, a validation set, and a test set. Step 2.3: Use the training set to train the semantic segmentation model. When the preset number of training times is reached, the semantic segmentation model training is completed. Step 2.4: Use the validation set to validate the trained semantic segmentation model. If the performance of the semantic segmentation model does not meet the preset evaluation index A, repeat steps 2.2 to 2.3 to train the semantic segmentation model. If the performance of the semantic segmentation model meets the preset evaluation index A, the semantic segmentation model validation passes. Step 2.5: Use the test set to test the trained semantic segmentation model. If the performance of the semantic segmentation model does not meet the preset evaluation index B, repeat steps 2.2 to 2.4 to train and verify the semantic segmentation model. If the performance of the semantic segmentation model meets the preset evaluation index B, the semantic segmentation model test passes.

Images