High-Resolution Real-Time Automatic Green Screen Keying Method and System Based on Attention Mechanism

Abstract

The present invention proposes a high-resolution, real-time, automatic green screen matting method and system based on an attention mechanism. The method comprises: utilizing a deep learning approach, combining a feature extraction model with an attention mechanism to remove manual adjustments and achieve automated processing. This is achieved by employing a lightweight encoder model and first performing green screen matting at low resolution, followed by restoring the original resolution through a high-resolution processing module. To achieve a more refined matting effect, skip connections are employed to mitigate the loss of low-level features. Furthermore, by using a specialized green screen matting dataset to train the network model, the feature extraction module also learns to remove features related to green overflow.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a high-resolution real-time automatic green screen matting method based on an attention mechanism. Background Art

[0002] Green screen keying involves shooting images or videos using a green screen as a background for later composite synthesis. This technique plays a key role in film and television post-production, especially in feature film. Green screen keying technology is not only used in the production of Hollywood blockbusters but also in everyday life, such as virtual studios and online live streaming. Due to its diverse applications and high commercial value, many scholars and companies at home and abroad have conducted in-depth research on green screen keying.

[0003] Currently, green screen keying methods can be divided into two categories: traditional methods and deep learning-based methods. Traditional green screen keying methods include chroma keying, color difference keying, luma keying, and triangulation keying, and most commercial software is based on these methods. Chroma keying estimates the opacity of each pixel based on a user-provided key color by estimating the chroma difference between the foreground object and the given key color. Chroma difference keying uses color differences to key out the green screen and is primarily suitable for scenes where the background color is much larger than the other two colors. Luma keying is a keying method based on differences in image brightness values. This method is often used to extract very bright or self-luminous foreground objects, such as smoke or sparks. Triangulation keying uses images of the same foreground against different backgrounds to key out the green screen. Deep learning-based methods are mostly based on natural scenes for keying, including trimap-based methods, background-based methods, and automatic methods. CF is an affinity-based method that assumes that local regions of an image are continuous, with adjacent pixels of similar colors having similar alpha values. Therefore, the alpha value of an unknown pixel can be estimated by sampling its neighboring pixels of known foreground and background. KNN is also an affinity-based method. This type of method assumes that local areas of the image have smooth characteristics and can be used to propagate alpha values from known areas to unknown areas. FBA proposes a low-cost modification of the Alpha matting network to predict foreground and background colors. LFP learns long-range context features outside the receptive field. BGMV2 (mobile) requires a background image as an additional input and aligns it with the input image. MODNet is an automatic real-time processing method for portraits, but the real-time processing resolution corresponds to a lower level. AIM designs a unified semantic representation module to guide the cutout to produce more accurate results.

[0004] Most commercial software that can achieve green screen keying usually uses traditional methods for green screen keying, but there are a large number of parameters that need to be adjusted by professionals, making it difficult to automate the process. When dealing with green screen keying problems, the higher the resolution, the longer the processing time required, making it difficult to achieve real-time processing speeds. Among the deep learning-based methods, both trimap-based methods and background-based methods require the user to input trimap or background as a priori. Although these methods have high accuracy, they are difficult to automate. Among the deep learning-based methods, there is a class of automatic methods that can achieve automatic and real-time processing at high resolution, but these methods are suitable for processing keying in natural scenes. When used for green screen keying, there is a problem of being unable to remove the green stain (green overflow) on the subject caused by the green screen reflecting onto the subject. Summary of the Invention

[0005] To address the above issues, this paper proposes a high-resolution, real-time, automatic green screen keying method based on an attention mechanism. This method combines a lightweight MobileNet model with an attention module for efficient feature extraction, eliminating manual adjustments and achieving automated processing. A lightweight feature encoder and a high-resolution processing module are employed to achieve real-time processing speed at high resolution. A dedicated green screen keying dataset is used for network training, enabling the network model to learn relevant features of green overflow, effectively eliminating the green overflow phenomenon.

[0006] When the inventors were researching methods for high-resolution, real-time automatic green screen matting, they found that traditional methods in the prior art usually require professionals to adjust a large number of parameters. After studying natural image matting, the inventors found that this defect can be solved by using a deep learning method. Combining a feature extraction model and an attention mechanism can eliminate the involvement of manual adjustment and achieve automated processing. It was found that traditional methods and deep learning-based methods in the prior art are usually slow when processing high-resolution materials and are difficult to achieve real-time processing. This defect can be solved by adopting a lightweight encoder model and first performing green screen matting at a low resolution and then restoring the original resolution through a high-resolution processing module. In order to achieve a more refined matting effect, a jump connection method is used to alleviate the phenomenon of low-level feature loss. It was found that deep learning-based methods in the prior art are difficult to remove green overflow. This defect can be solved by using a special green screen matting dataset to train the network model, so that the feature extraction module also learns the relevant features for removing green overflow.

[0007] Specifically, the present invention proposes a high-resolution, real-time, automatic green screen matting method based on an attention mechanism, which includes:

[0008] Step 1: Build a neural network model including a feature encoder, a dilated spatial convolutional pooling pyramid module, an attention module, and a feature decoder;

[0009] Step 2: The feature encoder downsamples the training image to obtain a low-definition image, extracts image features from the low-definition image, and generates intermediate features during the extraction process;

[0010] Step 3: The dilated spatial convolution pooling pyramid module samples the image features in parallel using dilated convolutions at different sampling rates, and the attention module extracts features from the sampling results to obtain attention features.

[0011] Step 4: The feature decoder decodes the attention feature according to the intermediate feature to obtain an intermediate result including a foreground image of the low-definition image after eliminating green overflow and the low-definition channel transparency image;

[0012] Step 5: Using the foreground image label and the channel transparency image label of the low-definition image as training targets, constructing a first loss based on the intermediate result, and training the neural network model;

[0013] Step 6: Add a high-resolution processing module to the output of the neural network model after training to obtain a matting model. The high-resolution processing module restores the intermediate result to the same resolution as the training image to obtain a matting result including the foreground image of the training image after eliminating green overflow and the channel transparent image of the training image;

[0014] Step 7: Take the foreground image label and channel transparency image label of the training image as training targets, construct a second loss based on the matting result, and train the matting model; input the green screen image into the trained matting model to obtain the foreground image and channel transparency image as the matting result of the green screen image.

[0015] The high-resolution real-time automatic green screen matting method based on the attention mechanism is described. The training image generation process is to select the foreground image with green overflow and the corresponding channel transparent image in the green screen dataset, and synthesize them with the green screen background image to obtain the training image.

[0016] The high-resolution, real-time, automatic green screen matting method based on the attention mechanism, wherein the feature encoder includes multiple convolutional layers, retaining the features after each convolutional layer as the intermediate features; the feature decoder includes multiple convolutional layers, each convolutional layer upsamples the features of the previous layer and concatenates them with the intermediate features;

[0017] The intermediate result also includes the error map and hidden feature hidden of the low-definition image. The first loss includes L1 loss, gradient loss and Laplace loss for the channel transparency map, L1 loss and Laplace loss for the foreground map, and L2 loss for the error map.

[0018] In the aforementioned high-resolution, real-time, automatic green screen matting method based on the attention mechanism, the training image generation process is to randomly select a foreground image F with green overflow and a corresponding channel transparency image α from a green screen matting dataset containing green overflow, and randomly select a background image B from a green screen background dataset. After unifying the resolution of the three, a composite image C is generated according to the synthesis formula:

[0019] C=F×α+B(1-α)

[0020] Downsampling the composite image C to obtain a low-resolution image C′;

[0021] The feature extraction process of the feature encoder is shown in the following formula, where Feature m Represents the image features extracted after passing through multiple convolution modules in the feature encoder, and Shortcuts represents the intermediate features of each convolution block;

[0022] Feature m ,Shortscuts=MobileNetV2(C′)

[0023] The sampling of the dilated spatial convolution pooling pyramid module is:

[0024] Feature aspp =ASPP(Feature m )

[0025] Feature aspp Represents the sampling result output by the dilated spatial convolutional pooling pyramid module;

[0026] The attention module extracts features through the following formula: se Represents the attention feature:

[0027] Feature se =SE(Feature aspp )

[0028] The feature decoder includes multiple convolutional layers. Except for the last layer, each convolutional layer is followed by a BN layer and a ReLU activation function. Before each convolutional layer, bilinear upsampling is used and concatenated with the intermediate features from the feature encoder. The decoding process of the feature decoder is shown in the following formula, where UpSample() represents upsampling, Concat represents feature concatenation, and Sharpcuts represents feature concatenation. i Represents the corresponding i-th layer intermediate features, ConvBlocki represents the i-th convolution module of the decoder;

[0029] Feature up=UpSample(Feature se )

[0030] Feature cat =Concat(Feature up , Shortcuts i )

[0031] Feature conv1 =ConvBlocki(Feature cat )

[0032] The intermediate result also includes the error map and hidden feature hidden of the low-definition image. The first loss is:

[0033]

[0034] The first loss includes the loss of channel transparency map Gradient loss and Laplace loss Loss of foreground image and Laplace loss And the loss L of the error graph err ;

[0035]

[0036] Use L1 loss to measure the predicted low-definition channel transparency image α lr And the foreground image label of the low-definition image is GroundTruth The difference between them, i represents the pixel position;

[0037]

[0038] in represents the gradient;

[0039]

[0040] in It represents the method of finding the pyramid, and s represents the pyramid level;

[0041]

[0042]

[0043] Represents the foreground image label of the low-definition image Ground Truth, F lrRepresents the predicted low-definition foreground image, and the standard L2 loss is used for the error map, where err represents the predicted error map. Represents the Ground Truth of the error graph;

[0044]

[0045]

[0046] The output of the second stage model at high resolution is F hr , α hr As shown in the following formula, DGF represents the depth-guided filtering module used, F lr , α lr , hidden represents the low-definition intermediate result:

[0047] F hr , α hr =DGF(C, C′, F lr , α lr ,hidden)

[0048] The second loss L hr As shown below, except for the first loss L lr In addition, it also includes the loss of channel transparency maps for high-resolution output results Gradient loss and Laplace loss Loss of foreground image and Laplace loss

[0049]

[0050] in:

[0051]

[0052]

[0053]

[0054]

[0055]

[0056] The present invention also proposes a high-resolution real-time automatic green screen matting system based on an attention mechanism, which includes:

[0057] Module 1 is used to build a neural network model including a feature encoder, a dilated spatial convolutional pooling pyramid module, an attention module, and a feature decoder;

[0058] Module 2 is used to enable the feature encoder to downsample the training image to obtain a low-definition image, and extract image features from the low-definition image, and generate intermediate features during the extraction process;

[0059] Module 3 is used to enable the dilated spatial convolution pooling pyramid module to sample the image features in parallel using dilated convolutions at different sampling rates, and the attention module to extract features from the sampling results to obtain attention features;

[0060] Module 4 is configured to enable the feature decoder to decode the attention feature according to the intermediate feature, to obtain an intermediate result including a foreground image of the low-definition image after eliminating green overflow and the low-definition channel transparency image;

[0061] Module 5 is used to train the neural network model by constructing a first loss based on the intermediate result using the foreground image label and the channel transparency image label of the low-definition image as training targets;

[0062] Module 6 is used to add a high-resolution processing module to the output of the neural network model after training to obtain a matting model. The high-resolution processing module restores the intermediate result to the same resolution as the training image to obtain a matting result including the foreground image of the training image after eliminating green overflow and the channel transparent image of the training image;

[0063] Module 7 is used to train the matting model by using the foreground image label and channel transparency image label of the training image as training targets, constructing a second loss based on the matting result, and inputting the green screen image into the trained matting model to obtain the foreground image and channel transparency image as the matting result of the green screen image.

[0064] The high-resolution real-time automatic green screen matting system based on the attention mechanism generates the training image by selecting the foreground image with green overflow and the corresponding channel transparent image in the green screen dataset, and synthesizing them with the green screen background image to obtain the training image.

[0065] The high-resolution, real-time, automatic green screen matting system based on the attention mechanism, wherein the feature encoder includes multiple convolutional layers, retaining the features after each convolutional layer as the intermediate features; the feature decoder includes multiple convolutional layers, each convolutional layer upsamples the features of the previous layer and connects them with the intermediate features;

[0066] The intermediate result also includes the error map and hidden feature hidden of the low-definition image. The first loss includes L1 loss, gradient loss and Laplace loss for the channel transparency map, L1 loss and Laplace loss for the foreground map, and L2 loss for the error map.

[0067] The high-resolution, real-time, automatic green screen matting system based on the attention mechanism is described. The training image generation process is as follows: a foreground image F with green overflow and a corresponding channel transparency image α are randomly selected from a green screen matting dataset containing green overflow, and a background image B is randomly selected from a green screen background dataset. After unifying the resolution of the three, a composite image C is generated according to the synthesis formula:

[0068] C=F×α+B(1-α)

[0069] Downsampling the composite image C to obtain a low-resolution image C′;

[0070] The feature extraction process of the feature encoder is shown in the following formula, where Feature m Represents the image features extracted after passing through multiple convolution modules in the feature encoder, and Shortcuts represents the intermediate features of each convolution block;

[0071] Feature m ,Shortscuts=MobileNetV2(C′)

[0072] The sampling of the dilated spatial convolution pooling pyramid module is:

[0073] Feature aspp =ASPP(Feature m )

[0074] Feature aspp Represents the sampling result output by the dilated spatial convolutional pooling pyramid module;

[0075] The attention module extracts features through the following formula: se Represents the attention feature:

[0076] Feature se =SE(Feature aspp )

[0077] The feature decoder includes multiple convolutional layers. Except for the last layer, each convolutional layer is followed by a BN layer and a ReLU activation function. Before each convolutional layer, bilinear upsampling is used and concatenated with the intermediate features from the feature encoder. The decoding process of the feature decoder is shown in the following formula, where UpSample() represents upsampling, Concat represents feature concatenation, and Sharpcuts represents feature concatenation. i Represents the corresponding i-th layer intermediate features, ConvBlocki represents the i-th convolution module of the decoder;

[0078] Feature up=UpSample(Feature se )

[0079] Feature cat =Concat(Feature up , Shortcuts i )

[0080] Feature conv1 =ConvBlocki(Feature cat )

[0081] The intermediate result also includes the error map and hidden feature hidden of the low-definition image. The first loss is:

[0082]

[0083] The first loss includes the loss of channel transparency map Gradient loss and Laplace loss Loss of foreground image and Laplace loss And the loss L of the error graph err ;

[0084]

[0085] Use L1 loss to measure the predicted low-definition channel transparency image α lr And the foreground image label of the low-definition image is GroundTruth The difference between them, i represents the pixel position;

[0086]

[0087] in represents the gradient;

[0088]

[0089] in represents the pyramid-seeking system, and s represents the pyramid level;

[0090]

[0091]

[0092] Represents the foreground image label of the low-definition image Ground Truth, F lrRepresents the predicted low-definition foreground image, and the standard L2 loss is used for the error map, where err represents the predicted error map. Represents the Ground Truth of the error graph;

[0093]

[0094]

[0095] The output of the second stage model at high resolution is F hr , α hr As shown in the following formula, DGF represents the depth-guided filtering module used, F lr , α lr , hidden represents the low-definition intermediate result:

[0096] F hr , α hr =DGF(C, C′, F lr , α lr ,hidden)

[0097] The second loss L hr As shown below, except for the first loss L lr In addition, it also includes the loss of channel transparency maps for high-resolution output results Gradient loss and Laplace loss Loss of foreground image and Laplace loss

[0098]

[0099] in:

[0100]

[0101]

[0102]

[0103]

[0104]

[0105] The present invention also proposes a storage medium for storing a program for executing any one of the high-resolution real-time automatic green screen matting methods based on the attention mechanism.

[0106] The present invention also proposes a client for use in any one of the aforementioned high-resolution, real-time, automatic green screen matting systems based on an attention mechanism.

[0107] From the above scheme, it can be seen that the advantages of the present invention are:

[0108] The present invention is particularly effective for real-time automatic green screen image matting at high resolution. The qualitative comparison with the latest matting methods CF, KNN, FBA, MODNet, LFP, BGMV2 (mobile), and AIM is listed in Figure 2 The quantitative comparison is listed in Table 1. A large number of comparative experiments show the effectiveness of the proposed scheme.

[0109] The present invention is compared with the above seven matting methods, among which CF and KNN are traditional methods for matting, both of which are based on trimap, and the remaining five methods are based on deep learning, among which FBA and LFP are based on trimap, BGMV2 (mobile) is based on background, MODNet and AIM are automatic matting methods. Figure 2 As shown, the CF method performs poorly for areas with holes, while the KNN method performs better but is unable to remove green spill. The FBA and LFP methods achieve more refined results, but they also struggle to remove green spill. Furthermore, since they require trimap as an auxiliary input, they cannot be used for video matting and are difficult to process in real time. The BGMV2 (mobile) method relies on the quality of the input background, which is subject to significant limitations. The AIM and MODNet methods are automatic, but they perform poorly and struggle to achieve high-quality results. MODNet only performs matting on portraits, which is a significant limitation. This method, however, can achieve realistic green screen matting effects. As shown in Table 1, the comparative experiments used multiple metrics. This method performed better across all metrics. Although the FBA and LFP methods outperformed this method, both methods are trimap-based, require additional manual intervention, are not real-time, and cannot remove green spill. Furthermore, as shown in Table 2, these two models have large parameter sizes and slow processing speeds. In terms of speed, as shown in Table 1, compared to other deep learning-based methods, this method has fewer model parameters and can achieve a keying speed of approximately 80 fps at 2K (the algorithm is running on a server configured with an Nvidia GeForce RTX 3090). In summary, the present invention's solution: a high-resolution, real-time, automatic green screen keying method based on an attention mechanism is very effective.

[0110] Table 1 Quantitative comparison with other matting methods

[0111]

[0112] Table 2 Comparison of model parameters and sizes

[0113] BRIEF DESCRIPTION OF THE DRAWINGS

[0114] Figure 1 It is the network model structure diagram;

[0115] Figure 2 Qualitative comparison of the matting method of the present invention and other matting methods. DETAILED DESCRIPTION

[0116] The purpose of this invention is to solve the problem that most existing methods require human participation, are difficult to automatically process, and have slow processing speed at high resolution. It provides a high-resolution, real-time, automatic green screen matting method based on an attention mechanism, including:

[0117] Step 1: read the input green screen image or green screen video frame and downsample it;

[0118] Step 2: Build a feature encoder and perform feature extraction on the downsampled green screen image;

[0119] Step 3: Add the dilated spatial convolutional pooling pyramid ASPP module and the attention SE module after the encoder to further process the extracted features;

[0120] Step 4: Build a feature decoder to process the acquired feature information to obtain high-level feature information rich in semantic information and object matting, and restore the feature map to the resolution after downsampling in step 1 through upsampling;

[0121] Step 5: The feature encoder, ASPP module, attention module, and feature decoder form the first-stage training model. The model outputs intermediate results and is trained using the first-stage loss function.

[0122] Step 6: The second stage of the model performs high-resolution processing on the intermediate results output by the first stage to restore the original resolution;

[0123] Step 7: Train the second-stage model and the DGF module, which will also update the first-stage training model and obtain the final green screen keying result;

[0124] During the training phase, the green screen image or green screen video frame in step 1 refers to a synthesized green screen image. Specifically, the green screen image or green screen video frame is synthesized by randomly selecting a corresponding foreground image and alpha image with green overflow from a dedicated green screen dataset and randomly selecting a green screen background image. During the testing phase, the green screen image or green screen video frame is actually captured by the camera. The downsampling method uses a bilinear method to reduce the resolution to half of the original value.

[0125] Building the feature encoder in step 2 refers to building an encoder structure based on the MobileNetV2 model officially provided by PyTorch. The encoder can also play a downsampling role, and its structure contains multiple convolutional layers, retaining the features after each convolutional layer as intermediate features for subsequent jump connections.

[0126] The ASPP module in step 3 is a dilated spatial convolutional pooling pyramid, which increases the network receptive field without changing the resolution. The attention module uses SENet, which enables the model to focus on the relationship between channels and automatically learn the importance of different channel features.

[0127] Building the feature decoder in step 4 means building four convolutional layers, each of which upsamples the features of the previous layer and connects (jumps) to the intermediate features corresponding to step 2. The upsampling uses a bilinear method to double the resolution.

[0128] The intermediate results in step 5 refer to the foreground image, alpha image, error image and hidden feature after removing green overflow at low resolution. The loss function refers to the loss, gradient loss and Laplace loss of the alpha image, the loss and Laplace loss of the foreground image and the loss of the error image, wherein the error image helps to improve the precision of the matte. The foreground image refers to the target image. The alpha image refers to Figure 2 In the black and white image, error refers to the difference between the predicted Alpha image and the Alpha image Ground Truth. The implicit feature refers to the redundant features of the output, which are used for subsequent high-resolution processing.

[0129] The high-resolution processing in step 6 refers to adding a depth-guided filter module (DGF) as a high-resolution processing module after the first-stage model.

[0130] Except for the loss function used in step 7 for training the second stage model, each loss term is also calculated for training at high resolution. The final green screen matting result is the foreground image and the corresponding alpha image after removing the green overflow at the original resolution.

[0131] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0132] This implementation is a high-resolution, real-time, automatic green screen keying method based on the attention mechanism. The main experimental environment is a GPU: Nvidia GeForce RTX 3090, PyTorch: 1.7.0, Python: 3.6. Follow the steps below to implement it:

[0133] Step 1:

[0134] Randomly select a foreground image F with green overflow and the corresponding alpha image α from the green screen matting dataset containing green overflow, and randomly select a background image B from the green screen background dataset. Unify the resolution of the three to (H, W), and then generate the composite image C according to the following synthesis formula:

[0135] C=F×α+B(1-α) (1)

[0136] The synthetic image is downsampled to half its original resolution, that is, the resolution is (0.5H, 0.5W). The downsampling formula is as follows, where DownSample() in C represents the downsampling method used, and C′ represents the image after downsampling. This method uses C′ as the input to build the network model.

[0137] C′=DownSample(C) (2)

[0138] Step 2:

[0139] Build the first-stage network model, specifically for green screen keying at low resolution. Use a lightweight MobileNet with an attention mechanism as the feature extraction encoder. Then, build a decoder corresponding to the encoder. The decoder uses skip connections to combine the encoder's features with those of the previous decoder as input. The decoder also outputs a foreground image, the corresponding alpha map, an error map, and hidden features for subsequent use.

[0140] a. Feature encoding

[0141] To achieve real-time processing speed, the feature encoder module uses a lightweight MobileNet as a feature extractor to improve the feature extraction speed while ensuring the quality of feature extraction. This method uses the MobileNetV2 model officially released by PyTorch and modifies it. The modifications include using expansion on the last module to keep the output stride at 16 and deleting the classifier module originally used for classification. As shown in the following formula, Feature m Represents the features extracted after multiple convolution modules of MobileNetV2, and Shortcuts represents the intermediate features of each convolution block for subsequent jump connection use.

[0142] Feature m ,Shortscuts=MobileNetV2(C′) (3)

[0143] To enhance the network model's ability to obtain multi-scale context, the ASPP module is added after the MobileNet encoder. The ASPP module refers to the dilated spatial convolution pooling pyramid, which samples the given input with dilated convolutions at different sampling rates in parallel, which is equivalent to capturing the context of the image at multiple scales, that is, increasing the network receptive field without changing the resolution. As shown in the following formula, where Feature aspp Represents the features after the dilated spatial convolution pooling pyramid.

[0144] Feature aspp =ASPP(Feature m ) (4)

[0145] b. Attention Mechanism

[0146] In order to achieve a more precise matting effect and accurately extract the foreground target object, the network model proposed in this method adds an attention mechanism after the initial feature extraction. The basic idea of the attention mechanism in computer vision is to enable the model to ignore irrelevant information and focus on key information. There are many methods of attention mechanism. This method uses SENet (Squeeze-and-Excitation Network) to extract attention features. The SENet module focuses on the relationship between channels, hoping that the model can automatically learn the importance of different channel features. A SENet module can be divided into two steps: compression and excitation: compression is performed by performing Global Average Pooling on the Feature Map layer to obtain the global compression feature of the current Feature Map; excitation is performed through a two-layer fully connected bottleneck structure to obtain the weight of each channel in the Feature Map, and the weighted Feature Map is used as the input of the next layer of the network. As shown in the following formula, Feature a Represents the features after attention extraction.

[0147] Feature se =SE(Feature aspp ) (5)

[0148] c. Feature decoding

[0149] The decoder network consists of four convolutional layers. Except for the last layer, each convolutional layer is followed by a batch normalization layer and a ReLU activation function. Before each convolutional layer, this scheme uses bilinear upsampling and concatenates it with the skip connection features from the encoder. Taking the first convolutional layer of the decoder as an example, as shown in the following formula, UpSample() represents the upsampling method used, Concat represents feature concatenation, and Shortcuts irepresents the encoder features of the corresponding skip connection, and ConvBlock1 represents the first convolutional module of the decoder.

[0150] Feature up =UpSample(Feature se ) (6)

[0151] Feature cat =Concat(Feature p , Shortcuts i ) (7)

[0152]

[0153] d. Model output and loss function

[0154] The decoder outputs four items: the foreground image after removing the green overflow, the corresponding alpha map, the error map, and the hidden feature for subsequent use. The output resolution is all at a low resolution (0.5H, 0.5W). The corresponding loss function is as follows:

[0155] Loss function L lr for:

[0156]

[0157] in:

[0158] For the Alpha graph, first, L1 loss is used to measure the entire Alpha graph α and its Ground Truth The difference between , i represents the pixel position.

[0159]

[0160] The second type of loss is called gradient loss.

[0161]

[0162] The third loss is the pyramid Laplace loss.

[0163]

[0164] For the predicted foreground F, the standard L1 loss and Laplace loss are also used. Represents Ground Truth. Only the loss of the foreground is calculated, which means that Ground Truth Greater than 0.

[0165]

[0166]

[0167] For the error map, the standard L2 loss is used. Represents the GroundTruth of the error map.

[0168]

[0169] Step 3:

[0170] Training is performed based on the first-stage model. After the model converges, a high-resolution processing module DGF is added to the first-stage model and training continues until convergence.

[0171] a. First stage model training

[0172] The first-stage network model built in step 2 is trained. The training dataset is a green screen keying dataset and a green screen background dataset. The ground truth of the green screen keying dataset includes the original green screen image, the foreground image after removing the green overflow, the corresponding alpha image, and the foreground image with green overflow. This method performs random data augmentation on the foreground image with green overflow, the corresponding alpha image, and the randomly selected green screen background image and then synthesizes them. The data augmentation methods include flipping, rotation, translation, chromaticity transformation, brightness transformation, saturation transformation, etc. The synthesized image is downsampled as needed and then input into the first-stage network model for training until convergence.

[0173] b. High-resolution module optimization

[0174] The first-stage network model can implement an automatic green screen keying method based on the attention mechanism. However, it struggles to achieve real-time processing speed when processing high-resolution footage. Therefore, this method divides model training into two stages. The first stage performs automatic green screen keying at low resolution. The second stage, based on the convergence of the first stage, adds a lightweight high-resolution processing module DGF to restore the low-resolution processing results to high resolution, thus achieving real-time automatic green screen keying at high resolution.

[0175] The high-resolution processing module uses a deep guided filter (DGF) module. The traditional guided filter algorithm can not only achieve edge smoothing of bilateral filtering, but also has good performance near the detected edge. It can be applied to scenes such as image enhancement, HDR compression, image matting, and image defogging. Deep guided filtering is a guided filtering method based on deep learning. It can be used to effectively generate high-resolution output under the corresponding low-resolution output and high-resolution guidance map. As shown in the following formula, DGF represents the depth guided filter module used, and Flr , α lr represents the output of the first stage model at low resolution, F hr , α hr Represents the model output results of the second stage at high resolution.

[0176] F hr , α hr =DGF(C, C′, F lr , α lr ,hidden) (16)

[0177] c. Model output and loss function

[0178] The output of the second-stage model is the foreground image with green overflow removed and the corresponding alpha image at high resolution. In addition to the loss function of the first stage, the loss function also includes several loss functions at high resolution, as shown in the following formula.

[0179]

[0180]

[0181]

[0182]

[0183]

[0184] Loss function L hr for:

[0185]

[0186] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiment.

[0187] The present invention also proposes a high-resolution real-time automatic green screen matting system based on an attention mechanism, which includes:

[0188] Module 1 is used to build a neural network model including a feature encoder, a dilated spatial convolutional pooling pyramid module, an attention module, and a feature decoder;

[0189] Module 2 is used to enable the feature encoder to downsample the training image to obtain a low-definition image, and extract image features from the low-definition image, and generate intermediate features during the extraction process;

[0190] Module 3 is used to enable the dilated spatial convolution pooling pyramid module to sample the image features in parallel using dilated convolutions at different sampling rates, and the attention module to extract features from the sampling results to obtain attention features;

[0191] Module 4 is configured to enable the feature decoder to decode the attention feature according to the intermediate feature, to obtain an intermediate result including a foreground image of the low-definition image after eliminating green overflow and the low-definition channel transparency image;

[0192] Module 5 is used to train the neural network model by constructing a first loss based on the intermediate result using the foreground image label and the channel transparency image label of the low-definition image as training targets;

[0193] Module 6 is used to add a high-resolution processing module to the output of the neural network model after training to obtain a matting model. The high-resolution processing module restores the intermediate result to the same resolution as the training image to obtain a matting result including the foreground image of the training image after eliminating green overflow and the channel transparent image of the training image;

[0194] Module 7 is used to train the matting model by using the foreground image label and channel transparency image label of the training image as training targets, constructing a second loss based on the matting result, and inputting the green screen image into the trained matting model to obtain the foreground image and channel transparency image as the matting result of the green screen image.

[0195] The high-resolution real-time automatic green screen matting system based on the attention mechanism generates the training image by selecting the foreground image with green overflow and the corresponding channel transparent image in the green screen dataset, and synthesizing them with the green screen background image to obtain the training image.

[0196] The high-resolution, real-time, automatic green screen matting system based on the attention mechanism, wherein the feature encoder includes multiple convolutional layers, retaining the features after each convolutional layer as the intermediate features; the feature decoder includes multiple convolutional layers, each convolutional layer upsamples the features of the previous layer and connects them with the intermediate features;

[0197] The intermediate result also includes the error map and hidden feature hidden of the low-definition image. The first loss includes L1 loss, gradient loss and Laplace loss for the channel transparency map, L1 loss and Laplace loss for the foreground map, and L2 loss for the error map.

[0198] The high-resolution, real-time, automatic green screen matting system based on the attention mechanism is described. The training image generation process is as follows: a foreground image F with green overflow and a corresponding channel transparency image α are randomly selected from a green screen matting dataset containing green overflow, and a background image B is randomly selected from a green screen background dataset. After unifying the resolution of the three, a composite image C is generated according to the synthesis formula:

[0199] C=F×α+B(1-α)

[0200] Downsampling the composite image C to obtain a low-resolution image C′;

[0201] The feature extraction process of the feature encoder is shown in the following formula, where Feature m Represents the image features extracted after passing through multiple convolution modules in the feature encoder, and Shortcuts represents the intermediate features of each convolution block;

[0202] Feature m ,Shortscuts=MobileNetV2(C′)

[0203] The sampling of the dilated spatial convolution pooling pyramid module is:

[0204] Feature aspp =ASPP(Feature m )

[0205] Feature aspp Represents the sampling result output by the dilated spatial convolutional pooling pyramid module;

[0206] The attention module extracts features through the following formula: a Represents the attention feature:

[0207] Feature se =SE(Feature aspp )

[0208] The feature decoder includes multiple convolutional layers. Except for the last layer, each convolutional layer is followed by a BN layer and a ReLU activation function. Before each convolutional layer, bilinear upsampling is used and concatenated with the intermediate features from the feature encoder. The decoding process of the feature decoder is shown in the following formula, where UpSample() represents upsampling, Concat represents feature concatenation, and Sharpcuts represents feature concatenation. i Represents the corresponding i-th layer intermediate features, ConvBlocki represents the i-th convolution module of the decoder;

[0209] Feature up=UpSample(Feature se )

[0210] Feature cat =Concat(Feature up , Shortcuts i )

[0211] Feature conv1 =ConvBlocki(Feature cat )

[0212] The intermediate result also includes the error map and hidden feature hidden of the low-definition image. The first loss is:

[0213]

[0214] The first loss includes the loss of channel transparency map Gradient loss and Laplace loss Loss of foreground image and Laplace loss And the loss L of the error graph err ;

[0215]

[0216] Use L1 loss to measure the predicted low-definition channel transparency image α lr And the foreground image label of the low-definition image is GroundTruth The difference between them, i represents the pixel position;

[0217]

[0218] in represents the gradient;

[0219]

[0220] in represents the pyramid-seeking system, and s represents the pyramid level;

[0221]

[0222]

[0223] Represents the foreground image label of the low-definition image Ground Truth, F lrRepresents the predicted low-definition foreground image, and the standard L2 loss is used for the error map, where err represents the predicted error map. Represents the Ground Truth of the error graph;

[0224]

[0225]

[0226] The output of the second stage model at high resolution is F hr , α hr As shown in the following formula, DGF represents the depth-guided filtering module used, F lr , α lr , hidden represents the low-definition intermediate result:

[0227] F hr , α hr =DGF(C, C′, F l r,α lr ,hidden)

[0228] The second loss L hr As shown below, except for the first loss L lr In addition, it also includes the loss of channel transparency maps for high-resolution output results Gradient loss and Laplace loss Loss of foreground image and Laplace loss

[0229]

[0230] in:

[0231]

[0232]

[0233]

[0234]

[0235]

[0236] The present invention also proposes a storage medium for storing a program for executing any one of the high-resolution real-time automatic green screen matting methods based on the attention mechanism.

[0237] The present invention also proposes a client for use in any one of the aforementioned high-resolution, real-time, automatic green screen matting systems based on an attention mechanism.

Claims

1. A high-resolution real-time automatic green screen matting method based on attention mechanism, characterized in that: include: Step 1: Build a neural network model including a feature encoder, a dilated spatial convolutional pooling pyramid module, an attention module, and a feature decoder; Step 2: The feature encoder downsamples the training image to obtain a low-definition image, extracts image features from the low-definition image, and generates intermediate features during the extraction process; Step 3: The dilated spatial convolution pooling pyramid module samples the image features in parallel using dilated convolutions at different sampling rates, and the attention module extracts features from the sampling results to obtain attention features. Step 4: The feature decoder decodes the attention feature according to the intermediate feature to obtain an intermediate result including a foreground image of the low-definition image after eliminating green overflow and the low-definition channel transparency image; Step 5: Using the foreground image label and the channel transparency image label of the low-definition image as training targets, constructing a first loss based on the intermediate result, and training the neural network model; Step 6: Add a high-resolution processing module to the output of the neural network model after training to obtain a matting model. The high-resolution processing module restores the intermediate result to the same resolution as the training image to obtain a matting result including the foreground image of the training image after eliminating green overflow and the channel transparent image of the training image; Step 7: Using the foreground image label and channel transparency image label of the training image as training targets, constructing a second loss based on the matting result, and training the matting model; inputting the green screen image into the trained matting model to obtain the foreground image and channel transparency image as the matting result of the green screen image; The intermediate result also includes the error map and hidden feature hidden of the low-definition image. The first loss includes L1 loss, gradient loss and Laplace loss for the channel transparency map, L1 loss and Laplace loss for the foreground map, and L2 loss for the error map. In addition to the first loss, the second loss also includes loss, gradient loss and Laplace loss for the channel transparency map of the high-resolution output result, and loss and Laplace loss for the foreground map.

2. In the high-resolution, real-time, automatic green screen matting method based on the attention mechanism as described in claim 1, the training image is generated by selecting a foreground image with green overflow and a corresponding channel transparent image from the green screen dataset, and synthesizing them with the green screen background image to obtain the training image.

3. The high-resolution, real-time, automatic green screen matting method based on the attention mechanism according to claim 1, characterized in that: The feature encoder includes multiple convolutional layers, and the features after each convolutional layer are retained as the intermediate features; the feature decoder includes multiple convolutional layers, and each convolutional layer upsamples the features of the previous layer and connects them with the intermediate features.

4. The high-resolution, real-time, automatic green screen matting method based on the attention mechanism according to claim 1, wherein: The training image generation process is to randomly select a foreground image F with green overflow and the corresponding channel transparency image α from the green screen matting dataset containing green overflow, and randomly select a background image B from the green screen background dataset. After unifying the resolution of the three, the composite image C is generated according to the synthesis formula: C=F×α+B(1-α) Downsampling the composite image C to obtain a low-resolution image C'; The feature extraction process of the feature encoder is shown in the following formula, where Feature m Represents the image features extracted after passing through multiple convolution modules in the feature encoder, and Shortcuts represents the intermediate features of each convolution block; Feature m ,Shortscuts=MobileNetV2(C') The sampling of the dilated spatial convolution pooling pyramid module is: Feature aspp =ASPP(Feature m ) Feature aspp Represents the sampling result output by the dilated spatial convolutional pooling pyramid module; The attention module extracts features through the following formula: se Represents the attention feature: Feature se =SE(Feature aspp ) The feature decoder consists of multiple convolutional layers. Except for the last layer, each convolutional layer is followed by a BN layer and a ReLU activation function. Before each convolutional layer, bilinear upsampling is used and concatenated with the intermediate features from the feature encoder. The decoding process of the feature decoder is shown in the following formula, where UpSample() represents upsampling, Concat represents feature concatenation, and Shortcuts i Represents the corresponding i-th layer intermediate features, ConvBlocki represents the i-th convolution module of the decoder; Feature up =UpSample(Feature se ) Feature cat =Concat(Feature up ,Shortcuts i ) Feature conv1 =ConvBlocki(Feature cat ) The intermediate result also includes the error map and hidden feature hidden of the low-definition image. The first loss is: The first loss includes the loss of channel transparency map Gradient loss and Laplace loss Loss of foreground image and Laplace loss And the loss L of the error graph err ; Use L1 loss to measure the predicted low-definition channel transparency image α lr And the foreground image label of the low-definition image is GroundTruth The difference between them, i represents the pixel position; in represents the gradient; in It represents the method of finding the pyramid, and s represents the pyramid level; Represents the foreground image label of the low-definition image Ground Truth, F lr Represents the predicted low-definition foreground image, and the standard L2 loss is used for the error map, where err represents the predicted error map. Represents the Ground Truth of the error graph; The output of the second stage model at high resolution is F hr ,α hr , as shown in the following formula, where DGF represents the depth-guided filtering module adopted, F lr ,α lr , hidden represents the low-definition intermediate result: F hr ,a hr =DGF(C,C',F lr ,a lr (hidden) The second loss L hr As shown below, L lr is the first loss, and also includes the high-resolution output results is the loss of channel transparency map, is the gradient loss and is the Laplace loss, is the loss of the foreground image and is the Laplace loss: in:

5. A high-resolution real-time automatic green screen keying system based on attention mechanism, characterized by: include: Module 1 is used to build a neural network model including a feature encoder, a dilated spatial convolutional pooling pyramid module, an attention module, and a feature decoder; Module 2 is used to enable the feature encoder to downsample the training image to obtain a low-definition image, and extract image features from the low-definition image, and generate intermediate features during the extraction process; Module 3 is used to enable the dilated spatial convolution pooling pyramid module to sample the image features in parallel using dilated convolutions at different sampling rates, and the attention module to extract features from the sampling results to obtain attention features; Module 4 is configured to enable the feature decoder to decode the attention feature according to the intermediate feature, to obtain an intermediate result including a foreground image of the low-definition image after eliminating green overflow and the low-definition channel transparency image; Module 5 is used to train the neural network model by constructing a first loss based on the intermediate result using the foreground image label and the channel transparency image label of the low-definition image as training targets; Module 6 is used to add a high-resolution processing module to the output of the neural network model after training to obtain a matting model. The high-resolution processing module restores the intermediate result to the same resolution as the training image to obtain a matting result including the foreground image of the training image after eliminating green overflow and the channel transparent image of the training image; Module 7 is configured to train the matting model using the foreground image label and the channel transparency image label of the training image as training targets, constructing a second loss based on the matting result, and inputting the green screen image into the trained matting model to obtain the foreground image and the channel transparency image as the matting result of the green screen image; The intermediate result also includes the error map and hidden feature hidden of the low-definition image. The first loss includes L1 loss, gradient loss and Laplace loss for the channel transparency map, L1 loss and Laplace loss for the foreground map, and L2 loss for the error map. In addition to the first loss, the second loss also includes loss, gradient loss and Laplace loss for the channel transparency map of the high-resolution output result, and loss and Laplace loss for the foreground map.

6. In the high-resolution, real-time, automatic green screen matting system based on the attention mechanism as described in claim 1, the training image is generated by selecting a foreground image with green overflow and a corresponding channel transparent image from a green screen dataset, and synthesizing them with a green screen background image to obtain the training image.

7. The high-resolution, real-time, automatic green screen matting system based on the attention mechanism according to claim 1, wherein: The feature encoder includes multiple convolutional layers, and the features after each convolutional layer are retained as the intermediate features; the feature decoder includes multiple convolutional layers, and each convolutional layer upsamples the features of the previous layer and connects them with the intermediate features.

8. The high-resolution, real-time, automatic green screen matting system based on the attention mechanism according to claim 1, wherein: The training image generation process is to randomly select a foreground image F with green overflow and the corresponding channel transparency image α from the green screen matting dataset containing green overflow, and randomly select a background image B from the green screen background dataset. After unifying the resolution of the three, the composite image C is generated according to the synthesis formula: C=F×α+B(1-α) Downsampling the composite image C to obtain a low-resolution image C'; The feature extraction process of the feature encoder is shown in the following formula, where Feature m Represents the image features extracted after passing through multiple convolution modules in the feature encoder, and Shortcuts represents the intermediate features of each convolution block; Feature m ,Shortscuts=MobileNetV2(C') The sampling of the dilated spatial convolution pooling pyramid module is: Feature aspp =ASPP(Feature m ) Feature aspp Represents the sampling result output by the dilated spatial convolutional pooling pyramid module; The attention module extracts features through the following formula: se Represents the attention feature: Feature se =SE(Feature aspp ) The feature decoder consists of multiple convolutional layers. Except for the last layer, each convolutional layer is followed by a BN layer and a ReLU activation function. Before each convolutional layer, bilinear upsampling is used and concatenated with the intermediate features from the feature encoder. The decoding process of the feature decoder is shown in the following formula, where UpSample() represents upsampling, Concat represents feature concatenation, and Shortcuts i Represents the corresponding i-th layer intermediate features, ConvBlocki represents the i-th convolution module of the decoder; Feature up =UpSample(Feature se ) Feature cat =Concat(Feature up ,Shortcuts i ) Feature conv1 =ConvBlocki(Feature cat ) The intermediate result also includes the error map and hidden feature hidden of the low-definition image. The first loss is: The first loss includes the loss of channel transparency map Gradient loss and Laplace loss Loss of foreground image and Laplace loss And the loss L of the error graph err ; Use L1 loss to measure the predicted low-definition channel transparency image α lr And the foreground image label of the low-definition image is GroundTruth The difference between them, i represents the pixel position; in represents the gradient; in represents the pyramid-seeking system, and s represents the pyramid level; Represents the foreground image label of the low-definition image Ground Truth, F lr Represents the predicted low-definition foreground image, and the standard L2 loss is used for the error map, where err represents the predicted error map. Represents the Ground Truth of the error graph; The output of the second stage model at high resolution is F hr ,α hr , as shown in the following formula, where DGF represents the depth-guided filtering module adopted, F lr ,α lr , hidden represents the low-definition intermediate result: F hr ,a hr =DGF(C,C',F lr ,a lr (hidden) The second loss L hr As shown below, L lr is the first loss, and also includes the high-resolution output results is the loss of channel transparency map, is the gradient loss and is the Laplace loss, is the loss of the foreground image and is the Laplace loss: in:

9. A storage medium for storing a program for executing any one of the attention mechanism-based high-resolution real-time automatic green screen matting methods as described in claims 1 to 4.

10. A client for use in any one of the attention mechanism-based high-resolution real-time automatic green screen matting systems of claims 5 to 8.

Images