A portrait cutout method based on general attention mechanism ESP-CA

By adopting a portrait cutout method based on the general attention mechanism ESP-CA in portrait cutout, combining the e-ASPP-a module and the ESP-CA module, the problems of low efficiency and insufficient advanced semantic representation ability in the existing technology are solved. , achieving more efficient portrait segmentation effect and performance improvement.

CN118115734BActive Publication Date: 2024-11-12浪潮智能终端有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410106567.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-01-25
Publication Date
2024-11-12
Estimated Expiration
2044-01-25

AI Technical Summary

Technical Problem

The existing portrait cutout method is less efficient in real-time applications, and the three-channel method is limited by insufficient label data and domain offset problems, resulting in poor performance of the model on real-world data and insufficient advanced semantic representation capabilities.

Method used

Using the portrait clap method based on the general attention mechanism ESP-CA, the MODNet-A model is constructed, combined with the e-ASPP-a module and the ESP-CA module, the advanced semantic representation ability is enhanced, the portrait segmentation effect is improved, and the channel attention is taken into account. and global space attention.

Benefits of technology

It significantly improves the performance of portrait cutout tasks, improves the efficiency and quality of neural network feature extraction, reduces network depth, enhances computing efficiency, and provides more intuitive model interpretation capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118115734B_ABST
    Figure CN118115734B_ABST
Patent Text Reader

Abstract

A portrait cutout method based on a general attention mechanism ESP‑CA relates to the field of image processing technology. The constructed e‑ASPP‑a module can enhance the ability of high-level semantic representation and improve the portrait segmentation effect. The ESP‑CA module in the e‑ASPP‑a module can realize the important feature attention that takes into account both channel attention and global spatial attention, improve the efficiency and quality of neural network feature extraction, reduce network depth and significantly improve the performance of tasks such as portrait cutout. The carefully designed weight distribution mechanism in the ESP‑CA module helps to reduce redundant calculations and improve the calculation efficiency of the model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a portrait cutout method based on a general attention mechanism ESP-CA. Background Art

[0002] As an important optimization strategy in the field of deep learning, the attention mechanism plays a key role in improving model performance and efficiency. From its origin to the current development process, it can be seen that the attention mechanism has been widely used. Researchers continue to explore how to improve the attention mechanism to cope with larger and more complex application scenarios. At the same time, the introduction of the attention mechanism also provides new ideas and methods for neural networks to extract effective features and improve recognition accuracy. Compared with convolutional neural networks, the attention mechanism is more similar to the human recognition method, which can make the network pay more attention to important features, thereby improving the accuracy and speed of recognition. Therefore, the attention mechanism is not only of great significance in theory, but also of great value in practical applications. As a simple and easy-to-use plug-in, the attention mechanism can be easily embedded in various convolutional neural networks and object detection networks, providing a simple and effective method for model optimization and improvement. Therefore, the development of the attention mechanism will provide more possibilities for the further development and application of the field of deep learning.

[0003] Currently, neural networks usually use SE modules as attention mechanisms, but ignore the consideration of position information. To address this problem, a new and efficient attention mechanism, Coordinate Attention (CA), is proposed. CA can encode horizontal and vertical position information into Channel Attention, allowing the neural network to pay attention to a wide range of position information while avoiding introducing too much computational burden. The advantages of CA are reflected in the following aspects: First, it not only considers direction-related position information while obtaining inter-channel information, but also helps the model to locate and identify targets more accurately; second, the mechanism is lightweight and flexible, and can be easily embedded in the core structure of the neural network; finally, CA can be applied to a variety of tasks, such as detection, tracking, and semantic segmentation, which can significantly improve performance. The introduction of CA fills the gaps in the existing attention mechanism and provides new possibilities for improving the performance of neural networks.

[0004] In the field of portrait matting, most current methods require auxiliary inputs or involve multiple stages of calculation, which makes them less efficient in real-time applications. In addition, existing three-channel-free methods are often limited by insufficient labeled data and domain shift problems, resulting in poor performance of the model on real-world data. Recently, some deep learning-based portrait matting methods, such as MODNet, use explicit constraints to simultaneously optimize a series of sub-goals as the guiding idea for their efficient design; at the same time, an e-ASPP module is proposed for fusing multi-scale features for semantic estimation to improve the effect of matting. However, there is still room for improvement in the processing method of convolutional fusion of multi-scale feature maps in the e-ASPP module. Summary of the invention

[0005] In order to overcome the shortcomings of the above technologies, the present invention provides a portrait cutout method based on a general attention mechanism ESP-CA, which enhances the ability of high-level semantic representation and improves the portrait segmentation effect.

[0006] The technical solution adopted by the present invention to overcome the technical problems is:

[0007] A portrait cutout method based on a general attention mechanism ESP-CA comprises the following steps:

[0008] S1. Obtain O images containing full-body portraits and half-body portraits as foreground images, and the i-th foreground image is F i , i∈{1,2,...,O}, get N background images, the jth background image is B j , j∈{1,2,...,N};

[0009] S2. Cut out the portraits in O foreground images to obtain O Alpha mask images, where the i-th mask image is α i , i∈{1,2,...,O}, randomly synthesize O foreground images with N background images to obtain an image set P consisting of O×N synthesized images, P={P1,P2,...,P i ,...,P O×N}, P i is the i-th composite image, i∈{1,2,...,O×N};

[0010] S3. Divide the image set P into a training set, a validation set, and a test set;

[0011] S4. Take the i-th synthetic image P in the training set i Perform random data enhancement and preprocessing to obtain the preprocessed image P i ′, i∈{1,2,...,O×N}, all preprocessed images constitute the image set P′, P′={P1′,P2′,...,P i′,...,P O×N '};

[0012] S5. Build the MODNet-A model and transform the i-th preprocessed image P i ′ is input into the MODNet-A model, and the portrait mask FM is output f .

[0013] Preferably, the ratio of full-body portraits to half-body portraits in the O images in step S1 is 5:1, the value of O in step S1 is 3600, the value of N is 40, the resolution of the O images is 1080P, and the resolution of the N background images is greater than or equal to 512P and less than or equal to 4K.

[0014] Preferably, step S2 comprises the following steps:

[0015] S2-1. Use Photoshop software to cut out the portraits in O foreground images to obtain O Alpha mask images, where the i-th mask image is α i ;

[0016] S2-2. Get the i-th foreground image F i Width w i and high h i , get the jth background image as B j Width w j and high h j , through the formula s = min (w i / w j ,h i / h j ) Calculate the image scaling factor s;

[0017] S2-3. Through the formula P i =φ(α i F i )+φ(1-α i )B j The calculation is scaled to (w i / s)×(h i / s) size of the composite image P i , where φ(·) is the resize function in OpenCV. Further, in step S3, the image set P is divided into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0018] Further, step S4 includes the following steps:

[0019] S4-1. Take the i-th synthetic image P in the training set iThe enhanced image P is obtained by using the ColorJitter function in Python and then performing random cropping, random scaling, random translation, and horizontal flipping operations in sequence. i z ;

[0020] S4-2. The enhanced image P i z Use Python to perform tensor transformation, normalization, dimension expansion, and dimension inversion operations in sequence to obtain the preprocessed image P i ′.

[0021] Further, step S5 includes the following steps:

[0022] S5-1. Construct a MODNet-A model consisting of the Encoder encoder of the MODNet model S branch and the e-ASPP-a module;

[0023] S5-2. The i-th preprocessed image P i ′ is input into the Encoder encoder of the S branch of the MODNet model, and the Shallow feature FM is output latent , is the real number space, c is the number of channels, and h is the characteristic FM latent The height of w is the characteristic FM latent width;

[0024] S5-3. The feature FM latent Input to the first convolutional layer of the e-ASPP-a module, and output the feature FM1.

[0025] S5-4. Split feature FM1 into c / 4 branches according to the number of channels c, and the i-th branch is FM 2-i , i∈{1,2,...,c / 4}, The i-th branch FM 2-i They are respectively input into the M dilated convolutional layers of the e-ASPP-a module, and the output results of the M dilated convolutional layers are concatenated to obtain the feature FM 3-n ,

[0026] S5-5. Feature FM 3-n Input into the ESP-CA module of the e-ASPP-a module, and output the weighted feature FM 4-n ,

[0027] S5-6. Feature FM 4-nInput to the second convolutional layer of the e-ASPP-a module, and output the intermediate result FM of the depthwise separable convolution 5-n ,

[0028] S5-7. Concatenate the intermediate results of the depthwise separable convolution of c / 4 branches to obtain feature FM6.

[0029] S5-8. Input feature FM6 into the third convolutional layer of the e-ASPP-a module, and output the semantically estimated portrait mask FM f .

[0030] Further, step S5-5 includes the following steps:

[0031] S5-5-1. FM 3-n Perform average pooling along the x direction of the feature space to obtain the feature FM x , FM 3-n Perform average pooling along the y direction of the feature space to obtain the feature FM y ,

[0032] S5-5-2. Set the feature FM x Perform an inversion operation on the width and height dimensions to obtain the inverted feature FM x ′, the feature FM x ′ and feature FM y After splicing operation according to high latitude, feature FM is obtained c1 ,

[0033] S5-5-3. Set the feature FM c1 All dimensions of are inverted to obtain the feature FM c2 ,

[0034] S5-5-4. Set the feature FM c2 Input into ESP Block, and output the characteristic FM esp ,

[0035] S5-5-5. Set the feature FM esp All dimensions of are inverted and then disassembled to obtain the feature FM c3 and feature FM c4 ,

[0036] S5-5-6. Through Formula FM mid =FM c4 ⊙FM3-n Calculate the characteristic FM mid , where ⊙ is the Hadamard product;

[0037] S5-5-7. Set the feature FM mid Perform global average pooling operation to obtain feature FM s-a , The feature FM mid Perform global maximum pooling operation to obtain feature FM s-m ,

[0038] S5-5-8. Set the feature FM s-a Input it into the 2-layer perceptron shared MLP for squeeze-excitation operation, and input the result of the squeeze-excitation operation into the SiLU activation function to obtain the feature FM s1 , The feature FM s-m Input it into the 2-layer perceptron shared MLP for squeeze-excitation operation, and input the result of the squeeze-excitation operation into the SiLU activation function to obtain the feature FM s2 ,

[0039] S5-5-9. Set the feature FM s1 With the characteristic FM s2 Perform Hadamard product operation to obtain weighted feature FM 4-n .

[0040] Preferably, the convolution kernel size of the first convolution layer of the e-ASPP-a module is 1×1, the convolution kernel size of the second convolution layer of the e-ASPP-a module is 1×1, and the convolution kernel size of the third convolution layer of the e-ASPP-a module is 1×1.

[0041] Further, step S5-5-4 includes the following steps:

[0042] S5-5-4-1.ESP Block consists of the first convolutional layer, encoder-1 module, encoder-2 module, latent module, decoder-1 module, decoder-2 module, sim module, decoder-up module, and the second convolutional layer;

[0043] S5-5-4-2. The encode-1 module of ESP Block is composed of convolutional layers, which transforms the feature FM c2 After being input into the first convolution layer of the ESP Block, it is input into the encoder-1 module and processed by (h+w) / s1 1×1 convolution kernels to obtain the feature FM. e1 , s1 is a hyperparameter,

[0044] S5-5-4-3. Set the feature FM e1 The input is sent to the encoder-2 module and processed by (h+w) / s2 1×1 convolution kernels to obtain the feature FM e2 , s2 is a hyperparameter,

[0045] S5-5-4-4. Set the feature FM e2 The input is processed by (h+w) / s3 1×1 convolution kernels to obtain the feature FM. la , s3 is a hyperparameter,

[0046] S5-5-4-5. Set the feature FM la The input is sent to the decode-1 module and processed by (h+w) / s2 1×1 convolution kernels to obtain the feature FM d1 ,

[0047] S5-5-4-6. Set the feature FM d1 The input is sent to the decode-2 module and processed by (h+w) / s1 1×1 convolution kernels to obtain the feature FM d2 ,

[0048] S5-5-4-7. Set the feature FM d2 The input is processed by (h+w) 1×1 convolution kernels to obtain the feature FM. sim ,

[0049] S5-5-4-8. Set the feature FM sim The input is sent to the decode-up module and processed by (h×w+h+w) / 2 1×1 convolution kernels to obtain the feature FM up ,

[0050] S5-5-4-9. Set the feature FM up The input is sent to the second convolution layer of ESP Block and processed by (h×w) 1×1 convolution kernels to obtain the feature FM. esp .

[0051] Preferably, the convolution kernel size of the first convolution layer of the ESP Block is 1×1.

[0052] The beneficial effects of the present invention are as follows: the e-ASPP-a module constructed by the present invention can enhance the high-level semantic representation capability and improve the portrait segmentation effect. The ESP-CA module in the e-ASPP-a module can realize the important feature attention that takes into account both channel attention and global spatial attention, improve the efficiency and quality of neural network feature extraction, reduce the network depth and significantly improve the performance of tasks such as portrait cutout. The carefully designed weight allocation mechanism in the ESP-CA module helps to reduce redundant calculations and improve the computational efficiency of the model. The design of the attention mechanism has a solid theoretical basis. Compared with the black box operation of the neural network, the attention mechanism is more intuitive in explaining the decision-making process of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a model structure diagram of the e-ASPP-a of the present invention;

[0054] Figure 2 is a structural diagram of the ESP-CA module of the present invention;

[0055] Figure 3 It is a structural diagram of the ESP Block of the present invention. DETAILED DESCRIPTION

[0056] The following is combined with Figure 1 To Attachment Figure 3 The present invention is further described.

[0057] A portrait cutout method based on a general attention mechanism ESP-CA comprises the following steps:

[0058] S1. Obtain O images containing full-body portraits and half-body portraits as foreground images, and the i-th foreground image is F i , i∈{1,2,...,O}, get N background images, the jth background image is B j , j∈{1,2,...,N}.

[0059] S2. Cut out the portraits in O foreground images to obtain O Alpha mask images, where the i-th mask image is α i , i∈{1,2,...,O}, randomly synthesize O foreground images with N background images to obtain an image set P consisting of O×N synthesized images, P={P1,P2,...,P i ,...,P O×N}, P i is the i-th composite image, i∈{1,2,...,O×N}.

[0060] S3. Divide the image set P into a training set, a validation set, and a test set.

[0061] S4. Take the i-th synthetic image P in the training set i Perform random data enhancement and preprocessing to obtain the preprocessed image P i ′, i∈{1,2,...,O×N}, all preprocessed images constitute the image set P′, P′={P1′,P2′,...,P i ′,...,P O×N ′}.

[0062] S5. Build the MODNet-A model and transform the i-th preprocessed image P i ′ is input into the MODNet-A model, and the portrait mask FM is output f .

[0063] It has been verified that although it is the deep shallow feature encoded by the encoder, it is still necessary to pay attention to the semantic position information in the semantic segmentation task. The contribution of the multi-scale feature map in the e-ASPP module of the MODNet network structure is controlled by SE-Block, which only pays attention to the attention on the channel, lacks the position information that is highly correlated with the semantic segmentation task, limits the effective extraction of its semantic information, and lacks the ability of high-level semantic representation. The portrait cutout semantic segmentation task is more sensitive to position information. According to the theoretical description of CA, it takes into account both channel information and spatial direction information, which brings a small improvement in model performance and reduces the complexity of the model. However, the spatial horizontal and vertical direction information it introduces is only coupled with the channel information, and the focus acts on the channel information with direction, and the spatial position information has not received essential attention. In fact, neither the local nor the global spatial self-attention is still extracted, which can be said to be pseudo-spatial attention. In essence, it is still channel attention, but it is embedded with weakly explanatory direction information. The present invention proposes an explicit spatial position coordinate attention mechanism ESP-CA with clear spatial position information based on CA improvement to enhance the ability of extracting spatial global long-range dependency information of the CA mechanism, and applies it to the e-ASPP module of the S (Semantic Estimation) branch of the MODNet network structure to enhance the high-level semantic representation capability and improve the portrait segmentation effect.

[0064] In this embodiment, preferably, the ratio of full-body portraits to half-body portraits in the O images in step S1 is 5:1, the value of O in step S1 is 3600, the value of N is 40, the resolution of the O images is 1080P, and the resolution of the N background images is greater than or equal to 512P and less than or equal to 4K.

[0065] In one embodiment of the present invention, step S2 comprises the following steps:

[0066] S2-1. Use Photoshop software to cut out the portraits in O foreground images to obtain O Alpha mask images, where the i-th mask image is α i .

[0067] S2-2. Get the i-th foreground image F i Width w i and high h i , get the jth background image as B j Width w j and high h j , through the formula s = min (w i / w j ,h i / h j ) calculates the image scaling factor s.

[0068] S2-3. Through the formula P i =φ(α i F i )+φ(1-α i )B j The calculation is scaled to (w i / s)×(h i / s) size of the composite image P i , where φ(·) is the resize function in OpenCV. In one embodiment of the present invention, in step S3, the image set P is divided into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0069] In one embodiment of the present invention, step S3 includes the following steps:

[0070] S4-1. Take the i-th synthetic image P in the training set i The enhanced image P is obtained by using the ColorJitter function in Python and then performing random cropping, random scaling, random translation, and horizontal flipping operations in sequence. i z ;

[0071] S4-2. The enhanced image P i z Use Python to perform tensor transformation, normalization, dimension expansion, and dimension inversion operations in sequence to obtain the preprocessed image P i ′.

[0072] In one embodiment of the present invention, step S5 comprises the following steps:

[0073] S5-1. Construct the MODNet-A model consisting of the Encoder encoder of the MODNet model S branch and the e-ASPP-a module.

[0074] S5-2. The i-th training image P in the training set i Input to the Encoder of the e-ASPP-a module, and output the Shallow feature FM latent , is the real number space, c is the number of channels, and h is the characteristic FM latent The height of w is the characteristic FM latent width.

[0075] S5-3. The feature FM latent The input is sent to the first convolutional layer of the e-ASPP-a module for channel dimension reduction to reduce the number of latent feature channels, and the output is feature FM1.

[0076] S5-4. Split feature FM1 into c / 4 branches according to the number of channels c, and the i-th branch is FM 2-i , i∈{1,2,...,c / 4}, The i-th branch FM 2-i The outputs of the M dilated convolutional layers are respectively input into the e-ASPP-a module for multi-scale feature extraction, and the output results of the M dilated convolutional layers are concatenated to obtain the feature FM 3-n ,

[0077] S5-5. Feature FM 3-n Input into the ESP-CA module of the e-ASPP-a module to obtain the weighted feature FM that takes into account both accurate location information and channel information 4-n ,

[0078] S5-6. Feature FM 4-n Input into the second convolutional layer of the e-ASPP-a module to obtain the intermediate result of the depthwise separable convolution, and output the intermediate result FM of the depthwise separable convolution. 5-n ,

[0079] S5-7. Concatenate the intermediate results of the depthwise separable convolution of c / 4 branches to obtain feature FM6.

[0080] S5-8. Input feature FM6 into the third convolutional layer of the e-ASPP-a module, and output the semantically estimated portrait mask FM f .

[0081] In one embodiment of the present invention, step S5-5 includes the following steps:

[0082] S5-5-1. FM 3-n Perform average pooling along the x direction of the feature space to obtain the feature FM x , FM 3-n Perform average pooling along the y direction of the feature space to obtain the feature FM y , The operation is consistent with CA, but the purpose is different from CA. The operation of CA is to embed directional information on the channel, while the operation of ESP-CA borrows the idea of ​​spatially separable convolution but is different. Spatially separable convolution is achieved by separating the spatial directions of the convolution kernels. For example, 3×3 convolution is split into 3×1 and 1×3 convolutions, while ESP-CA is achieved by separating the global spatial directions of the feature map, ultimately achieving the purpose of reducing the amount of computation and scale of the attention structure.

[0083] S5-5-2. Set the feature FM x Perform an inversion operation on the width and height dimensions to obtain the inverted feature FM x ′, the feature FM x ′ and feature FM y After splicing operation according to high latitude, feature FM is obtained c1 ,

[0084] S5-5-3. Set the feature FM c1 All dimensions of are inverted to obtain the feature FM c2 ,

[0085] S5-5-4. Set the feature FM c2 Input into ESP Block, and output the characteristic FM esp , It is brought into the ESP Block to obtain spatial attention with additional explicit spatial position information and embedded channel information. This module is converted from the channel attention codec of CA to a (h×w)-dimensional spatial attention codec with explicit position information.

[0086] S5-5-5. Set the feature FM esp All dimensions of are inverted and then disassembled to obtain the feature FM c3 and feature FM c4 ,

[0087] S5-5-6. Through Formula FM mid =FM c4 ⊙FM 3-n Calculate the characteristic FM mid , where ⊙ is the Hadamard product.

[0088] S5-5-7. Set the feature FM mid Perform global average pooling operation to obtain feature FM s-a , The feature FM mid Perform global maximum pooling operation to obtain feature FM s-m ,

[0089] S5-5-8. Set the feature FM s-a Input it into the 2-layer perceptron shared MLP for squeeze-excitation operation, and input the result of the squeeze-excitation operation into the SiLU activation function to obtain the feature FM s1 , The feature FM s-m Input it into the 2-layer perceptron shared MLP for squeeze-excitation operation, and input the result of the squeeze-excitation operation into the SiLU activation function to obtain the feature FM s2 , Compared with the sigmoid activation function of CA, the SiLU activation function can avoid the gradient disappearance problem.

[0090] S5-5-9. Set the feature FM s1 With the characteristic FM s2 Perform Hadamard product operation to obtain weighted feature FM 4-n .

[0091] The ESP-CA module performs channel attention on the feature map after spatial attention processing, which has a slightly improved effect compared to the standard CBAM. Therefore, ESP-CA first performs spatial attention and then channel attention calculation. Spatial attention borrows the idea of ​​directional embedding in CA to reduce the amount of calculation, and adapts the style of the decoding network structure to achieve an alternative encoding and decoding structure. Specifically, the encoders in the standard encoding and decoding structure are in the form of space reduction and channel increase or space unchanged and channel reduction. For example, the encoding in CA is a structure of space unchanged and channel reduction. In the spatial attention of ESP-CA, due to the use of the directional embedding method, the input space dimension (h+w) of the 1×1 convolution is much smaller than the actual position encoding amount (h×w). Therefore, it is necessary to increase the channel (actually the number of positions) while keeping the space (actually the number of channels) unchanged, so as to achieve the purpose of introducing explicit position information in the output layer of ESP Block. The channel attention method of CBAM is inherited in the channel attention.

[0092] Preferably in this embodiment, the convolution kernel size of the first convolution layer of the e-ASPP-a module is 1×1, the convolution kernel size of the second convolution layer of the e-ASPP-a module is 1×1, and the convolution kernel size of the third convolution layer of the e-ASPP-a module is 1×1.

[0093] In one embodiment of the present invention, step S5-5-4 includes the following steps:

[0094] S5-5-4-1.ESP Block consists of the first convolutional layer, encode-1 module, encode-2 module, latent module, decode-1 module, decode-2 module, sim module, decode-up module and the second convolutional layer.

[0095] S5-5-4-2. The encode-1 module of ESP Block is composed of convolutional layers, which transforms the feature FM c2 After being input into the first convolution layer of the ESP Block, it is input into the encoder-1 module and processed by (h+w) / s1 1×1 convolution kernels to obtain the feature FM. e1 , s1 is a hyperparameter, This operation implements preliminary spatial direction encoding.

[0096] S5-5-4-3. Set the feature FM e1 The input is sent to the encoder-2 module and processed by (h+w) / s2 1×1 convolution kernels to obtain the feature FM e2 , s2 is a hyperparameter, This operation achieves further spatial direction encoding.

[0097] S5-5-4-4. Set the feature FM e2 The input is processed by (h+w) / s3 1×1 convolution kernels to obtain the feature FM. la , s3 is a hyperparameter, This operation realizes the final spatial direction encoding to obtain the latent feature space.

[0098] S5-5-4-5. Set the feature FM la The input is sent to the decode-1 module and processed by (h+w) / s2 1×1 convolution kernels to obtain the feature FM d1 , This operation achieves preliminary spatial position decoding.

[0099] S5-5-4-6. Set the feature FM d1 The input is sent to the decode-2 module and processed by (h+w) / s1 1×1 convolution kernels to obtain the feature FM d2 , This operation achieves further spatial position decoding.

[0100] S5-5-4-7. Set the feature FM d2 The input is processed by (h+w) 1×1 convolution kernels to obtain the feature FM. sim , This operation ensures that the dimension of the spatial position after decoding is consistent with the initial spatial direction encoding dimension.

[0101] S5-5-4-8. Set the feature FM sim The input is sent to the decode-up module and processed by (h×w+h+w) / 2 1×1 convolution kernels to obtain the feature FM up , This operation is used to achieve the amplification of spatial position and serves as an intermediate representation for encoding precise position information.

[0102] S5-5-4-9. Set the feature FM up The input is sent to the second convolution layer of ESP Block and processed by (h×w) 1×1 convolution kernels to obtain the feature FM. esp This operation achieves the final precise position information encoding.

[0103] In this embodiment, preferably, the convolution kernel size of the first convolution layer of the ESP Block is 1×1. Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A portrait cutout method based on the general attention mechanism ESP-CA, It is characterized in that The steps include: S1. Obtain O images containing full-body portraits and half-body portraits as foreground images, and the i-th foreground image is F i , i∈{1,2,…,O}, get N background images, the jth background image is B j , j∈{1,2,…,N}; S2. Cut out the portraits in O foreground images to obtain O Alpha mask images, where the i-th mask image is α i , i∈{1,2,...,O}, randomly synthesize O foreground images with N background images to obtain an image set P consisting of O×N synthesized images, P={P 1 ,P 2 ,...,P i ,...,P O×N }, P i is the i-th composite image, i∈{1,2,...,O×N}; S3. Divide the image set P into a training set, a validation set, and a test set; S4. Take the i-th synthetic image P in the training set i Perform random data enhancement and preprocessing to obtain the preprocessed image P i ′, i∈{1,2,...,O×N}, all preprocessed images constitute the image set P′, P′={P 1 ′,P 2 ′,...,P i ′,...,P O×N '}; S5. Build the MODNet-A model and transform the i-th preprocessed image P i ′ is input into the MODNet-A model, and the portrait mask FM is output f ; Step S5 includes the following steps: S5-1. Construct a MODNet-A model consisting of the Encoder encoder of the MODNet model S branch and the e-ASPP-a module; S5-2. The i-th preprocessed image P i ′ is input into the Encoder encoder of the S branch of the MODNet model, and the Shallow feature FM is output latent , is the real number space, c is the number of channels, and h is the characteristic FM latent The height of w is the characteristic FM latent width; S5-3. Feature FM latent Input into the first convolutional layer of the e-ASPP-a module, and output the feature FM 1 , S5-4. Feature FM 1 According to the number of channels c, it is split into c / 4 branches, and the i-th branch is FM 2-i , i∈{1,2,...,c / 4}, The i-th branch FM 2-i They are respectively input into the M dilated convolutional layers of the e-ASPP-a module, and the output results of the M dilated convolutional layers are concatenated to obtain the feature FM 3-n , S5-5. Feature FM 3-n Input into the ESP-CA module of the e-ASPP-a module, and output the weighted feature FM 4-n , S5-6. Feature FM 4-n Input to the second convolutional layer of the e-ASPP-a module, and output the intermediate result FM of the depthwise separable convolution 5-n , S5-7. Concatenate the intermediate results of the depthwise separable convolution of c / 4 branches to obtain the feature FM 6 , S5-8. Feature FM 6 Input into the third convolutional layer of the e-ASPP-a module, and output the semantically estimated portrait mask FM f ; Step S5-5 includes the following steps: S5-5-1. FM 3-n Perform average pooling along the x direction of the feature space to obtain the feature FM x , FM 3-n Perform average pooling along the y direction of the feature space to obtain the feature FM y , S5-5-2. Set the feature FM x Perform an inversion operation on the width and height dimensions to obtain the inverted feature FM x ′, the feature FM x ′ and feature FM y After splicing operation according to high latitude, feature FM is obtained c1 , S5-5-3. Set the feature FM c1 All dimensions of are inverted to obtain the feature FM c2 , S5-5-4. Set the feature FM c2 Input into ESP Block, and output the characteristic FM esp , S5-5-5. Set the feature FM esp All dimensions of are inverted and then disassembled to obtain the feature FM c3 and feature FM c4 , S5-5-6. Through Formula FM mid =FM c4 ⊙FM 3-n Calculate the characteristic FM mid , where ⊙ is the Hadamard product; S5-5-7. Set the feature FM mid Perform global average pooling operation to obtain feature FM s-a , The feature FM mid Perform global maximum pooling operation to obtain feature FM s-m , S5-5-8. Set the feature FM s-a Input it into the 2-layer perceptron shared MLP for squeeze-excitation operation, and input the result of the squeeze-excitation operation into the SiLU activation function to obtain the feature FM s1 , The feature FM s-m Input it into the 2-layer perceptron shared MLP for squeeze-excitation operation, and input the result of the squeeze-excitation operation into the SiLU activation function to obtain the feature FM s2 , S5-5-9. Set the feature FM s1 With the characteristic FM s2 Perform Hadamard product operation to obtain weighted feature FM 4-n .

2. The portrait cutout method based on the general attention mechanism ESP-CA according to claim 1, Features: In step S1, the ratio of full-body portraits to half-body portraits in the O images is 5:1, the value of O in step S1 is 3600, the value of N is 40, the resolution of the O images is 1080P, and the resolution of the N background images is greater than or equal to 512P and less than or equal to 4K.

3. The portrait cutout method based on the general attention mechanism ESP-CA according to claim 1, It is characterized in that Step S2 includes the following steps: S2-1. Use Photoshop software to cut out the portraits in O foreground images to obtain O Alpha mask images, where the i-th mask image is α i ; S2-2. Get the i-th foreground image F i Width w i and high h i , get the jth background image as B j Width w j and high h j , through the formula s = min (w i / w j ,h i / h j ) Calculate the image scaling factor s; S2-3. Through the formula P i =φ(α i F i )+φ(1-α i )B j The calculation is scaled to (w i / s)×(h i / s) size of the composite image P i , where φ(·) is the resize function in OpenCV.

4. The portrait cutout method based on the general attention mechanism ESP-CA according to claim 1, Features: In step S3, the image set P is divided into a training set, a validation set, and a test set in a ratio of 8:1:

1.

5. The portrait cutout method based on the general attention mechanism ESP-CA according to claim 1, It is characterized in that Step S4 includes the following steps: S4-1. Take the i-th synthetic image P in the training set i The enhanced image P is obtained by using the ColorJitter function in Python and then performing random cropping, random scaling, random translation, and horizontal flipping operations in sequence. i z ; S4-2. The enhanced image P i z Use Python to perform tensor transformation, normalization, dimension expansion, and dimension inversion operations in sequence to obtain the preprocessed image P i ′.

6. The portrait cutout method based on the general attention mechanism ESP-CA according to claim 1, Features: The convolution kernel size of the first convolution layer of the e-ASPP-a module is 1×1, the convolution kernel size of the second convolution layer of the e-ASPP-a module is 1×1, and the convolution kernel size of the third convolution layer of the e-ASPP-a module is 1×1.

7. The portrait cutout method based on the general attention mechanism ESP-CA according to claim 1, It is characterized in that Step S5-5-4 includes the following steps: S5-5-4-1.ESP Block consists of the first convolutional layer, encoder-1 module, encoder-2 module, latent module, decoder-1 module, decoder-2 module, sim module, decoder-up module, and the second convolutional layer; S5-5-4-2. The encode-1 module of ESP Block is composed of convolutional layers, which transforms the feature FM c2 After being input into the first convolutional layer of the ESP Block, it is input into the encoder-1 module and passes through (h+w) / s 1 1×1 convolution kernel processing to obtain the feature FM e1 ,s 1 is a hyperparameter, S5-5-4-3. Set the feature FMe 1 Input to the encode-2 module and pass through (h+w) / s 2 1×1 convolution kernel processing to obtain the feature FM e2 ,s 2 is a hyperparameter, S5-5-4-4. Set the feature FM e2 Input to the latent module passes through (h+w) / s 3 1×1 convolution kernel processing to obtain the feature FM la ,s 3 is a hyperparameter, S5-5-4-5. Set the feature FM la Input to decode-1 module through (h+w) / s 2 1×1 convolution kernel processing to obtain the feature FM d1 , S5-5-4-6. Set the feature FM d1 Input to decode-2 module through (h+w) / s 1 1×1 convolution kernel processing to obtain the feature FM d2 , S5-5-4-7. Set the feature FM d2 The input is processed by (h+w) 1×1 convolution kernels to obtain the feature FM. sim , S5-5-4-8. Set the feature FM sim The input is sent to the decode-up module and processed by (h×w+h+w) / 2 1×1 convolution kernels to obtain the feature FM up , S5-5-4-9. Set the feature FM up The input is sent to the second convolution layer of ESP Block and processed by (h×w) 1×1 convolution kernels to obtain the feature FM. esp .

8. The portrait cutout method based on the general attention mechanism ESP-CA according to claim 7, Features: The convolution kernel size of the first convolution layer of ESP Block is 1×1.