A portrait matting method, device, and related medium based on coordinate attention

Abstract

The application discloses a portrait matting method and device based on coordinate attention and related media, and the method comprises the following steps: obtaining a portrait image to be matting; wherein the portrait image is an RGB image; extracting high-level semantic features of the portrait image by using a Mobilenetv2 network to obtain candidate feature maps of different sizes; selecting the candidate feature map with the smallest size as a target feature map and inputting the target feature map into a coordinate attention network, and then outputting a first intermediate image by the coordinate attention network; combining the portrait image and the first intermediate image, and performing detail generation processing on the first intermediate image by using a detail generation network to obtain a second intermediate image; and fusing the portrait image, the first intermediate image and the second intermediate image to obtain a final matting result. The position information of the portrait image is embedded into the channel attention through the coordinate attention mechanism, so that the prediction ability for the portrait foreground can be improved, and the accuracy of the matting result is improved.

Description

Technical Field

[0001] This invention relates to the field of computer software technology, and in particular to a method, apparatus and related medium for portrait matting based on coordinate attention. Background Technology

[0002] An image can be simply viewed as consisting of a foreground and a background. Image matting is the process of separating the foreground and background of a given image. Portrait matting uses the person as the foreground and non-person areas as the background. With the development of the internet, portrait matting technology has been widely applied in many fields, such as advertising design, image compositing, virtual and augmented reality, image editing, and film special effects. Furthermore, with continuous technological advancements, the application scenarios of portrait matting technology are constantly expanding. As a pixel-level computer vision task, portrait matting requires the matting model to identify each pixel and predict the probability that that pixel is in the foreground. Therefore, the matting model's ability to identify each coordinate in the image is crucial. However, currently, few matting models can embed this module that focuses on coordinate position information into their network structure. This results in the final matting result not being particularly accurate. Summary of the Invention

[0003] This invention provides a portrait matting method, apparatus, computer device, and storage medium based on coordinate attention, aiming to improve the predictive ability of portrait foregrounds, thereby improving the accuracy of matting results.

[0004] In a first aspect, embodiments of the present invention provide a portrait matting method based on coordinate attention, including:

[0005] Obtain the portrait image to be cut out; wherein, the portrait image is an RGB image;

[0006] High-level semantic features are extracted from the portrait image using the Mobilenetv2 network to obtain candidate feature maps of different sizes;

[0007] The candidate feature map with the smallest size is selected as the target feature map and input into the coordinate attention network. Then, the coordinate attention network outputs the first intermediate image.

[0008] By combining the portrait image and the first intermediate image, a detail generation network is used to perform detail generation processing on the first intermediate image to obtain the second intermediate image;

[0009] The portrait image, the first intermediate image, and the second intermediate image are fused to obtain the final cutout result.

[0010] Secondly, embodiments of the present invention provide a portrait matting device based on coordinate attention, comprising:

[0011] An image acquisition unit is used to acquire a portrait image to be cut out; wherein, the portrait image is an RGB image;

[0012] The semantic feature extraction unit is used to extract high-level semantic features from the portrait image using the Mobilenetv2 network to obtain candidate feature maps of different sizes;

[0013] The coordinate attention unit is used to select the smallest candidate feature map as the target feature map and input it into the coordinate attention network, which then outputs the first intermediate image.

[0014] The detail generation unit is used to combine the portrait image and the first intermediate image, and use the detail generation network to perform detail generation processing on the first intermediate image to obtain the second intermediate image;

[0015] The image fusion unit is used to fuse the portrait image, the first intermediate image, and the second intermediate image to obtain the final cutout result.

[0016] Thirdly, embodiments of the present invention provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the portrait matting method based on coordinate attention as described in the first aspect.

[0017] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the portrait matting method based on coordinate attention as described in the first aspect.

[0018] This invention provides a method, apparatus, computer device, and storage medium for portrait matting based on coordinate attention. The method includes: acquiring a portrait image to be matted; wherein the portrait image is an RGB image; extracting high-level semantic features from the portrait image using a MobileNetv2 network to obtain candidate feature maps of different sizes; selecting the smallest candidate feature map as the target feature map and inputting it into a coordinate attention network, then outputting a first intermediate image from the coordinate attention network; combining the portrait image and the first intermediate image, performing detail generation processing on the first intermediate image using a detail generation network to obtain a second intermediate image; and fusing the portrait image, the first intermediate image, and the second intermediate image to obtain the final matting result. This invention first uses a MobileNetv2 network to represent the semantic features of the portrait image, then embeds the positional information of the portrait image into channel attention using a coordinate attention mechanism, then generates more details using a detail generation network, and finally performs image fusion using a fusion module. This improves the predictive ability for the portrait foreground, thereby improving the accuracy of the matting result. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating a portrait matting method based on coordinate attention, provided in an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of a sub-process of a portrait matting method based on coordinate attention provided in an embodiment of the present invention;

[0022] Figure 3 A diagram illustrating the overall network architecture of a portrait matting method based on coordinate attention, provided in an embodiment of the present invention.

[0023] Figure 4 This invention provides a network architecture diagram for coordinate attention in a portrait matting method based on coordinate attention, as shown in the embodiments of the present invention.

[0024] Figure 5 A schematic block diagram of a portrait matting device based on coordinate attention provided in an embodiment of the present invention;

[0025] Figure 6 This is a schematic diagram of a portrait matting device based on coordinate attention, provided as an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0028] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0029] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0030] Please see below. Figure 1 , Figure 1 The flowchart of a portrait matting method based on coordinate attention provided in an embodiment of the present invention specifically includes steps S101 to S105.

[0031] S101. Obtain the portrait image to be cut out; wherein, the portrait image is an RGB image;

[0032] S102. High-level semantic features are extracted from the portrait image using the Mobilenetv2 network to obtain candidate feature maps of different sizes;

[0033] S103. Select the candidate feature map with the smallest size as the target feature map and input it into the coordinate attention network. Then, the coordinate attention network outputs the first intermediate image.

[0034] S104. Combining the portrait image and the first intermediate image, the first intermediate image is processed by a detail generation network to obtain a second intermediate image.

[0035] S105. The portrait image, the first intermediate image, and the second intermediate image are fused to obtain the final cutout result.

[0036] In this embodiment, for the portrait image to be cut out, high-level semantic features are first extracted through the Mobilenetv2 network to obtain multiple candidate feature maps of different sizes. Then, the smallest one among these candidate feature maps is selected as the target feature map and input into the coordinate attention network to embed the position information of the portrait image into the channel attention. Subsequently, the output image of the coordinate attention network is combined with the original portrait image to achieve detail generation. For the image after detail generation, it is fused with the previous portrait image and the first intermediate image output by the coordinate attention network to obtain the final cutout result.

[0037] Combination Figure 3 In this embodiment, the semantic feature representation of the portrait image is first performed by the Mobilenetv2 network in the semantic estimation module. Then, the position information of the portrait image is embedded into the channel attention through the coordinate attention mechanism. Next, more details are generated through the detail module, and the image is fused through the fusion module to obtain the alpha image. This can improve the prediction ability of the portrait foreground, thereby improving the accuracy of the matting result.

[0038] In one embodiment, step S102 includes:

[0039] Dilated convolution is used to map the features in the portrait image from a low-dimensional space to a high-dimensional space;

[0040] By using depthwise convolution in depthwise separable convolution to perform convolution operations on each input channel, and by using pointwise convolution in depthwise separable convolution to linearly combine the feature maps in each channel, candidate feature maps of different sizes are obtained.

[0041] In this embodiment, an RGB image is used as input for semantic feature extraction to locate the position of the person. The input image is then encoded by the MobileNetv2 backbone network to extract high-level semantic features, outputting three sets of candidate feature maps of different sizes. These are designed to locate objects of different sizes; larger candidate feature maps are better for locating small objects, while smaller candidate feature maps are better for locating large objects. Here, this embodiment inputs the smallest candidate feature map into the Coordinate Attention (CA) module, thus enhancing the ability to perceive and predict the position of the person.

[0042] The MobileNetv2 backbone network processing steps can be understood as the input image entering the MobileNetv2 network structure to obtain the output feature map. In this embodiment, the input image is the aforementioned portrait image, which is an RGB image. MobileNetv2 employs a network structure called "Inverted Residuals with Linear Bottlenecks." This structure improves feature representation capabilities by using depthwise convolution with a linear bottleneck layer. The dilated convolution captures broader contextual information by increasing the receptive field of the convolutional kernel. For convolution operations, MobileNetv2 uses a series of depthwise separable convolutional layers to reduce computational complexity. This depthwise separable convolutional structure allows MobileNetv2 to maintain relatively high accuracy while reducing computational cost.

[0043] Depthwise separable convolution consists of two steps: depthwise convolution and pointwise convolution. Depthwise convolution performs convolution operations on each input channel separately, using small convolution kernels. This reduces the number of model parameters and computational cost. Pointwise convolution uses 1x1 convolution kernels to linearly combine the feature maps of each channel, increasing the interactivity between features.

[0044] In one embodiment, such as Figure 2 As shown, step S103 includes steps S201 to S205.

[0045] S201. Input the target feature map into the residual module, and obtain the first sub-feature map by outputting the residual module;

[0046] S202. Perform global average pooling on the first sub-feature map in the X-axis and Y-axis directions respectively, and combine the global average pooling results in the X-axis and Y-axis directions to form the second sub-feature map.

[0047] S203. The second sub-feature map is processed sequentially by 2D convolution, batch normalization, and ReLU-based nonlinear operators to obtain the third sub-feature map.

[0048] S204. The third sub-feature map is split into two fourth sub-feature maps according to the number of channels, and the weights of the two fourth sub-feature maps are obtained by sequentially passing them through 2D convolution and the Sigmoid function.

[0049] S205. The first sub-feature map is multiplied and weighted based on the two weights to obtain the first intermediate image.

[0050] Research on visual attention has shown that channel attention is beneficial for improving model performance; however, most current visual attention methods typically ignore positional information, which is crucial for image matting tasks. Coordinate attention can embed positional information into channel attention. Unlike channel attention, which converts feature tensors into single feature vectors through two-dimensional global pooling, coordinate attention decomposes channel attention into two one-dimensional feature encoding processes, aggregating features along two spatial directions respectively. In this way, long-range dependencies can be captured along one spatial direction while retaining precise positional information along the other. The resulting feature maps are then encoded separately into a pair of orientation-aware and position-sensitive attention maps, which can be complementaryly applied to the input feature map to enhance the representation of the object of interest, thereby better predicting the location of a person. The coordinate attention module structure diagram is shown below. Figure 4 As shown.

[0051] In this embodiment, the target feature map is first input into the residual module, which outputs the first sub-feature map. Then, global average pooling (XAvg Pool, YAvgPool) is performed on this feature map along both the X and Y axes. The results of XAvgPool and YAvgPool are then concatenated to form the second sub-feature map. This second sub-feature map is then processed by a 2D convolution (Conv2d), batch normalization (BatchNorm), and a ReLU6-based non-linear operator to obtain the third sub-feature map. This third sub-feature map is then split into two fourth sub-feature maps based on the number of channels. Each fourth sub-feature map is then processed by a Conv2d operation, and two weights are obtained using the Sigmoid function. Finally, these two weights are multiplied and weighted with the original first sub-feature map to obtain the final output of the coordinate attention module. It should be noted that an image can be understood as a coordinate system, with each pixel having coordinates (x, y). To better capture the positional information of the person, global average pooling is performed along both the X and Y axes. Here, assuming we have an m x n matrix M, global average pooling along the X-axis means averaging each column of matrix M to form a 1 x n matrix. Similarly, global average pooling along the Y-axis means averaging each row of matrix M to form an m x 1 matrix.

[0052] In one embodiment, step S104 includes:

[0053] The portrait image and the first intermediate image are processed using 13 Conv2dIBNormRelu modules, and the second intermediate image is output.

[0054] The Conv2dIBNormReLU module includes, in sequence, a convolution operator, instance batch normalization, and a ReLU operator.

[0055] In this embodiment, since the edges of portrait images are often difficult to distinguish between foreground and background, a detail generation network is used to predict the edges of the portrait image, thereby increasing the edge prediction capability of the entire matting model. The detail generation network takes the RGB image and the output of the semantic estimation module (i.e., the first intermediate image) as input. Specifically, the RGB image and the first intermediate image are taken as input, and then the output is obtained by using 13 Conv2dIBNormReLU modules. The Conv2dIBNormReLU module specifically includes: convolution operator, instance batch normalization (IBNorm), and ReLU operator.

[0056] In one embodiment, step S105 includes:

[0057] The portrait image, the first intermediate image, and the second intermediate image are fused using four Conv2dIBNormRelu modules, and a fused feature map is output.

[0058] The fused feature map is output to the Sigmoid operator, and the Sigmoid operator outputs the seeded matting result.

[0059] Specifically, the step of outputting the fused feature map to the Sigmoid operator, and having the Sigmoid operator output the final matting result, includes:

[0060] The final image cutout result is obtained by calculating using the following formula:

[0061]

[0062] Where S(x) represents the Sigmoid function, and x represents the fused feature map.

[0063] In this embodiment, the initial portrait image, the first intermediate image, and the second intermediate image are fused to obtain the corresponding fused feature map. Then, the Sigmoid operator is used to output the hidden layer neurons to obtain the final matting result, i.e., the alpha image.

[0064] Figure 5 A schematic block diagram of a portrait matting device 500 based on coordinate attention provided in an embodiment of the present invention. The device 500 includes:

[0065] Image acquisition unit 501 is used to acquire a portrait image to be cut out; wherein, the portrait image is an RGB image;

[0066] The semantic feature extraction unit 502 is used to extract high-level semantic features from the portrait image using the Mobilenetv2 network to obtain candidate feature maps of different sizes.

[0067] The coordinate attention unit 503 is used to select the smallest candidate feature map as the target feature map and input it into the coordinate attention network, and then the coordinate attention network outputs the first intermediate image.

[0068] The detail generation unit 504 is used to combine the portrait image and the first intermediate image, and use a detail generation network to perform detail generation processing on the first intermediate image to obtain a second intermediate image.

[0069] The image fusion unit 505 is used to fuse the portrait image, the first intermediate image, and the second intermediate image to obtain the final cutout result.

[0070] In one embodiment, the semantic feature extraction unit 502 includes:

[0071] A spatial mapping unit is used to map features in the portrait image from a low-dimensional space to a high-dimensional space using dilated convolution;

[0072] The linear combination unit is used to perform convolution operations on each input channel using depthwise convolution in depthwise separable convolution, and to linearly combine the feature maps in each channel using pointwise convolution in depthwise separable convolution to obtain candidate feature maps of different sizes.

[0073] In one embodiment, such as Figure 6 As shown, the coordinate attention unit 503 includes:

[0074] The residual output unit 601 is used to input the target feature map into the residual module and output the first sub-feature map from the residual module.

[0075] The global pooling unit 602 is used to perform global average pooling on the first sub-feature map in the X-axis and Y-axis directions respectively, and combine the global average pooling results in the X-axis and Y-axis directions into a second sub-feature map.

[0076] The feature processing unit 603 is used to process the second sub-feature map sequentially through 2D convolution, batch normalization and ReLU-based nonlinear operators to obtain the third sub-feature map;

[0077] The feature splitting unit 604 is used to split the third sub-feature map into two fourth sub-feature maps according to the number of channels, and then output the weights of the two fourth sub-feature maps by sequentially passing them through 2D convolution and the Sigmoid function.

[0078] The weighting unit 605 is used to perform multiplication and weighting processing on the first sub-feature map based on the two weights to obtain the first intermediate image.

[0079] In one embodiment, the detail generation unit 504 includes:

[0080] The detail processing unit is used to perform detail processing on the portrait image and the first intermediate image using 13 Conv2dIBNormRelu modules, and output the second intermediate image.

[0081] In one embodiment, the Conv2dIBNormReLU module sequentially includes a convolution operator, instance batch normalization, and a ReLU operator.

[0082] In one embodiment, the image fusion unit 505 includes:

[0083] The fusion processing unit is used to perform fusion processing on the portrait image, the first intermediate image and the second intermediate image using four Conv2dIBNormRelu modules, and output the fused feature map.

[0084] The fusion output unit is used to output the fused feature map to the Sigmoid operator, and the Sigmoid operator outputs the matting result.

[0085] In one embodiment, the fusion output unit includes:

[0086] The result calculation unit is used to calculate the final image matting result according to the following formula:

[0087]

[0088] Where S(x) represents the Sigmoid function, and x represents the fused feature map.

[0089] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.

[0090] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed, can perform the steps provided in the above embodiments. The storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0091] This invention also provides a computer device, which may include a memory and a processor. The memory stores a computer program, and when the processor calls the computer program in the memory, it can implement the steps provided in the above embodiments. Of course, the computer device may also include various network interfaces, power supplies, and other components.

[0092] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0093] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A portrait matting method based on coordinate attention, characterized in that, The method comprises the following steps: An image to be cut out is obtained; wherein the image is an RGB image; High-level semantic features of the image are extracted by using a Mobilenetv2 network to obtain candidate feature maps of different sizes; The smallest candidate feature map is selected as a target feature map, which is input into a coordinate attention network, and then a first intermediate image is output by the coordinate attention network; Details of the first intermediate image are generated by using a detail generation network in combination with the image, to obtain a second intermediate image; The image, the first intermediate image and the second intermediate image are fused to obtain a final cut-out result; The smallest candidate feature map is selected as a target feature map, which is input into a coordinate attention network, and then a first intermediate image is output by the coordinate attention network, which comprises the following steps: The target feature map is input into a residual module, and a first sub-feature map is obtained by outputting the residual module; Global average pooling is performed on the first sub-feature map in X and Y directions, and the global average pooling results in X and Y directions are spliced into a second sub-feature map; The second sub-feature map is processed by 2D convolution, batch normalization and relu-based nonlinear operators in sequence to obtain a third sub-feature map; The third sub-feature map is split into two fourth sub-feature maps according to the number of channels, and the weights of the two fourth sub-feature maps are obtained by outputting the two fourth sub-feature maps by 2D convolution and Sigmoid function in sequence; The first sub-feature map is multiplied and weighted based on the two weights to obtain the first intermediate image; The first intermediate image is processed by using a detail generation network in combination with the image to obtain a second intermediate image, which comprises the following steps: The image and the first intermediate image are processed by using 13 Conv2dIBNormRelu modules, and the second intermediate image is output; The Conv2dIBNormRelu module comprises a convolution operator, an instance batch normalization and a ReLu operator in sequence.

2. The portrait matting method based on coordinate attention according to claim 1, characterized in that, The image is mapped from a low-dimensional space to a high-dimensional space by using a dilated convolution; The features in each input channel are convolved by using a depth convolution in a depth separable convolution, and the feature maps in each channel are linearly combined by using a point-wise convolution in the depth separable convolution to obtain candidate feature maps of different sizes. The image, the first intermediate image and the second intermediate image are fused to obtain a final cut-out result, which comprises the following steps:

3. The portrait matting method based on coordinate attention according to claim 1, characterized in that, The image, the first intermediate image and the second intermediate image are fused by using 4 Conv2dIBNormRelu modules, and a fusion feature map is output; The fusion feature map is output to a Sigmoid operator, and the final cut-out result is output by the Sigmoid operator. ​ 4. The portrait matting method based on coordinate attention according to claim 3, characterized in that, The fusion feature map is output to a Sigmoid operator, and a final matting result is output by the Sigmoid operator, comprising: The final matting result is calculated according to the following formula: Wherein, S(x) represents a Sigmoid function, and x represents the fusion feature map.

5. An apparatus for portrait matting based on coordinate attention, characterized in that, Comprise: An image acquisition unit is configured to acquire a portrait image to be matting; wherein the portrait image is an RGB image; A semantic feature extraction unit is configured to extract high-level semantic features from the portrait image by using a Mobilenetv2 network to obtain candidate feature maps of different sizes; A coordinate attention unit is configured to select a candidate feature map of the smallest size as a target feature map, and input the target feature map into a coordinate attention network, and then output a first intermediate image by the coordinate attention network; A detail generation unit is configured to combine the portrait image and the first intermediate image, and perform detail generation processing on the first intermediate image by using a detail generation network to obtain a second intermediate image; An image fusion unit is configured to fuse the portrait image, the first intermediate image and the second intermediate image to obtain a final matting result; The coordinate attention unit comprises: A residual output unit is configured to input the target feature map into a residual module, and output a first sub-feature map by the residual module; A global pooling unit is configured to perform global average pooling on the first sub-feature map in X-axis and Y-axis directions respectively, and combine the global average pooling results in the X-axis and Y-axis directions into a second sub-feature map; A feature processing unit is configured to sequentially process the second sub-feature map by 2D convolution, batch normalization and relu-based nonlinear operator to obtain a third sub-feature map; A feature splitting unit is configured to split the third sub-feature map into two fourth sub-feature maps according to the number of channels, and sequentially process the two fourth sub-feature maps by 2D convolution and Sigmoid function output to obtain the weights of the two fourth sub-feature maps; A weight weighting unit is configured to multiply and weight process the first sub-feature map based on the two weights to obtain the first intermediate image; The detail generation unit comprises: A detail processing unit is configured to perform detail processing on the portrait image and the first intermediate image by using 13 Conv2dIBNormRelu modules, and output the second intermediate image; The Conv2dIBNormRelu module sequentially comprises a convolution operator, an instance batch normalization and a ReLu operator.

6. A computer device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the portrait matting method based on coordinate attention according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the portrait matting method based on coordinate attention according to any one of claims 1 to 4.

Images