Image matting method based on foreground and background decoupling

Abstract

This invention relates to a matting method based on foreground-background decoupling. It includes foreground segmentation, background segmentation, and disambiguation, and designs a novel foreground-background decoupled matting (FBDM) deep neural network model driven by new subtasks. The invention consists of four parts: the first part designs a more reasonable set of matting subtasks; the second part designs a nested attention mechanism to decouple the backbone features; the third part is the semantic decoding part, which uses two independent progressive semantic decoders to complete the foreground and background segmentation subtasks using the backbone features; the fourth part is a frequency-division local disambiguation module, which corrects errors in different frequency domains by predicting low-frequency and high-frequency information separately, ultimately improving the matting accuracy. A series of experimental results show that the algorithm of this invention significantly improves the matting accuracy.

Description

Technical Field

[0001] This invention belongs to the field of image matting, specifically relating to a matting method based on foreground-background decoupling. Background Technology

[0002] Image matting assumes that the input image is an image I formed by linearly combining a foreground image F and a background image B, as shown below:

[0003] I = α i *F i +(1-α i )*B i

[0004] Where α∈[0,1], and i is the pixel index. The purpose of image matting is to extract specific foregrounds with complex hair details and diverse appearance challenges. Image matting has many applications in real-world scenarios, such as video editing and poster creation.

[0005] Predicting the foreground image F, background image B, and α value from image I is a typical maladaptive problem.

[0006] Currently, a common practice in image matting is to introduce additional input as auxiliary information, such as embellishments, doodles, and backgrounds. However, generating additional input usually requires manual operation, which limits its use cases in the real world. For example, it is impossible to manually provide trimming for real-time matting in video conferencing, and it is difficult for users to draw trimmings on mobile devices.

[0007] To reduce reliance on additional input, some methods requiring no additional input have recently emerged. For example, TOM-Net. [1] The object mask, attenuation mask, and refractive flow field generated by the encoder-decoder network are connected to produce a coarse prediction, which is then refined using a residual network. 2 Net [2] A two-level nested U-shaped structure was designed to decode more background information. (HAtMatting) [3] By employing spatial and channel attention to integrate appearance cues and pyramid features, alpha matte can be perceived from refined boundaries and adaptive semantics. MODNet [5] A lightweight extinction target decomposition network is proposed for real-time portrait extinction of a single input image. GFM [4] A parallel framework inspired by a comprehensive empirical analysis of the component pipeline in image extinction is proposed. It designs a component route called RSSN to reduce the domain gap caused by differences in resolution, sharpness, and noise. More and more works are focusing on privacy protection, such as P3Mnet. [6]GFM is further extended to privacy benchmarks by increasing information exchange between parallel branches. However, these methods model boundless hidden line culling as global segmentation and detail culling, such as... Figure 1 As shown.

[0008] As can be seen, the field of image matting has developed towards matting without additional input. Most methods have evolved to perform repeatable matting of fixed object categories without additional input. Currently, most matting methods without additional input can be divided into serial methods, serial methods with global information, and parallel methods. The serial method connects global segmentation and detail ablation, where global segmentation aims to obtain a trimap image. [7,8] Or a rough foreground image [9] The latter performs image matting with the help of the previous segmentation. A drawback of this pipeline is that subsequent extinction steps cannot correct semantic errors. [4] Furthermore, it can never perform end-to-end training. A serial approach with global information... [4,5] This approach introduces global information for detail extinction. While this pipeline solves the problem of phased modeling and training, correcting errors from previous steps remains a challenge due to its serial design. Recently, parallel approaches have emerged that treat global segmentation and detail extinction as two independent subtasks. However, parallelism can introduce bottlenecks for these subtasks. Intuitively, the lower bound of extinction performance is logically limited by the lower bound of the subtask. But the lower bounds of global segmentation and detail extinction are not stable because their objectives are not aligned. This can lead to a sharp drop in overall performance when the performance of one subtask deteriorates.

[0009] This invention proposes a new set of subtasks, decomposing the image matting task into foreground segmentation, background segmentation, and disambiguation tasks. For example... Figure 1As shown in (d), the foreground and background segmentation tasks are parallel, while the disambiguation task is serial. The foreground and background segmentation tasks aim to learn semantic information, which can be viewed as subtasks of global segmentation. A parallel design to mine uncertain regions is clearly more effective, helping to reduce the difficulty of correction. Since the goals of foreground and background segmentation are similar to those of matting, i.e., they have equivalent lower bounds, this avoids the bottleneck problem. Setting up a disambiguation subtask not only reduces uncertainty between semantic predictions but also effectively achieves the goal of detail disambiguation, as the uncertain regions between foreground and background semantics typically include both semantic and detail uncertainties. Furthermore, when complex backgrounds lower the lower bound of background segmentation performance, the disambiguation task evaluates the outputs of foreground and background segmentation to maintain the stability of the lower bound. Inspired by the new set of subtasks, we propose a novel foreground-background decoupling network model called FBDM. Specifically, the proposed FBDM first uses a nested self-attention (NSA) mechanism to automatically decouple backbone features into foreground and background features. Then, the decoupled features are passed through a progressive semantic decoder to recover foreground and background semantics in a coarse-to-fine manner. To eliminate the uncertainty in foreground and background segmentation, we propose a frequency-division local disambiguation module, which uses local retention and local multi-scale convolution to refine low-frequency and high-frequency regions respectively. The low-frequency region represents the semantic uncertainty region, and the high-frequency region represents the detail uncertainty region. Summary of the Invention

[0010] The purpose of this invention is to provide a matting method based on foreground-background decoupling, which can improve matting accuracy.

[0011] To achieve the above objectives, the technical solution of this invention is: a matting method based on foreground-background decoupling. First, a set of decomposed subtasks is proposed, including foreground segmentation, background segmentation, and disambiguation tasks. Second, inspired by the decomposed subtask set, a foreground-background decoupled matting network (FBDM) is proposed. This network decouples the backbone features through a nested attention mechanism. Then, the backbone features are used to complete the foreground and background segmentation tasks using two independent progressive semantic decoders. Finally, a frequency-division local disambiguation module is used to predict low-frequency and high-frequency information to correct errors in different frequency domains.

[0012] Compared to existing technologies, this invention offers the following advantages: It utilizes a more stable set of decomposition subtasks without additional input, namely foreground segmentation, background segmentation, and disambiguation. Inspired by this set of subtasks, we propose a novel Foreground-Background Decoupled Matching Network (FBDM) for image matting. FBDM employs nested self-attention (NSA) to address long-distance dependencies across layers, decoupling backbone features into foreground and background features, thereby enabling the proposed progressive semantic decoder to predict the semantics of both the foreground and background. Then, FBDM uses a proposed frequency-division local disambiguation module to refine the uncertainties in low-frequency semantics and high-frequency details. Attached Figure Description

[0013] Figure 1 This is a subtask breakdown diagram.

[0014] Figure 2 This is the network model architecture of the present invention.

[0015] Figure 3 This is a framework diagram for nested self-attention mechanisms (NAS).

[0016] Figure 4 Module diagram for dividing local ambiguity (FBLD) by frequency.

[0017] Figure 5 Image α is the prediction result of the image matting algorithm. Detailed Implementation

[0018] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0019] This invention presents a matting method based on foreground-background decoupling. First, it proposes a set of decomposed subtasks, including foreground segmentation, background segmentation, and disambiguation. Second, inspired by the decomposed subtasks, it proposes a foreground-background decoupled matting network (FBDM), which decouples the backbone features through a nested attention mechanism. Then, it uses the backbone features to complete the foreground and background segmentation tasks using two independent progressive semantic decoders. Finally, it employs a frequency-division local disambiguation module to predict low-frequency and high-frequency information separately to correct errors in different frequency domains.

[0020] The following is a detailed implementation process of the present invention.

[0021] Figure 1 This is a network model architecture diagram of the present invention. Specifically, the present invention provides a method for image matting based on foreground-background decoupling, including:

[0022] 1. Collection of background removal subtasks

[0023] To address the performance bottleneck caused by parallel subtask sets, this invention proposes a novel set of subtasks, decomposing image matting into foreground segmentation, background segmentation, and disambiguation. Foreground and background segmentation tasks remain parallel, while disambiguation is performed sequentially with the other two tasks. Foreground and background segmentation aim to learn the semantic information of the image, which can be viewed as a subtask of global segmentation. The parallel design mines uncertain regions, thus reducing the difficulty of correction. Furthermore, the objectives of foreground and background segmentation are similar to those of disambiguation, namely, having an equivalent lower bound, thus avoiding the bottleneck problem. The disambiguation task not only reduces the uncertainty between semantic predictions but also effectively achieves detail reduction, as the uncertain regions between foreground and background semantics typically include both semantic and detail uncertainties. Moreover, when complex backgrounds lower the lower bound of background segmentation performance, the disambiguation task evaluates the outputs of foreground and background segmentation to maintain the stability of the lower bound.

[0024] 2. Backbone Network

[0025] like Figure 2 As shown, the algorithm consists of a backbone network, two parallel semantic branches, and a disambiguation branch based on defined subtasks. The algorithm is applicable to any backbone network; we tested it using MobilenetV2 or ResNet-34 pre-trained on the ImageNet dataset. Furthermore, two prediction heads with 1×1 convolutions are input from the highest-level semantic features output from the backbone network to predict coarse foreground and background images. The loss for this coarse semantic information is:

[0026]

[0027] in, and These are the predictions from the foreground predictor and the background predictor. and These are tags for foreground segmentation and background segmentation. It is a roughly prospective supervised loss. It is a roughly supervised loss of background information.

[0028] L S(·,·) The loss is the binary cross-entropy value, as follows:

[0029]

[0030] Where H and W are the length and width of the predicted image, p i ∈[0,1],y i ∈[0,1].

[0031] 3. Nested self-attention mechanism

[0032] Self-attention mechanism

[10] Significant success has been achieved in the field of image processing. However, it has resulted in enormous computational demands, making it unacceptable for many tasks. To address this issue, the algorithm of this invention proposes a nested self-attention (NSA) mechanism to mine effective information at different levels, such as... Figure 3 As shown.

[0033] A nested self-attention (NSA) mechanism is used to automatically decouple backbone features into foreground and background features. Then, the decoupled features are passed through a progressive semantic decoder to recover foreground and background semantics in a coarse-to-fine manner.

[0034] like Figure 3 As shown, we will use the feature map E of each layer in the backbone network. i The new feature map E' is obtained by inputting into the NAS module. i The NSA module consists of an internal self-attention module and an external self-attention module, which operate on image patches and layers, respectively. For internal self-attention, we add a GAU to the backbone features.

[11] We use layers to learn long-range dependencies between image patches in the inner layers. To reduce computation, we utilize an adaptive average pooling layer to reduce the feature size to 1 / 16 gram before the GAU layer, and then use bilinear interpolation to restore it to the original size. For external self-attention, we use coarse semantic p e Guided by this principle, a masked self-attention mechanism is introduced to learn the relationships between different layers. A new feature layer E' is computed through masked self-attention. i It can be described as follows:

[0035]

[0036] Where K is the number of backbone network layers, E i V is the feature map output by the i-th layer backbone network. j Q is the result of internal self-attention in the j-th layer. i and K i It is for feature map E i The result is obtained through a linear layer after interpolation; Linear() represents the linear layer operation. The height of the feature map of the i-th layer, Let be the width of the feature map of the i-th layer.

[0037] 4. Semantic decoder

[0038] Because NSA is capable of modeling long-range dependencies at multiple levels, all features have the opportunity to learn global knowledge. To reduce the risk of overfitting, the algorithm in this invention aims for the network model to learn sparse solutions. When encoded features contain sufficient decision information, reusing them in the decoder often results in a sparse parameter matrix. Therefore, we introduce skip connections as UNet.

[12] To enhance the effective information content that satisfies the sparsity constraint, the semantic decoding layer first scales the features from the preceding decoding layer to twice their size, then uses a 1×1 convolutional kernel to fuse the encoded features and the scaled features using a skip-connection approach. To extract contextual information, an ASPP is inserted after the highest-level features in the foreground and background branches. [3] Layers. The algorithm of this invention uses 1×1 convolutions to predict the semantics of the current level in each layer of the decoder. The loss function is:

[0039]

[0040] Where M is the number of layers in the decoder, L S It is a binary cross-entropy function. and These are the predicted values ​​of the i-th layer of the decoder, and These are the labels for foreground segmentation and background segmentation, respectively.

[0041] 5. Distributed Disambiguation Module

[0042] Uncertainty, derived from Bayesian theory, is an effective way to find fuzzy regions. However, due to multifaceted requirements, it may not be a perfect deployment plan. To address this issue, we obtain uncertainty based on foreground and background semantics. Furthermore, this method preserves the lower bounds of foreground and background segmentation because the uncertainty map only reflects the fuzzy regions of both, while retaining regions with consistent performance. We identify three cases of uncertainty, namely the fuzzy region U between foreground and background semantics. bf Low-confidence region U in foreground semantics f and the low-confidence region U in the background b .

[0043] Our proposed uncertainty scheme is a single, flexible approach. The three uncertainty maps are normalized and averaged to obtain the final uncertainty map U. To avoid redundant calculations of low-uncertainty samples, a sampling strategy is necessary to filter them. Corresponding to the uncertain regions, we use OTSU...

[13] To generate an adaptive threshold.

[0044] The uncertain regions include semantic regions and detail regions, referring to low-frequency and high-frequency information, respectively. Therefore, we propose a multi-stage disambiguation process that decomposes multiple frequencies into layers of Local Disambiguation Layers (FDLD). The structure of an FDLD layer can be described using... Figure 4 (a) describes the FDLD layer, which consists of a low-frequency disambiguation module (such as...). Figure 4 (b) shown), high-frequency disambiguation module (as shown in the image) Figure 4 (c) shows the composition of the fusion module. We found that the self-attention mechanism is a low-pass filter, while the convolutional network is a high-pass filter. Therefore, we use the self-attention mechanism to simulate the low-frequency disambiguation module and multi-scale convolution to simulate the high-frequency disambiguation module.

[0045] For the low-frequency disambiguation module, we concatenate the input image and semantic features through a linear layer. Then, according to the sampling strategy, we crop the features using 1 padding to obtain feature patch X. Since X contains semantic information of the input image, we use it to generate a distance map D along the [Q,K] generated by following the self-attention pipeline. Subsequently, we concatenate the backbone features from the previous stage with the semantic features, which are obtained by passing a convolution with a kernel size of 3 and a span of 2 through a linear layer, using the same strategy as the generation process of X to obtain image patch Z. Then, V is generated by passing Z through a linear layer. A new feature pixel. It is obtained by the dot product of D and V. Finally, a 1×1 convolution is used to predict the new result, replacing the original prediction at this position. The algorithm of this invention uses binary cross-entropy to supervise low-frequency predictions that are not equal to 0.5 in the trimap graph, as shown in the following formula:

[0046]

[0047] Where ~T represents the non-transition region. It is a refined result of the low-frequency disambiguation module. It is a label that matches the size of p in the current layer.

[0048] For the high-frequency disambiguation module, we first double the size of the previous output and the previous feature. Then, we concatenate the previous output, the previous feature, and the previous level backbone feature using a 1×1 convolution. Unlike the low-frequency cropping strategy, we crop two feature blocks within an uncertain pixel, one with half the cropped size padding and the other without padding. We double the size of the unpadded patch and merge the padding into the channel size, as shown below. Figure 4 As shown in (c). Subsequently, the combined image patch is reduced twice by a convolutional block with a kernel size of 3×3 and a stride of 2 to obtain a more refined feature Z. ij HFinally, high-frequency results are predicted using 1×1 kernels to replace the original positions. In the transition region, an L1 loss function is used to supervise high-frequency predictions; the loss function is as follows:

[0049]

[0050] Where T is the transition region. It is the refined result of the high-frequency disambiguation module. Is the current layer and The size of the label matches the label.

[0051] Furthermore, we integrate low-frequency disambiguation and high-frequency disambiguation from a feature perspective. We will Double the size and then The two sequences are concatenated and then subjected to a 1×1 convolution to predict the result at pixel position ij. The loss function for fusion is as follows:

[0052]

[0053] in, It is the result of refinement after fusion. Is the current layer and The matching tags.

[0054] 6. Training Objectives

[0055] The FBDM proposed in this invention is an end-to-end network that can be optimized using a cooperative loss term. The entire training objective can be divided into a coarse semantic loss. Foreground / Background Segmentation Loss Disambiguation loss L D and a predicted matted image p m L of the image M Combined in a weighted manner:

[0056]

[0057] Here, λ1, λ2, λ3, and λ4 are all balancing parameters. We set λ1 to 0.1, and the other parameters to 1.

[0058] Loss of disambiguation L D as follows:

[0059]

[0060] Where K is the number of layers in the backbone network, and M is the number of layers in the decoder, for the final predicted matted image p m We use the following loss:

[0061]

[0062] Where H and W are the predicted image p m Length and width, Sobel gradient operation is used to enhance the predicted image p. m The details are y, which is the real label.

[0063] To evaluate the effectiveness of our algorithm on real-world image matting data, we conducted experiments on the P3M-10k dataset (containing 10,000 high-resolution portrait privacy-preserving images) and compared it with several recently published algorithms. We also explored the interaction between the various modules on the PPM-100 dataset (containing 100 high-resolution portrait images). We compared it with SHM... [1] U 2 Net [8] MODNet [5] GFM [4] P3Mnet [6] SHM and MODNet use MobileNet as their backbone network, while GFM and P3MNet use ResNet-34 as their backbone network. Due to U... 2 Net is a U-shaped nested network structure without a backbone network. The algorithm in this invention uses SAD (Sum of Absolute Differences), MSE (Mean Trivial Error), and MAD (Mean Absolute Difference) as the main evaluation metrics. Furthermore, we report the SAD... f SAD b SAD t As additional metrics, they are separately listed as SAD for the foreground area, background area, and transition area.

[0064] 1. Parameter Selection

[0065] We investigated the role of each module in PPM-100, and chose FBDM with the ResNet-34 backbone as our comparative experimental method because it performed best in Table 2. In addition, we explored pure FBDM to verify the effectiveness of the proposed subtask set. Pure FBDM refers to sharing the same coding features in semantic branches and using standard skip connections instead of FDLD. As shown in Table 1, pure FBDM achieves a comparative result that demonstrates the effectiveness of the proposed subtask set. We also explored the effectiveness of the NSA and FDLD modules in our algorithm. Compared to pure FBDM, pure FBDM with NSA can improve the MAD of PPM from 0.0154 to 0.0139, and the FDLD module can also improve the MAD of PPM by 0.0022 and the MAD of PPT by 0.0016.

[0066] Table 1 Ablation experiments of the FBDM module

[0067]

[0068] Table 2 Quantitative comparison of image matting algorithms on the P3M-10k dataset.

[0069]

[0070] 2. Qualitative comparison

[0071] To quantitatively compare our algorithm with several recently published matting algorithms, we output the predicted matted images on the P3M-10k test set.

[0072] Figure 5 This demonstrates the alpha image predicted by the matting algorithm on P3M-10k. Overall, the qualitative evaluations of GFM, P3MNet, and FBDM are better than U... 2 FBDM outperforms GFM, P3MNet, and MODNet significantly. Meanwhile, FBDM demonstrates more stable performance than GFM and P3MNet. In the first image, GFM, P3MNet, and FBDM show almost identical good performance in detail, but GFM has semantic errors. In the second image, P3MNet has semantic errors. FBDM's stronger semantic predictions highlight the advantages of the proposed subtask. Furthermore, in the third image, FBDM shows a greater advantage in the hand, and in the final image, it shows more detail at the hairline, indicating that the proposed FDLD can recover hard samples belonging to both semantically and detail-uncertain regions.

[0073] 3. Quantitative comparison

[0074] Table 2 shows the evaluation results on the P3M-10k test set. 2 Net utilizes skip connections to increase the amount of effective information needed to satisfy sparsity constraints; however, a single decoder cannot simultaneously learn the global semantic features and local detail features of all types of images. Therefore, U 2 P3Mnet underperforms FBDM across all evaluation metrics. P3Mnet, a matting algorithm focused on human privacy images, shows competitive results on P3M-10k. Utilizing the same backbone network ResNet34, FBDM maintains comparable performance to P3Mnet. In particular, decomposing the resolution into foreground segmentation, background segmentation, and resolution prevents the model from overemphasizing either the background or foreground; for example, FBDM's SAD... f It is 1.59, SAD b It is 2.56, while P3M's SAD is 2.56. f SAD is 1.28. bThe SAD is 4.67. FBDM prioritizes the overall picture when segmenting foreground and background, rather than favoring either the foreground or background, which is a key reason for its high performance. Furthermore, under the same backbone network, FBDM demonstrates significant superiority over GFM, with SAD, MSE, and MAD of 10.82, 0.0044, and 0.0066 respectively, compared to GFM's 12.73, 0.0058, and 0.0079.

[0075] References:

[0076] [1]Chen, Guanying, Kai Han, and Kwan-Yee K.Wong. "Tom-net: Learningtransparent object matting from a single image." Proceedings of the IEEEconference on computer vision and pattern recognition. 2018.

[0077] [2]Qin, Xuebin, et al. "U2-Net: Going deeper with nested U-structure forsalient object detection." Pattern recognition 106(2020):107404.

[0078] [3]Qiao, Yu, et al. "Attention-guided hierarchical structure aggregation for image matting." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020.

[0079] [4] Li, Jizhizi, et al. "Bridging composite and real:towards end-to-enddeep image matting." International Journal of ComputerVision 130.2(2022):246-266.

[0080] [5]Ke,Zhanghan,et al."MODNet:real-time trimap-free portrait mattingvia objective decomposition."Proceedings oftheAAAI Conference onArtificialIntelligence.Vol.36.No.1.2022.

[0081] [6]Li,Jizhizi,et al."Privacy-preserving portrait matting."Proceedingsof the 29th ACM International Conference on Multimedia.2021.

[0082] [7]Chen,Quan,et al."Semantic human matting."Proceedings of the 26thACM international conference onMultimedia.2018..

[0083] [8]Shen,Xiaoyong,et al."Deep automatic portrait matting."Europeanconference on computer vision.Springer,Cham,2016.

[0084] [9]Zhang,Yunke,et al."A late fusion cnn for digital matting."Proceedings of the IEEE / CVF conference on computer visionandpatternrecognition.2019.

[0085]

[10] Dosovitskiy,Alexey,et al."An image is worth 16x16 words:Transformers for image recognition at scale."arXivpreprint arXiv:2010.11929(2020)..

[0086]

[11] Hua, Weizhe, et al. "Transformer quality in linear time." International Conference on Machine Learning. PMLR, 2022.

[0087]

[12] Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation." International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015.

[0088]

[13] Otsu, Nobuyuki. "A threshold selection method from gray-level histograms." IEEE transactions on systems, man, and cybernetics 9.1 (1979): 62-66.

[0089] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for image matting based on foreground-background decoupling, characterized in that, First, a set of decomposed subtasks is proposed, including foreground segmentation, background segmentation, and disambiguation. Second, inspired by the decomposed subtasks, a foreground-background decoupled matting network (FBDM) is proposed. This network decouples the backbone features through a nested attention mechanism, and then uses the backbone features to complete the foreground and background segmentation tasks using two independent progressive semantic decoders. Finally, a frequency-division local disambiguation module is used to predict low-frequency and high-frequency information to correct errors in different frequency domains. The Foreground-Background Decoupled Maturing Network (FBDM) consists of a backbone network, two parallel semantic branches, and a disambiguation branch according to a set subtask. The two prediction heads with 1×1 convolutions take the highest-level semantic features output from the backbone network as input to predict coarse foreground and background images, i.e. coarse semantic information. Feature maps of each layer in the backbone network of the Foreground-Background Decoupling Maturing Network (FBDM) New feature maps are obtained through nested attention mechanisms. The nested attention mechanism consists of an inner self-attention layer and an outer self-attention layer, which operate on image patches and layers, respectively. For the inner self-attention layer, a GAU layer is added to the backbone features to learn the long-distance dependencies between image patches in the inner layers. To reduce computation, an adaptive average pooling layer is used to reduce the feature size to 1 / 16 gram before the GAU layer, and then bilinear interpolation is used to restore it to its original size. For the outer self-attention layer, coarse semantic information is used... Guided by this, a masked self-attention mechanism is introduced to learn the relationships between different layers; The frequency division local disambiguation module is composed of layers of local disambiguation FDLD, and the FDLD is composed of a low-frequency disambiguation module, a high-frequency disambiguation module and a fusion module. For the low-frequency disambiguation module, the input image and semantic features are concatenated through a linear layer; then, according to the sampling strategy, features are cropped using 1-padded material to obtain feature blocks. ;because It contains semantic information about the input image, which is used in the generation following the self-attention pipeline. To generate distance map D, and This is obtained by interpolating the feature maps and then passing them through a linear layer; subsequently, the backbone features from the previous level are concatenated with the semantic features, the latter being passed through a linear layer using a convolutional layer with a kernel size of 3 and a span of 2, and then using... The same strategy is used to obtain image patch Z during the generation process; then... It is generated by Z through a linear layer; a new feature pixel. It is composed of D and It is obtained by dot product; finally, a The convolution is used to predict new results, replacing the original prediction at this position; For the high-frequency disambiguation module, first double the size of the previous output and the previous feature; then, connect the previous output, the previous feature, and the previous stage backbone feature, and then... The convolution is performed; unlike low-frequency cropping strategies, two feature blocks are cropped from an uncertain pixel, one with half the cropped size padding and the other without padding; the size of the unpadded patch is doubled, and the padded patch is merged into the channel size; subsequently, the combined image patch is processed by a convolution kernel with a kernel size of... The convolutional blocks with a stride of 2 are used to reduce the size twice, resulting in more refined features Z. ij H Finally, using a convolution kernel size of... Convolutions are used to predict high-frequency results, replacing the original positions; Furthermore, low-frequency disambiguation and high-frequency disambiguation are integrated from a feature perspective; Double the size and then spliced ​​together through a Convolution to predict at pixel location The result.

2. The image matting method based on foreground-background decoupling according to claim 1, characterized in that, The foreground segmentation task and the background segmentation task are parallel, while the foreground segmentation task, the background segmentation task, and the disambiguation task are serial.

3. The image matting method based on foreground-background decoupling according to claim 1, characterized in that, The loss of coarse semantic information is: in, and These are the predictions from the foreground predictor and the background predictor. and These are tags for foreground segmentation and background segmentation. It is a roughly prospective supervised loss. It is a supervised loss with a rough background; The loss is the binary cross-entropy value, as follows: Where H and W are the length and width of the predicted image, .

4. The image matting method based on foreground-background decoupling according to claim 1, characterized in that, New feature map calculated through mask self-processing The description is as follows: Where K is the number of backbone network layers. For the first Feature maps output by the backbone network For the first The result of internal self-attention within the layer, and It is a feature map The result is obtained through a linear layer after interpolation; Linear() represents the linear layer operation. For the first The height of the layer feature map, For the first The width of the layer feature map.

5. The image matting method based on foreground-background decoupling according to claim 3, characterized in that, The semantic decoder first scales the features of the decoding layer to twice their size, and then uses a convolutional kernel with a size of [missing value]. The convolutions are used to fuse encoded features and adjusted features using a skip-connection method; to extract contextual information, an ASPP layer is inserted after the highest-level features of the foreground and background branches; the loss function of the semantic decoder is: Where M is the number of layers in the semantic decoder. It is a binary cross-entropy function. and These are the predicted values ​​of the i-th layer of the decoder, and These are the labels for foreground segmentation and background segmentation, respectively.

6. The image matting method based on foreground-background decoupling according to claim 5, characterized in that, In the non-transition region, binary cross-entropy is used to supervise low-frequency predictions, and the loss function formula is as follows: in Represents a non-transitional region. It is the refined result of the low-frequency disambiguation module. It is a label that matches the size of p in the current layer; In the transition area, use The loss function is used to supervise high-frequency predictions. The formula for the loss function is as follows: Where T is the transition region. It is the refined result of the high-frequency disambiguation module. Is the current layer and Labels that match the size; Will Double the size and then spliced ​​together through a Convolution to predict at pixel location The resulting loss function for fusion is as follows: in, It is the result of refinement after fusion. Is the current layer and The size of the label matches the label.

7. The image matting method based on foreground-background decoupling according to claim 6, characterized in that, The foreground-background decoupling matting network FBDM is optimized using a collaborative loss term; the entire training objective is divided into a coarse semantic loss. Foreground / background segmentation loss Disambiguation loss and a predictive matting image loss Combined in a weighted manner: in, These are all balance parameters; Disambiguation loss as follows: Where K is the number of layers in the backbone network, which affects the final predicted matted image. The following loss is used: in Sobel gradient operation is used to enhance the predicted image. Details It's a real label.

Images