Method and device for matting based on tri-graph visual transformer semantic information decoder

Abstract

The application discloses a matting method based on a trilateral graph visual Transformer semantic information decoder, and comprises the following steps: a light-weighted detail feature extraction layer is constructed by using continuous convolution layers, and the picture I is processed by using the feature extraction layer to obtain a detail feature map group {D1, D2, D3, D4}; the trilateral graph visual Transformer semantic information decoder is used to process the detail feature map D4 and a trilateral graph Trimap to obtain a semantic feature map S4; the upper-layer semantic feature map and the corresponding detail feature map are processed by continuously using the matting decoder to obtain a fused semantic feature map and a hierarchical matting output; the matting network based on the trilateral graph visual Transformer decoder is trained on the training data set until convergence; the trained matting network is used to process the picture to be matting, and the hierarchical matting output result output by the last matting decoder is the final matting result. The application also provides a corresponding matting device based on the trilateral graph visual Transformer semantic information decoder.

Description

Technical Field

[0001] This invention belongs to the field of deep learning and computer vision technology, and more specifically, relates to a method and apparatus for image matting based on a three-part visual Transformer semantic information decoder. Background Technology

[0002] Image matting is an important task in computer vision. It can be widely used in tasks such as image editing, image compositing, and special effects creation. Over the past few decades, a large number of matting methods have emerged, such as sampling-based methods or propagation-based methods. However, because these methods usually use manually constructed features, which are often limited by pixel color, distance, etc., it is difficult for them to handle complex situations that require more semantic information.

[0003] The image matting task can be expressed by the formula: I = αF + (1 - α)B, where I represents the original image, F represents the foreground image, B represents the background image, and α represents the foreground opacity. Solving the image matting problem means solving for the value of α in the above formula. However, since both the foreground image F and the background image B are unknown, this problem is considered an underconstrained problem in traditional approaches. Therefore, the concept of a tripartite image has been proposed. (See below) Figure 1 As shown, a tripartite diagram is a manually drawn cue image that divides an image into a defined foreground, a defined background, and an uncertain region. In the defined region α = 1 / 0, the uncertain region is the area to be solved. By using a tripartite diagram, the above equation becomes solvable. The use of tripartite diagrams has also been adopted in later deep learning-based image matting methods.

[0004] In recent years, with the development of deep learning, the field of computer vision has made groundbreaking progress. In the downstream field of image matting, numerous matting methods based on convolutional neural networks have emerged, driving the development of image matting. However, with the rapid development of visual Transformers, they have gradually replaced convolutional neural networks and become the mainstream trend in computer vision. On the one hand, the attention mechanism of visual Transformers can significantly increase the receptive field of the network, enhancing its ability to extract semantic information and thus improving the accuracy of the algorithm. On the other hand, visual Transformers can be pre-trained on large amounts of data, and pre-trained visual Transformers can exhibit superior performance in downstream tasks. Summary of the Invention

[0005] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides an image matting method based on a tripartite visual Transformer semantic information decoder. By using the tripartite visual Transformer semantic information decoder and matting decoder designed in this invention, matting performance is improved.

[0006] To achieve the above objectives, according to one aspect of the present invention, a method for image matting based on a three-part visual Transformer semantic information decoder is provided, comprising the following steps:

[0007] (1) A lightweight detail feature extraction layer is constructed using consecutive convolutional layers. This feature extraction layer is used to process image I to obtain the detail feature map group {D1,D2,D3,D4}.

[0008] (2) Use the Trimap visual Transformer semantic information decoder to process the detail feature map D4 and the Trimap to obtain the semantic feature map S4;

[0009] (3) By continuously using the matting decoder, the upper semantic feature map and the corresponding detail feature map are processed to obtain the fused semantic feature map and the hierarchical matting output;

[0010] (4) Train the matting network based on the three-part visual Transformer decoder constructed in steps (1)-(3) on the training dataset until convergence;

[0011] (5) Use the matting network trained in step (4) to process the image to be matted. The layer matting output of the last layer matting decoder is the final matting result.

[0012] In one embodiment of the present invention, the triangulation visual Transformer semantic information decoder is implemented through the following steps:

[0013] (2.1) Let the dimension of the input original image I be (H, W, 3), then the dimension of the input original trimap is (H, W, 1), and the dimension of the detail feature map D4 is... Trimap is downsampled using nearest neighbor interpolation, and then its third dimension is increased to match the dimension of the detail feature map D4 to obtain Triamp′, which has the following dimension.

[0014] (2.2) The processed triamp image Triamp′ is added to the detail feature map D4 and passed through a linear layer; the output of the linear layer is added to the detail feature map D4 to construct a residual structure based on the triamp image, and finally the semantic feature map S is obtained. tri ;

[0015] (2.3) Cascade Naive Visual Transformer layers to construct Naive Visual Transformer;

[0016] (2.4) Based on the existing mask autoencoder method, pre-train the Naive Vision Transformer on the image classification dataset to obtain the pre-trained weights of the model.

[0017] In one embodiment of the present invention, the image matting decoder is implemented through the following steps:

[0018] (3.1) In the matting decoder MD i In the first step, the upper-layer semantic feature map S, derived from the upper-layer matting decoder or the three-part visual Transformer semantic information decoder, is processed. i+1 Bilinear interpolation upsampling is performed, and the semantic feature map S is obtained through a convolutional layer. i+1 ′;

[0019] (3.2) Transform the semantic feature map S i+1 ′ and detail feature map D i By concatenating the images, we obtain the intermediate feature map mS. i , will mS i A new fused semantic feature map S is generated through convolutional layers. i ; and will mS i Perform deconvolution, followed by three lightweight convolutions with a normalized exponential function to obtain the hierarchical matting output α. i ;

[0020] (3.3) During training, the loss function is used to supervise the matting output of all layers. For the trained network, the matting output α1 of the last layer is used as the final matting result.

[0021] In one embodiment of the present invention, the calculation formula for step (2.2) is S. tri =D4+linear(D4+Triamp′), where linear represents a linear layer.

[0022] In one embodiment of the present invention, the linear layer is initialized with all zeros.

[0023] In one embodiment of the present invention, in step (2.3), each Naive Visual Transformer layer block_n converts the intermediate semantic feature map MS. n-1 Mapping to matrix query matrix Q n Key matrix K n Sum matrix V n By using a self-attention mechanism and a linear layer, a new intermediate semantic feature map (MS) is obtained. n .

[0024] In one embodiment of the present invention, during training in step (2.4), the Naive Vision Transformer module is initialized using the pre-trained MAE Naive Vision Transformer weights.

[0025] In one embodiment of the present invention, the calculation formula for step (3.2) is as follows:

[0026]

[0027] S i =Conv(mS i )

[0028] α i =softmax(Convs(TransConv(mS)) i )))

[0029] Where upsample represents bilinear interpolation upsampling. The symbol represents a concatenation operation, TransConv represents deconvolution, Conv represents convolution, Convs represents continuous convolution, and softmax represents the normalized exponential function.

[0030] In one embodiment of the present invention, in step (3), a deep-supervised training strategy is adopted for the hierarchical matting output.

[0031] According to another aspect of the present invention, a matting device based on a tripartite visual Transformer semantic information decoder is also provided, including at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor, wherein the instructions, after being executed by the processor, are used to complete the above-described matting method based on a tripartite visual Transformer semantic information decoder.

[0032] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0033] This invention provides an image matting method based on a tri-graph visual Transformer semantic information decoder. The tri-graph visual Transformer semantic information decoder designed in this invention significantly improves the semantic information extraction and tri-graph information extraction capabilities of the matting network. Furthermore, this invention also designs a matting decoder that can more efficiently fuse semantic and detail information and provide deep supervision to the network, further enhancing the matting performance. Attached Figure Description

[0034] Figure 1 This is an example diagram of a three-part diagram in an embodiment of the present invention;

[0035] Figure 2 This is a schematic diagram illustrating the principle of a matting network based on a three-part visual Transformer semantic information decoder in an embodiment of the present invention.

[0036] Figure 3 This is a schematic diagram illustrating the principle of a three-part visual Transformer semantic information decoder in an embodiment of the present invention.

[0037] Figure 4 This is a schematic diagram illustrating the principle of a cutout decoder in an embodiment of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0039] Figure 2 This is a schematic diagram illustrating the principle of a matting network based on a tri-graph visual Transformer semantic information decoder in an embodiment of the present invention. Figure 2 As shown, this invention provides a method for image matting based on a three-part visual Transformer semantic information decoder, including:

[0040] (1) A lightweight detail feature extraction layer is constructed using consecutive convolutional layers. This feature extraction layer is used to process image I to obtain the detail feature map group {D1,D2,D3,D4}.

[0041] (2) Use the Trimap visual Transformer semantic information decoder to process the detail feature map D4 and the Trimap to obtain the semantic feature map S4;

[0042] (3) By continuously using the matting decoder, the upper semantic feature map and the corresponding detail feature map are processed to obtain the fused semantic feature map and the hierarchical matting output;

[0043] (4) Train the matting network based on the three-part visual Transformer semantic information decoder constructed in steps (1)-(3) on the training dataset until convergence;

[0044] (5) Use the matting network trained in step (4) to process the image to be matted. The layer matting output of the last layer matting decoder is the final matting result.

[0045] Furthermore, such as Figure 3 As shown, the tri-graph visual Transformer semantic information decoder in step (2) is implemented through the following steps:

[0046] (2.1) Let the dimension of the input original image I be (H, W, 3), then the dimension of the input original trimap is (H, W, 1), and the dimension of the detail feature map D4 is... Trimap is downsampled using nearest neighbor interpolation, and then its third dimension is increased to match the dimension of the detail feature map D4, resulting in Triamp′. Its dimension is...

[0047] (2.2) The processed triamp image Triamp′ is added to the detail feature map D4 and passed through a linear layer; the output of the linear layer is added to the detail feature map D4 to construct a residual structure based on the triamp image. The final semantic feature map S is obtained. tri As shown in the formula below, where linear represents a linear layer:

[0048] S tri =D4 + linear(D4 + Triamp′)

[0049] Step (2.1) and this step together construct the following: Figure 2 The semantic information extraction module based on the tripartite graph is shown. During training, the linear layers of this module are initialized with all zeros.

[0050] (2.3) Based on existing methods, cascade Naive Visual Transformer layers to construct a Naive Visual Transformer. Each Naive Visual Transformer layer, block_n, stores the intermediate semantic feature maps MS. n-1 Mapping to matrix query matrix Q n Key matrix K n Sum matrix V n By using a self-attention mechanism and a linear layer, a new intermediate semantic feature map (MS) is obtained. n .

[0051] (2.4) Based on the existing masked autoencoders (MAE) method, a naive visual Transformer is pre-trained on an image classification dataset to obtain the model's pre-trained weights. During training, the naive visual Transformer module corresponding to this invention is initialized using the MAE-pre-trained weights. Thus, we have completed the construction of the semantic information decoder function of the tri-image visual Transformer.

[0052] Furthermore, such as Figure 4 As shown, the image matting decoder in step (2) is implemented through the following steps:

[0053] (3.1) In the matting decoder MD i In the first step, the upper-layer semantic feature map S, derived from the upper-layer matting decoder or the three-part visual Transformer semantic information decoder, is processed. i+1 Bilinear interpolation upsampling is performed, and the semantic feature map S is obtained through a convolutional layer. i+1 ′;

[0054] (3.2) Transform the semantic feature map S i+1 ′ and detail feature map D i By concatenating the images, we obtain the intermediate feature map mS. i mS i A new fused semantic feature map S is generated through convolutional layers. i At the same time, mS i Perform deconvolution, followed by three lightweight convolutions with a normalized exponential function to obtain the hierarchical matting output α. i The formula is shown below, where upsample represents bilinear interpolation upsampling. The concatenation operation is represented by TransConv, deconvolution by Conv, convolution by Convs, and softmax by softmax.

[0055]

[0056] S i =Conv(mS i )

[0057] α i =softmax(Convs(TransConv(mS)) i )))

[0058] (3.3) During training, a loss function is used to supervise the matting outputs of all layers. For the trained network, the matting output α1 of the last layer is used as the final matting result. This completes the function of the matting decoder.

[0059] Furthermore, the present invention also provides a matting device based on a tripartite visual Transformer semantic information decoder, including at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to complete the above-mentioned matting method based on a tripartite visual Transformer semantic information decoder.

[0060] This invention, through the use of a network architecture based on a tri-graph visual Transformer semantic information decoder, was tested on the compositional-1k dataset based on tri-graphs and outperformed the state-of-the-art matting methods, achieving the best matting performance to date.

[0061]

[0062] Table 1 shows the test results in compositional-1k.

[0063] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for image matting based on a trisection visual Transformer semantic information decoder, characterized in that, Includes the following steps: (1) A lightweight detail feature extraction layer is constructed using consecutive convolutional layers, and this feature extraction layer is used to process images. , obtain detailed feature map group ; (2) Use the three-part visual Transformer semantic information decoder to process the detail feature map. and the three-part chart Obtain semantic feature map The triangulation visual Transformer semantic information decoder is implemented through the following steps: (2.1) Describe the input original image The dimension is Then the input original triangulation The dimension is Detailed feature map The dimension is Nearest neighbor interpolation downsampling And in the third dimension, it rises to the level of detailed feature maps. With the same dimensions, we get Its dimensions are ; (2.2) The processed three-part diagram With detailed feature map The results are added together and passed through a linear layer; the output of the linear layer is then compared with the detail feature map. By adding the results, a residual structure based on the three-part graph is constructed, ultimately yielding the semantic feature map. ; (2.3) Concatenate Naive Visual Transformer layers to construct Naive Visual Transformer; (2.4) Pre-train Naive Visual Transformer on an image classification dataset based on the existing mask autoencoder method to obtain the pre-trained weights of the model. (3) By continuously using the matting decoder, the upper semantic feature map and the corresponding detail feature map are processed to obtain the fused semantic feature map and the hierarchical matting output; (4) Train the matting network based on the three-part visual Transformer decoder constructed in steps (1)-(3) on the training dataset until convergence; (5) Use the matting network trained in step (4) to process the image to be matted. The layer matting output of the last layer matting decoder is the final matting result.

2. The image matting method based on a triangulated visual Transformer semantic information decoder as described in claim 1, characterized in that, The image matting decoder is implemented through the following steps: (3.1) In the image matting decoder In the first step, the upper-layer semantic feature map from the upper-layer matting decoder or the three-part visual Transformer semantic information decoder is processed. Bilinear interpolation upsampling is performed, and the semantic feature map is obtained through a convolutional layer. ; (3.2) Semantic feature map and detailed feature map By splicing, an intermediate feature map is obtained. ,Will New fused semantic feature maps are generated through convolutional layers. ; And will Perform deconvolution, followed by three lightweight convolutions with a normalized exponential function to obtain the hierarchical matting output. ; (3.3) During training, the loss function is used to supervise the matting output of all layers. For the trained network, the matting output of the last layer is used. This is the final result of the image cutout.

3. The image matting method based on a triangulated visual Transformer semantic information decoder as described in claim 1, characterized in that, The calculation formula for step (2.2) is as follows: ,in This indicates a linear layer.

4. The image matting method based on a triangulation visual Transformer semantic information decoder as described in claim 3, characterized in that, The linear layer is initialized with all zeros.

5. The image matting method based on a triangulated visual Transformer semantic information decoder as described in claim 1, characterized in that, In step (2.3), each Naive Visual Transformer layer block_n will convert the intermediate semantic feature map... Mapping to a matrix query matrix Key matrix Sum matrix New intermediate semantic feature maps are obtained through a self-attention mechanism and a linear layer. .

6. The image matting method based on a triangulated visual Transformer semantic information decoder as described in claim 1, characterized in that, During training in step (2.4), the Naive Vision Transformer module is initialized using the pre-trained MAE Naive Vision Transformer weights.

7. The image matting method based on a triangulated visual Transformer semantic information decoder as described in claim 2, characterized in that, The calculation formula for step (3.2) is as follows: in, This indicates bilinear interpolation upsampling. This indicates a splicing operation. Indicates deconvolution. Represents convolution. Indicates continuous convolution. This represents the normalized exponential function.

8. The image matting method based on a triangulated visual Transformer semantic information decoder as described in claim 2, characterized in that, In step (3), a deep-supervised training strategy is used for the hierarchical matting output.

9. A matting device based on a tripartite visual Transformer semantic information decoder, characterized in that: It includes at least one processor and a memory, which are connected via a data bus. The memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to complete the matting method based on the triangulation visual Transformer semantic information decoder as described in any one of claims 1-8.

Images