Transparent object image matting and foreground prediction method

Abstract

The application discloses a transparent object image matting and foreground prediction method, comprising the following steps: 1) designing a training set image standard; 2) designing a general model of matting and foreground prediction; 3) designing a model loss for restraining the extraction of background features; and 4) an alpha auxiliary foreground prediction module combines the shallow color features and the deep structure features of the foreground, can obtain an accurate foreground image, can effectively reduce the background, and completes the foreground prediction task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a transparent object image matting and foreground prediction method. BACKGROUND

[0002] The present application mainly faces the derived problem of transparent object image matting: transparent object foreground prediction. Most existing image matting methods only consider predicting the alpha mask of the image to separate the image foreground and background. Existing technologies can be divided into traditional methods and deep learning methods.

[0003] (1) Traditional method: can be divided into sampling-based method and propagation-based method. The sampling-based method estimates the alpha mask by utilizing the color similarity between the unknown region and the known region (foreground and background), combined with the statistical information of the foreground and background colors. The propagation-based method, also known as affinity-based method, estimates the alpha mask by propagating the alpha value from the known region (foreground and background) to the unknown region based on the affinity of adjacent pixels.

[0004] (2) Deep learning method: the application in the field of image matting mainly relies on the construction of complex convolutional neural network models. These models automatically learn to extract key features through the training of a large number of image data sets, and realize the accurate separation of foreground and background in the image. When designing these models, multiple layers of convolution, pooling, activation and full connection are used to gradually extract and abstract image features. In order to improve the performance of the model, the key is to collect diversified and accurately labeled training data, and to improve the generalization ability of the model through data enhancement techniques. After training, the model can quickly and accurately extract the foreground on new images. In addition, by optimizing the model architecture and computational efficiency, the practicability of the model in real-time processing scenarios is ensured.

[0005] Existing matting methods are all for non-transparent objects, and when facing transparent object foreground, accurate foreground images cannot be obtained. When performing image matting, only the alpha mask of the target foreground is obtained, and no accurate foreground is used for new synthesis. In actual application, when a new synthesis of the target transparent object foreground is performed, only the image with the background can be used for synthesis, and this synthesis method will cause great error and visual impact.

[0006] In addition, the existing matting model always contains the features of the background when extracting the features of the transparent object, and the foreground mask obtained will have background influence when synthesizing. SUMMARY

[0007] In view of the problems existing in the prior art, the purpose of the present application is to provide a transparent object image matting and foreground prediction method.

[0008] To solve the above problems, the application adopts the technical solutions as follows.

[0009] A transparent object image matting and foreground prediction method, comprising the following steps:

[0010] 1) Design training set image standard;

[0011] 2) Design overall model of matting and foreground prediction;

[0012] 3) Design model loss for suppressing background feature extraction;

[0013] 4) The alpha auxiliary foreground prediction module combines the shallow color features and deep structure features of the foreground.

[0014] In step 1), based on the principle of image synthesis, the specific formula is as follows:

[0015] I = alpha F + (1-alpha) B, alpha belongs to [0, 1] (1)

[0016] When the background is pure black, the background value B is 0, and the above formula is simplified as:

[0017] I = alpha F (2)

[0018] By dividing the image I by alpha, the foreground image F is approximately obtained.

[0019] In step 2), Swin Transformer is selected as the encoder of the deep model, and the designed double decoder outputs three foreground matrices from different intermediate layers, with output sizes of 1 / 8, 1 / 4 and 1 / 1 of the input resolution, and then the output is adjusted to the size of the input resolution, and then the output is selectively fused to gradually refine the uncertain area.

[0020] In step 3), the trained preliminary model is retrained again to better separate the foreground and the background, so that the preliminary model extracts features from the real foreground, and the features extracted from the real foreground are defined as V i F , and the features extracted by the model during retraining are k represents the number of different feature extraction sizes, and the loss function is:

[0021]

[0022] The total loss function is:

[0023]

[0024] L total = Lalpha +L foreground +L suppress (6)

[0025] wherein L l1 is the absolute difference between the real foreground and the predicted foreground, L comp represents the absolute difference between the ground truth image and the synthesized image, L lap measures the difference of the Laplacian pyramid representation of the foreground and captures local and global differences, L foreground is the loss of the foreground in the dual decoder branch.

[0026] In step 4), the alpha mask is multiplied by the result of I, and then down-sampling is performed to align the channels and sizes required for feature fusion, and then it is connected with the structure features to be decoded, the deep layer semantic and structure features are fused with the shallow layer color and texture features through convolution connection, and the fused features are V F , and finally the fused features are decoded:

[0027] V F = Relu(Cat(Down(Conv(alpha*I)), V i F )) (7). BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is the overall flowchart of the present application.

[0029] Figure 2 is the foreground selection diagram of the training set of the present application.

[0030] Figure 3 is the model diagram of the encoder of the present application. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application; obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments, and all other embodiments obtained by those skilled in the art without creative labor on the basis of the embodiments in the present application are within the protection scope of the present application.

[0032] A transparent object image matting and foreground prediction method, comprising the following basic steps:

[0033] 2.1 Basic steps

[0034] 1) Design training set image standard;

[0035] 2) Matting and foreground prediction overall model;

[0036] 3) Design model loss to suppress the extraction of background features;

[0037] 4) Alpha auxiliary foreground prediction module combines the shallow color features and deep structural features of the foreground;

[0038] 2.2 Detailed steps

[0039] The present application will be further described in conjunction with the drawings and specific implementation steps. As shown in the drawings, an image matting and foreground prediction method includes the following detailed steps: Figure 1

[0040] Step 1: Design training set image standard

[0041] Step 1.1: When constructing a completely standardized training dataset, it usually requires extremely high cost and stringent conditions, especially for the acquisition of foreground images. To solve this challenge, we decided to use an innovative method to create another training set of foreground images. This method is based on the principle of image synthesis, and the specific formula is as follows:

[0042] I = αF + (1-α)B, α ∈ [0, 1] (1)

[0043] As shown in the drawings, when the background is pure black, the background value B is 0. Therefore, we can simplify the above formula as: Figure 2 I = αF (2)

[0044] Since the foreground image resources of transparent objects are very scarce, we choose to use images with black backgrounds as training data. By dividing the image I by α, we can approximately obtain the foreground image F. This method can to some extent alleviate the lack of transparent object foreground images in the training dataset, while also avoiding the high cost of artificially constructing such a foreground dataset. In this way, we can effectively balance the cost and resource constraints while maintaining the quality of the dataset.

[0045] Step 2: Matting and foreground prediction overall model

[0046]

[0047] ​​Step 2.1: Overall we adopt a single encoder dual decoder architecture, since Swin Transformer has a good performance in modeling global information. Therefore we choose Swin Transformer as the encoder of our deep model. The decoder we designed outputs three foreground matrices from different intermediate layers, with output sizes of 1 / 8, 1 / 4 and 1 / 1 of the input resolution, and then adjusts them to the size of the input resolution. Then, the outputs are selectively fused to gradually refine uncertain areas. The decoder has a gradual refinement design idea, which is very effective for our fine matting and foreground prediction. Our single encoder dual decoder is shown in Figure 3 .

[0048] Step 3: Design model loss to suppress background feature extraction

[0049] Step 3.1: Since the foreground features in the transparent object image are not prominent enough, and it is extremely difficult to separate the foreground and background after synthesis, the encoder often captures background features when extracting features. Therefore, as shown in Figure 2 , we designed a background suppression feature extraction loss. We trained the model to predict the foreground to obtain a preliminary model that can extract features.

[0050] Step 3.2: According to the preliminary model trained in step 3.1, the model is retrained again to better separate the foreground and background. To achieve this, we let the preliminary model extract features from the real foreground, and use L1 loss with the features extracted from the foreground by the untrained model. This loss helps to constrain the model to extract features mainly from the foreground rather than the background. Here, we define the features extracted from the real foreground by the preliminary model as V i F , and the features extracted by the model during retraining as k represents the number of different feature extraction sizes. The loss function is:

[0051]

[0052] The total loss function is:

[0053]

[0054] L total = L alpha + L foreground + L suppress (6)

[0055] where L l1 is the absolute difference between the real foreground and the predicted foreground. L comp represents the absolute difference between the ground truth image and the synthesized image. Llap The difference of the Laplacian pyramid representation of the foreground is measured and both local and global differences are captured. The loss representing the alpha mask in the dual decoder branch, L foreground is the loss for the foreground in the dual decoder branch.

[0056] Step 4: The alpha-assisted foreground prediction module combines the shallow color features and the deep structural features of the foreground

[0057] Step 4.1: Since alpha plays an effective role in foreground-background separation, it is natural to make full use of this powerful auxiliary input when designing the network to obtain better foreground prediction results. In the multi-scale feature pyramid structure, the deep features contain more global information, while the shallow features contain rich local information such as texture and color similarity. The fusion of these features is crucial for accurately predicting the foreground of high-transparency objects. Therefore, we propose an alpha-assisted foreground prediction module in the decoding process, which effectively integrates high-level semantic information and low-level color information under the guidance of non-background information. Multiplying the alpha mask predicted by the existing model with the original image I can effectively reduce the background, but it is still not accurate enough. Therefore, the foreground color carried by the result of alpha mask multiplied by I is combined with the feature set V i F of the encoder. The shallow color features are used to supplement the deep color features of the model, and the deep semantic features are used to reduce the background features contained in the result of alpha mask multiplied by I, so as to achieve a complementary effect. Specifically, the result of alpha mask multiplied by I is convolved and then down-sampled to align the channels and sizes required for feature fusion. Then it is connected with the structural features to be decoded. The fusion of deep semantic and structural features and shallow color and texture features is achieved through convolution connection, and the fused features are V F , and finally the fused features are decoded.

[0058] V F = Relu(Cat(Down(Conv(alpha*I)), V i F )) (7)

[0059] Since our decoder part is dual decoder, we use one of the decoders to predict the alpha mask and the other decoder to predict the foreground. The feature fusion is performed by inputting the alpha mask predicted by the decoder as part of the alpha-assisted foreground prediction module. Finally, the foreground prediction task is completed using the foreground decoder. That is, the alpha mask and the foreground of the model predicted image can be obtained.

[0060] The above merely provides the preferred embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art, according to the technical solution of the present application and the improved concept thereof, makes equivalent replacement or change within the technical range disclosed by the present application, and should be covered within the protection scope of the present application.

Claims

1. A method for transparent object image matting and foreground prediction, characterized in that, Comprise the following steps: 1) design training set image standard; 2) design overall model of matting and foreground prediction; 3) design model loss for suppressing extraction of background features; 4) design alpha auxiliary foreground prediction module for combining shallow color features and deep structural features of foreground; In step 1), based on the principle of image synthesis, the specific formula is as follows: I = alpha F + (1-alpha) B, alpha belongs to [0, 1] (1) When the background is pure black, the background value B is 0, and the above formula is simplified as: I = alpha F (2) By dividing the image I by alpha, the foreground image F is approximately obtained; In step 2), Swin Transformer is selected as the encoder of the deep model, and the designed double decoder outputs three foreground matrices from different intermediate layers, with output sizes of 1 / 8, 1 / 4 and 1 / 1 of the input resolution respectively, then the output is adjusted to the size of the input resolution, then, the output is selectively fused, and the uncertain area is gradually refined; In step 3), the trained preliminary model re-trains the model again to better separate the foreground and background, let the preliminary model extract features from the real foreground, and use L1 loss to define the features extracted from the real foreground by the preliminary model as V i F , the features extracted by the model during re-training are k represents the number of different feature extraction sizes, and the loss function is: The total loss function is: L total = L alpha + L foreground + L suppress (6) where L alpha represents the loss of the alpha mask in the dual decoder branch, L foreground is the loss of the foreground in the dual decoder branch; In step 4), the alpha mask is multiplied by the result of the convolution of I, and then down-sampling is performed to align the channels and sizes required for feature fusion, and then it is connected with the structure features to be decoded, the deep semantic and structure features are fused with the shallow color and texture features through convolution connection, and the fused features are V F , and finally the fused features are decoded: V F = Relu(Cat(Down(Conv(alpha * I)), V i F )) (7).

Images