Image matting method based on deep learning

Abstract

A kind of image matting method based on deep learning, comprising the following steps: step 1: obtaining matting dataset, including original graph, label graph, according to label graph to generate tripartite graph, the sample in the matting dataset is divided into training set, verification set and test set;Step 2: build the image matting network model based on the encoder-decoder structure of GAN;Step 3: the training set is input into model for training, after training every round, it is verified once with verification set, loss is calculated, and the best model is saved;Step 4: the image to be tested and tripartite graph are input into model for testing.The purpose of the present application is to solve the technical problems of shallow feature loss, insufficient multi-scale feature extraction and complex background information confusion in complex natural image matting task, and a natural image matting method is provided, which can effectively retain shallow features, improve the background discrimination ability of the target and obtain rich multi-scale features.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a computer vision technology, and specifically relates to a matting network and a matting method. BACKGROUND

[0002] Image matting is a process of extracting the part needed in an image. The extracted part is called foreground, and the remaining part is called background. Porter and Duff proposed a mathematical principle of a composite image in matting, that is, formula (1):

[0003] I i =α i F i +(1-α i )B i ,α i ∈[0,1] (1)

[0004] Wherein, I represents a natural image, F represents a foreground, B represents a background, and alpha represents opacity, which represents the proportion of foreground and background in each pixel. The process of matting is the inverse process of the composite image. That is, given an image I, solve the foreground F, the background B and the opacity alpha. The alpha of the foreground pixel is determined as 1, the alpha of the background pixel is determined as 0, and the alpha of the pixel mixed with the foreground and the background is between 0 and 1. For a natural image, F, B and I each pixel point has three color components, and it can be known from formula (1) that the matting problem consists of three equations and seven unknown arrays, which leads to this problem without an exact solution. In order to solve this problem, many matting methods nowadays generate trimap, that is, a trimap to help solve it. The trimap consists of three parts of known foreground area, background area and foreground-background mixed area. The main purpose of the matting task is to roughly obtain the opacity alpha of the pixel in the mixed area. The application prospect of the matting task is very wide, and it is gradually applied to various computer vision tasks, such as AI face changing, movie background synthesis and the like.

[0005] Traditional image matting methods mainly include sampling-based and propagation-based methods, or a combination of the two. Traditional methods mainly target the surface information such as texture and color of the image, and cannot well meet the task requirements. In recent years, methods based on deep learning have been gradually applied to image matting tasks. Deep neural networks can better capture features through deep structures, can well utilize low-level appearance space information of images, extract high-level semantic information, and iteratively reuse image feature information at the same level, but there are still some problems in existing methods. For example, in the article "Full-automatic portrait matting algorithm based on deep learning", an attention mechanism is used to filter important channels of image features, and a multi-scale feature is extracted using the attention mechanism, which optimizes the results, but there is still a technical problem of insufficient use of shallow features and multi-scale features. SUMMARY

[0006] The purpose of the present application is to solve the technical problems of shallow feature loss, insufficient multi-scale feature extraction and complex background information confusion in complex natural image matting tasks, and to propose an image matting method that can effectively preserve shallow features, improve background discrimination ability for targets and obtain rich multi-scale features.

[0007] In order to solve the above technical problems, the technical scheme adopted by the present application is as follows:

[0008] A deep learning-based image matting method, comprising the following steps:

[0009] Step 1: Obtain a matting dataset, including an original image and a label image, generate a tripartite image according to the label image, and divide the samples in the matting dataset into a training set, a validation set and a test set;

[0010] Step 2: build an image matting network model based on the encoder-decoder structure of GAN;

[0011] Step 3: input the training set into the model for training, and after each round of training, validate it with the validation set, calculate the loss, and save the best model, that is, the model with the lowest loss function;

[0012] Step 4: input the image to be tested and the tripartite image into the model for testing.

[0013] In step 2, when building an image matting network model based on the encoder-decoder structure of GAN, the following method is used:

[0014] 2-1: build a generator network based on Unet to generate an alpha mask infinitely close to a real image;

[0015] 2-2: Construct a five-layer structure discriminator network, use the generated signature generator results and label to calculate the loss, and distinguish the fake image from the real image.

[0016] In step 2-1, a Unet-based generator network is constructed to generate an alpha mask infinitely close to the real image. The Unet-based generator network is composed of an encoder, a decoder and a skip dense link. The specific steps include:

[0017] 2-1-1: The encoder performs four times of down-sampling on the input feature map. The first layer obtains feature map F1.1 through a DS-ASPP module and a convolution, standardization and activation layer from feature map F1, and then fuses feature map F1 with feature map F1.1 to obtain feature map F1.2;

[0018] The second layer obtains feature map F2 by down-sampling the feature map F1.1 output by the first layer, and then inputs the feature map F2 into a DS-ASPP module and a convolution, standardization and activation layer to obtain feature map F2.1. Then, the feature map F2 is fused with the feature map F2.1 to obtain the feature map F2.2;

[0019] The third layer obtains feature map F3 by down-sampling the feature map F2.1 output by the second layer, and then inputs the feature map F3 into a DS-ASPP module and a convolution, standardization and activation layer to obtain feature map F3.1. Then, the feature map F3 is fused with the feature map F3.1 to obtain the feature map F3.2;

[0020] The fourth layer obtains feature map F4 by down-sampling the feature map F3.1 output by the third layer, and then inputs the feature map F4 into a DS-ASPP module and a convolution, standardization and activation layer to obtain feature map F4.1. Then, the feature map F4 is fused with the feature map F4.1 to obtain the feature map F4.2;

[0021] The fifth layer obtains feature map F5 by down-sampling the feature map F4.1 output by the fourth layer, and then inputs the feature map F5 into a DS-ASPP module and adjusts the channel through convolution to obtain feature map F5.1;

[0022] 2-1-2: The feature maps F1.2, F2.2, F3.2 and F4.2 fused by the first layer, the second layer, the third layer and the fourth layer, respectively, are input into an S-BAM module. S-BAM combines the input features through two separate attention branches of channel Mc and space Ms to generate the final 3D feature Mf. The 3D Mf is multiplied element by element with the input feature map, and then added to the original input feature map to output feature maps F1.3, F2.3, F3.3 and F4.3, respectively.

[0023] 2-1-3: In the skip dense link, the features output by encoding each layer are respectively densely fused. The feature map F4.3 of the fourth layer of the encoder is input to the third layer after upsampling, the feature map F4.3 is fused with the feature map F3.3 of the third layer of the encoder to obtain the feature map F3.4, and then the feature map F3.3 and the feature map F3.4 are respectively transmitted to the second layer after upsampling; the feature map F3.3 and the feature map F2.3 of the second layer of the encoder are fused to obtain the feature map F2.4, the feature map F3.4 is fused with the feature map F2.3 and the feature map F2.4 of the second layer of the encoder to obtain the feature map F2.5, and then the feature map F2.3, the feature map F2.4 and the feature map F2.5 are transmitted to the first layer after upsampling; the feature map F2.3 and the feature map F1.3 of the first layer of the encoder are fused to obtain the feature map F1.4, the feature map F2.4 is fused with the feature map F1.3 and the feature map F1.4 of the first layer of the encoder to obtain the feature map F1.5, and the feature map F2.5 is fused with the feature map F1.3 and the feature map F1.5 of the first layer of the encoder to obtain the feature map F1.6;

[0024] 2-1-4: The decoder fuses the feature map output by the DS-ASPP four times after upsampling, and fuses the feature map output by the skip link in the form of channel splicing after each upsampling. First, the feature map F5.1 output by the fifth layer of the encoder DS-ASPP is transmitted to the second layer of the decoder after upsampling and fused with the feature map F4.3 of the first layer of the skip link to obtain the feature map F4.4, the feature map F4.4 is subjected to convolution, standardization and activation layer to obtain the feature map F4.5, then the feature map F4.5 is transmitted to the third layer of the decoder after upsampling and fused with the feature map F3.4 and the feature map F3.3 of the second layer of the skip link to obtain the feature map F3.5, the feature map F3.5 is subjected to convolution, standardization and activation layer to obtain the feature map F3.6, then the feature map F3.6 is transmitted to the fourth layer of the decoder after upsampling and fused with the feature map F2.3 and the feature map F2.5 of the third layer of the skip link to obtain the feature map F2.6, the feature map F2.6 is subjected to convolution, standardization and activation layer to obtain the feature map F2.7, then the feature map F2.7 is transmitted to the fifth layer of the decoder after upsampling and fused with the feature map F1.3 and the feature map F1.6 of the fourth layer of the skip link to obtain the feature map F1.7, the feature map F1.7 is subjected to convolution, standardization and activation layer to obtain the output feature map F.

[0025] In step 2-2, a five-layer convolutional network structure discriminator is constructed, and the loss is calculated by using the generated signature generator result and the label to distinguish the pseudo image and the real image, and the following steps are further included:

[0026] 2-2-1: Obtain the F generated by the generator, and input F and the image true label into the discriminator respectively. The discriminator improves the ability to judge the segmentation result of the generated image as false through a five-layer convolutional network. The discriminator values corresponding to the generator segmentation image and the segmentation label image are calculated respectively. The cross-entropy is used to calculate the loss value of the discriminator network.

[0027] 2-2-2: The generator uses the training loss of the discriminator to continuously optimize to fit the label image so that the discriminator judges it as true. The two continuously game in the training process to form a confrontation. After network training, the accuracy of the generator image segmentation result is improved, and accurate image segmentation results can be obtained in testing and actual use.

[0028] In step 3, the loss function for network training is as follows:

[0029] 1) Generator loss function

[0030] The generator loss function uses a weighted fusion method of prediction loss and adversarial loss, and the calculation formula is as follows:

[0031] L G = λ1L α + λ2L ad

[0032] Wherein, λ1 and λ2 are weights, L α is the α prediction loss, and L ad is the adversarial loss.

[0033] The α measurement loss represents the absolute value difference between the true α value and the predicted α value at each pixel point, and the calculation formula is as follows:

[0034]

[0035] Wherein, represents the predicted value of a certain pixel point i, represents the true value of a certain pixel point i, and ε takes 10 -6 .

[0036] The adversarial loss comes from the adversarial mechanism of the generator and the discriminator, so as to optimize the image mask generated by the generator. The adversarial loss is calculated as the mean square error value of the result matrix obtained by inputting the image mask generated by the generator into the discriminator and the all-1 matrix tensor of the same size. When the result matrix of the image mask is close to the all-1 matrix, the discriminator cannot judge the authenticity of the image, so as to realize the dynamic balance of the generator and the discriminator. The calculation formula is as follows:

[0037]

[0038] Wherein, M is the number of pixel points, i is a pixel point sequence number, and xi is input, G(x i ) represents the image mask output by the generator network, D(G(x i )) represents the output matrix of the discriminator network, I one represents the matrix tensor, and MSE is the mean square error;

[0039] 2) Discriminator loss function:

[0040] When the image mask generated by the generator and the real image label synthesized image are input into the discriminator, the discriminator should judge the image mask as false, and the real label should be judged as true. The MSE is calculated using the output matrix of the all-0 and all-1 matrix tensor synthesized with the generated image and the real alpha image, and the loss function is defined as follows:

[0041]

[0042] Where M is the number of pixels, i is a pixel index, x i , y i represent the input of the generator and the discriminator respectively, G(x i ) represents the output of the generator network for x i , D(y i ) represents the output of the discriminator network for y i , D(G(x i )) represents the output of the discriminator network for G(x i ), I one represents the all-1 matrix tensor, Z zero represents the all-0 matrix, and MSE is the mean square error.

[0043] The image matting network model based on the GAN encoder-decoder structure built is as follows:

[0044] Encoder first layer→encoder second layer→encoder third layer→encoder fourth layer→encoder fifth layer→decoder first layer;

[0045] Encoder fourth layer→first S-BAM module→first layer of skip dense link, decoder first layer→decoder second layer;

[0046] Encoder third layer→second S-BAM module→second layer of skip dense link, decoder second layer→decoder third layer;

[0047] Encoder second layer→third S-BAM module→third layer of skip dense link, decoder third layer→decoder fourth layer;

[0048] Encoder first layer→fourth S-BAM module→fourth layer of skip dense link, decoder fourth layer→decoder fifth layer.

[0049] S-BAM is specifically as follows:

[0050] input feature image → channel dimension feature 1x1xC → channel feature coefficient Mc;

[0051] input feature image → spatial dimension feature HxWx1 → spatial feature coefficient Ms;

[0052] channel feature coefficient Mc, spatial feature coefficient Ms → addition fusion → fusion feature coefficient Mf;

[0053] input feature image, fusion feature coefficient Mf → feature map weighting → output feature image.

[0054] The image matting network model based on the encoder-decoder structure of GAN uses the following steps when in use:

[0055] The first layer obtains feature map F1.1 through a DS-ASPP module and a convolution, standardization and activation layer from feature map F1, and then fuses feature map F1 and feature map F1.1 to obtain feature map F1.2;

[0056] The second layer obtains feature map F2 by performing pooling downsampling on the feature map F1.1 output by the first layer, and then inputs the feature map F2 into a DS-ASPP module and a convolution, standardization and activation layer to obtain feature map F2.1, and then fuses feature map F2 and feature map F2.1 to obtain feature map F2.2;

[0057] The third layer obtains feature map F3 by performing pooling downsampling on the feature map F2.1 output by the second layer, and then inputs the feature map F3 into a DS-ASPP module and a convolution, standardization and activation layer to obtain feature map F3.1, and then fuses feature map F3 and feature map F3.1 to obtain feature map F3.2;

[0058] The fourth layer obtains feature map F4 by performing pooling downsampling on the feature map F3.1 output by the third layer, and then inputs the feature map F4 into a DS-ASPP module and a convolution, standardization and activation layer to obtain feature map F4.1, and then fuses feature map F4 and feature map F4.1 to obtain feature map F4.2;

[0059] The fifth layer obtains feature map F5 by performing pooling downsampling on the feature map F4.1 output by the fourth layer, and then inputs the feature map F5 into a DS-ASPP module and adjusts the channel through convolution to obtain feature map F5.1;

[0060] The fused feature maps F1.2, F2.2, F3.2 and F4.2 of the first, second, third and fourth layers are respectively input into an S-BAM module. The S-BAM combines the input features through two separate attention branches of channels Mc and space Ms to generate the final 3D feature Mf. The 3D Mf is multiplied element by element with the input feature map, and then added to the original input feature map to output the feature maps F1.3, F2.3, F3.3 and F4.3 respectively.

[0061] The feature map F4.3 of the fourth layer of the encoder is input into the third layer after upsampling. The feature map F4.3 is fused with the feature map F3.3 of the third layer of the encoder to obtain the feature map F3.4. Then the feature maps F3.3 and F3.4 are respectively transmitted into the second layer after upsampling. The feature map F2.3 of the second layer of the encoder is fused with the feature map F3.3 to obtain the feature map F2.4. The feature map F2.3 is fused with the feature map F2.4 of the second layer of the encoder to obtain the feature map F2.5. Then the feature maps F2.3, F2.4 and F2.5 are transmitted into the first layer after upsampling. The feature map F1.3 of the first layer of the encoder is fused with the feature map F2.3 to obtain the feature map F1.4. The feature map F1.3 of the first layer of the encoder is fused with the feature map F1.4 to obtain the feature map F1.5. The feature map F1.3 of the first layer of the encoder is fused with the feature map F1.5 to obtain the feature map F1.6.

[0062] The feature map F5.1 output by the fifth layer DS-ASPP of the encoder is transmitted into the second layer of the decoder and fused with the feature map F4.3 of the first layer of the jump link to obtain the feature map F4.4. The feature map F4.4 is subjected to convolution, standardization and activation layer to obtain the feature map F4.5. Then the feature map F4.5 is transmitted into the third layer of the decoder and fused with the feature map F3.4 and F3.3 of the second layer of the jump link to obtain the feature map F3.5. The feature map F3.5 is subjected to convolution, standardization and activation layer to obtain the feature map F3.6. Then the feature map F3.6 is transmitted into the fourth layer of the decoder and fused with the feature map F2.3 and F2.5 of the third layer of the jump link to obtain the feature map F2.6. The feature map F2.6 is subjected to convolution, standardization and activation layer to obtain the feature map F2.7. Then the feature map F2.7 is transmitted into the fifth layer of the decoder and fused with the feature map F1.3 and F1.6 of the fourth layer of the jump link to obtain the feature map F1.7. The feature map F1.7 is subjected to convolution, standardization and activation layer to obtain the output feature map F.

[0063] After obtaining the feature map F generated by the generator, the feature map F and the image real label are input into the discriminator respectively, the discriminator improves the ability to distinguish the segmentation result of the generated image as false through a five-layer convolutional network, the discriminator values corresponding to the generator segmentation image and the segmentation label image are calculated respectively, and the cross-entropy is used to calculate the discriminator network loss value;

[0064] The generator uses the training loss of the discriminator to continuously optimize to fit the label image so that the discriminator discriminates it as true, and the two continuously game in the training process to form a confrontation, after network training, the accuracy of the generator image segmentation result is improved, and accurate image segmentation result can be obtained in testing and actual use.

[0065] An image matting network model based on GAN encoder-decoder structure, the model network is as follows:

[0066] The first layer of the encoder→the second layer of the encoder→the third layer of the encoder→the fourth layer of the encoder→the fifth layer of the encoder→the first layer of the decoder;

[0067] The fourth layer of the encoder→the first S-BAM module→the first layer of the skip dense link, the first layer of the decoder→the second layer of the decoder;

[0068] The third layer of the encoder→the second S-BAM module→the second layer of the skip dense link, the second layer of the decoder→the third layer of the decoder;

[0069] The second layer of the encoder→the third S-BAM module→the third layer of the skip dense link, the third layer of the decoder→the fourth layer of the decoder;

[0070] The first layer of the encoder→the fourth S-BAM module→the fourth layer of the skip dense link, the fourth layer of the decoder→the fifth layer of the decoder.

[0071] S-BAM is as follows:

[0072] Input feature image→channel dimension feature 1x1xC→channel feature coefficient Mc;

[0073] Input feature image→spatial dimension feature HxWx1→spatial feature coefficient Ms;

[0074] Channel feature coefficient Mc, spatial feature coefficient Ms→additive fusion→fusion feature coefficient Mf;

[0075] Input feature image, fusion feature coefficient Mf→feature image weighting→output feature image.

[0076] The image matting network model based on GAN encoder-decoder structure adopts the following steps when in use:

[0077] The first layer obtains a feature map F1.1 through a DS-ASPP module and a convolution, standardization and activation layer from the feature map F1, and then fuses the feature map F1 and the feature map F1.1 to obtain a feature map F1.2;

[0078] The second layer obtains a feature map F2 through pooling down-sampling from the feature map F1.1 output by the first layer, and then inputs the feature map F2 to a DS-ASPP module and a convolution, standardization and activation layer to obtain a feature map F2.1, and then fuses the feature map F2 and the feature map F2.1 to obtain a feature map F2.2;

[0079] The third layer obtains a feature map F3 through pooling down-sampling from the feature map F2.1 output by the second layer, and then inputs the feature map F3 to a DS-ASPP module and a convolution, standardization and activation layer to obtain a feature map F3.1, and then fuses the feature map F3 and the feature map F3.1 to obtain a feature map F3.2;

[0080] The fourth layer obtains a feature map F4 through pooling down-sampling from the feature map F3.1 output by the third layer, and then inputs the feature map F4 to a DS-ASPP module and a convolution, standardization and activation layer to obtain a feature map F4.1, and then fuses the feature map F4 and the feature map F4.1 to obtain a feature map F4.2;

[0081] The fifth layer obtains a feature map F5 through pooling down-sampling from the feature map F4.1 output by the fourth layer, and then inputs the feature map F5 to a DS-ASPP module and adjusts the channel through convolution to obtain a feature map F5.1;

[0082] The feature maps F1.2, F2.2, F3.2 and F4.2 fused by the first layer, the second layer, the third layer and the fourth layer respectively are input into an S-BAM module, the S-BAM combines the input features through two separate attention branches of channels Mc and spaces Ms to generate a final 3D feature Mf, element-wise multiplies the 3D Mf with the input feature map, and then adds it to the original input feature map to respectively output the feature maps F1.3, F2.3, F3.3 and F4.3;

[0083] The feature map F4.3 of the fourth layer of the encoder is input to the third layer after upsampling, the feature map F4.3 is fused with the feature map F3.3 of the third layer of the encoder to obtain the feature map F3.4, and then the feature map F3.3 and the feature map F3.4 are respectively transmitted to the second layer after upsampling; the feature map F3.3 is fused with the feature map F2.3 of the second layer of the encoder to obtain the feature map F2.4, the feature map F3.4 is fused with the feature map F2.3 and the feature map F2.4 of the second layer of the encoder to obtain the feature map F2.5, and then the feature map F2.3, the feature map F2.4 and the feature map F2.5 are transmitted to the first layer after upsampling; the feature map F2.3 is fused with the feature map F1.3 of the first layer of the encoder to obtain the feature map F1.4, the feature map F2.4 is fused with the feature map F1.3 and the feature map F1.4 of the first layer of the encoder to obtain the feature map F1.5, and the feature map F2.5 is fused with the feature map F1.3 and the feature map F1.5 of the first layer of the encoder to obtain the feature map F1.6;

[0084] The feature map F5.1 output by the fifth layer DS-ASPP of the encoder is transmitted to the second layer of the decoder and fused with the feature map F4.3 of the first layer of the jump link to obtain the feature map F4.4, the feature map F4.4 is subjected to convolution, standardization and an activation layer to obtain the feature map F4.5, then the feature map F4.5 is transmitted to the third layer of the decoder and fused with the feature map F3.4 and the feature map F3.3 of the second layer of the jump link to obtain the feature map F3.5, the feature map F3.5 is subjected to convolution, standardization and an activation layer to obtain the feature map F3.6, then the feature map F3.6 is transmitted to the fourth layer of the decoder and fused with the feature map F2.3 and the feature map F2.5 of the third layer of the jump link to obtain the feature map F2.6, the feature map F2.6 is subjected to convolution, standardization and an activation layer to obtain the feature map F2.7, then the feature map F2.7 is transmitted to the fifth layer of the decoder and fused with the feature map F1.3 and the feature map F1.6 of the fourth layer of the jump link to obtain the feature map F1.7, and the feature map F1.7 is subjected to convolution, standardization and an activation layer to obtain the output feature map F.

[0085] Compared with the prior art, the present application has the following technical effects:

[0086] The present application can effectively preserve shallow features, fuse the shallow features of each layer of the encoder, introduce the shallow detail features in the backbone network into the deep network through dense jump links, thereby capturing more detail texture information, and use the DS-ASPP in each layer of the encoder in the model to perform multi-scale extraction and fusion on the features, use the S-BAM to allocate weights to the features, focus on foreground information, and improve the background and foreground distinguishing ability of the model. BRIEF DESCRIPTION OF DRAWINGS

[0087] The application will be further described below with reference to the accompanying drawings and examples:

[0088] Figure 1 Flow chart of the application;

[0089] Figure 2 Network flow chart of the application;

[0090] Figure 3 Network structure chart of the application;

[0091] Figure 4 S-BAM structure chart of the application DETAILED DESCRIPTION

[0092] The purpose of the application is to solve the technical problems of shallow feature loss, insufficient multi-scale feature extraction and complex background information confusion in complex natural image matting tasks, and to provide an image matting method which can effectively preserve shallow features, improve the background discrimination ability of the target and obtain rich multi-scale features. Figure 1 As shown in the figure, an image matting method based on deep learning includes the following steps:

[0093] Step 1: Obtain a matting dataset including an original image and a label image, generate a trimap according to the label image, and divide the samples in the matting dataset into a training set, a validation set and a test set;

[0094] Step 2: Build an image matting network model based on the encoder-decoder structure of GAN;

[0095] Step 3: Input the training set into the model for training, verify it with the validation set every epoch, calculate the loss, and save the best model;

[0096] Step 4: Input the image to be tested and the trimap into the model for testing

[0097] In step 1, the matting dataset composed of 431 unique foreground objects and corresponding alpha maps needs to be data-augmented. First, 100 images are randomly selected from the MSCOCO dataset as background images, each foreground image is combined with a background image to form a new image, and methods such as rotation and scaling are used to increase the complexity of the dataset to obtain the final training set and test set.

[0098] Specifically, in step 1, step 1: obtain the matting dataset, respectively make the training set and test set suitable for training and testing, and perform data enhancement. The method uses the Adobe Deep Matting dataset published by Xu et al. in 2017. The Adobe Deep Matting dataset only provides the foreground image and alpha mask of the training set, and the foreground image, alpha mask and ternary graph of the test set need to be additionally provided to synthesize the training graph and test graph, and the COCO dataset needs to be downloaded to combine each foreground image with 100 COCO dataset images for training and testing; in addition, since the Adobe Deep Matting dataset only contains 431 alpha masks for training and 20 alpha masks for testing, which is not enough for neural network learning, data enhancement is needed; the specific implementation steps are as follows:

[0099] The operations of data enhancement on the picture mainly include the following:

[0100] (1) Rotation: random rotation with the image center as the rotation center;

[0101] (2) Translation: random translation of the image along the X-axis or Y-axis;

[0102] (3) Scaling: random scaling of the image according to the scale to achieve multi-scale training;

[0103] (4) Color jittering: random change of the exposure, saturation and hue of the image to form pictures under different light and color;

[0104] (5) Gaussian noise: adding noise to make the image blurred.

[0105] In step 2: the original image and the ternary graph are input into the generator to train, and the image mask is obtained, and the image mask and the image label are input into the discriminator to train, the ability of the discriminator to distinguish the segmentation result of the generated image as false is continuously improved, and the discriminator values corresponding to the generator segmentation image and the segmentation label image are calculated respectively.

[0106] In step 2, the image matting network model based on the encoder-decoder structure of GAN is built, which further includes the following sub-steps:

[0107] 2-1: Construct a generator network based on Unet to generate an alpha mask that is infinitely close to a real image;

[0108] 2-2: Construct a five-layer structure discriminator network to calculate the loss using the generated signature image and the label to distinguish between fake images and real images;

[0109] In step 2-1, a Unet-based generator network is constructed to generate an alpha mask infinitely close to the real image.

[0110] The Unet-based generator network is composed of an encoder, a decoder and a skip link, and further includes the following steps:

[0111] 2-1-1: The encoder performs four times of down-sampling on the input feature map. In each layer of the encoder, a shallow layer detail feature fusion module is added, i.e. the shallow layer features before convolution are spliced with the features after convolution to obtain a new feature that retains the shallow layer features.

[0112] Specifically:

[0113] 2-1-1: The encoder performs four times of down-sampling on the input feature map. The first layer obtains F1.1 through a DS-ASPP module and a convolution, standardization and activation layer, and then fuses F1 with F1.1 to obtain F1.2; the second layer obtains F2 by down-sampling the output of the first layer F1.1 through pooling, and then inputs F2 to a DS-ASPP module and a convolution, standardization and activation layer to obtain F2.1, and then fuses F1 with F1.1 to obtain F1.2; the third layer obtains F3 by down-sampling the output of the second layer F2.1 through pooling, and then inputs F3 to a DS-ASPP module and a convolution, standardization and activation layer to obtain F3.1, and then fuses F3 with F3.1 to obtain F3.2; the fourth layer obtains F4 by down-sampling the output of the third layer F3.1 through pooling, and then inputs F4 to a DS-ASPP module and a convolution, standardization and activation layer to obtain F4.1, and then fuses F4 with F4.1 to obtain F4.2; the fifth layer obtains F5 by down-sampling the output of the fourth layer F4.1 through pooling, and then inputs F5 to a DS-ASPP module and adjusts the channels through convolution to obtain F5.1.

[0114] 2-1-2: An S-BAM module is added in the dense skip link of each layer. S-BAM combines the new features that retain the shallow layer features in each layer of the encoder through two separate attention branches of channels Mc and space Ms to generate the final 3D feature Mf, element-wise multiplies the 3D Mf with the input feature map, and then adds it to the original input feature map to obtain a refined feature map.

[0115] Specifically: 2-1-2: the features F1.2, F2.2, F3.2, F3.2 fused respectively in the first layer, the second layer, the third layer, the fourth layer are input into an S-BAM module, S-BAM combines the input features through two separate attention branches of channels Mc and space Ms to generate the final 3D feature Mf, element-wise multiplies the 3D Mf with the input feature map, and then adds it to the original input feature map to output F1.3, F2.3, F1.3, F2.3, F3.3, F4.3 respectively.

[0116] 2-1-3: a DS-ASPP module is added between the encoder and the decoder to extract features of high-level semantic information at multiple scales. DS-ASPP performs 1x1 convolution on the feature map output by the encoder, three times of 3x3 depth separable convolution with a hole rate of 1, 2, and 5 respectively, and once average pooling, and then fuses the five layers of extracted features in the form of channel splicing and inputs them into the decoder.

[0117] Specifically: 2-1-3: in the skip dense link, the features output by each layer of the encoder are fused respectively. F4.3 of the fourth layer of the encoder is input into the third layer after upsampling, F4.3 and F3.3 of the third layer of the encoder are fused to obtain F3.4, then F3.3 and F3.4 are respectively transmitted to the second layer after upsampling; F2.3 of the second layer of the encoder is fused to obtain F2.4, F3.4 and F2.3, F2.4 of the second layer of the encoder are fused to obtain F2.5, then F2.3, F2.4, and F2.5 are transmitted to the first layer after upsampling; F1.3 of the first layer of the encoder is fused to obtain F1.4, F2.4 and F1.3 of the first layer of the encoder are fused to obtain F1.5, and F1.5 and F1.3 of the first layer of the encoder are fused to obtain F1.6.

[0118] 2-1-4: the decoder performs four times of upsampling on the feature map output by the DS-ASPP, and fuses the feature map output by the skip link in the form of channel splicing after each upsampling. After four times of upsampling, the alpha mask is obtained.

[0119] Specifically: 2-1-4: The decoder upsamples the feature map output by DS-ASPP four times, and after each upsampling, it merges it with the feature map output by the skip link in a channel-by-channel concatenation manner. First, the feature F5.1 output from the fifth layer of the encoder's DS-ASPP is upsampled and fed into the second layer of the decoder, where it is fused with F4.3 from the first layer of the skip link to obtain F4.4. F4.4 is then processed through convolution, normalization, and activation layers to obtain F4.5. F4.5 is then upsampled and fed into the third layer of the decoder, where it is fused with F3.4 and F3.3 from the second layer of the skip link to obtain F3.5. F3.5 is then processed through convolution, normalization, and activation layers to obtain F3.6. F3.6 is then upsampled and fed into the fourth layer of the decoder, where it is fused with F2.3 and F2.5 from the third layer of the skip link to obtain F2.6. F2.6 is then processed through convolution, normalization, and activation layers to obtain F2.7. F2.7 is then upsampled and fed into the fifth layer of the decoder, where it is fused with F1.3 and F1.6 from the fourth layer of the skip link to obtain F1.7. F1.7 is then processed through convolution, normalization, and activation layers to obtain the output F.

[0120] In step 2-2, a discriminator with a five-layer convolutional network structure is constructed. The loss is calculated using the results of the generated image tagger and the label, and the fake image is distinguished from the real image. Specifically, the following steps are also included:

[0121] 2-2-1: Obtain the α mask output by the generator. Input the α mask and the true image label into the discriminator. Through training, the discriminator's ability to classify the segmentation result of the generated image as false continuously improves. Calculate the discriminator values ​​corresponding to the segmented image and the segmented label image. Then, use cross-entropy to calculate the discriminator network loss value.

[0122] 2-2-2: The generator continuously optimizes its image segmentation by utilizing the discriminator's training loss to fit the labeled image so that the discriminator classifies it as true. The two continuously engage in a game-like process during training, creating an adversarial relationship. After network training, the generator's image segmentation accuracy improves, achieving precise image segmentation results in both testing and real-world applications.

[0123] Table 1 Ablation Experiment

[0124]

[0125] The ablation experiment first used PatchGAN as the discriminator as the base network on the dataset. Then, each layer of the encoder was fused, and the DS-ASPP structure was added. Finally, S-BAM was added for the experiment. In the experimental results, the absolute error, SAD and mean squared error MSE were all reduced compared with the initial case, and the image matting effect was improved.

[0126] In step 3: the generator constantly optimizes the image mask to fit the label image so that the discriminator will judge it to be true, and the two constantly play against each other in the training process, forming a confrontation. After network training, the accuracy of the generator image segmentation result is improved, and the optimal result is obtained.

[0127] In step 3, the loss function for network training is as follows:

[0128] 1) Generator loss function

[0129] The generator loss function uses a weighted fusion method of prediction loss and adversarial loss, and the calculation formula is as follows:

[0130] L G = λ1L α + λ2L ad

[0131] Where λ1 and λ2 are weights, L α is the α prediction loss, and L ad is the adversarial loss.

[0132] The α prediction loss represents the absolute value difference between the true α value and the predicted α value at each pixel point, and the calculation formula is as follows:

[0133]

[0134] Where, represents the predicted value at a certain pixel point i, represents the true value at a certain pixel point i, and ε takes 10 -6 .

[0135] The adversarial loss comes from the adversarial mechanism of the generator and the discriminator, so as to optimize the image mask generated by the generator. The adversarial loss is calculated as the mean square error value of the result matrix obtained by inputting the image mask generated by the generator into the discriminator and the all-1 matrix tensor of the same size. When the result matrix of the image mask is close to the all-1 matrix, the discriminator cannot judge the authenticity of the image, thereby realizing the dynamic balance of the generator and the discriminator. The calculation formula is as follows:

[0136]

[0137] Where M is the number of pixel points, i is a pixel point sequence number, x i is the input, G(x i ) represents the image mask output by the generator network, D(G(x i )) represents the result matrix output by the discriminator network, I one represents the matrix tensor, and MSE is the mean square error.

[0138] Wherein, the adversarial loss is derived from the adversarial mechanism of the generator and the discriminator, so as to optimize the image mask generated by the generator, and the adversarial loss is calculated as the mean square error value of the result matrix obtained by inputting the image mask generated by the generator into the discriminator and the all-1 matrix tensor of the same size, when each element of the result matrix of the image mask is close to the all-1 matrix, the discriminator cannot distinguish the authenticity of the image, so that the dynamic balance of the generator and the discriminator is realized.

[0139] 2) Discriminator loss function:

[0140] When the image mask generated by the generator and the image synthesized by the real image label are input into the discriminator, the discriminator should distinguish the image mask as false, and the real label should be distinguished as true, the MSE is calculated by using the all-0 and all-1 matrix tensors and the output matrix of the synthesized image of the generated image and the real alpha image, and the loss function is defined as follows:

[0141]

[0142] Wherein, M is the number of pixel points, i is a pixel point sequence number, x i ,y i respectively represent the input of the generator and the discriminator, G(x i ) represents the output of the generator network for x i , D(y i ) represents the output of the discriminator network for y i , D(G(x i )) represents the output of the discriminator network for G(x i ), I one represents the all-1 matrix tensor, Z zero represents the all-0 matrix, and MSE is the mean square error.

[0143] Wherein, when the image mask generated by the generator and the image synthesized by the real image label are input into the discriminator, the discriminator should distinguish the image mask as false, and the real label should be distinguished as true, the MSE is calculated by using the all-0 and all-1 matrix tensors and the output matrix of the synthesized image of the generated image and the real alpha image.

[0144] In step 4, for the image to be tested, the corresponding tripartite graph is generated, and the original image and the tripartite Figure 1 image are input into the model for testing to obtain the test result.

[0145] The application also provides an image matting network model based on a GAN encoder-decoder structure, and the model network is as follows:

[0146] Encoder first layer 1→encoder second layer 2→encoder third layer 3→encoder fourth layer 4→encoder fifth layer 5→decoder first layer 6.

[0147] Encoder fourth layer 4→ first S-BAM module 13→ skip dense link first layer 14, decoder first layer 6→ decoder second layer 7;

[0148] Encoder third layer 3→ second S-BAM module 12→ skip dense link second layer 15, decoder second layer 7→ decoder third layer 8;

[0149] Encoder second layer 2→ third S-BAM module 11→ skip dense link third layer 16, decoder third layer 8→ decoder fourth layer 9;

[0150] Encoder first layer 1→ fourth S-BAM module 10→ skip dense link fourth layer 17, decoder fourth layer 9→ decoder fifth layer 10.

[0151] The S-BAM is as follows:

[0152] Input feature image 18→ channel dimension feature 1×1×C 19→ channel feature coefficient Mc 21;

[0153] Input feature image 18→ spatial dimension feature H×W×1 20→ spatial feature coefficient Ms 22;

[0154] Channel feature coefficient Mc 21, spatial feature coefficient Ms 22→ addition fusion 23→ fusion feature coefficient Mf 24;

[0155] Input feature image 18, fusion feature coefficient Mf 24→ feature map weighting 25→ output feature image 26.

[0156] The image matting network model based on the encoder-decoder structure of GAN uses the following steps when in use:

[0157] The first layer obtains feature map F1.1 through a DS-ASPP module and a convolution, standardization and activation layer from feature map F1, and then fuses feature map F1 and feature map F1.1 to obtain feature map F1.2;

[0158] The second layer obtains feature map F2 by performing pooling downsampling on the feature map F1.1 output by the first layer, and then inputs the feature map F2 into a DS-ASPP module and a convolution, standardization and activation layer to obtain feature map F2.1, and then fuses feature map F2 and feature map F2.1 to obtain feature map F2.2;

[0159] The third layer obtains feature map F3 by performing pooling downsampling on the feature map F2.1 output by the second layer, and then inputs the feature map F3 into a DS-ASPP module and a convolution, standardization and activation layer to obtain feature map F3.1, and then fuses feature map F3 and feature map F3.1 to obtain feature map F3.2;

[0160] The fourth layer obtains a feature map F4 by pooling and down-sampling the feature map F3,1 output by the third layer, inputs the feature map F4 into a DS-ASPP module and a convolution, standardization and activation layer to obtain a feature map F4,1, and then fuses the feature map F4 and the feature map F4,1 to obtain a feature map F4,2;

[0161] The fifth layer obtains a feature map F5 by pooling and down-sampling the feature map F4,1 output by the fourth layer, inputs the feature map F5 into a DS-ASPP module and obtains a feature map F5,1 after convolution and channel adjustment;

[0162] The feature maps F1,2, F2,2, F3,2 and F4,2 fused by the first layer, the second layer, the third layer and the fourth layer, respectively, are input into an S-BAM module, the S-BAM combines the input features through two separate attention branches of channels Mc and spaces Ms to generate a final 3D feature Mf, element-wise multiplies the 3D Mf and the input feature map, and then adds it to the original input feature map to output the feature maps F1,3, F2,3, F3,3 and F4,3, respectively;

[0163] The feature map F4,3 of the fourth layer of the encoder is input into the third layer after up-sampling, the feature map F4,3 is fused with the feature map F3,3 of the third layer of the encoder to obtain a feature map F3,4, then the feature map F3,3 and the feature map F3,4 are respectively transmitted to the second layer after up-sampling; the feature map F2,4 is obtained by fusing the feature map F3,3 and the feature map F2,3 of the second layer of the encoder, the feature map F2,5 is obtained by fusing the feature map F3,4 and the feature map F2,3 and the feature map F2,4 of the second layer of the encoder, then the feature map F2,3, the feature map F2,4 and the feature map F2,5 are transmitted to the first layer after up-sampling; the feature map F1,4 is obtained by fusing the feature map F2,3 and the feature map F1,3 of the first layer of the encoder, the feature map F1,5 is obtained by fusing the feature map F2,4 and the feature map F1,3 and the feature map F1,4 of the first layer of the encoder, and the feature map F1,6 is obtained by fusing the feature map F2,5 and the feature map F1,3 and the feature map F1,5 of the first layer of the encoder;

[0164] The feature map F5.1 output by the fifth layer DS-ASPP of the encoder is transmitted into the feature map F4.3 of the second layer of the decoder and the first layer of the skip connection after upsampling to obtain the feature map F4.4, the feature map F4.4 is obtained through convolution, standardization and activation layer to obtain the feature map F4.5, then the feature map F4.5 is transmitted into the feature map F3.4, the feature map F3.3 of the third layer of the decoder and the second layer of the skip connection after upsampling to obtain the feature map F3.5, the feature map F3.5 is obtained through convolution, standardization and activation layer to obtain the feature map F3.6, then the feature map F3.6 is transmitted into the feature map F2.3, the feature map F2.5 of the fourth layer of the decoder and the third layer of the skip connection after upsampling to obtain the feature map F2.6, the feature map F2.6 is obtained through convolution, standardization and activation layer to obtain the feature map F2.7, then the feature map F2.7 is transmitted into the feature map F1.3, the feature map F1.6 of the fifth layer of the decoder and the fourth layer of the skip connection after upsampling to obtain the feature map F1.7, the feature map F1.7 is obtained through convolution, standardization and activation layer to obtain the output feature map F.

[0165] The model can effectively preserve shallow features, fuse the shallow features of each layer in the encoder, introduce the shallow detail features in the backbone network to the deep network through dense skip connection, thereby capturing more detail texture information, and use DS-ASPP in each layer of the encoder in the model to perform multi-scale extraction and fusion on the features; the s-BAM is used to allocate weights to the features, focus on foreground information, and improve the background and foreground distinguishing ability of the model.

Claims

1. A deep learning-based image matting method, characterized in that, Includes the following steps: Step 1: Obtain the image matting dataset, including the original image and the label image. Generate a three-part image based on the label image and divide the samples in the image matting dataset into a training set, a validation set, and a test set. Step 2: Build an image matting network model based on a GAN encoder-decoder structure; Step 3: Input the training set into the model for training. After each training round, validate the model with the validation set, calculate the loss, and save the best model, that is, keep the model with the lowest loss function. Step 4: Input the image to be tested and the triangulation into the model for testing; The image matting network model based on the GAN encoder-decoder structure is as follows: Encoder layer 1 (1) → Encoder layer 2 (2) → Encoder layer 3 (3) → Encoder layer 4 (4) → Encoder layer 5 (5) → Decoder layer 1 (6); Encoder layer 4 (4) → first S-BAM module (13) → skip dense link layer 1 (14), decoder layer 1 (6) → decoder layer 2 (7); Encoder layer 3 (3) → second S-BAM module (12) → skip dense link layer 2 (15), decoder layer 2 (7) → decoder layer 3 (8); Encoder layer 2 (2) → third S-BAM module (11) → skip dense link layer 3 (16), decoder layer 3 (8) → decoder layer 4 (9); Encoder layer 1 (1) → fourth S-BAM module (10) → skip dense link layer 4 (17), decoder layer 4 (9) → decoder layer 5 (10). S-BAM is detailed below: Input feature image (18) → Channel dimension feature 1×1×C (19) → Channel feature coefficient Mc (21); Input feature image (18) → Spatial dimension feature H×W×1 (20) → Spatial feature coefficient Ms (22); Channel characteristic coefficient Mc (21), spatial characteristic coefficient Ms (22) → addition and fusion (23) → fusion characteristic coefficient Mf (24); Input feature image (18), fuse feature coefficients Mf (24) → feature map weighting (25) → output feature image (26).

2. The method according to claim 1, characterized in that, In step 2, the following method is used when building the image matting network model based on the GAN encoder-decoder structure: 2-1: Construct a generator network based on Unet to generate alpha masks that are infinitely close to real images; 2-2: Construct a five-layer discriminator network, calculate the loss using the generator results and label map, and distinguish between fake images and real images.

3. The method according to claim 2, characterized in that, In step 2-1, a Unet-based generator network is constructed to generate an alpha mask that is infinitely close to the real image. The Unet-based generator network consists of three parts: an encoder, a decoder, and skip dense connections. Specifically, it also includes the following steps: 2-1-1: The encoder downsamples the input feature map four times; the first layer passes the feature map F1 through a DS-ASPP module and a convolution, normalization and activation layer to obtain feature map F1.1, and then the feature map F1 and feature map F1.1 are fused to obtain feature map F1.2; The second layer takes the feature map F1.1 output from the first layer and downsamples it through pooling to obtain feature map F2. Then, feature map F2 is input into a DS-ASPP module and a convolution, normalization and activation layer to obtain feature map F2.

1. Finally, feature map F2 and feature map F2.1 are fused to obtain feature map F2.

2. The third layer takes the feature map F2.1 output from the second layer and downsamples it through pooling to obtain feature map F3. Feature map F3 is then input into a DS-ASPP module and a convolution, normalization and activation layer to obtain feature map F3.

1. Finally, feature map F3 and feature map F3.1 are fused to obtain feature map F3.

2. The fourth layer takes the feature map F3,1 output by the third layer and downsamples it through pooling to obtain the feature map F4. The feature map F4 is then input into a DS-ASPP module and a convolution, normalization and activation layer to obtain the feature map F4.

1. Finally, the feature map F4 and the feature map F4.1 are fused to obtain the feature map F4.

2. The fifth layer takes the feature map F4,1 output from the fourth layer and downsamples it through pooling to obtain feature map F5. Feature map F5 is then input into a DS-ASPP module and adjusted through convolution to obtain feature map F5.

1. 2-1-2: The feature maps F1.2, F2.2, F3.2, and F4.2 obtained by fusing the first, second, third, and fourth layers respectively are input into an S-BAM module. The S-BAM combines the input features through two separate attention branches, channel Mc and spatial Ms, to generate the final 3D feature Mf. The 3D Mf is then multiplied element-wise with the input feature map and added to the original input feature map, resulting in the output feature maps F1.3, F2.3, F3.3, and F4.3 respectively. 2-1-3: In the skip-dense link, the features output from each encoding layer are densely fused. The feature map F4.3 of the encoder's fourth layer is upsampled and input into the third layer. Feature map F4.3 is fused with the feature map F3.3 of the encoder's third layer to obtain feature map F3.

4. Then, feature maps F3.3 and F3.4 are upsampled and input into the second layer. Feature map F3.3 is fused with the feature map F2.3 of the encoder's second layer to obtain feature map F2.

4. Feature map F3.4 is then fused with the feature map F2.3 of the encoder's second layer. 2.

3. Feature map F2.4 is fused to obtain feature map F2.

5. Then, feature maps F2.3, F2.4, and F2.5 are upsampled and fed into the first layer. Feature map F2.3 is fused with feature map F1.3 of the first layer encoder to obtain feature map F1.

4. Feature map F2.4 is fused with feature maps F1.3 and F1.4 of the first layer encoder to obtain feature map F1.

5. Feature map F2.5 is fused with feature maps F1.3 and F1.5 of the first layer encoder to obtain feature map F1.

6. 2-1-4: The decoder upsamples the feature map output by DS-ASPP four times. After each upsampling, it is fused with the feature map output by the skip link via channel concatenation. First, the feature map F5.1 output by the fifth layer of the encoder's DS-ASPP is upsampled and fed into the second layer of the decoder, where it is fused with the feature map F4.3 from the first layer of the skip link to obtain feature map F4.

4. Feature map F4.4 is then processed through convolution, normalization, and activation layers to obtain feature map F4.

5. Finally, feature map F4.5 is upsampled and fed into the third layer of the decoder, where it is fused with the feature maps F3.4 and F3.3 from the second layer of the skip link to obtain the final feature map. F3.5 is processed through convolution, normalization, and activation layers to obtain feature map F3.

6. Then, feature map F3.6 is upsampled and fed into the fourth layer of the decoder, where it is fused with feature maps F2.3 and F2.5 from the third layer of the skip link to obtain feature map F2.

6. Feature map F2.6 is then processed through convolution, normalization, and activation layers to obtain feature map F2.

7. Then, feature map F2.7 is upsampled and fed into the fifth layer of the decoder, where it is fused with feature maps F1.3 and F1.6 from the fourth layer of the skip link to obtain feature map F1.

7. Feature map F1.7 is then processed through convolution, normalization, and activation layers to obtain output feature map F.

4. The method according to claim 2, characterized in that, In step 2-2, a five-layer convolutional network discriminator is constructed. The loss is calculated using the generator results and the label map to distinguish between fake and real images. Specifically, the steps include: 2-2-1: Obtain the generator output F, and input F and the real image label into the discriminator. The discriminator continuously improves its ability to distinguish the segmentation result of the generated image as false through a five-layer convolutional network. Calculate the discriminator values ​​corresponding to the segmented image and the segmented label image, and use cross-entropy to calculate the discriminator network loss value. 2-2-2: The generator uses the training loss of the discriminator to continuously optimize and fit the labeled image so that the discriminator classifies it as true. The two constantly play against each other during the training process. After network training, the accuracy of the generator's image segmentation results is improved, and accurate image segmentation results can be obtained in testing and actual use.

5. The method according to claim 1, characterized in that, In step 3, the loss function used for network training is as follows: 1) Generator Loss Function The generator loss function uses a weighted fusion of prediction loss and adversarial loss, and the calculation formula is as follows: ； Where λ1 and λ2 are weights, Lα is the α-prediction loss, and Lad is the adversarial loss; The alpha loss represents the absolute difference between the true alpha value and the predicted alpha value at each pixel, and is calculated using the following formula: ； in, This represents the predicted value at a certain pixel i. This represents the actual value at a given pixel i. Take 10 -6 ; Adversarial loss originates from the adversarial mechanism between the generator and the discriminator, thereby optimizing the image mask generated by the generator. The adversarial loss is calculated as the mean squared error between the resulting matrix of the generator's image mask input to the discriminator and a tensor of all-one matrices of the same size. When each element of the resulting matrix of the image mask is close to an all-one matrix, the discriminator cannot distinguish between genuine and fake images, thus achieving a dynamic balance between the generator and the discriminator. The calculation formula is as follows: ； Where M is the number of pixels, i is the pixel index, and x is the number of pixels. i It is the input, G(x) i ) represents the image mask output by the generator network, D(G(x) i )) represents the result matrix output by the discriminator network, I one This represents a matrix tensor, and MSE is the mean squared error. 2) Discriminator loss function: When the image synthesized from the generator's image mask and the real image label is input into the discriminator, the discriminator should classify the image mask as false and the real label as true. The MSE is calculated using the output matrix of the synthesized image from the generator image and the real α image, respectively, using all-zero and all-one matrix tensors. The loss function is defined as follows: ； Where M is the number of pixels, i is the pixel index, and x i ,y i Let G(x) represent the inputs to the generator and discriminator, respectively. i ) indicates that for x i The output of the generator network, D(y) i ) indicates that for y i The output of the discriminator network, D(G(x) i )) represents G(x i The output of the discriminator network, I one Let Z represent a matrix tensor consisting entirely of 1s. zero This represents a matrix of all zeros, and MSE is the mean squared error.

6. An image matting system based on a GAN encoder-decoder structure, characterized in that, The system is as follows: Encoder layer 1 (1) → Encoder layer 2 (2) → Encoder layer 3 (3) → Encoder layer 4 (4) → Encoder layer 5 (5) → Decoder layer 1 (6); Encoder layer 4 (4) → first S-BAM module (13) → skip dense link layer 1 (14), decoder layer 1 (6) → decoder layer 2 (7); Encoder layer 3 (3) → second S-BAM module (12) → skip dense link layer 2 (15), decoder layer 2 (7) → decoder layer 3 (8); Encoder layer 2 (2) → third S-BAM module (11) → skip dense link layer 3 (16), decoder layer 3 (8) → decoder layer 4 (9); Encoder layer 1 (1) → fourth S-BAM module (10) → skip dense link layer 4 (17), decoder layer 4 (9) → decoder layer 5 (10). S-BAM is detailed below: Input feature image (18) → Channel dimension feature 1×1×C (19) → Channel feature coefficient Mc (21); Input feature image (18) → Spatial dimension feature H×W×1 (20) → Spatial feature coefficient Ms (22); Channel characteristic coefficient Mc (21), spatial characteristic coefficient Ms (22) → addition and fusion (23) → fusion characteristic coefficient Mf (24); Input feature image (18), fuse feature coefficients Mf (24) → feature map weighting (25) → output feature image (26).

Images