A weakly supervised natural image matting method based on patch clustering

Abstract

The present invention discloses a weakly supervised natural image matting method based on block clustering. Specifically, the user first uses data containing real transparency mask annotations to train the teacher network; secondly, the transparency mask similarity metric is used to train the block clustering module; then, the trained block clustering module is used to cluster the data containing real transparency mask annotations and optimize the clustering results; finally, all the data are used to supervise the student network using traditional image matting loss and clustering prior loss function respectively. When in use, the user inputs the image to be matted and its corresponding ternary map into the student network to obtain the final matting result. For the first time, the present invention improves the matting effect by using data without real transparency mask annotations under the premise of only a small amount of real transparency mask annotations, which can overcome the problem that the previous method requires a large amount of labeled training data.

Description

Technical Field

[0001] The present invention relates to the technical field of natural image matting, and in particular to a weakly supervised natural image matting method based on block clustering. Background Art

[0002] The goal of natural image matting is to extract the pixel transparency of a specified object at each position in the image. Traditional algorithms constrain the solution space by artificially providing priors (such as ternary images) and establishing color and spatial relationship models. Recently proposed matting methods are often based on deep convolutional neural networks, which predict transparency masks by learning multi-scale high-dimensional semantic features. Compared with traditional algorithms, these methods have achieved more significant results, because deep neural networks can learn more effective features from a large amount of labeled data. However, due to the high precision requirements of the matting task, it is very difficult to annotate the training data required, and professionals are often required to estimate the transparency of the image pixel by pixel. It can be seen that data annotation for natural image matting requires a lot of manpower costs, which is a major obstacle to the widespread application of deep neural network-based matting methods. Summary of the invention

[0003] In view of the shortcomings of the prior art, the present invention proposes a weakly supervised natural image cutout method based on block clustering. This invention is the first to improve the cutout effect by using data without real transparency mask annotations under the premise of only a small amount of real transparency mask annotations. This can overcome the problem that previous methods require a large amount of labeled training data, and greatly reduce the application difficulty of deep neural network-based cutouts.

[0004] In order to achieve the above object, the present invention adopts the following technical solution: a weakly supervised natural image matting method based on block clustering, which comprises the following steps:

[0005] S1: Obtain training data, including a data set L with a true transparency mask annotation and a data set U without a true transparency mask annotation; wherein the data set L includes three parts: a natural image, a ternary map corresponding to the natural image, and a true transparency mask corresponding to the natural image; the data set U includes two parts: a natural image and its corresponding ternary map; wherein the ternary map divides all pixels in the natural image into three categories, including foreground pixels, background pixels, and pixels to be solved; the true transparency mask represents the transparency of the corresponding pixel at each position in the natural image;

[0006] S2: Use the dataset L and the deep learning cutout loss function to train the teacher network T. The network T is a deep neural network based on the encoder-decoder architecture, which includes an encoder and a decoder. A natural image and its corresponding ternary image are input. The encoder maps the input into high-dimensional semantic feature data. The decoder decodes the real transparency mask from the high-dimensional semantic feature data as the prediction result. The network T can be trained by calculating the difference between the prediction result and the real transparency mask using the cutout loss function.

[0007] S3: Use the dataset L and the transparency mask similarity loss to train the patch clustering module; the patch clustering module includes the encoder of the network T trained in S2 and a mapper network M; the network M is a multi-layer perceptron (MLP), which is responsible for mapping the feature data output by the encoder of the network T to feature data for patch clustering; during training, a natural image and its corresponding ternary image are input, and the predicted patch clustering feature vector is output after passing through the encoder of the network T and the mapper network M; finally, the transparency mask similarity loss is used to train the network M, and the encoder of the network T does not participate in the training;

[0008] S4: Use the tile clustering module trained in S3 to predict the clustering feature data of the data set L, perform clustering operations on the predicted results and optimize the clustering results; wherein, firstly, the natural images in the data set L and their corresponding ternary image data are sent to the tile clustering module to obtain the corresponding tile clustering feature data; after all the tile clustering feature data are obtained, the feature data are clustered using the K-means algorithm and the cluster center is obtained;

[0009] S5: further optimizing the cluster centers obtained in S4 and collecting transparency masks near the cluster centers; wherein the cluster center optimization includes filtering invalid cluster centers and aggregating similar cluster centers;

[0010] S6: Use dataset L, dataset U, and the optimized cluster centers and their nearby transparency masks in S5 to train the student network S; wherein, network S is a deep neural network based on an encoder-decoder architecture with the same structure as network T, and is initialized using the weights of network T trained in S2; during training, the data of datasets L and dataset U are simultaneously fed into network S, and network S outputs the prediction result of the transparency mask; for the data in dataset L, a cutout loss function is used and the known real transparency mask in dataset L is used for training; for the data in dataset U, the data is first fed into the trained tile clustering module in S3 to obtain the tile clustering feature data corresponding to the data, and then the nearest cluster center optimized by S5 and its nearby transparency mask are searched based on this feature data, and finally the clustering prior loss is used to train using the found transparency mask;

[0011] S7: Use the network S trained in S6 to predict the transparency mask; input the natural image to be cutout and its corresponding ternary image to the network S, and the network S predicts the transparency mask corresponding to the natural image.

[0012] As a preferred solution of the present invention, the teacher network T is trained using the data set L and the deep learning cutout loss function in S2, specifically:

[0013] Firstly, an image classification model trained on a large image classification dataset is used as the network initialization, and then the dataset L and the corresponding data augmentation techniques are used to train the network T using a commonly used cutout loss function; the data augmentation techniques include random data augmentation techniques, random non-rigid body deformation and transparency jitter; wherein, the random non-rigid body deformation includes elastic deformation (Elastic Transform) and thin plate spline deformation (Thin Plate Spline Transform), and the transparency jitter is a random gamma transform of the real transparency mask.

[0014] As a preferred solution of the present invention, the transparency mask similarity loss in S3 is specifically:

[0015] Given a natural image dataset L And its corresponding ternary graph data and the actual transparency mask data Will and After being sent to the encoder of network T for encoding, feature data F is obtained L , transparency mask similarity loss function L as It can be calculated as follows:

[0016] L as =τ(s p ,s n )

[0017]

[0018]

[0019]

[0020] s p ={φ(M T (f i ), M T (f j ))},

[0021]

[0022] s n ={φ(M T (f i ), M T (f j ))},

[0023]

[0024] Among them, f i , f j ∈F L is the characteristic data F L The feature data at position i and position j in Is the real transparency mask data The corresponding transparency mask blocks at positions i and j; M T is the network M; τ is the Circle loss, γ, θ m is the Circle loss related parameter, s p and n They are the intra-class pairs and inter-class pairs in Circle loss, respectively; φ is the cosine similarity, and H and G are two indicators of transparency mask similarity measurement, respectively; is the threshold used when selecting intra-class and inter-class pairs.

[0025] As a preferred solution of the present invention, the transparency mask similarity metrics H and G are calculated as follows:

[0026]

[0027]

[0028] where α i , α j are two transparency mask blocks of the same size, ||·||1 is the L1 paradigm, is the image gradient operator, is a sort operation.

[0029] As a preferred solution of the present invention, the block clustering module is trained using the dataset L and the transparency mask similarity loss in S3, specifically:

[0030] The encoder in the network T is used as the initialization of the encoder of the tile clustering module, and the mapper is randomly initialized; then the data set L and the corresponding data augmentation techniques are used to train the network M pair using the transparency mask similarity loss; the data augmentation techniques include random data augmentation techniques, random non-rigid body deformation and transparency jitter; wherein the random non-rigid body deformation includes elastic deformation (Elastic Transform) and thin plate spline deformation (Thin Plate Spline Transform), and the transparency jitter is a random gamma transform of the real transparency mask; the encoder of the tile clustering module does not participate in the training.

[0031] As a preferred solution of the present invention, in S5, a cluster center that satisfies any one of the following two conditions will be judged as an invalid cluster center:

[0032] C1, after being mapped by the tile clustering module in the dataset L, the number of data closest to the cluster center is less than 0.1% of the total number of data in the dataset L;

[0033] C2: In the transparency mask set of the tiles corresponding to the 5% feature data closest to the cluster center, less than half of the transparency mask tile pairs are intra-class pairs, that is, s p The number of elements in the set is less than 50% of all pairs.

[0034] As a preferred solution of the present invention, in S5, the method for determining similar cluster centers is specifically as follows:

[0035] D1. First, the similarity between the two categories is calculated. The similarity is obtained by calculating the ratio of the transparency masks of the tiles corresponding to the 5% of feature data closest to the two cluster centers that belong to the same class. Specifically:

[0036]

[0037]

[0038]

[0039] k∈n i , l∈n j

[0040] Among them, J is the cluster center similarity matrix, J ij represents the similarity between cluster centers i and j, |·| is a counting operation, and n i , n j are the numbers of the 5% of feature data closest to cluster centers i and j respectively;

[0041] D2. After obtaining J, use the threshold θJ The similarity matrix J can be converted into an adjacency matrix B, namely:

[0042]

[0043] In this case, the matrix B can represent an undirected graph G = (V, E), where V = {1..K}, E = {(i, j) | J ij >θ J}; When there is an edge directly connecting two cluster centers, that is, B ij =1, the two cluster centers are considered similar and can be aggregated into the same group; on the contrary, if the two cluster centers are still not connected after one hop in the graph, that is, (B 2 ) ij When =0, the two cluster centers are considered to be dissimilar.

[0044] As a preferred solution of the present invention, in S6, the clustering prior loss is specifically:

[0045] The transparency mask of the image block corresponding to the nearest N feature data of the cluster center is used as the anchor patch, and the transparency mask similarity metric is used to calculate the difference between the prediction result of the network S and the anchor patch and train the network S accordingly; specifically:

[0046] Given K cluster centers, the anchor blocks The data of dataset U is clustered by the clustering label c output by the tile clustering module, the transparency mask prediction α output by the network S, and the clustering prior loss L cp It can be written as:

[0047]

[0048]

[0049]

[0050]

[0051]

[0052] D(α i , α j ) = [1-χ(H(αx, α j ))][1-χ(G(α i , α j ))]

[0053]

[0054]

[0055] where f i , c i , α i They correspond to the feature data of a certain block i in the dataset U after passing through the encoder of the network T, the clustering label output by the block clustering module, and the transparency mask predicted by the network S; e j represents the jth cluster center; is the cluster label c i Corresponding similar cluster centers and non-similar cluster centers; D(α i , α j ) is an indicator based on the transparency mask similarity measurement, and χ is a linear normalization function that can compress the original indicator to the interval [0, 1]; are the similarity indices of the image patch i to its nearest anchor blocks of similar and dissimilar cluster centers, respectively; θ c Yes p and d n The margin difference threshold of the indicator.

[0056] As a preferred solution of the present invention, the student network S is trained using the dataset L, the dataset U, and the optimized cluster center and the transparency mask near it in S5 in S6, specifically:

[0057] First, the network T trained in S2 is used as the initialization of the network S, and then the data sets L and U and the corresponding data augmentation techniques are used to train the network T using the commonly used cutout loss function and the clustering prior loss function; the data augmentation techniques include random data augmentation techniques, random non-rigid body deformation and transparency jitter; among them, the random non-rigid body deformation includes elastic deformation (Elastic Transform) and thin plate spline deformation (Thin Plate Spline Transform), and the transparency jitter is a random gamma transform of the real transparency mask.

[0058] As a preferred embodiment of the present invention, during the training process, the overall loss function of the network S is the weighted sum of the cutout loss function and the clustering prior loss; specifically, the overall loss function L S It can be written as:

[0059] L S =L im +θ S L cp

[0060] Among them, θ S is the weight of the clustering prior loss, L im is the image matting loss function.

[0061] The beneficial effects of the present invention are:

[0062] 1. An effective differentiable indicator that can measure the similarity between transparency mask blocks is proposed. It can not only be used for the training of weakly supervised image matting tasks, but also can be used as an auxiliary loss function in the training of supervised matting tasks to improve the training effect.

[0063] 2. In addition, a new random data augmentation technique based on random non-rigid deformation and transparency jitter is used to further improve the utilization of data with real transparency mask annotations to achieve better training results;

[0064] 3. A new weakly supervised training process designed for the cutout task is proposed. Under the premise that there is only a small amount of data labeled with real transparency masks, the cutout effect is improved by using data without real transparency mask annotations. This can overcome the problem that previous methods require a large amount of labeled training data, and greatly reduce the application difficulty of deep neural network-based cutout. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A flow chart of a weakly supervised natural image matting method based on block clustering provided by an embodiment of the present invention;

[0066] Figure 2 A schematic diagram of data collection and annotation provided by an embodiment of the present invention;

[0067] Figure 3 A schematic diagram of the training process of a weakly supervised natural image matting method based on block clustering provided in an embodiment of the present invention;

[0068] Figure 4 The present invention provided for the embodiment of the present invention uses only 20% of the data as the data set L and the remaining 80% of the data as the data set U for training, and compares the cutout results after using all the data as the data set L for training. DETAILED DESCRIPTION

[0069] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention, and are not used to limit the present invention.

[0070] like Figure 1 As shown, a flow chart of a weakly supervised natural image matting method based on block clustering provided by an embodiment of the present invention comprises the following steps:

[0071] (1) Data collection, such as Figure 2 shown. Figure 2 The left is an RGB natural image, which can be taken with a digital camera or shot with a green screen and then cut out and merged with other backgrounds using a professional green screen cutout tool. Figure 2 The middle one is a real transparency mask, which can be obtained by using a professional green screen cutout tool after shooting with a green screen. Figure 2 The right is a ternary image, where white (value 255) represents pixels belonging to the pure foreground, black (value 0) represents pixels belonging to the pure background, and gray (value 128) represents pixels that are to be solved. When there is a true transparency mask, the ternary image can be generated directly from the true transparency mask. Specifically, when the transparency mask pixel value is 0, the pixel belongs to the pure background of the ternary image, and when the transparency mask pixel value is 255, the pixel belongs to the pure foreground of the ternary image. In other cases, the pixel belongs to the solution. Finally, the ternary image is obtained by morphologically dilating the solution pixel area. When there is no true transparency mask, you can use software such as Paint or Photoshop to draw the ternary image manually, or you can use deep learning-based image segmentation technology to automatically generate the ternary image.

[0072] In the specific collection process, green screen shooting can be used and professional green screen cutout tools (such as the Chroma Key tool of Adobe AfterEffects software) can be used to process a small amount of data to obtain the real transparency mask and three-value map to form a data set L; then a large amount of natural image data can be obtained by using digital camera shooting or network crawling and annotating the three-value map to form a data set U.

[0073] (2) Use the dataset L and image matting loss to train the network T, such as Figure 3 As shown in the left box (1). Among them, the network T includes an encoder and a decoder. In this embodiment, the encoder and decoder use the "tiny" branch and "UperHead" architecture of the Swin Transformer method respectively (see Liu, Ze, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. "Swin transformer: Hierarchical vision transformer using shifted windows." In Proceedings of the IEEE / CVFInternational Conference on Computer Vision, pp. 10012-10022.2021.); For the image matting loss, we follow the image loss function used by the FBA Matting method (see Forte, Marco, and Pitié. "$F$,$B$,Alpha Matting." arXiv preprint arXiv:2003.07711(2020).). In this embodiment, the network T uses an image classification model trained on ImageNet as initialization, and uses data augmentation techniques during training, including random scaling, rotation, flipping, cropping, color jitter, JPEG compression, transparency jitter, and random non-rigid deformation. Among them, random non-rigid deformation includes elastic deformation (Elastic Transform) and thin plate spline deformation (Thin Plate Spline Transform), and transparency jitter is a random gamma transform of the true transparency mask. During training, the single batch data size is 24, the training time is 150 epochs, the learning rate is 6e-5, and the "cosine" learning rate decreasing strategy is used.

[0074] (3) Use the dataset L and the transparency mask similarity loss function to train the patch clustering module, such as Figure 3 As shown in the left box (2). The tile clustering module includes the encoder and mapper (network M) of network T. In this embodiment, network M is a multi-layer perceptron with 16 fully connected layers, and each layer follows the structure of "fully connected + LayerNorm + LeakyReLU" and gradually reduces its feature dimension to reduce the computational complexity of the subsequent clustering algorithm. The transparency mask similarity loss is specifically: given the natural images in the dataset L And its corresponding ternary graph data and the actual transparency mask data Will and After being sent to the encoder of network T for encoding, feature data F is obtained L , then the transparency mask similarity loss function L as It can be calculated as follows:

[0075] L as =τ(s p ,s n )

[0076]

[0077]

[0078]

[0079] s p ={φ(M T (f i ), M T (f j ))},

[0080]

[0081] s n ={φ(M T (f i ), M T (f j ))},

[0082]

[0083] Among them, f i , f j ∈F L is the characteristic data F L The feature data at position i and position j in Is the real transparency mask data The corresponding transparency mask blocks at positions i and j. T is the mapper of the patch clustering module. τ is the Circle loss, γ, θ m are parameters related to Circle loss, which are 0.4 and 80 respectively in this embodiment. p and n are the intra-class pairs (positive sample pairs) and inter-class pairs (negative sample pairs) in Circle loss. φ is the cosine similarity. H and G are the two indicators of transparency mask similarity measurement. are the thresholds used when selecting intra-class and inter-class pairs. In this example, these thresholds are 0.1, 0.3, 0.05, and 0.1, respectively.

[0084] Specifically, the transparency mask similarity metrics H and G are calculated as follows:

[0085]

[0086]

[0087] where α i , α j are two transparency mask blocks of the same size, ||·||1 is the L1 paradigm, is the image gradient operator, is a sort operation.

[0088] In this embodiment, the encoder of network T is used as the initialization of the encoder of the tile clustering module, and the mapper is randomly initialized. Then, using the data set L and the corresponding data augmentation techniques, the network M is trained using the transparency mask similarity loss. Specifically, in addition to the common random data augmentation techniques (such as rotation, translation, etc.), the data augmentation techniques also include random non-rigid body deformation and transparency jitter. Among them, random non-rigid body deformations include elastic deformation (ElasticTransform) and thin plate spline deformation (Thin Plate Spline Transform), and transparency jitter is a random gamma transform of the real transparency mask. The encoder of the tile clustering module does not participate in training. During training, the single batch data size is 24, the training time is 200 epochs, the learning rate is 1e-4, and the "cosine" learning rate decreasing strategy is used.

[0089] (4) Use the tile clustering module to cluster the data in the dataset L, such as Figure 3 As shown in (3) in the left box, in this embodiment, the number of cluster centers of the K-means algorithm is set to 100, and the K-means++ algorithm is used for initialization.

[0090] (5) Further optimize the clustering results, including filtering invalid cluster centers and cluster centers that are similar to cluster centers. Specifically, cluster centers that meet any of the following two conditions will be identified as invalid cluster centers:

[0091] C1, after being mapped by the tile clustering module in the dataset L, the number of data closest to the cluster center is less than 0.1% of the total number of data in the dataset L;

[0092] C2: In the transparency mask set of the tiles corresponding to the 5% feature data closest to the cluster center, less than half of the transparency mask tile pairs are intra-class pairs (i.e., s p The number of elements in the set is less than 50% of all pairs).

[0093] The specific criteria for judging similar cluster centers are:

[0094] D1. First, calculate the similarity between the two categories. This similarity is obtained by calculating the ratio of the transparency masks of the tiles corresponding to the 5% of feature data closest to the two cluster centers that belong to the same class. Specifically:

[0095]

[0096]

[0097]

[0098] k∈n i, l∈n j

[0099] Among them, J is the cluster center similarity matrix, J ij represents the similarity between cluster centers i and j, |·| is a counting operation, and n i , n j are the numbers of the 5% of feature data closest to cluster centers i and j respectively.

[0100] D2. After obtaining J, use the threshold θ J The similarity matrix J can be converted into an adjacency matrix B, namely:

[0101]

[0102] In this case, the matrix B can represent an undirected graph G = (V, E), where V = {1..K}, E = {(i, j) | J ij >θ J}. When there is an edge directly connecting two cluster centers (i.e., two nodes in G) (i.e., B ij =1), the two cluster centers are considered similar and can be aggregated into the same group. On the contrary, if the two cluster centers are still not connected after one hop in the graph (i.e. (B 2 ) ij =0), the two cluster centers are considered to be dissimilar. j is 0.05.

[0103] (6) Use dataset L and dataset U to jointly train network S using the cutout loss function and clustering prior loss function. In this embodiment, the network T trained in S2 is used as the initialization of network S, and then the dataset L and dataset U and the corresponding data augmentation techniques are used to train network S using the cutout loss function and clustering prior loss function. Specifically, the clustering prior loss is to first use the transparency mask of the image block corresponding to the N nearest feature data of the cluster center as the anchor patch, and then use the transparency mask similarity metric to calculate the difference between the prediction result of network S and the anchor patch and use it to train network S. Specifically, given the anchor patches of K cluster centers The data of dataset U is clustered by the cluster label c output by the tile clustering module and the transparency mask prediction α output by the network S, then the clustering prior loss L cp It can be written as:

[0104]

[0105]

[0106]

[0107]

[0108]

[0109] D(α i , α j )=[1-χ(H(α i , α j ))][1-χ(G(α i , α j ))]

[0110]

[0111]

[0112] where f i , c i , α i They correspond to the feature data of a certain block i in the dataset U after passing through the encoder of the network T, the clustering label output by the block clustering module, and the transparency mask predicted by the network S. j represents the jth cluster center. is the cluster label c i The corresponding similar cluster centers and non-similar cluster centers. D(α i , α j ) is an indicator based on the transparency mask similarity measurement, and χ is a linear normalization function that can compress the original indicator to the [0, 1] interval. are the similarity indices of patch i to its nearest similar and non-similar anchor patches, respectively. c Yes p and d n The margin difference threshold of the indicator.

[0113] In this embodiment, χ is the reciprocal of the largest H and G in the current single batch of data, θ c is 1. During the training process, the overall loss function of the network S is the weighted sum of the cutout loss function and the clustering prior loss. Specifically, the overall loss function L S It can be written as:

[0114] L S =L im +θ S L cp

[0115] Among them, θ S is the weight of the clustering prior loss, which is 1 in this embodiment, L imis the image matting loss function, which is consistent with the image matting loss function used in training network T. During training, the single batch size is 16, the learning rate is a constant 3e-5, and the training time is 20 epochs.

[0116] (7) Use the trained network S to predict the transparency mask. The user inputs the natural image to be cut out and its corresponding ternary image data to the network S, and the network S can predict the transparency mask corresponding to the natural image.

[0117] Figure 4 The present invention provides a comparison of the matting results after using only 20% of the data as the data set L and the remaining 80% of the data as the data set U for training and the matting results after using all the data as the data set L for training. The first column is a natural image, the corresponding three-value map is in the box in the upper right corner of the image, the second column is the real transparency mask, the third column is the matting result after using the implementation method to train with all the data as the data set L, and the fourth column is the matting result after using the implementation method to train with only 20% of the data as the data set L and the remaining 80% as the data set U. The box in the upper left corner of all pictures is a partial enlargement.

[0118] In one embodiment, a computer device is proposed, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the indoor scene virtual roaming method based on reflection decomposition in the above-mentioned embodiments.

[0119] In one embodiment, a storage medium storing computer-readable instructions is provided, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps in the indoor scene virtual roaming method based on reflection decomposition in the above embodiments. The storage medium may be a non-volatile storage medium.

[0120] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0121] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A weakly supervised natural image matting method based on patch clustering, characterized in that: The following steps are involved: S1: Obtain training data, including a data set L with a true transparency mask annotation and a data set U without a true transparency mask annotation; wherein the data set L includes three parts: a natural image, a ternary map corresponding to the natural image, and a true transparency mask corresponding to the natural image; the data set U includes two parts: a natural image and its corresponding ternary map; wherein the ternary map divides all pixels in the natural image into three categories, including foreground pixels, background pixels, and pixels to be solved; the true transparency mask represents the transparency of the corresponding pixel at each position in the natural image; S2: Use the dataset L and the deep learning cutout loss function to train the teacher network T. The network T is a deep neural network based on the encoder-decoder architecture, which includes an encoder and a decoder. A natural image and its corresponding ternary image are input. The encoder maps the input into high-dimensional semantic feature data. The decoder decodes the real transparency mask from the high-dimensional semantic feature data as the prediction result. The cutout loss function is used to calculate the difference between the prediction result and the real transparency mask, thereby realizing the training of the network T. The method of using the data set L and the deep learning cutout loss function to train the teacher network T is as follows: first, an image classification model trained on a large image classification data set is used as the initialization of the network, and then the data set L and the corresponding data augmentation techniques are used to train the network T using a commonly used cutout loss function; the data augmentation techniques include random data augmentation techniques, random non-rigid body deformation and transparency jitter; wherein the random non-rigid body deformation includes elastic deformation (Elastic Transform) and thin plate spline deformation (Thin Plate Spline Transform), and the transparency jitter is a random gamma transform of a real transparency mask; S3: Use the dataset L and the transparency mask similarity loss to train the patch clustering module; the patch clustering module includes the encoder of the network T trained in S2 and a mapper network M; the network M is a multi-layer perceptron (MLP), which is responsible for mapping the feature data output by the encoder of the network T to feature data for patch clustering; during training, a natural image and its corresponding ternary image are input, and the predicted patch clustering feature vector is output after passing through the encoder of the network T and the mapper network M; finally, the transparency mask similarity loss is used to train the network M, and the encoder of the network T does not participate in the training; The transparency mask similarity loss is specifically: Given a natural image dataset L And its corresponding ternary graph data and the actual transparency mask data Will and After being sent to the encoder of network T for encoding, feature data F is obtained L , transparency mask similarity loss function L as Calculated as follows: L as =τ(s p ,s n ) Among them, f i ,f j ∈F L is the characteristic data F L The feature data at position i and position j in a i , Is the real transparency mask data The corresponding transparency mask blocks at positions i and j; M T is the network M; τ is the Circle loss, γ,θ m is the Circle loss related parameter, s p and n They are the intra-class pairs and inter-class pairs in Circle loss, respectively; φ is the cosine similarity, and H and G are two indicators of transparency mask similarity measurement, respectively; is the threshold used when selecting intra-class and inter-class pairs; S4: Use the tile clustering module trained in S3 to predict the clustering feature data of the data set L, perform clustering operations on the predicted results and optimize the clustering results; wherein, firstly, the natural images in the data set L and their corresponding ternary image data are sent to the tile clustering module to obtain the corresponding tile clustering feature data; after all the tile clustering feature data are obtained, the feature data are clustered using the K-means algorithm and the cluster center is obtained; S5: further optimizing the cluster centers obtained in S4 and collecting transparency masks near the cluster centers; wherein the cluster center optimization includes filtering invalid cluster centers and aggregating similar cluster centers; S6: Use dataset L, dataset U, and the optimized cluster centers and their nearby transparency masks in S5 to train the student network S; wherein, network S is a deep neural network based on an encoder-decoder architecture with the same structure as network T, and is initialized using the weights of network T trained in S2; during training, the data of datasets L and dataset U are simultaneously fed into network S, and network S outputs the prediction result of the transparency mask; for the data in dataset L, a cutout loss function is used and the known real transparency mask in dataset L is used for training; for the data in dataset U, the data is first fed into the trained tile clustering module in S3 to obtain the tile clustering feature data corresponding to the data, and then the nearest cluster center optimized by S5 and its nearby transparency mask are searched based on this feature data, and finally the clustering prior loss is used to train using the found transparency mask; S7: Use the network S trained in S6 to perform cutout; input the natural image to be cutout and its corresponding ternary image to the network S, and the network S predicts the transparency mask corresponding to the natural image, that is, the cutout result.

2. The method according to claim 1, characterized in that The transparency mask similarity metrics H and G are calculated as follows: where α i ,α j are two transparency mask blocks of the same size, ||·||1 is the L1 paradigm, is the image gradient operator, is a sort operation.

3. The method according to claim 1, characterized in that The block clustering module in S3 is trained using the dataset L and the transparency mask similarity loss, specifically: The encoder in the network T is used as the initialization of the encoder of the tile clustering module, and the mapper is randomly initialized; then the data set L and the corresponding data augmentation techniques are used to train the network M pair using the transparency mask similarity loss; the data augmentation techniques include random data augmentation techniques, random non-rigid body deformation and transparency jitter; wherein the random non-rigid body deformation includes elastic deformation (Elastic Transform) and thin plate spline deformation (Thin Plate Spline Transform), and the transparency jitter is a random gamma transform of the real transparency mask; the encoder of the tile clustering module does not participate in the training.

4. The method according to claim 1, characterized in that: In S5, a cluster center that satisfies any one of the following two conditions will be judged as an invalid cluster center: C1, after being mapped by the tile clustering module in the dataset L, the number of data closest to the cluster center is less than 0.1% of the total number of data in the dataset L; C2: In the transparency mask set of the tiles corresponding to the 5% feature data closest to the cluster center, less than half of the transparency mask tile pairs are intra-class pairs, that is, s p The number of elements in the set is less than 50% of all pairs.

5. The method according to claim 1, characterized in that: In S5, the method for determining similar cluster centers is specifically as follows: D1. First, the similarity between the two categories is calculated. The similarity is obtained by calculating the ratio of the transparency masks of the tiles corresponding to the 5% of feature data closest to the two cluster centers that belong to the same class. Specifically: k∈n i ,l∈n j Among them, J is the cluster center similarity matrix, J ij represents the similarity between cluster centers i and j, |·| is a counting operation, and n i ,n j are the numbers of the 5% of feature data closest to cluster centers i and j respectively; D2. After obtaining J, use the threshold θ J Convert the similarity matrix J into an adjacency matrix B, that is: At this time, the matrix B represents an undirected graph G = (V, E), where V = {1..K}, E = {(i,j)|J ij >θ J }; When there is an edge directly connecting two cluster centers, that is, B ij =1, the two cluster centers are considered similar and aggregated into the same group; on the contrary, if the two cluster centers are still not connected after one hop in the graph, that is, (B 2 ) ij When =0, the two cluster centers are considered to be dissimilar.

6. The method according to claim 1, characterized in that In S6, the clustering prior loss is specifically: The transparency mask of the image block corresponding to the nearest N feature data of the cluster center is used as the anchor patch, and the transparency mask similarity metric is used to calculate the difference between the prediction result of the network S and the anchor patch and train the network S accordingly; specifically: Given K cluster centers, the anchor blocks The data of dataset U is clustered by the clustering label c output by the tile clustering module, the transparency mask prediction α output by the network S, and the clustering prior loss L cp Written as: D(a i ,a j )=[1-χ(H(α i ,a j ))][1-x(G(a i ,a j ))] where f i ,c i ,α i They correspond to the feature data of a certain block i in the dataset U after passing through the encoder of the network T, the clustering label output by the block clustering module, and the transparency mask predicted by the network S; e j represents the jth cluster center; is the cluster label c i Corresponding similar cluster centers and non-similar cluster centers; D(α i ,α j ) is an indicator based on the transparency mask similarity measurement, and χ is a linear normalization function that can compress the original indicator to the [0,1] interval; are the similarity indices of the image patch i to its nearest anchor blocks of similar and dissimilar cluster centers, respectively; θ c Yes p and d n The margin difference threshold of the indicator.

7. The method according to claim 1, characterized in that The method of using the dataset L, the dataset U, and the optimized cluster center and the transparency mask near it in S5 to train the student network S in S6 is as follows: First, the network T trained in S2 is used as the initialization of the network S, and then the data sets L and U and the corresponding data augmentation techniques are used to train the network T using the commonly used cutout loss function and the clustering prior loss function; the data augmentation techniques include random data augmentation techniques, random non-rigid deformation and transparency jitter; among them, the random non-rigid deformation includes elastic deformation (Elastic Transform) and thin plate spline deformation (Thin Plate Spline Transform), and the transparency jitter is a random gamma transform of the real transparency mask.

8. The method according to claim 7, characterized in that During the training process, the overall loss function of the network S is the weighted sum of the cutout loss function and the clustering prior loss. Specifically, the overall loss function L S Written as: L S =L im +θ S L cp Among them, θ S is the weight of the clustering prior loss, L im is the image matting loss function.

Images