Semi-supervised semantic segmentation method based on mutual training

Abstract

The invention discloses a semi-supervised semantic segmentation method based on mutual training. The method comprises the following steps: firstly, acquiring a group of labeled data sets and a group of unlabeled data sets; and then constructing a semi-supervised semantic segmentation model comprising a non-image semantic error correction module and an image semantic error correction module, and carrying out semantic segmentation based on the annotated data set and the unannotated data set. And secondly, designing an iterative training strategy to train the semi-supervised semantic segmentation model, and outputting a semantic segmentation result. According to the method, the mutual training strategy is introduced, the problem that a self-training method lacks a self-error detection mechanism is effectively solved, the problem of false label expansion is relieved through the random iteration training strategy, and the accuracy and robustness of final semantic segmentation are improved.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision and semantic segmentation, and specifically provides a semi-supervised semantic segmentation method based on mutual training. Background Art

[0002] Traditionally, semantic segmentation problems were solved through classical image processing and CV techniques, such as thresholding techniques or clustering algorithms. However, with the emergence of deep learning methods, the quality of segmentation results has taken a leap. Semi-supervised learning lies between fully supervised learning and unsupervised learning, and is committed to more effectively utilizing limited labeled data in semantic segmentation tasks, combining unlabeled data to improve model performance. Its main purpose is to reduce the acquisition cost of labeled data, because large-scale annotation is a time-consuming and expensive task. By adopting semi-supervised learning methods, both labeled and unlabeled image data can be comprehensively utilized during the training process, thereby more fully mining information in model training.

[0003] The current problem faced by the method based on pseudo-label training in semi-supervised semantic segmentation is that there will be classification errors in the pseudo-labels. The errors in the pseudo-labels are due to the prediction errors of the model for images. Retraining the model with pseudo-labels with misclassifications will cause the errors to be further strengthened, ultimately leading the model to be optimized in the wrong direction. In particular, when using the method based on pseudo-label training, no matter what pseudo-label selection metrics are adopted, it is difficult for a single model to discover its own errors. During the iterative training of pseudo-labels, the iterative refinement of labeled data may lead to overfitting of the training set, ultimately resulting in pseudo-label inflation, where a single dominant class prediction will eventually spread to cover the entire image. Using a combined dataset of labeled images and pseudo-labels for training can slow down the occurrence of pseudo-label inflation, but it cannot completely solve it. Summary of the Invention

[0004] The present invention solves the problems of the lack of self-error detection in the self-training method in semi-supervised semantic segmentation, the iterative refinement of pseudo-labels leading to the continuous strengthening of the feedback loop and compounding the error predictions in previous iterations, and the semantic category errors in pseudo-labels

[0005] To solve the above problems, the present invention provides a semi-supervised semantic segmentation method based on mutual training, including:

[0006] Step 1, obtain a set of M labeled datasets D l ={(X l , Y l )} M and a set of N unlabeled datasets D u ={X u} N , where N>M, X lIt is labeled image data, Y l is X l annotation, X u is unlabeled image data.

[0007] Step 2: Construct a semi-supervised semantic segmentation model, which is a dynamic mutual training model composed of two DeeplabV3+ network models F and G with Resnet101 as the backbone network. In addition, the model also includes a non-image class semantic error correction module based on the student-teacher architecture and an image class semantic error correction module based on DeepLabV3+ of ResNet101.

[0008] Step 3: Set different initial values for the parameters of semantic segmentation models F and G and conduct an initial training. Use the labeled dataset D l input the network models F and G for a supervised iterative training, thereby calculating the supervised losses L lf and L lg , and update the parameters of network models F and G according to the losses L lf and L lg respectively.

[0009] Step 4: Use the labeled dataset D l and the unlabeled dataset D u to pre-train the non-image class semantic error correction module. Specifically, the student model in the student-teacher architecture is trained on the labeled dataset D l using the categorical cross-entropy loss L ce , while on the unlabeled dataset, during training, it is carried out by minimizing the consistency loss L con between the student and teacher models. The pseudo-labels obtained by the teacher model are input into the student model for the student model to learn. The parameters of the teacher model are updated using the exponential moving average of the student model parameters.

[0010] Step 5: On the labeled dataset D l , the image class semantic error correction module is trained by minimizing the mean squared error loss L mse . The image class semantic error correction module combines the pseudo-labels and the corresponding confidence maps filtered by the non-image class semantic error correction module, calculates the standard difference map between the annotation and the pseudo-labels, and optimizes the parameters of the image class semantic error correction module.

[0011] Step 6: Input the unlabeled dataset D u into the network models F and G respectively to obtain the pseudo-labels f(D u ) generated based on the network model F and the pseudo-labels g(D u ) generated based on the network model G.

[0012] Step 7: Input the pseudo-labels f(D u ) and g(D u ) into the teacher network of the non-image semantic error correction module respectively, and set the confidence threshold η to filter out the categories with confidence lower than the confidence threshold η, obtaining the filtered pseudo-labels and

[0013] Step 8: Concatenate the pseudo-labels and with the unlabeled dataset Du in the channel dimension and then input them into the image semantic error correction module to obtain the confidence maps fc(D u ) and gc(D u ). Obtain the final optimized pseudo-labels Y u ) and gc(D u ) of the unlabeled dataset D u according to the confidence of the confidence maps fc(D u ).

[0014] Step 9: Adopt a random iterative training strategy to solve the problems of pseudo-label inflation and overfitting. Each time during training, randomly select all the pseudo-labels Y u of the unlabeled dataset D u or use the labeled data Y l of the labeled dataset D l and combine them with some pseudo-labels to form the labeled data Y * for training, where the pseudo-label data is divided into two parts Y u1 and Y u2 . In one iteration, when using the combination of the labeled data of the labeled dataset and some pseudo-labels for training and the used labeled data Y * = Y l + Y u1 , then the next time when using the combination of the labeled data of the labeled dataset and some pseudo-labels for training, the labeled data used for training is Y * = Y l + Y u2 .

[0015] Step 10: Use the updated labeled data Y * to train the network models F and G, calculate the cross-entropy losses L f and L g and update the parameters of the F and G models according to the training losses L f and L g .

[0016] Step 11: Determine whether the current iteration training count cur_iter is less than or equal to the total iteration training count total_iter. If so, return to Step 6 to continue the iterative training; if not, the iterative training ends, output the updated network model, and proceed to Step 12.

[0017] Step 12: Input the image into the trained F and G models to generate their respective semantic prediction probability maps P f and P g , concatenate the two probability maps in the channel dimension, and select the semantic label with the highest prediction probability among the two for each pixel point as the final semantic segmentation result.

[0018] The parameters of the teacher model in the non-image class semantic error correction module in Step 4 are the exponential moving average of the student model. The exponential moving average function is specifically:

[0019] ρ′ t = αρ′ t-1 +(1 - α)ρ t

[0020] where ρ t is the parameter of the student network at time t, ρ′ t is the parameter of the teacher network, and α is the smoothing coefficient hyperparameter.

[0021] On the labeled dataset D l ={(X l ,Y l )} M the student model is trained using the categorical cross-entropy loss L ce . The categorical cross-entropy loss function is specifically:

[0022]

[0023] On the unlabeled dataset D u ={X u} N the student model and the teacher model are constrained by the mean squared error loss L con . The mean squared error loss function is specifically:

[0024] L con = ||ρ t (x l ) - ρ′ t (x l )||

[0025] The image class semantic error correction module in Step 5 on the labeled dataset D l ={(X l , Y l )}M The mean squared error loss \(L\) is used above mse to train the model. The mean squared error loss \(L\) mse function is specifically:

[0026]

[0027] where \(\odot\) represents the image and the pseudo - label after being corrected by the non - image - class semantic error correction module are concatenated in the channel dimension. \(y\) p is a binary map. When the pseudo - label is the same as the ground truth \(y\) l it is 1, otherwise 0.

[0028] In step 7, the non - image - class semantic error correction module sets a confidence threshold \(\eta\) and corrects the classes with confidence lower than \(\eta\). The specific function is as follows:

[0029]

[0030] where \(Y(x)\) is the semantic probability map and \(c\) is the number of image channels. The corrected probability map is then used to obtain the pseudo - label corrected by the non - image - class semantic error correction module through the argmax function The specific function is as follows:

[0031]

[0032] In step 8, through the image - class semantic error correction modules of two different models, the corrected semantic correction confidence maps are compared to obtain the final pseudo - label for training. The specific formula is as follows:

[0033]

[0034] where \(i\) represents the semantic category of the pixel in the image, \(y_c\) i is the semantic category corresponding to each pixel on the final pseudo - label, and represent the pseudo - labels corrected by the non - image - class semantic error correction module of two different models respectively.

[0035] In step 10, mutual training is carried out between two different models through cross - entropy loss, and the differences between different models are used to detect the errors of the other model. The specific formula is as follows:

[0036]

[0037] where \(C\) represents the total number of categories, \(y\) * represents the set of the true labels of the labeled data used for training in this iteration and the pseudo - labels of the unlabeled data generated by another model, \(x\)* represents the corresponding image data, y a represents the prediction result obtained by running the model on this dataset.

[0038] Beneficial effects: Based on the pseudo-label method, this solution introduces a co-training strategy. By using two models with different initial weights, each model can retrain using unlabeled images and the corresponding pseudo-labels generated by the other model. This design effectively solves the problem of the lack of self-error detection mechanism in the self-training method. Through this co-training method, the two models can correct each other, thereby enhancing the robustness of the model. In addition, this solution adopts a random iterative training strategy, randomly selecting different proportions of labeled data and pseudo-labels of unlabeled data during each training process. This method effectively alleviates the problem of pseudo-label inflation and fully exploits the potential of unlabeled data by progressively using all unlabeled data and their pseudo-labels, thereby reducing the risk of overfitting. From the perspective of improving the quality of pseudo-labels, this solution introduces two semantic error correction modules during the pseudo-label generation process. First, a semi-supervised corrector based on the teacher-student architecture is used to correct errors that do not belong to the image semantic category. Second, a fully supervised corrector based on DeepLabV3+ corrects pixel errors in the boundary regions of the pseudo-labels. By improving the quality and stability of the pseudo-labels, the model can generate more reliable training data, thereby improving the accuracy and robustness of the final semantic segmentation. Description of the Drawings

[0039] Figure 1 is a schematic diagram of the overall framework of the semi-supervised semantic segmentation method based on co-training;

[0040] Figure 2 is a schematic diagram of the two semantic correction modules added during the pseudo-label generation process of the present invention. Detailed Implementation Manner

[0041] To enable those skilled in the art to better understand the solution of the present invention, the present invention will be further described in detail below in conjunction with specific implementation manners.

[0042] As Figure 1 and Figure 2 shown, a semi-supervised semantic segmentation method based on co-training includes:

[0043] Step 1, Dataset acquisition. In this embodiment, a set of M labeled datasets D l ={(X 1 , Y l )} M and a set of N unlabeled datasets D u ={X u}N , where N > M. Both the labeled dataset and the unlabeled dataset used are from the PASCAL VOC 2012 dataset. This dataset contains images for semantic segmentation tasks, with a total of 6,000 images, which are divided into 4 major categories and 20 minor categories according to the image content. In the PASCAL VOC 2012 dataset, the image data is divided into a training set, a test set, and a validation set, containing 1,464 images, 1,449 images, and 2,913 images respectively. During the training process, the training set is used to train the model, and the validation set is used to evaluate the performance of the model. This method divides the entire training set of PASCAL VOC 2012 into two groups according to different ratios, where 1 / 4, 1 / 8, and 1 / 30 of the training data are used as the labeled dataset D l , and the remaining images are used as the unlabeled dataset D u , and the validation set is used to verify the model effect.

[0044] Step 2, Construct a semi-supervised semantic segmentation model. In this specific embodiment, two DeeplabV3+ network models with ResNet-101 as the backbone network are used to construct the semi-supervised semantic segmentation model. During the pseudo-label generation process of the semi-supervised semantic segmentation model, it will be filtered by two semantic correction modules, including a non-image class semantic error correction module based on the student-teacher architecture and an image class semantic error correction module based on DeepLabV3+ of ResNet101.

[0045] Step 3, Model initialization and initial training. The parameters of the semantic segmentation models F and G are initialized differently and then initially trained. Specifically, use the labeled dataset D l to input into the network models F and G for one round of supervised iterative training, and calculate the supervised losses L lf and L lg , and the supervised segmentation loss functions L lf and L lg Specifically:

[0046]

[0047] And update the parameters of the network models F and G respectively according to the losses L lf and L lg .

[0048] Step 4, Pre-training of the non-image class semantic error correction module. Use the labeled dataset D l and the unlabeled dataset D u to pre-train the non-image class semantic error correction module. Specifically, the student model in the student-teacher architecture uses the categorical cross-entropy loss L l on the labeled dataset D ceTrain with the categorical cross-entropy loss L ce The formula is as follows:

[0049]

[0050] On the unlabeled dataset, during training, minimize the consistency loss L con between the student and teacher models. The consistency loss L con The formula is as follows:

[0051] L con = ||ρ t (x l ) - ρ' t (x l )||

[0052] Input the pseudo-labels obtained by the teacher model into the student model for the student model to learn. The teacher model is updated using the exponential moving average of the student model's parameters. The exponential moving average function is specifically:

[0053] ρ' t = αρ' t-1 + (1 - α)ρ t

[0054] Step 5: Training of the image class semantic error correction module. On the labeled dataset D l , the image class semantic error correction module is trained by minimizing the mean squared error loss L mse . The specific calculation formula of the mean squared error loss L mse is as follows:

[0055]

[0056] This module combines the pseudo-labels output by the non-image class semantic error correction module and the corresponding confidence maps, calculates the standard difference map between the annotations and the pseudo-labels, and then optimizes the parameters of the image class semantic error correction module.

[0057] Step 6: Generate pseudo-labels. Input the unlabeled dataset D u into the network models F and G respectively to obtain the pseudo-labels f(D u ) generated based on the network model F and the pseudo-labels g(D u ) generated based on the network model G.

[0058] Step 7: Input the pseudo-labels f(D u ) and g(D u ) into the teacher network of the non-image class semantic error correction module respectively, and set the confidence threshold η to correct the categories with low confidence. The specific function is as follows:

[0059]

[0060] The corrected probability map is obtained, and then the pseudo-label corrected by the non-image class semantic error correction module is obtained through the argmax function. and The specific function is as follows:

[0061]

[0062] Step 8, Image class semantic error correction and pseudo-label optimization. The pseudo-label and are concatenated with the unlabeled dataset Du in the channel dimension and then input into the image class semantic error correction module to obtain the confidence maps fc(D u ) and gc(D u ). By the image class semantic error correction modules of two different models, the corrected semantic correction confidence maps are compared to obtain the final pseudo-labels for training. The specific formula is as follows:

[0063]

[0064] Step 9, Random iterative training. To avoid the problems of pseudo-label inflation and overfitting, a random iterative training strategy is adopted. When training each time, the randomly selected dataset combinations are divided into two cases:

[0065] (1) Select the unlabeled dataset D u all the pseudo-labels Y u for training, and do not use the labeled data Y l in the labeled dataset.

[0066] (2) Select the labeled data Y l of the labeled dataset D l and a part of the pseudo-labels to form the labeled data Y * for training. The Y * used in one iteration is Y l +Y u1 , then the labeled data used in the next training is Y * =Y l +Y u2 .

[0067] Step 10, Use the updated labeled data Y * to train the network models F and G, calculate the cross-entropy losses L f and L g and update the parameters of the F and G models according to the training losses L f and L g . The cross-entropy losses L f 、L gThe specific formula is as follows;

[0068]

[0069] Step 11: Determine whether the current iteration training times cur_iter is less than or equal to the total iteration training times total_iter. If so, return to Step 6 to continue the iterative training; if not, the iterative training ends, output the updated network models F and G, and enter Step 12.

[0070] Step 12: Input the image into the trained F and G models to generate their respective semantic prediction probability maps P f and P g , splice the two probability maps in the channel dimension, and select the semantic label with the highest prediction probability at each pixel point as the final semantic segmentation result.

[0071] This method is first compared with the DeeplabV3+ baseline network model with ResNet-101 as the backbone network. The experimental results show that when the amount of labeled data decreases, the performance improvement of this method compared to the baseline is more significant. When using the PASCAL VOC 2012 dataset and the proportion of labeled data in the total data is 1 / 2, 1 / 4, 1 / 8, 1 / 16, the performance of this method model exceeds the baseline by 1.41%, 2.91%, 4.32%, 5.01%. This paper also conducts experimental comparisons with the latest semi-supervised semantic segmentation methods. Under the experimental conditions where the proportion of labeled data in the total data is 1 / 8, the performances of models F and G are 61.54 and 61.88 respectively. After being corrected by the non-image class semantic error correction module, the performances are 62.33 and 62.51. After being further corrected by the image class semantic error correction module, the performances are improved to 63.10 and 63.35. Compared with other current semi-supervised semantic segmentation methods, such as the performance of GCT is 61.92, the performance of CutMix is 61.86, and the performance of CPS is 60.94, the performance of this method is significantly improved compared to other methods, and through the optimization of the two semantic correction modules, the segmentation results are more accurate.

[0072] The present invention introduces random iterative training in the co-training to solve the problem of pseudo-label inflation, and progressively utilizes unlabeled images and their pseudo-labels, making full use of all image data and reducing the risk of overfitting. From the perspective of improving the quality of pseudo-labels, two semantic error correction modules are added in the process of generating pseudo-labels. The semi-supervised non-image class semantic error correction module based on the teacher-student architecture is used to correct the errors that do not belong to the semantic category of this image, and the fully supervised image class semantic error correction module based on DeepLabv3 is used to reduce the pixel errors on the boundary in the pseudo-labels.

Claims

1. A semi-supervised semantic segmentation method based on mutual training, characterized in that: The following steps are involved: Step 1: Obtain a set of labeled data sets and a set of unlabeled data sets; Step 2: construct a semi-supervised semantic segmentation model including a non-image semantic error correction module and an image semantic error correction module, and perform semantic segmentation based on annotated datasets and unannotated datasets; Step 3: Design an iterative training strategy to train the semi-supervised semantic segmentation model and output the semantic segmentation results.

2. The semi-supervised semantic segmentation method based on mutual training according to claim 1, characterized in that: The specific implementation process of step 2 is as follows: Step 2.1, build a semi-supervised semantic segmentation model, which is a dynamic mutual training model consisting of two DeeplabV3+ network models F and G with Resnet101 as the backbone network. The semi-supervised semantic segmentation model also includes a non-image semantic error correction module based on a student-teacher architecture and an image semantic error correction module based on DeepLabV3+ of ResNet101; Step 2.2: Set different initialization values ​​for the parameters of the semantic segmentation models F and G and perform an initial training; use the labeled dataset D l Input network models F and G for a supervised iterative training and calculate the supervised loss L lf and L lg , and according to the loss L lf and L lg Update the parameters of network models F and G respectively; Step 2.3: Use the labeled dataset D l And the unlabeled dataset D u Pre-training of the non-image semantic error correction module: The student model in the student-teacher architecture is trained on the labeled dataset D l The classification cross entropy loss L is used ce Training is performed on unlabeled datasets by minimizing the consistency loss L between the student and teacher models. con The pseudo labels obtained by the teacher model are input into the student model to let the student model learn; the parameters of the teacher model are updated using the exponential moving average of the student model parameters; Step 2.4: In the labeled dataset D l The image class semantic error correction module minimizes the mean square error loss L mse During training, the image semantic error correction module combines the pseudo-labels and corresponding confidence maps output after filtering by the non-image semantic error correction module, calculates the standard difference map between the annotations and the pseudo-labels, and optimizes the parameters of the image semantic error correction module; Step 2.5: Unlabeled dataset D u Input into the network models F and G to obtain the pseudo labels f(D u ) and the pseudo-label g(D u ); Step 2.6: Set the pseudo label f(D u ) and g(D u ) are respectively input into the teacher network of the non-image semantic error correction module, and the confidence threshold η is set to filter out the categories with confidence lower than the confidence threshold η, and the pseudo labels after filtering are obtained. and Step 2.7: Pseudo-label and With the unlabeled dataset D u After concatenation in the channel dimension, it is input into the image semantic error correction module to obtain the pseudo-label confidence map fc(D u ) and gc(D u ); According to the confidence map fc(D u ) and gc(D u ) to obtain the final optimized unlabeled dataset D u The pseudo label Y u .

3. The semi-supervised semantic segmentation method based on mutual training according to claim 2, characterized in that: In the non-image semantic error correction module, in the labeled dataset D l ={(X l ,Y l )} M The classification cross entropy loss L is used ce Train the student model, X l is the labeled image data, Y l For X l 's marking; In the unlabeled dataset D u ={X u } N The mean square error loss L is used above con Consistency loss constraints are imposed on the student model and the teacher model. The number of datasets N>M, X u It is unlabeled image data.

4. The semi-supervised semantic segmentation method based on mutual training according to claim 3, characterized in that: The non-image semantic error correction module sets a confidence threshold η and corrects the categories with confidence lower than η. The specific function is as follows: Where Y(x) is the semantic probability map, c is the number of image channels; the corrected probability map is then passed through the argmax function to obtain the pseudo label corrected by the non-image semantic error correction module The specific functions are as follows:

5. The semi-supervised semantic segmentation method based on mutual training according to claim 4, characterized in that: The specific implementation process of step 2.7 is: through the image semantic error correction modules of two different models, the corrected semantic correction confidence map is compared to obtain the pseudo label used for training. The specific formula is as follows: Where ⊙ represents splicing, i represents the semantic category of pixels in the image, and yc i is the semantic category corresponding to each pixel on the final pseudo-label, and They respectively represent the pseudo labels corrected by two different models through the non-image semantic error correction module.

6. The semi-supervised semantic segmentation method based on mutual training according to claim 5, characterized in that: The specific implementation process of step 3 is as follows: Step 3.1: Use a random iterative training strategy to solve the problems of pseudo-label expansion and overfitting. During each training, randomly select an unlabeled dataset D u All pseudo labels Y u Or use a labeled dataset D l The labeled data Y l And some pseudo labels are combined to form the label data Y used for training * , where the pseudo-label data is divided into two parts Y u1 and Y u2 ; In one iteration, when the labeled data of the labeled dataset is combined with some pseudo labels for training, the labeled data Y used * =Y l +Y u1 When , the next time you use the labeled data of the labeled data set plus some pseudo labels for training, the labeled data used for training is Y * =Y l +Y u2 ; Step 3.2: Use the updated label data Y * Train the network models F and G and calculate the cross entropy loss L f and L g And according to the training loss L f and L g Update the parameters of the F and G models; Step 3.3: Determine whether the number of iterative training cur_iter is less than or equal to the total number of iterative training total_iter; if so, return to continue iterative training; if not, the iterative training ends, outputs the updated network model, and proceeds to step 3.4; Step 3.4: Input the image into the trained F and G models to generate their respective semantic prediction probability maps P f and P g , the two probability maps are concatenated in the channel dimension, and the semantic label with the highest prediction probability for each pixel is selected as the final semantic segmentation result.

Images