Semi-supervised image segmentation method based on uncertainty guidance

Abstract

The application discloses a semi-supervised image segmentation method based on uncertainty guidance, which comprises the following steps: 1) data preprocessing; 2) obtaining a prediction result through two peer segmentation networks; 3) uncertainty estimation and dynamic weight calculation; and 4) optimization model training. The method can realize noise-resistant training and reduce the problem of segmentation performance decline caused by pseudo-label noise to the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the image processing technology, specifically to a semi-supervised image segmentation method based on uncertainty guidance. BACKGROUND

[0002] In the field of semi-supervised learning and deep learning, significant progress has been made in recent years. Semi-supervised learning methods can improve model performance by making full use of unlabeled data, which has important application value in cases where data is limited. However, pseudo-supervision noise is a major challenge faced by semi-supervised learning methods. In pseudo-supervised learning, pseudo-labels obtained by model prediction are used for training from unlabeled data. However, since these pseudo-labels are generated by the model itself, their quality may not be as good as that of real labels manually labeled, resulting in the problem of pseudo-supervision noise. The existence of pseudo-supervision noise can affect the generalization ability and performance of the model, especially in cases where there is a large amount of unlabeled data, the problem of pseudo-supervision noise is particularly prominent. In addition, in the task of semantic segmentation, due to the differences between images of different classes, some classes may be easier to learn than others, resulting in performance imbalance. This performance imbalance can affect the effectiveness of the semantic segmentation model in real-world applications, especially in complex scenarios. Existing methods to solve the problem of pseudo-supervision noise mainly include selective screening and weighting based on the confidence of pseudo-labels, and introducing additional supervision information for noise correction. However, these methods may not be completely effective in some scenarios and are relatively complex.

[0003] In addition, some existing semi-supervised learning methods mainly focus on image classification tasks, and there is still a lack of more efficient and effective solutions for the task of semantic segmentation. SUMMARY

[0004] The purpose of the present application is to overcome the shortcomings of the prior art and provide a semi-supervised image segmentation method based on uncertainty guidance. This method can achieve noise-resistant training and reduce the problem of performance degradation caused by pseudo-label noise.

[0005] The technical solution to achieve the purpose of the present application is:

[0006] A semi-supervised image segmentation method based on uncertainty guidance, comprising the following steps:

[0007] 1) Data preprocessing: data preprocessing includes:

[0008] 1-1) Use the public segmentation dataset PASCAL VOC 2012 and Cityscapes of natural images, set the image size in PASCAL VOC 2012 to 321*321, and set the image size in Cityscapes to 720*720;

[0009] 1-2) Divide the dataset into labeled data and unlabeled data, and the labeled data is 1 / 16, 1 / 8, 1 / 4 and 1 / 2 of the dataset respectively, and the remaining data in the dataset is unlabeled data;

[0010] 1-3) Weak data augmentation is performed on the labeled images, and the weak data augmentation is random flipping and adjusting the size of the training image between 0.5 and 2.0; strong data augmentation is performed on the unlabeled images, and color jittering, grayscale, and random cropping are used; the weak data augmentation performed on the labeled images and the strong data augmentation performed on the unlabeled images are both using the random number function provided by the pytorch framework, for each data augmentation operation, a probability value p is generated, when p>0.5, the corresponding data augmentation operation is performed, otherwise the corresponding data augmentation is not performed, for example, for the weak data augmentation operation of random flipping, the generated probability value is 0.6>0.5, then the labeled images and their corresponding labels are randomly flipped;

[0011] 2) Obtain the prediction results through two peer segmentation networks: the two peer segmentation networks have different initial weights and the same structure, both of which are DeepLabV3+ with atrous convolution, DeepLabV3+ is composed of an encoder and a main decoder, and the backbone network is a pre-trained ResNet101; the labeled and unlabeled images are fed into two peer segmentation networks DeepLabV3+ with the same structure but different initializations, then feature extraction is performed through a convolutional neural network CNN, DeepLabV3+ uses a deep convolutional neural network ResNe101 to learn high-level feature representations in images, finally, a decoder is used to map high-level features to pixel-level semantic segmentation results, and a classifier is used to classify each pixel, specifically:

[0012] 2-1) Use the labeled and unlabeled data sets divided in step 1), and denote the labeled data set as and the unlabeled data set as where N<<M;

[0013] 2-2) For a given input image x i , input it into the peer model DeepLabV3+ to generate two prediction probability maps p1 and p2, p1 and p2 are respectively the outputs of the two parallel peer segmentation networks, and the probability map represents the predicted probability of each pixel belonging to different classes after softmax normalization:

[0014]

[0015] Two peer segmentation networks have the same network structure but different weight initialization parameters, denoted as and and denote two peer segmentation networks composed of DeepLabV3+ respectively;

[0016] 2-3) For the prediction probabilities p1 and p2 obtained in step 2-2), an argmax operation is performed to obtain pseudo-labels for cross-teaching, as shown in equation (2):

[0017]

[0018] 3) Uncertainty estimation and dynamic weight calculation: according to the difference between the prediction probability maps p1 and p2 of the peer networks, uncertainty estimation is performed, and dynamic weights are calculated, including:

[0019] 3-1) Compare the probability maps p1 and p2 of the two peer networks to quantify the divergence, where a larger divergence indicates the presence of errors and corresponds to a lower weight value, where the uncertainty map is calculated as shown in equation (3):

[0020] U = |Max(p1) - Max(p2)| (3),

[0021] where U represents the uncertainty map, and Max represents the maximum value function of p1 and p2 at the corresponding position;

[0022] 3-2) According to the uncertainty map U obtained in step 3-1), a dynamic weight map is further obtained, as shown in equation (4):

[0023] Weight = (1 - U) λ (4),

[0024] U represents the uncertainty map, ranging from 0 to 1, Weight is the weight map, and λ controls the variability range of the weight;

[0025] 4) Model training optimization: based on the supervised loss function and the dynamic cross-loss function, the model training optimization is performed, and the optimization function includes the supervised loss L s and the dynamic cross-teaching loss L ct Two loss functions, namely:

[0026] 4-1) For labeled data, the supervised loss L s is formulated based on the standard pixel cross-entropy loss on the labeled images of the two peer processes, as shown in equation (5):

[0027]

[0028] where l ce is the cross-entropy loss function, y ij is the true label, W and H represent the width and height of the input image, and ω is applied to the labeled image weak augmentation function, which randomly flips and adjusts the size of the training image between 0.5 and 2.0;

[0029] 4-2) For unlabeled data, an uncertainty-guided dynamic cross-teaching loss L ct is trained as shown in equation (6):

[0030]

[0031] where s represents a strong data augmentation function applied to the unlabeled image;

[0032] 4-3) The total loss function Ltotal is shown in equation (7):

[0033] L total = L s + λL ct (7),

[0034] where λ is a hyperparameter with a value between 0 and 1.

[0035] In the prior art, semi-supervised learning has made significant progress in computer vision tasks such as image classification and semantic segmentation, however, due to the need for large amounts of labeled data for deep learning methods to learn generalized representations, the labor cost of labeling these data has become unbearable, in practical applications, in order to reduce the labeling cost, semi-supervised learning makes full use of unlabeled data to assist learning, and generates pseudo-labels through pseudo-supervision methods for training, however, the existence of pseudo-supervision noise leads to unstable quality of pseudo-labels, affecting the generalization ability of the model,

[0036] The technical solution introduces an uncertainty measurement method, which uses the difference between models to reweight the loss function and realizes noise-resistant training.

[0037] The technical solution directly predicts the divergence using a quantitative counterpart model, estimates uncertainty, realizes noise-resistant training, and reduces the problem of segmentation performance degradation caused by pseudo-label noise to the model.

[0038] This method can realize noise-resistant training and reduce the problem of segmentation performance degradation caused by pseudo-label noise to the model. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 The method flowchart of the embodiment is shown in the figure;

[0040] Figure 2 The experimental effect diagram of the embodiment is shown in the figure. DETAILED DESCRIPTION

[0041] The application will be further described in conjunction with the accompanying drawings and examples, but is not limited to the application.

[0042] Embodiments:

[0043] Referring to Figure 1 A semi-supervised image segmentation method based on uncertainty guidance, comprising the following steps:

[0044] 1) Data preprocessing: data preprocessing includes:

[0045] 1-1) Using the public segmentation dataset PASCAL VOC 2012 and Cityscapes of natural images, the image size in PASCAL VOC 2012 is set to 321*321, and the image size in Cityscapes is set to 720*720;

[0046] 1-2) Divide the dataset into labeled data and unlabeled data, the labeled data is 1 / 16, 1 / 8, 1 / 4 and 1 / 2 of the dataset respectively, and the remaining data in the dataset is unlabeled data;

[0047] 1-3) Weak data augmentation is performed on the labeled images, which is to randomly flip and adjust the size of the training images between 0.5 and 2.0; strong data augmentation is performed on the unlabeled images, which is to use color jittering, grayscale, and random cropping; the weak data augmentation performed on the labeled images and the strong data augmentation performed on the unlabeled images are both using the random number function provided by the pytorch framework, for each data augmentation operation, a probability value p is generated, when p>0.5, the corresponding data augmentation operation is performed, otherwise the corresponding data augmentation is not performed;

[0048] 2) Obtain the prediction results through two peer segmentation networks: the two peer segmentation networks have different initial weights and the same structure, both of which are DeepLabV3+ with a hollow convolution, DeepLabV3+ is composed of an encoder and a main decoder, the backbone network is a pre-trained ResNet101, the labeled and unlabeled images are fed into two peer segmentation networks DeepLabV3+ with the same structure but different initializations, then the features are extracted through a convolutional neural network CNN, DeepLabV3+ uses a deep convolutional neural network ResNe101 to learn high-level feature representations in images, finally, the decoder is used to convert the high-level feature mapping into a pixel-level semantic segmentation result, and the classifier is used to classify each pixel, specifically:

[0049] 2-1) Use the labeled and unlabeled datasets divided in step 1), denote the labeled dataset as The unlabeled dataset is where N << M;

[0050] 2-2) For a given input image x i , input into the peer model DeepLabV3+, generate two prediction probability maps p1 and p2, p1 and p2 are respectively the outputs of two parallel peer segmentation networks, the probability map represents the prediction probability of each pixel belonging to different categories after softmax normalization:

[0051]

[0052] Two peer segmentation networks have the same network structure, but have different weight initialization parameters, respectively represented as and and respectively represent two peer segmentation networks composed of DeepLabV3+;

[0053] 2-3) For the prediction probability p1 and p2 obtained in step 2-2), an argmax operation is performed to obtain the pseudo-label for cross-teaching, as shown in equation (2):

[0054]

[0055] 3) Uncertainty estimation and dynamic weight calculation: according to the difference between the peer network prediction probability maps p1 and p2, uncertainty estimation is performed, and dynamic weight is calculated, including:

[0056] 3-1) Compare the probability maps p1 and p2 of the two peer networks to quantify the divergence, where a larger divergence indicates that there may be an error, and corresponds to a lower weight value, where the uncertainty map is calculated as shown in equation (3):

[0057] U = |Max(p1) - Max(p2)| (3),

[0058] where U represents the uncertainty map, Max represents the maximum value function of p1 and p2 corresponding positions;

[0059] 3-2) According to the uncertainty map U obtained in step 3-1), further obtain the dynamic weight map, as shown in equation (4):

[0060] Weight = (1 - U) λ (4),

[0061] U represents the uncertainty map, ranging from 0 to 1, Weight is the weight map, and λ controls the variability range of the weight;

[0062] 4) Optimize model training: optimize model training according to supervised loss function and dynamic cross loss function, and the optimization function includes supervised loss L s and dynamic cross teaching loss L ct Two loss functions, namely:

[0063] 4-1) For labeled data, the supervised loss L is formulated according to the standard pixel cross entropy loss on the labeled images of the two peer processes s As shown in formula (5):

[0064]

[0065] Where lce is the cross entropy loss function, y ij is the real label, W and H represent the width and height of the input image, and ω is applied to the labeled image weak augmentation function, which randomly flips and adjusts the size of the training image between 0.5 and 2.0;

[0066] 4-2) For unlabeled data, uncertainty-guided dynamic cross teaching loss L ct is used for training as shown in formula (6):

[0067]

[0068] Where s represents the strong data augmentation function applied to the unlabeled image;

[0069] 4-3) The total loss function Ltotal is shown in formula (7):

[0070] L toal = L s + λL ct (7),

[0071] Where λ is a hyperparameter, and its value is between 0 and 1.

[0072] The method is used for experimental verification:

[0073] I. Quantitative results: experiments show that the method does not discard any pixels, because the method can learn from pixels in uncertain areas, which avoids the loss of useful information;

[0074] Training: Unlabeled data from the dataset is fed into two equivalent segmentation models for training. For PASCAL VOC 2012, in this example, the number of images in a training batch is set to 16, and the total number of training iterations is 80. For the Cityscapes dataset, in this example, the number of images in a training batch is set to 16, and the total number of training iterations is 240. To prevent overfitting, online data augmentation is performed during training. Weak data augmentation for labeled images and strong data augmentation for unlabeled images are performed with a certain probability. Specifically, the random number function provided by the PyTorch framework generates a probability value p for each data augmentation operation. If p > 0.5, the corresponding data augmentation operation is executed; otherwise, no data augmentation is performed. For example, for the weak data augmentation operation of random flipping, the generated probability value is 0.6 > 0.5, so the labeled image and its corresponding label are randomly flipped. The experimental results of this method are shown in Table 1. Table 1 presents the performance comparison results of this method and the state-of-the-art semi-supervised learning methods on PASCAL VOC 2012 and Cityscapes. Semantic segmentation mIOU is used to evaluate the segmentation performance.

[0075] Table 1:

[0076]

[0077] For the PASCAL VOC 2012 and Cityscapes datasets, this example method achieved the highest miou segmentation performance across all different labeled ratios. This example method employs uncertainty-guided training, which continuously improves the model's segmentation performance and robustness.

[0078] II. Qualitative Results:

[0079] like Figure 2 As shown, Figure 2 The diagram illustrates the segmentation prediction results of the method in this example and other semi-supervised learning methods on PASCAL VOC 2012 with labeled data comprising 50% of the dataset. The segmentation prediction results show that the method in this example has better scalability for segmentation instances of different shapes.

Claims

1. A semi-supervised image segmentation method based on uncertainty-guided, characterized in that, Comprising the following steps: 1) data preprocessing: data preprocessing includes: 1-1) using the public segmentation dataset PASCAL VOC 2012 and Cityscapes of natural images, setting the image size in PASCAL VOC 2012 to 321*321, and setting the image size in Cityscapes to 720*720; 1-2) dividing the dataset into labeled data and unlabeled data, the labeled data being 1 / 16, 1 / 8, 1 / 4 and 1 / 2 of the dataset respectively, and the remaining data in the dataset being unlabeled data; 1-3) weak data augmentation on labeled images, weak data augmentation being random flipping and adjusting the size of training images between 0.5 and 2.0; strong data augmentation on unlabeled images, using color jittering, grayscale, and random cropping, weak data augmentation on labeled images and strong data augmentation on unlabeled images both using the random number function provided by the pytorch framework, for each data augmentation operation, generating a probability value p, when p>0.5, executing the corresponding data augmentation operation, otherwise not performing the corresponding data augmentation; 2) obtaining prediction results through two peer segmentation networks: the two peer segmentation networks have different initial weights and the same structure, both using DeepLabV3+ with atrous convolution, DeepLabV3+ being composed of an encoder and a main decoder, the backbone network being a pre-trained ResNet101, labeled and unlabeled images being fed into two peer segmentation networks DeepLabV3+ with the same structure but different initializations, then using a convolutional neural network (CNN) for feature extraction, DeepLabV3+ using a deep convolutional neural network, ResNe101 to learn high-level feature representations in images, finally using a decoder to map high-level features to pixel-level semantic segmentation results, and classifying each pixel through a classifier, specifically: 2-1) Using the labeled and unlabeled data sets divided in step 1), denoted as the unlabeled data set is where N << M. 2-2) For a given input image x i , two prediction probability maps pi and p2 are generated by feeding into the peer model DeepLabV3+, pi and p2 are the outputs of two parallel peer segmentation networks respectively, the probability map represents the predicted probability of each pixel belonging to different categories after softmax normalization: The two peer segmentation networks have the same network structure but have different weight initialization parameters, denoted as and and respectively represent two peer segmentation networks composed of DeepLabV3+. 2-3) performing an argmax operation on the prediction probabilities p1 and p2 obtained in step 2-2) to obtain pseudo-labels for cross-teaching, as shown in formula (2): 3) uncertainty estimation and dynamic weight calculation: according to the difference between the prediction probability maps p1 and p2 of the peer networks, uncertainty is estimated and a dynamic weight is calculated, including: 3-1) comparing the probability maps p1 and p2 of the two peer networks to quantify the divergence, where a larger divergence indicates a possible error and corresponds to a lower weight value, where the uncertainty map is calculated as shown in formula (3): U = |Max(p1) - Max(p2)| (3), where U represents the uncertainty map, Max represents the maximum value function of p1 and p2 at the corresponding position; 3-2) further obtaining a dynamic weight map according to the uncertainty map U obtained in step 3-1), as shown in formula (4): Weight = (1 - U) λ (4), U represents the uncertainty map, ranging from 0 to 1, Weight is the weight map, and λ controls the variability range of the weight. 4) Optimize model training: optimize model training according to supervised loss function and dynamic cross loss function, optimization function includes supervised loss L s and dynamic cross teaching loss L ct Two loss functions, namely: 4-1) For labeled data, the supervision loss L is formulated as the standard pixel cross-entropy loss on the labeled images of the two peer processes s As shown in equation (5): where lceis the cross-entropy loss function, y ij is the true label, W and H represent the width and height of the input image, and ω applies a weak augmentation function to the labeled images that randomly flips and resizes the training images between 0.5 and 2.0; 4-2) For unlabeled data, adopt uncertainty-guided dynamic cross-teaching loss L ct Training is performed as shown in equation (6): where s denotes a strong data augmentation function applied to the unlabelled images; 4-3) The total loss function Ltotal is shown in equation (7): L total = L s + λL ct (7), where λ is a hyperparameter and takes a value between 0 and 1.

Images