A semi-supervised semantic segmentation method based on adaptive pseudo-label correction

Abstract

The present invention discloses a semi-supervised semantic segmentation method based on adaptive pseudo-label correction, comprising the following steps: selecting GTA 5. Dataset construction source domain, selection Cityscapes The target domain is constructed by the data set; the source domain image is input into the deep convolutional neural network for training to obtain a pre-trained semantic segmentation model; the prediction probability matrix generated based on the target image is used to construct a selection strategy for the uncertainty region using information entropy and density clustering algorithm; an adaptive pseudo-label correction strategy is constructed to obtain the final pseudo-label as supervision to train the semi-supervised semantic segmentation model; the target image in the target domain verification set is input into the trained semi-supervised semantic segmentation model to verify the performance of semantic segmentation. The present invention realizes online updating of pseudo-labels, solves the confirmation bias problem, alleviates the class imbalance problem, overcomes the shortcomings of full convolution, and improves the semantic segmentation effect of the model on the target domain.

Description

Technical Field

[0001] The present invention belongs to the field of self-supervised domain adaptation semantic segmentation, and specifically relates to a semi-supervised semantic segmentation method based on adaptive pseudo-label correction. Background Art

[0002] The goal of semantic segmentation is to assign semantic-level labels to each pixel in an image. It is widely used in real-world applications such as autonomous driving, robotic manipulation, and medical analysis. However, learning segmentation models relies heavily on a large amount of data with pixel annotations, and manual annotation is time-consuming and costly. In addition, the generalization ability of the model on different data is also a major challenge. Various research works have been carried out to address the above issues, and domain adaptation is a promising approach.

[0003] Recently, self-supervised training has promoted domain adaptation, using pseudo labels generated by target domain prediction as supervised training networks. For example, Cheng et al. proposed a domain adaptation semantic segmentation method based on dual path learning, which aligns the source domain and target domain through two complementary and interactive single domain adaptation pipelines, so that the target domain generates more reliable pseudo labels, and improves the performance of semantic segmentation networks in a self-supervised training manner (Yi ting Chen, Fangyun Wei, Ji anmin Bao, Dong Chen, Fang Wen, and Wenqiang Zhang. Dual Path Learning for DomainadAptation of Semantic Segmentation. In ICCV, 9082-9091, 2021). Zheng et al. proposed a domain-adaptive semantic segmentation method that corrects pseudo-label learning through uncertainty estimation. They modeled uncertainty by predicting variance and incorporated uncertainty into the optimization objective to improve the performance of semantic segmentation (Zhengdong Zheng and YiYang.RectifyingPseudoLabelLearningViaUncertaintyEstimationforDomainAdaptiveSemanticSegmentation.InINTERNATIONALJOURNAL OFCOMPUTERVISION,129(4):1106-1120,2021). However, these semantic segmentation models will gradually generate pseudo-labels that are biased against the dominant class during training. Most of the current adaptive models pay more attention to pseudo-labels with high confidence and discard pseudo-labels with low confidence, making the error irreversible. Therefore, the semantic segmentation network may never be able to learn certain pixels during the entire training process of self-supervised training, resulting in a confirmation bias.

[0004] In order to make full use of the unlabeled target image data, every pixel should be properly utilized. Wang et al. proposed a semi-supervised semantic segmentation method using unreliable pseudo-labels, which separated reliable pixels from unreliable pixels by predicting entropy, pushed each unreliable pixel into a category queue consisting of negative samples, and tried to train the model with all candidate pixels (Yuchao Wang, Haochen Wang, Yujun Shen, Jingjing Fei, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, Xinyi Le. Semi-Supervised Semantic Segmentation Using Unerliable Pseudo-Label. In CVPR, 4248-4257, 2022). Although this method makes full use of unlabeled data, it does not consider the impact of downsampling and upsampling on the detailed information between image feature levels, and the amount of calculation is also very large in the process of contrastive learning. Therefore, the algorithm based on semi-supervised domain adaptation semantic segmentation needs further research, and the algorithm performance needs to be improved. Summary of the invention

[0005] The purpose of the present invention is to provide a semi-supervised semantic segmentation method based on adaptive pseudo-label correction, based on the prediction probability matrix generated by the target image, using information entropy and density clustering algorithm to construct an uncertainty region selection strategy and an adaptive pseudo-label correction strategy to realize online correction of the target image pseudo-label. The present invention makes full use of the pixel points of the unlabeled target image data by correcting the target image pseudo-label online, avoids overfitting of incorrect pseudo-labels, and solves the problem of confirmation bias of the semantic segmentation model for the dominant class during training; by improving the resolution of the image in the uncertain region and performing classification prediction again, the problem of loss of detail information between the target images is fully considered, the problem of class imbalance is alleviated, and the shortcomings of full convolution are overcome at the same time, and the classification performance of the semi-supervised semantic segmentation model and the generalization ability in the target domain are improved as a whole.

[0006] The technical solution to achieve the purpose of the present invention is: a semi-supervised semantic segmentation method based on adaptive pseudo-label correction, comprising the following steps:

[0007] Step 1: Select the GTA5 dataset to construct the source domain, select the Cityscapes dataset to construct the target domain, divide the target images in the target domain into a training set and a validation set, and proceed to step 2;

[0008] Step 2: Input the image in the source domain into the deep convolutional neural network to obtain a pre-trained semantic segmentation model, and then proceed to step 3;

[0009] Step 3: Input the target image in the training set of the target domain into the pre-trained semantic segmentation model to generate the corresponding prediction probability matrix of the target image, use the information entropy and density clustering algorithm to construct the selection strategy of the uncertainty region, obtain the uncertainty region in the prediction probability matrix of the target image, and proceed to step 4;

[0010] Step 4: find the same position on the target image corresponding to the uncertain area on the target image prediction probability matrix, cut the position on the target image to obtain the uncertain area image, and enlarge it as the secondary target image, input the secondary target image into the pre-trained semantic segmentation model, and upsample the size of the uncertain area to generate a secondary pseudo label and fuse it with the target image pseudo label, build an adaptive pseudo label correction strategy to obtain the final pseudo label as the supervision of the target image, and input the source domain image and the target domain image in the same batch to jointly train the semi-supervised semantic segmentation model; when the predetermined number of training times is reached, obtain the trained semi-supervised semantic segmentation model, and proceed to step 5;

[0011] Step 5: Input the target image in the target domain verification set into the trained semi-supervised semantic segmentation model to generate pseudo labels to verify the semantic segmentation performance of the network.

[0012] Compared with the prior art, the advantages of the present invention are:

[0013] 1) Compared with the existing semantic segmentation methods, firstly, most semantic segmentation methods only consider labels with high confidence and ignore labels with low confidence, which leads to overfitting of incorrect pseudo-labels, making the errors irreversible and causing confirmation bias; secondly, most semantic segmentation methods use full convolution to encode and decode images, which reduces the resolution of feature maps during the encoding process, which means that some detail information is lost. During the decoding process, the encoding model must be very powerful to restore the image information well, which means that a larger model and calculation amount are required during the encoding process. In order to solve the above two problems, the present invention proposes an uncertain region selection strategy and an adaptive pseudo-label correction strategy, which improves the classification performance of the semi-supervised semantic segmentation model and its generalization ability in the target domain.

[0014] 2) The uncertain region selection strategy based on information entropy and density clustering proposed in the present invention changes the traditional training method of using only high-confidence pseudo labels as supervision. This method not only uses the high-confidence labels on the target image but also fully considers the low-confidence labels, so that every pixel on the target image can be fully utilized;

[0015] 3) The adaptive pseudo-label correction strategy proposed in the present invention, on the one hand, realizes the online correction of the pseudo-labels of the target image, avoids overfitting of incorrect pseudo-labels in the process of semantic segmentation model training, prevents irreversible errors, solves the problem of confirmation bias of the semantic segmentation model for the dominant class during training, and improves the performance of the semantic segmentation model; on the other hand, it adopts bilinear interpolation to proportionally amplify the low-resolution uncertain region image, which not only improves the resolution of the uncertain region image but also is equivalent to expanding the target image containing difficult-to-classify samples, avoids the loss of detail information between feature levels of the uncertain region image caused by downsampling, and alleviates the problem of imbalanced categories in the target domain training set;

[0016] 4) In the process of online correction of pseudo-labels of target images, the present invention uses the enlarged secondary target image as input for classification prediction again. In the decoding process, upsampling is performed according to the size of the uncertain area, thereby reducing the upsampling multiple, which means that the requirements for the encoding model are reduced and the amount of calculation is reduced, thereby overcoming the shortcomings of full convolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of the semi-supervised semantic segmentation method based on adaptive pseudo-label correction of the present invention.

[0018] Figure 2 This is a model diagram of the semi-supervised semantic segmentation method based on adaptive pseudo-label correction in the present invention. DETAILED DESCRIPTION

[0019] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0020] Combination Figure 1-Figure 2 , a semi-supervised semantic segmentation method based on adaptive pseudo-label correction, comprising the following steps:

[0021] Step 1: Select GTA5 as the source domain dataset, which has 24,966 images and labels, and the pixel size is 1914×1052; select Cityscapes urban landscape dataset as the target domain dataset, which has 5,000 images and labels, including 2,975 training sets, 500 validation sets and 1,525 test sets, and the pixel size is 1024×2048. Due to the different resolutions of the source domain and the target domain, normalization processing is required. In this paper, the pixel size is unified to 1024×512; the source domain and the target domain have 19 common classes, so the final classification is 19 categories. Let the source domain be represented as Among them, S represents the image in the source domain, represents the ground truth label corresponding to S, H represents the height of S, that is, H = 1024, W represents the width of S, that is, W = 512, H×W represents the resolution of S, and also represents the total number of pixels of S. The 3 in H×W×3 represents the three color channels of RGB. Let the target domain be represented by D T ={T|T∈R H×W×3}, where T represents the target image in the target domain and has no corresponding semantic label, and then proceed to step 2;

[0022] Step 2: Input the source domain image into the deep convolutional neural network to obtain a pre-trained semantic segmentation model, as follows:

[0023] The present invention uses ResNet101-Deeplabv2 joint encoding as a semantic segmentation model, wherein ResNet101 is a backbone network used to extract features. The network is divided into 5 convolution layers with 101 convolutions. The convolution part is used as a feature extractor, and there is a fully connected layer at the end as a classifier. In the present invention, the fully connected layer is discarded and only the first 5 convolution layers are retained as encoders to extract features. The fully connected layer is replaced by Deeplabv2 as a classifier; Deeplabv2 has a trackless spatial pyramid ASPP collection scheme, which applies parallel expansion convolutions of different rates to the input feature map and then fuses them together. Since objects of the same category may have different sizes in the image, ASPP helps to explain different object sizes. In the present invention, Deeplabv2 is used as a classifier to obtain the predicted probability matrix of pixel points. The network has four branches, each of which is composed of 3 fully connected layers, but with different void rates, respectively [6, 12, 28, 24].

[0024] Load the image in the source domain as input and input it into the ResNet101-Deeplabv2 semantic segmentation model. Use the ResNet101 encoder to extract the feature vector of the source domain image, and then input it into the Deeplabv2 classifier to obtain the prediction probability matrix of each pixel in the source domain image belonging to 19 categories. The channel index value where the pixel has the largest predicted probability is used as the classification category to generate the source domain image pseudo label. The predicted probability matrix and the real ground label are used to perform cross entropy loss to optimize the segmentation performance of the semantic segmentation model. Finally, the pre-trained semantic segmentation model is obtained, as shown in the following formula (1):

[0025]

[0026] in, Represents the source domain image S∈R H×W The cross entropy loss, C represents the total number of classification categories, The true ground label of The hot encoding of the i-th pixel of represents the predicted probability that the i-th pixel of the source domain image belongs to category c (c∈C).

[0027] Step 3: Input the target image in the training set in the target domain into the pre-trained semantic segmentation model to generate the corresponding prediction probability matrix of the target image. Use the information entropy and density clustering algorithm to construct a selection strategy for the uncertainty region and obtain the uncertainty region in the prediction probability matrix of the target image. The details are as follows:

[0028] Step 3.1: Take the target image T∈R in the target domain training set H×W Input into the pre-trained semantic segmentation model to generate the corresponding prediction probability matrix of the target image The information entropy is used to calculate the dispersion of the prediction probability of each pixel on the target image. When the entropy of a pixel is less than the entropy threshold, the pixel is regarded as an uncertain point, as shown in formula (2) and formula (3):

[0029]

[0030] in, Represents the entropy map of the i-th pixel of the target image, represents the prediction probability matrix of the i-th pixel of the target image, X n represents the nth uncertain point, n∈{1,2,...,N}, N represents the total number of uncertain points, (x,y) represents the coordinate position of the uncertain point on the target image, γ t represents the minimum threshold of information entropy at the tth iteration, and γ t Set to α t The corresponding quantile, α t is the proportion of selected uncertain points, which is adjusted by linear strategy, as shown in formula (4):

[0031]

[0032] Among them, α0 represents the proportion of uncertain points initially selected and is set to 20%, iter represents the current number of iterations, and total iter represents the predetermined number of iterations.

[0033] Step 3.2, density clustering is essentially a concept of density for clustering, and the essence of density comes from the distance between two points. There are many clustering algorithms, such as K-means clustering, spectral clustering, etc., but for K-means clustering, it is necessary to first determine the specific number of clusters to be clustered. It is unknown how many clusters the uncertain points of the present invention are clustered into, so it is not applicable; spectral clustering and density clustering can automatically determine how many clusters to cluster in the clustering process, but the result of spectral clustering needs to determine the center point position and shearing direction when dividing the uncertain area, while density clustering can help us better determine the center position and the shearing area. The most important thing is that density clustering has the advantage of anti-noise, that is, it refers to objects that do not belong to any cluster, which means that noise information can be removed in the process of clustering uncertain points; therefore, the present invention chooses to use density clustering to cluster the positions of uncertain points.

[0034] Based on the selected uncertain points, a density clustering algorithm is used to find the prediction probability matrix of the target image. The uncertainty region T un The sample set input by the density clustering algorithm is a set of uncertain points D = {X1, X2, ..., X N}, the input domain parameters are (ε, M), ε is the radius determined by density clustering, the samples in the sample set whose distance to the core object is not greater than ε are called ε-domains, and M is the minimum number of samples contained in the ε-domain; the output of the density clustering algorithm is the cluster partition A = {A1, A2, ..., A K}, A represents the set of all uncertain points divided into K clusters, A K represents the Kth cluster, as shown in formula (5):

[0035] N ε (X j )={X i ∈D|D dist(X i ,X j )≤ε} (5)

[0036] Among them, N ε (X j ) represents the number of samples contained in the ε-domain, X i , X j Represents the core object, X i With X j The difference is X j By X i Density direct, if X j Located in X i In the ε-field, and X i is also a core object, then it is called X j By X i Density direct access, dist(X i ,Xj ) represents the distance between two core points.

[0037] The density clustering algorithm finds all core objects according to the given neighborhood parameters (ε, M). First, a core object in the data set is randomly selected as the "seed", and then the core object is used as the starting point to find the samples that can be reached by its density to generate clusters. For X j With X i , there exists a sample sequence R1,R2,...,R Z And R1=X j , R Z =X i And R i+1 By R i The density is directly proportional to X j By X i The density is reached until all core objects are visited and clustering is completed.

[0038] Step 3.3: According to the parameter requirements of the density clustering algorithm, input the set of uncertain points D = {X1, X2, ..., X N} as the sample set and set the domain parameters (ε, M), the output is the cluster partition A = {A1, A2, ..., A K}, select the cluster with the largest density among the K clusters Then The cluster center is the center of the uncertainty region X o , the width w is set to 2ε, and the height h is set to 4ε. If the uncertain region exceeds the range of the target image, the cluster with the second best density in the K clusters is used to select the uncertain region, and so on. When none of the K clusters meet the above conditions, the next uncertain region selection strategy is entered.

[0039] The present invention sets the input target image size to 1024×512×3, and encodes the input image through the pre-trained semantic segmentation model in sequence. The order of the feature vector sizes output by different convolutional layers is: 1024×512×3→512×256×64→256×128×64→256×128×256→128×64×512→128×64×

[0040] 1024→128×64×2048→128×64×C. In order to restore the input target image size to obtain the prediction probability matrix after the scaled feature vector is restored, 8 times upsampling is required during decoding. In order to make the enlarged image directly restore the size of the uncertain area image by upsampling in the adaptive pseudo-label correction process, instead of restoring the input secondary target image size, the present invention sets the domain radius ε to 128, 64, and 32 respectively. Since the width w of the uncertain area is set to 2ε and the height h is set to 4ε, it can be seen that the size of the uncertain area can be 512×256, 256×128 or 128×64. Because the final size of the downsampling of the pre-trained semantic segmentation model is 128×64×C, the size of the uncertain area image directly restored by upsampling can be set to 4 times upsampling, 2 times upsampling, and no sampling, respectively, so as to obtain a secondary probability matrix and a secondary pseudo-label.

[0041] According to the size of the uncertainty region, the number of pixels contained in it is 131072, 3276 and 8192 respectively. We hope that regions of different sizes contain at least half of the uncertainty points to be called uncertain regions, so the minimum number of uncertain points contained in the ε-domain is set to 65536, 16384 and 4096 respectively. Through the continuous optimization of the semi-supervised semantic segmentation model, if the number of uncertain points in the uncertain region is less than 4096, it means that the semi-supervised semantic segmentation model has been well optimized, and the uncertainty region selection strategy is stopped.

[0042] In summary, the present invention sets three sets of domain parameters (ε1, M1), (ε2, M2), ..., (ε m ,M m ), that is, (128,65536), (64,16384), (32,4096), and satisfy ε1>ε2>ε3. The selection strategies of the uncertain regions are as follows:

[0043] Strategy 1: When N>M1, input the sample set D={X1,X2,...,X N} and domain parameters (ε1,M1), output uncertainty region T un =[X o ,w,h].

[0044] Strategy 2: When M1≥N>M2 or when entering the next selection strategy, input the sample set D={X1,X2,...,X N} and domain parameters (ε2, M2), output uncertainty region T un =[X o ,w,h].

[0045] Strategy 3: When M2≥N>M3 or when entering the next selection strategy, input the sample set D={X1,X2,...,X N} and domain parameters (ε3, M3), output uncertainty region T un =[X o ,w,h].

[0046] When the uncertain area does not satisfy the above strategy, the uncertain area selection strategy is stopped.

[0047] Step 4: Build an adaptive pseudo-label correction strategy to obtain the final pseudo-label as the supervision of the target domain, and use the same batch of source domain images and target domain images to jointly train the semi-supervised semantic segmentation model, as follows:

[0048] Step 4.1, for semantic segmentation, objects that are difficult to classify are mostly because their categories account for a small proportion in the data set and the resolution in the image is relatively small, which can also be called short-tail categories, such as people, motorcycles, bicycles, traffic lights, etc. The present invention proposes an adaptive pseudo-label correction strategy to fully utilize every pixel of the target image and these short-tail categories. On the one hand, the bilinear interpolation method is used to proportionally enlarge the image to 1024×512, so that the image can be used as a supplement to the target domain training set, balancing the proportion of short-tail categories, thereby alleviating the problem of category imbalance; on the other hand, we proportionally enlarge the image and input it into the pre-trained semantic segmentation model as a secondary target image for encoding, thereby overcoming the loss of detail information in the encoding process, and the decoding process only needs to upsample the encoded feature vector to the size of the secondary target image before enlargement, which reduces the requirements for the encoding model and reduces the amount of calculation.

[0049] Find the same position on the target image that corresponds to the uncertain region on the pseudo label of the target image, cut the position on the target image to obtain the uncertain region image, and use bilinear interpolation to proportionally enlarge the uncertain region image to 1024×512 to enhance the resolution of the uncertain region image. The enlargement factor is The enlarged image is used as the secondary target image and input into the pre-trained semantic segmentation model. During the encoding process, the feature vector of the secondary target image that is reduced by 8 times is obtained, and its size is 128×64×C. According to the size of the uncertain region h×w, the upsampling times, restore the feature vector of the same size as the uncertain region image, and after softmax activation, obtain the prediction probability matrix and pseudo label of the uncertain region image, which are used as the secondary prediction probability matrix and secondary pseudo label, respectively, as shown in formula (6) and formula (7):

[0050]

[0051] Among them, P un Represents the secondary prediction probability matrix P un ∈R h×w , F un The feature vector F representing the secondary target image un ∈R h×w×C , Represents secondary pseudo-label

[0052] Step 4.2: Zero-fill the surrounding area of ​​the generated secondary pseudo-label, and use the filled pseudo-label as the third pseudo-label The uncertain region T is selected by the uncertain region selection strategy. un =[X o ,w,h], combining the target image and the uncertain region to generate a mask Mask∈R H×W , as shown in formula (8):

[0053]

[0054] In step 4.3, the target image pseudo label and the secondary pseudo label are fused as the final pseudo label of the target image, as shown in formula (9):

[0055]

[0056] in, represents the final pseudo label of the target image, represents the target image pseudo label.

[0057] The final pseudo-label is used as supervision, as shown in formula (10):

[0058]

[0059] in, Denotes the target image T∈R H×W The cross entropy loss is Represents the final pseudo label The hot encoding of the i-th pixel of represents the predicted probability that the i-th pixel of the target image belongs to category c.

[0060] In the process of training the semi-supervised semantic segmentation model, the batches of input source domain images and target domain images are both 1, and the model parameters are optimized by self-training. Therefore, the overall loss function L of the semi-supervised semantic segmentation model with adaptive pseudo-label correction is shown in formula (11):

[0061]

[0062] Among them, λ Tis the weight of the target loss. During the training process of the semi-supervised semantic segmentation model, the number of uncertain points becomes smaller and smaller, that is, the entropy is less than the threshold γ t The percentage of pixels with γ is getting smaller and smaller, so we hope that the target domain will have an increasing impact on the semi-supervised semantic segmentation model. The weight is defined as the entropy in the current batch training that is less than the threshold γ. t The inverse of the percentage of pixels, as shown in formula (12):

[0063]

[0064] Among them, |B T | indicates the number of images input in the current batch, set to 1, is an indicator function, if X n It is 1 if it is an uncertain point, otherwise it is 0.

[0065] In the process of training the semi-supervised semantic segmentation model, the optimizer used is SGD, the weight decay is 0.0005, and the learning rate adjustment method is "poly". Its decay mechanism is base_lr is the initial learning rate set to 0.001, iter is the current number of iterations, total_iter is the maximum number of iterations set to 2k times, and power is set to 0.9 to adjust the learning rate.

[0066] Step 5: Input 500 target images in the target domain validation set into the trained semi-supervised semantic segmentation model to generate pseudo labels. Because the target images in the target domain validation set have real ground labels, the generated pseudo labels are used to calculate the average intersection-over-union ratio with the real ground labels of the images to verify the segmentation performance of the semi-supervised semantic segmentation model.

Claims

1. A semi-supervised semantic segmentation method based on adaptive pseudo-label correction, characterized in that: The following steps are involved: Step 1: Select the GTA5 dataset to construct the source domain, select the Cityscapes dataset to construct the target domain, divide the target images in the target domain into a training set and a validation set, and proceed to step 2; Step 2: Input the image in the source domain into the deep convolutional neural network to obtain a pre-trained semantic segmentation model, and then proceed to step 3; Step 3: Input the target image in the training set of the target domain into the pre-trained semantic segmentation model to generate the corresponding prediction probability matrix of the target image, use the information entropy and density clustering algorithm to construct the selection strategy of the uncertainty region, obtain the uncertainty region in the prediction probability matrix of the target image, and proceed to step 4; Step 4: find the same position on the target image corresponding to the uncertain area on the target image prediction probability matrix, cut the position on the target image to obtain the uncertain area image, and enlarge it as the secondary target image, input the secondary target image into the pre-trained semantic segmentation model, and upsample the size of the uncertain area to generate a secondary pseudo label and fuse it with the target image pseudo label, build an adaptive pseudo label correction strategy to obtain the final pseudo label as the supervision of the target image, and input the source domain image and the target domain image in the same batch to jointly train the semi-supervised semantic segmentation model; when the predetermined number of training times is reached, obtain the trained semi-supervised semantic segmentation model, and proceed to step 5; Step 5: Input the target image in the target domain verification set into the trained semi-supervised semantic segmentation model to generate pseudo labels to verify the semantic segmentation performance of the network.

2. According to claim 1, a semi-supervised semantic segmentation method based on adaptive pseudo-label correction is characterized in that: In step 2, the image in the source domain is input into the deep convolutional neural network to obtain a pre-trained semantic segmentation model, as shown in formula (1): in, Represents the source domain image S∈R H ×W cross entropy loss, H represents the height of the source domain image, W represents the width of the source domain image, H×W represents the total number of pixels on the source domain image, and C represents the total number of classified categories. The true ground label of The hot encoding of the i-th pixel of represents the predicted probability that the i-th pixel of the source domain image belongs to category c, c∈C.

3. The semi-supervised semantic segmentation method based on adaptive pseudo-label correction according to claim 2, characterized in that: In step 3, the selection strategy of using information entropy and density clustering algorithm to construct uncertainty regions is as follows: Step 3.1: Take the target image T∈R in the target domain training set H×W Input into the pre-trained semantic segmentation model to generate the corresponding prediction probability matrix of the target image The information entropy is used to calculate the dispersion of the prediction probability of each pixel on the target image. When the entropy of a pixel is less than the entropy threshold, the pixel is regarded as an uncertain point, as shown in formula (2) and formula (3): in, Represents the entropy map of the i-th pixel of the target image, represents the prediction probability matrix of the i-th pixel of the target image, X n represents the nth uncertain point, n∈{1, 2, ..., N}, N represents the total number of uncertain points, (x, y) represents the coordinate position of the uncertain point on the target image, γ t represents the minimum threshold of information entropy at the tth iteration, and γ t Set to α t The corresponding quantile, α t is the proportion of selected uncertain points, which is adjusted by linear strategy, as shown in formula (4): Among them, α0 represents the proportion of uncertain points initially selected and is set to 20%, iter represents the current number of iterations, and totaliter represents the predetermined number of iterations; Step 3.2: Use density clustering algorithm to find the prediction probability matrix of the target image based on the selected uncertain points The uncertainty region T un The sample set input by the density clustering algorithm is a set of uncertain points D = {X1, X2, ..., X N }, the input domain parameters are (ε, M), ε is the radius determined by density clustering, the samples in the sample set whose distance to the core object is not greater than ε are called ε-domains, and M is the minimum number of samples contained in the ε-domain; the output of the density clustering algorithm is cluster partition A = {A1, A2, ..., A K }, A represents the set of all uncertain points divided into K clusters, A K represents the Kth cluster, as shown in formula (5): N ε (X j )={X i ∈D|D dist(X i ,X j )≤ε} (5) Among them, N ε (X j ) represents the number of samples contained in the ε-domain, X i , X j Represents the core object, X i With X j The difference is X j By X i Density direct, if X j Located in X i In the ε-field, and X i is also a core object, then it is called X j By X i Density direct access, dist(X i , X j ) represents the distance between two core points; The density clustering algorithm finds all core objects according to the given neighborhood parameters (ε, M). First, a core object in the data set is randomly selected as the "seed", and then the core object is used as the starting point to find the samples that can be reached by its density to generate clusters. For X j With X i , there exists a sample sequence R1, R2, ..., R Z And R1=X j , R Z =X i And R i+1 By R i The density is directly proportional to X j By X i The density is reachable until all core objects are visited and clustering is completed; Step 3.3: According to the parameter requirements of the density clustering algorithm, input the set of uncertain points D = {X1, X2, ..., X N } as the sample set and set the domain parameters (ε, M), the output is the cluster partition A = {A1, A2, ..., A K }, select the cluster with the largest density among the K clusters Then The cluster center is the center of the uncertainty region X o , the width w is set to 2ε, and the height h is set to 4ε. If the uncertain region exceeds the range of the target image, the cluster with the second best density in the K clusters is used to select the uncertain region. By analogy, when none of the K clusters meet the above conditions, the next uncertain region selection strategy is entered. Therefore, our uncertain region selection strategy can set multiple sets of domain parameters, namely (ε1, M1), (ε2, M2), ..., (ε m , M m ), and needs to satisfy ε1>ε2>…>ε m , the selection strategies of the uncertain region are as follows: Strategy 1: When N>M1, input the sample set D={X1, X2, ..., X N } and domain parameters (ε1, M1), output uncertainty region T un =[X o ,w,h]; Strategy 2: When M1≥N>M2 or when entering the next selection strategy, input the sample set D={X1,X2,...,X N } and domain parameters (ε2, M2), output uncertainty region T un =[X o ,w,h]; ...... Strategy m: When M m-1 ≥N>M m Or when entering the next selection strategy, input the sample set D = {X1, X2, ..., X N } and domain parameters (ε m , M m ), output uncertainty region T un =[X o ,w,h]; When the uncertain area does not satisfy the above strategy, the iterative training is stopped.

4. The semi-supervised semantic segmentation method based on adaptive pseudo-label correction according to claim 3, characterized in that: In step 4, the construction method of the adaptive pseudo-label correction strategy is as follows: Step 4.1, find the same position on the target image corresponding to the uncertain region on the target image prediction probability matrix, cut at this position on the target image to obtain the uncertain region image, and use bilinear interpolation to proportionally enlarge the uncertain region image to increase its resolution, the magnification factor is The enlarged image is used as the secondary target image and input into the pre-trained semantic segmentation network. During the encoding process, the feature vector of the secondary target image that is reduced by 8 times is obtained, and its size is 128×64×C. According to the size of the uncertain area h×w, the upsampling times, restore the feature vector of the same size as the uncertain region image, and after softmax activation, obtain the prediction probability matrix and pseudo label of the uncertain region image, which are used as the secondary prediction probability matrix and secondary pseudo label, respectively, as shown in formula (6) and formula (7): Among them, P un Represents the secondary prediction probability matrix P un ∈R h×w , F un The feature vector F representing the secondary target image un ∈R h ×w×C , Represents secondary pseudo-label Step 4.2: Zero-fill the surrounding area of ​​the generated secondary pseudo-label, and use the filled pseudo-label as the third pseudo-label The uncertain region T is selected by the uncertain region selection strategy. un =[X o , w, h], combining the target image and the uncertain region to generate a mask Mask∈R H×W , as shown in formula (8): In step 4.3, the target image pseudo label and the secondary pseudo label are fused as the final pseudo label of the target image, as shown in formula (9): in, represents the final pseudo label of the target image, represents the target image pseudo label; The final pseudo-label is used as supervision, as shown in formula (10): in, Denote the target image T∈R H×W The cross entropy loss is Represents the final pseudo label The hot encoding of the i-th pixel of Represents the predicted probability that the i-th pixel of the target image belongs to category c; In the process of training the semi-supervised semantic segmentation model, the source domain images and target domain images are input in the same batch, and the model parameters are optimized by self-training. Therefore, the overall loss function of the semi-supervised semantic segmentation model with adaptive pseudo-label correction is As shown in formula (11): Among them, λ T is the weight of the target loss, which is defined as the entropy of the current batch training being less than the threshold γ t The inverse of the percentage of pixels, as shown in formula (12): Among them, |B T | indicates the number of images input in the current batch, is an indicator function, if X n It is 1 if it is an uncertain point, otherwise it is 0.

Images