An unsupervised domain-adaptive semantic segmentation method

By self-training, generating pseudo-labels and constraining category correlations in the output space, the problem of large differences between source and target domains in adaptive semantic segmentation in unsupervised fields is solved, and the accuracy and generalization ability of semantic segmentation are improved, which is suitable for automatic Applications such as driving and robot navigation.

CN112699892BActive Publication Date: 2024-11-08BEIJING UNIV OF TECH
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202110026447.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-01-08
Publication Date
2024-11-08
Estimated Expiration
2041-01-08

AI Technical Summary

Technical Problem

In the adaptive semantic segmentation of unsupervised fields, traditional methods are difficult to effectively reduce the differences between the source and target domains, resulting in a degradation of classifier performance, especially in application scenarios such as autonomous driving and robot navigation. , it is difficult to learn commonalities among fields.

Method used

An unsupervised domain adaptive semantic segmentation method is proposed, which generates high confidence target domain pseudo-labels through self-training methods, and constrains the consistent category correlation between source domain and target domain in the output space, and utilizes cross entropy loss and category correlation. Sexual loss optimization network, reduces inter-domain differences, and improves semantic segmentation accuracy.

Benefits of technology

Through the self-training method, the learning ability of target domain data is improved, the class-level domain differences between the source domain and the target domain are reduced, and the accuracy and generalization ability of semantic segmentation are significantly improved. The performance is better than that of other unsupervised fields. Semantic segmentation method.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112699892B_ABST
    Figure CN112699892B_ABST
Patent Text Reader

Abstract

The present invention discloses an unsupervised domain adaptive semantic segmentation method, which trains a neural network based on source domain images; uses the trained network to calculate pseudo labels of target domain images; and uses source domain images and target domain images with pseudo labels to retrain the network, further improving the accuracy of pseudo labels and optimizing the generalization ability of the network. This method uses a self-training method to extract high-confidence target domain pseudo labels using a trained network, making up for the shortcoming of lack of supervision information in the target domain. Compared with other methods, this method enriches the information of target domain data and improves the learning ability of the network for target domain data. This method focuses on the inter-domain differences based on categories, performs class correlation measurement for the predictions of the source domain and the target domain, constrains the class correlations of the two domains to be consistent, reduces the inter-domain differences of the two domain class levels, and improves the generalization ability of the network. The performance of the present invention is better than other unsupervised domain adaptive semantic segmentation methods.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of pattern recognition and computer vision, and can be applied to autonomous driving and robot visual navigation. Background Art

[0002] Semantic segmentation is to classify each pixel in the image, determine the category of each point, and then divide the area. With the development of machine learning algorithms, semantic segmentation models have achieved good segmentation effects. However, traditional machine learning algorithms assume that training data and test data follow the same distribution, which is often not true in real applications. If this assumption is not met, new training data is needed to rebuild most statistical models. However, collecting and annotating data sets for each new task or new field is a very expensive and time-consuming process, and the data that meets the training requirements may not be sufficient. If the distribution of training data and test data is very different, the performance of the classifier trained by traditional machine learning algorithms will be greatly reduced, especially the advantages of deep network methods that rely on large-scale annotated data sets will disappear. Unsupervised domain adaptive semantic segmentation is based on training models with labeled source domain data and unlabeled target domain data, and learns a model that can adapt to the distribution of target domain data and has good semantic segmentation effects. It is difficult to obtain large-scale high-quality annotated data in application scenarios such as autonomous driving, robot navigation, and video surveillance, and there is a strong demand for unsupervised domain adaptive semantic segmentation models with good performance.

[0003] How to learn the commonalities between domains, reduce the differences between the target domain (unlabeled dataset) and the source domain (labeled dataset), and adapt the classifier from the source domain to the target domain is the main problem to be solved in unsupervised domain adaptation. The commonly used strategies include: image-level adaptation, feature-level adaptation, and output-level adaptation. Image-level adaptation makes the distribution of the source domain and the target domain closer in the input space by transferring the style of the input image, but the judgment conditions of this style transfer are too subjective, and the style transfer process will also have some adverse effects on the samples, such as shape shift, thus affecting the adaptive effect; feature-level adaptation considers the mutual fusion of features. The typical method is to use domain discriminators for adversarial training in order to enable the feature extractor to obtain features with domain-invariant properties. The classification network is usually used to judge the domain invariance, which cannot guarantee the effectiveness of its transfer; output-level adaptation reduces the domain differences between the two domains based on the prediction results of the model output, but this strategy usually relies on pixel-level or region-level domain alignment. Local alignment may widen the distribution distance of the originally relatively matching categories, resulting in "negative transfer".

[0004] Semi-supervised learning uses a large amount of unlabeled data and some labeled data for pattern recognition, which is similar to the task of using a labeled source domain learning model to perform high-precision labeling on an unlabeled target domain in unsupervised domain adaptation. Therefore, some mature methods in semi-supervised learning, such as self-training methods, can be used as reference to solve unsupervised domain adaptation. Self-training methods use pseudo-labels for supervised training, which can effectively alleviate the problem of unlabeled data. However, this method also has obvious drawbacks when applied to unsupervised domain adaptation, that is, the setting of pseudo-labels is too dependent on source domain data and artificial threshold limits. Summary of the invention

[0005] To address the problem of reducing the inter-domain difference between the source domain and the target domain in unsupervised domain adaptation tasks, the present invention proposes to constrain the category correlation of the source domain and the target domain to be consistent in the output space, thereby reducing the inter-domain difference between the source domain and the target domain in the output space.

[0006] This paper proposes a new unsupervised domain adaptive semantic segmentation method. On the one hand, it uses self-training to improve the model's learning ability for target domain data; on the other hand, it constrains the category correlation of source domain prediction and target domain prediction to be consistent, reduces the domain difference between the source domain and the target domain, and thus improves the semantic segmentation accuracy of the target domain. Specifically, it includes the following three steps:

[0007] Step 1: training a neural network based on source domain images;

[0008] Step 1.1, randomly initialize network parameters;

[0009] Step 1.2: Input the source domain image and its corresponding label into the network, use the source domain label for supervised training, calculate the cross entropy loss, and optimize the network parameters through back propagation, so that the network has a higher semantic segmentation ability for the source domain image;

[0010] Step 1.3, save the trained network parameters.

[0011] Step 2: Use the trained network to calculate the pseudo label of the target domain image

[0012] Step 2.1, load the trained network parameters

[0013] Step 2.2: Input the target domain image into the network, output the target domain prediction map, and count the prediction probability values ​​of each category of all target domain prediction maps.

[0014] In step 2.3, the predicted probability values ​​of each category are sorted from large to small, and the top 20% values ​​of each category are taken as the pseudo-label threshold.

[0015] In step 2.4, the target domain prediction map is limited by using the pseudo-label threshold, and pixels whose maximum predicted category probability value is higher than the threshold are marked and saved as pseudo-labels of the target domain image.

[0016] Step 3: Retrain the network using source domain images and target domain images with pseudo labels

[0017] Step 3.1, load the trained network parameters;

[0018] Step 3.2, randomly match the labeled or pseudo-labeled source domain image and the target domain image, input them into the network in pairs, and obtain the features and prediction graphs of the two domains;

[0019] Step 3.3, using the two domain prediction graphs and their respective labels or pseudo labels, calculate the cross entropy loss of the two domains respectively;

[0020] Step 3.4, input the two domain features and their respective labels or pseudo labels into the category-level relevance measurement module;

[0021] Step 3.5, count the common categories of the input labels and pseudo labels, and for the common categories, calculate the category-level one-hot masks of the two domains respectively;

[0022] Step 3.6: transform the dimensions of the two domain features and their corresponding category-level unique hot masks, unify the dimensions and multiply them to obtain the two domain class representation matrix;

[0023] Step 3.7, calculate the class correlation matrix of the two domain class representation matrices, and subtract the corresponding elements of the two domain class correlation matrices, and calculate the sum of the absolute values, which is used as the class correlation loss;

[0024] Step 3.8, combine the losses of step 3.3 and step 3.7 as the total loss, perform back propagation, and expect the total network loss to be minimized, optimizing the network's semantic segmentation ability for the target domain image;

[0025] Step 3.9, save the retrained network parameters;

[0026] Step 4: Repeat steps 2 and 3 to further improve the accuracy of pseudo labels and optimize the generalization ability of the network.

[0027] Compared with the prior art, the present invention has the following advantages:

[0028] 1) This method uses a self-training method to extract high-confidence pseudo-labels of the target domain using a trained network, which makes up for the lack of supervision information in the target domain. Compared with other methods, it enriches the information of the target domain data and improves the network's learning ability for the target domain data;

[0029] 2) Compared with other traditional methods, this method focuses on the category-based domain differences, measures the class correlation of the source domain and the target domain, constrains the class correlation of the two domains to be consistent, reduces the domain differences at the class level of the two domains, and improves the generalization ability of the network.

[0030] 3) Experiments have shown that the performance of the present invention is better than other unsupervised domain adaptive semantic segmentation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 The network structure for the unsupervised domain adaptive semantic segmentation method.

[0032] Figure 2 The structure of the classifier σ

[0033] Figure 3 is the category-level correlation loss L between the source domain image and the target domain image corr Calculation process diagram. DETAILED DESCRIPTION

[0034] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0035] 1. Network structure:

[0036] The overall network structure of the unsupervised domain adaptive semantic segmentation method proposed in this invention is as follows: Figure 1 As shown in the figure, the main network adopts the Deeplab-V2 framework based on ResNet101. ResNet101 as the basic network is the feature extractor G, and the ASPP module is the classifier σ. The structural parameters of the feature extractor are shown in Table 1, where the convolution layer Conv_1 contains 64 7×7 filters with stride=2 and padding=3; Conv_2, Conv_3, Conv_4, and Conv_5 are four blocks, each of which includes 3, 4, 23, and 3 residual modules respectively, and uses ReLU as the activation function.

[0037] The features output by Conv_5 are passed to the classifier σ. The final feature map output by σ is 1 / 8 of the original image. The feature map output by σ is restored to the original image size using bilinear interpolation. Softmax is used to calculate the predicted probability values ​​of each category to obtain the final semantic segmentation result. The structure of the classifier σ is as follows: Figure 2 Its detailed parameters are shown in Table 2.

[0038] 2. Network training process

[0039] 1) Train the network using source domain images.

[0040] The network input is the source domain image and its label (I s, Y s ),in represents the source domain image, Recording Image I s The label of , that is, if the nth pixel belongs to the cth class, The network loss function is the cross entropy loss function defined by formula (1):

[0041]

[0042] Where N = H × W, θ represents the network parameter, n represents the pixel index, and K represents the number of categories. Represents the source domain image I s Whether the nth pixel is marked as category c, if it is marked as category c, its value is 1, otherwise it is 0. Represents the image I output by the network s The probability value of the nth pixel being predicted as the cth class.

[0043] 2) Use the network to compute pseudo labels for target domain images.

[0044] The network input is the target domain image, and the output is the predicted label of the target domain image. For any target domain image definition Its pseudo label, the value of the pseudo label is generated according to the network's prediction of the pixel, and Represents the network's target domain image I t The prediction of the nth pixel is where c∈{1, 2, ..., K}.

[0045] According to the target domain image I output by the network t The probability value of the nth pixel predicted as class c calculate The specific calculation method is as follows:

[0046]

[0047] Among them, α c represents the threshold of class c.

[0048] α c The calculation method is: sort all the pixels predicted to be the cth class in the target domain image from large to small according to the predicted probability value, and the predicted probability value of the pixels in the first 20% position is α c .

[0049] 3) Train the network using source domain images and target domain images with pseudo labels.

[0050] The network input is a pair of randomly matched source domain images I s and the target domain image I t The network loss function is L(I s , I t )=L sce +L tce +λ a L corr , where L sce is the source domain image I s The cross entropy loss is calculated as in formula (1); L tce is the target domain image I t The cross entropy loss is calculated as shown in formula (3); a is a hyperparameter, L corr is the class-related loss between the source domain image and the target domain image, and is calculated as shown in formula (10).

[0051] Formula (3) defines the target domain image I t The cross entropy loss is:

[0052]

[0053] in Represents the target domain image I t The nth pixel of is predicted as category c by the network in step 2). If it is predicted as category c, its value is 1, otherwise it is 0. Indicates that the current network converts image I t The probability value of the nth pixel being predicted as the cth class.

[0054] Figure 3 The category-level correlation loss L between the source domain image and the target domain image is shown. corr The calculation process. First, calculate the source domain image I s and the target domain image I t Public tag collection of Let the number of elements in set U be K U K U =|U|.

[0055] Source domain image I s via Figure 1 The features obtained by the ASPP module of the network are recorded as Target domain image I t via Figure 1 The features obtained by the ASPP module of the network are recorded as

[0056] Then, using feature F s The corresponding true label Y of the source domain image s Making a unique thermal mask Similarly, using feature F t The corresponding pseudo-label of the target domain image Making a unique thermal mask Where u∈U.

[0057]

[0058]

[0059] Using a unique thermal mask M s and M t For feature F s and F t Carry out the separation of each category and s and F t Reshape it to have the following dimensions: Denoted as and The same applies to M s and M t Also reshape it to have the dimension Denoted as and Using formulas (6) and (7), we can obtain the class representation matrix and

[0060]

[0061]

[0062] Calculate the class correlation matrix A for the source domain and the target domain respectively s and A t , as shown in formulas (8) and (9), E(X) represents the expectation of X.

[0063]

[0064]

[0065] Where 1≤i≤K U And 1≤i≤K U , based on the class correlation matrix A of the source domain and the target domain s and A t , calculate the class-related loss L corr .

[0066]

[0067] 4) Repeat steps 2) to 3) until the maximum number of network training rounds.

[0068] Experimental details

[0069] 1. Experimental Dataset

[0070] The proposed method is experimented on the commonly used unsupervised adaptive dataset GTA5-Cityscapes, where the synthetic dataset GTA5 is used as the source domain and the real dataset Cityscapes is used as the target domain. The model is evaluated on the Cityscapes validation set.

[0071] GTA5: The synthetic dataset GTA5 contains 24966 synthetic images with a resolution of 1914×1052 and the corresponding ground-truth. These synthetic images are collected from a cityscape video game based on the city of Los Angeles. The automatically generated ground-truth contains 33 categories. Methods experimenting on GTA5-Cityscapes generally only consider 19 categories compatible with the Cityscapes dataset, and this invention is no exception.

[0072] Cityscapes: As a dataset collected from the real world, Cityscapes provides 3975 images with fine segmentation annotations. The training set contains 2975 images and the validation set contains 500 images.

[0073] 2. Experimental evaluation indicators

[0074] The present invention uses the Intersection-over-Union (IoU) to evaluate the performance of semantic segmentation. The IoU value is between [0, 1]. The larger the value, the better the segmentation effect. The IoU is defined as follows:

[0075] IoU = TP / (TP+FP+FN)

[0076] Where TP, FP and FN are the number of true positive, false positive and false negative pixels respectively. The mIoU in Table 3 is the average IoU of 19 categories.

[0077] 3. Network training parameters

[0078] The present invention sets batch_size=2, the input image resolution of the source domain and the target domain is 512*1024, the test resolution is 1024*2048, random gradient descent is used for optimization, and the learning rate is set to 5*10 -5 Before training begins, first, the initial network is used to evaluate the training set of the target domain, see step 1); then, α is extracted based on the evaluation results. cand pseudo labels, see step 2); finally, the source domain and target domain data are simultaneously input into the network for network parameter training, see step 3). According to step 4), the self-training process is trained continuously for three rounds, and the model is saved in each round, and the one with the best performance is selected.

[0079] 4. Experimental results

[0080] The experimental results of the present invention, AdaptSegNet[1], and AdvEnt+MinEnt[2] on the commonly used unsupervised adaptive dataset GTA5-Cityscapes show that the experimental results of the present invention are significantly better than those of the comparison methods. a When θ is 0.005, our method is 0.66 higher than the baseline method.

[0081] Table 1: Feature generator structure parameters

[0082]

[0083] Table 2: ASPP structural parameters

[0084]

[0085] [1] Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang, and Manmohan Chandraker. Learning to adapt structured output space forsemantic segmentation. In CVPR, 2018.

[0086] [2]Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. ADVENT: adversarial entropy minimization for domain adaptation insemantic segmentation. CoRR, abs / 1811.12833, 2018.

Claims

1. An unsupervised domain adaptive semantic segmentation method, characterized by: The specific steps include: Step 1: training a neural network based on source domain images; Step 2: Use the trained network to calculate the pseudo label of the target domain image; Step 3: Retrain the network using source domain images and target domain images with pseudo labels. Step 4: Repeat steps 2 and 3 to improve the accuracy of pseudo labels and optimize the generalization ability of the network; In step 1, step 1.1, randomly initialize the network parameters; Step 1.2: Input the source domain image and its corresponding label into the network, use the source domain label for supervised training, calculate the cross entropy loss, and optimize the network parameters through back propagation, so that the network has a higher semantic segmentation ability for the source domain image; Step 1.3, save the trained network parameters; In step 2, step 2.1, load the trained network parameters; Step 2.2, input the target domain image into the network, output the target domain prediction map, and count the prediction probability values ​​of each category of all target domain prediction maps; Step 2.3, sort the predicted probability values ​​of each category from large to small, and take the top 20% values ​​of each category as the pseudo-label threshold; Step 2.4: Use the pseudo-label threshold to limit the target domain prediction map, annotate the pixels whose maximum prediction category probability value is higher than the threshold, save and use it as the target domain image pseudo-label; In step 3, step 3.1, load the trained network parameters; Step 3.2: randomly match the labeled or pseudo-labeled source domain image and the target domain image, input them into the network in pairs, and obtain the features and prediction graphs of the two domains; Step 3.3, using the two domain prediction graphs and their respective labels or pseudo labels, calculate the cross entropy loss of the two domains respectively; Step 3.4, input the two domain features and their respective labels or pseudo labels into the category-level relevance measurement module; Step 3.5, count the common categories of the input labels and pseudo labels, and for the common categories, calculate the category-level one-hot masks of the two domains respectively; Step 3.6: transform the dimensions of the two domain features and their corresponding category-level unique hot masks, unify the dimensions and multiply them to obtain the two domain class representation matrix; Step 3.7, calculate the class correlation matrix of the two domain class representation matrices, and subtract the corresponding elements of the two domain class correlation matrices, and calculate the sum of the absolute values, which is used as the class correlation loss; Step 3.8, combine the losses of step 3.3 and step 3.7 as the total loss, perform back propagation, and expect the total network loss to be minimized, optimizing the network's semantic segmentation ability for the target domain image; Step 3.9, save the retrained network parameters.

Citation Information

Patent Citations

  • Unsupervised domain adaptive semantic segmentation method based on maximum square loss

    CN110222690A

  • Domain adaptive semantic segmentation method based on similarity space alignment

    CN110322446A

  • Image segmentation network training method and device, image segmentation method and device and storage medium

    CN111199550A

  • Gini exponent-based domain adaptive semantic segmentation method

    CN112116593A