An adaptive image semantic segmentation method based on pixel dual consistency and energy score

Abstract

The application relates to a kind of unsupervised domain adaptation street view semantic segmentation method, establish PixEL unsupervised domain adaptation network model;First, the pre-training segmentation model is obtained by training the segmentation network deeplabv2_multi to source domain data set;In target domain, the features obtained by deeplabv2_multi encoder are added to the disturbance of image, and the enhanced version of the features of the image is obtained;The strong enhancement version, weak enhancement version and feature disturbance version of the target domain image are trained by using the segmentation model;In target domain, the closeness of the energy score of the unlabeled sample to the current training distribution is evaluated by calculating the energy score, the sample with higher energy score is selected to generate pseudo label, and the model is further optimized and trained by using the pseudo label.The application compared with the existing unsupervised domain adaptation method obtains better semantic segmentation effect on the benchmark data set from synthesis to reality, whether in subjective vision or in objective evaluation index is obviously improved, effectively solves the unsupervised domain adaptation problem in the field of street view image semantic segmentation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and semantic segmentation, specifically, to an unsupervised domain adaptive street image semantic segmentation method based on pixel double consistency and energy score. BACKGROUND

[0002] In the field of image semantic segmentation, unsupervised domain adaptation (UDA) methods play an increasingly important role. UDA methods are particularly important in image semantic segmentation because they can utilize unlabeled data to improve the generalization ability of the model, thereby reducing the need for labeled data. They are also useful for intelligent transportation systems and autonomous driving systems.

[0003] The current mainstream solution for unsupervised domain adaptive street image semantic segmentation is to use a strong-weak consistency framework. However, this approach may limit the model's ability to adapt to various scene changes, and the pseudo-label reliability is low, making it difficult to effectively utilize unlabeled data and resulting in poor model generalization. Exploring a broader perturbation space can enable the model to better adapt to various scene changes, and using energy scores to filter labels can make more efficient use of unlabeled data. Therefore, it is necessary to study the application of the above methods in UDA semantic segmentation. SUMMARY

[0004] To overcome the shortcomings of the prior art, the present application provides an unsupervised domain adaptive street image semantic segmentation method based on pixel double consistency and energy score, which achieves high-precision semantic segmentation of input images and overcomes the problems of the current strong-weak consistency framework. It can be widely used in urban intelligent transportation and autonomous driving systems.

[0005] The technical solution adopted by the present application to achieve the above-mentioned purposes is as follows:

[0006] An adaptive image semantic segmentation method based on pixel double consistency and energy score, comprising the following steps:

[0007] 1) Obtain labeled and unlabeled street image data from the source domain and the target domain, respectively;

[0008] 2) Train a semantic segmentation model using the labeled data from the source domain;

[0009] 3) Perform three different image enhancement operations on the unlabeled data from the target domain, and use the semantic segmentation model to predict the three enhanced images to obtain the corresponding probability distribution and logit value;

[0010] 4) Calculate the energy score of each pixel, and filter out the pixels that meet the threshold condition according to the energy score threshold, and generate pseudo-labels for them;

[0011] 5) The consistency loss is calculated by using the pseudo-label of the image-level weakly augmented image and the prediction results of the other two augmented images.

[0012] 6) The semantic segmentation model is jointly optimized using the labeled data of the source domain and the pseudo-label data of the target domain, and the model parameters are updated.

[0013] 7) Repeat steps 3) to 6) until the model converges or reaches the preset number of iterations, obtain the final semantic segmentation model, and use the model for image segmentation.

[0014] The semantic segmentation model adopts a DeepLab-v2 structure, including an encoder based on ResNet-101 and a decoder based on atrous convolution.

[0015] The step 3) includes the following steps:

[0016] 3.1) Translate, crop, and color transform the original image to obtain an image-level weakly augmented image;

[0017] 3.2) Perform noise injection, MixMatch, and CutMatch operations on the original image to obtain an image-level strongly augmented image;

[0018] 3.3) Perform random dropout or noise addition operations on the output features of the encoder to obtain a feature-level augmented image;

[0019] 3.4) Input the three augmented images into the semantic segmentation model to output logit values representing the relative scores of each pixel belonging to each class;

[0020] 3.5) Input the logit values into a softmax function to obtain the corresponding probability distribution.

[0021] The energy score is:

[0022]

[0023] where x is the input data, represents the corresponding logit value of the i-th class, is the number of classes, is the adjustable temperature.

[0024] The consistency loss is:

[0025]

[0026] where, is the batch size of unlabeled data, is an image-level weak augmentation operation, is an image-level strong augmentation or feature-level augmentation operation, is an energy fraction, is an energy fraction threshold, is a cross-entropy loss, is a probability distribution of strong augmentation or feature-level augmentation prediction, is a probability distribution of weak augmentation prediction, is an indicator function, is a segmentation model, is a ground truth label.

[0027] The step 6) comprises the following steps:

[0028] 6.1) using a cross-entropy loss as a supervised loss of the source domain :

[0029]

[0030] wherein, is a batch size of the source domain data, is an input image of the source domain, is a ground truth label of the source domain, is a predicted probability distribution, is a number of samples in the batch, is a cross-entropy loss;

[0031] 6.2) using a consistency loss as an unsupervised loss of the target domain;

[0032] 6.3) using a joint optimization loss as a total loss function :

[0033]

[0034] wherein, is a consistency loss of the th augmentation manner of the target domain, is a hyperparameter;

[0035] 6.4) updating the model parameters using a stochastic gradient descent method to minimize the joint optimization loss.

[0036] An adaptive image semantic segmentation system based on pixel double consistency and energy fraction, comprising:

[0037] an image acquisition module, configured to acquire labeled and unlabeled street view image data from a source domain and a target domain respectively;

[0038] a semantic segmentation model training module, configured to train a semantic segmentation model using labeled data of the source domain;

[0039] an image enhancement module, configured to perform three different image enhancement operations on the unlabeled data of the target domain, and use the semantic segmentation model to predict the three enhanced images to obtain corresponding probability distribution and logit value;

[0040] an energy score calculation module, configured to calculate the energy score of each pixel, filter out the pixels meeting the threshold condition according to the energy score threshold, and generate pseudo labels for the pixels;

[0041] a consistency loss calculation module, configured to use the pseudo labels of the image-level weak enhanced image and the prediction results of the other two enhanced images for consistency constraint, and calculate the consistency loss;

[0042] a semantic segmentation model updating module, configured to use the labeled data of the source domain and the pseudo label data of the target domain to jointly optimize the semantic segmentation model, and update the model parameters.

[0043] An adaptive image semantic segmentation device based on pixel double consistency and energy score, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to realize the adaptive image semantic segmentation method based on pixel double consistency and energy score when the computer program is executed.

[0044] A computer readable storage medium, the storage medium stores a computer program, when the computer program is executed by a processor, the adaptive image semantic segmentation method based on pixel double consistency and energy score is realized.

[0045] The present application has the following advantages and benefits:

[0046] 1. The present application proposes an unsupervised domain adaptive semantic segmentation method based on pixel double consistency, which expands the perturbation space by performing different perturbations at the image level and feature level, and improves the prediction reliability of the model in the target domain.

[0047] 2. The present application proposes a pseudo label generation strategy based on energy score, which calculates the energy score of unlabeled samples to evaluate their proximity to the current training distribution, selects samples with low energy score to generate pseudo labels, and improves the overall reliability of the pseudo labels. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is the schematic diagram of the overall framework of the method of the present application;

[0049] Figure 2 is the schematic diagram of the target domain branch of the method of the present application;

[0050] Figure 3is a schematic diagram of a double consistency framework of the method of the present application;

[0051] Figure 4 is a schematic diagram of an energy score of the method of the present application. DETAILED DESCRIPTION

[0052] The present application will be further described below in conjunction with the accompanying drawings and embodiments.

[0053] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort belong to the scope of protection of the present application.

[0054] A street view semantic segmentation method of unsupervised domain adaptation, a PixEL unsupervised domain adaptation network model is established; first, the pre-training segmentation model is obtained by training the segmentation network deeplabv2_multi through the source domain data set; the features obtained by the image through the deeplabv2_multi encoder are added to the disturbance in the target domain to obtain the feature disturbance enhanced version of the image; the segmentation model is used to perform consistency training on the strong enhanced version, the weak enhanced version and the feature disturbance version of the target domain image; the energy score of the unlabeled sample is calculated in the target domain to evaluate the closeness of the unlabeled sample to the current training distribution, the sample with a higher energy score is selected to generate a pseudo label, and the model is further optimized and trained by using the pseudo label. Compared with the existing unsupervised domain adaptation method, the present application achieves better semantic segmentation effect on the synthetic-to-real benchmark data set, and obviously improves the subjective vision and objective evaluation index, effectively solving the problem of unsupervised domain adaptation in the field of street view image semantic segmentation

[0055] The method of the present application mainly includes two parts of a source domain branch and a target domain branch, as shown in Figure 1 The source domain branch uses the labeled data of the source domain to train an initial semantic segmentation model, the model adopts a DeepLab-v2 structure, including an encoder based on ResNet-101 and a decoder based on an empty convolution. The target domain branch uses the unlabeled data of the target domain for unsupervised domain adaptation, and the branch includes two main innovations: one is a consistency framework based on pixel double consistency, and the other is a pseudo label generation strategy based on energy score.

[0056] As shown in Figure 2 The specific process of the target domain branch is as follows:

[0057] Step 1: For each image in the target domain, we generate three image augmentation versions, namely image-level weak augmentation, image-level strong augmentation, and feature-level augmentation. Image-level weak augmentation is a small perturbation or augmentation operation on the original image, such as translation, cropping, color transformation, etc., which does not change the semantic information of the image. Image-level strong augmentation is a large perturbation or augmentation operation on the original image, such as noise injection, MixMatch, CutMatch, etc., which changes the feature expression of the image and increases the diversity of the target domain image. Feature-level augmentation is to perform random dropping or adding noise on the output features of the encoder, which increases the difference and diversity between features and expands the perturbation space of features.

[0058] Step 2: Use the semantic segmentation model to predict the three augmented images to get the corresponding probability distribution and logit value. The probability distribution is obtained by applying the softmax function to the logit value, which represents the probability of each pixel belonging to each class. The logit value is the original output of the model, which represents the relative score of each pixel belonging to each class.

[0059] Step 3: Calculate the energy score of each pixel, and according to the energy score threshold, filter out the pixels close to the current training distribution and generate pseudo labels for them. The energy score is an index that measures the distance between unlabeled samples and data distribution, which is based on the probability density function and reflects the probability density of a sample in the model. The lower the energy score, the more likely the sample belongs to the data distribution covered by the current model; otherwise, it is more likely to be an abnormal sample. The energy score is defined as:

[0060]

[0061] where x is the input data, represents the corresponding logit value of the i-th class, is the number of classes, is the adjustable temperature. As Figure 4 shown, we can set an energy score threshold τe according to the distribution of the energy score, and consider the pixels with energy score lower than the threshold as in-distribution pixels and generate pseudo labels for them. The pseudo label is obtained by taking the maximum value of the probability distribution, which represents the most likely class of each pixel. In this way, we can filter out samples close enough to the current training data and assign pseudo labels, making the model's update more robust and avoiding the instability factors that may be caused by the softmax confidence score.

[0062] Step 4: The pseudo-label of the image-level weak augmented image is used to constrain the consistency of the prediction results of the other two augmented images, and the consistency loss is calculated. The purpose of consistency constraint is to require the model to keep consistent output for the same target domain image under different perturbations. This step is to make the model learn to maintain robustness under different perturbations. As shown in Figure 3 , we use a double consistency framework, that is, consistency constraint is performed at both image level and feature level. We use cross-entropy loss as the consistency loss, defined as:

[0063]

[0064] where Bu is the batch size of unlabeled data, is the image-level weak augmentation operation, is the image-level strong augmentation or feature-level augmentation operation, is the energy score, is the energy score threshold, is the cross-entropy loss, is the probability distribution of strong augmentation or feature-level augmentation prediction, is the probability distribution of weak augmentation prediction. We only perform consistency constraint on pixels with energy score below the threshold, which can avoid interference with abnormal samples and improve the quality of consistency.

[0065] Step 5: Segmentation model can be decomposed into encoder g and decoder h. In addition to and mentioned in FixMatch framework, we also add feature perturbation , which is obtained by the following formula:

[0066]

[0067] where, is the extracted feature of , and is the feature perturbation, which can be dropout or adding noise, etc.

[0068] In summary, the pixel strong-weak consistency constraint framework is as shown in Figure 4 , each small batch of unlabeled data maintains three image augmentation methods: (1) image-level weak augmentation: -> -> (2) image-level strong augmentation: -> -> (3) feature-level augmentation: ->g-> ->h-> .

[0069] Finally, the loss of pixel intensity consistency constraint can be simply summarized as follows:

[0070]

[0071] Image-level augmentation and feature-level augmentation have their own properties and advantages, respectively. By combining feature-level perturbation with image-level perturbation, a more effective consistency framework is constructed.

[0072] Step 6: Use the labeled data of the source domain and the pseudo-label data of the target domain to jointly optimize the semantic segmentation model and update the model parameters. The purpose of joint optimization is to make the model achieve good performance on both the source domain and the target domain, while reducing the difference between the two domains. We use cross-entropy loss as the supervised loss of the source domain, defined as:

[0073]

[0074] where is the batch size of the source domain data, is the input image of the source domain, is the true label of the source domain, is the predicted probability distribution. We use consistency loss as the unsupervised loss of the target domain, defined as:

[0075]

[0076] where the meanings of the symbols are the same as before. We use joint optimization loss as the total loss function, defined as:

[0077]

[0078] where is the supervised cross-entropy loss of the source domain, is the consistency loss of the th augmentation method of the target domain, is a hyperparameter used to control the relative weight of the source domain and target domain loss. We use the stochastic gradient descent method to update the model parameters to minimize the joint optimization loss and improve the performance of the model on the source domain and target domain;

[0079] Step 7: Repeat the above steps until the model converges or reaches the preset number of iterations.

[0080] The present application is oriented to real street view images, and GTA-to-Cityscapes is a widely recognized large-scale UDA segmentation benchmark, which takes the Cityscapes street view dataset as the target domain, and the dataset contains 2975 training images and 500 validation images, and the resolution of each image is 2 As source domain is the GTA dataset, which contains 24966 Zhang Yuhe Dataset consisting of synthetic images, resolution of 1 In addition, the Synthia dataset can also be selected, which contains 9400 synthetic pictures, and the resolution is .

[0081] In the experiment, a DeepLab-v2 segmentation model with a ResNet-101 backbone is used. The training batch Batch_size is 1, the number of categories num_classes is 19, the number of iterations iterations is 150000, the learning strategy adopts a stochastic gradient descent (SGD) algorithm, the momentum is set to 0.9, the weight decay is set to 0.0005, and the polynomial learning decay rate is: All experiments are performed on a single NVIDIA RTX 2080 Ti GPU with 11 GB of VRAM.

[0082] The experimental results of the present application are shown in Table 1:

[0083] Table 1

[0084]

Claims

1. An adaptive image semantic segmentation method based on pixel dual consistency and energy score, characterized in that, The method comprises the following steps: 1) obtaining labeled and unlabeled street view image data from source and target domains respectively; 2) training a semantic segmentation model using labeled data from the source domain; 3) performing three different image enhancement operations on the unlabeled data from the target domain, and using the semantic segmentation model to predict the three enhanced images to obtain corresponding probability distribution and logit value; 4) calculating the energy score of each pixel, screening out pixels meeting the threshold condition according to the energy score threshold, and generating pseudo-labels for the pixels; 5) using the pseudo-labels of the image-level weakly enhanced image and the prediction results of the other two enhanced images to perform consistency constraint, and calculating consistency loss; 6) jointly optimizing the semantic segmentation model using labeled data from the source domain and pseudo-label data from the target domain, and updating model parameters; 7) repeating steps 3) to 6) until the model converges or reaches a preset number of iterations, obtaining a final semantic segmentation model, and using the model for image segmentation; Step 3) comprises the following steps: 3.1) performing translation, cropping and color transformation operations on the original image to obtain an image-level weakly enhanced image; 3.2) performing noise injection, MixMatch and CutMatch operations on the original image to obtain an image-level strongly enhanced image; 3.3) performing random dropout or noise addition operations on the output features of the encoder to obtain a feature-level enhanced image; 3.4) inputting the three enhanced images into the semantic segmentation model to output logit values representing the relative scores of each pixel belonging to each class; 3.5) inputting the logit values into a softmax function to obtain the corresponding probability distribution.

2. The adaptive image semantic segmentation method based on pixel dual consistency and energy score according to claim 1, characterized in that, The semantic segmentation model adopts a DeepLab-v2 structure, including an encoder based on ResNet-101 and a decoder based on atrous convolution.

3. The adaptive image semantic segmentation method based on pixel dual consistency and energy score according to claim 1, characterized in that, The energy fraction is: ； where x is the input data, denotes the respective logit value for the i-th class, is the number of classes, is the adjustable temperature.

4. The adaptive image semantic segmentation method based on pixel dual consistency and energy score according to claim 1, characterized in that, The consistency loss Is: ； wherein, is the batch size of unlabelled data, is an image-level weak augmentation operation, is an image-level strong augmentation or feature-level augmentation operation, is an energy fraction, is an energy fraction threshold, is a cross-entropy loss, is a probability distribution of strong augmentation or feature-level augmentation prediction, is a probability distribution of weak augmentation prediction, is an indicator function, is a segmentation model, is a ground truth label.

5. The adaptive image semantic segmentation method based on pixel dual consistency and energy score according to claim 1, characterized in that, Step 6) comprises the following steps: 6.1) Use cross-entropy loss as supervised loss for source domain : ； wherein, is a batch size of the source domain data, is an input image of the source domain, is a ground truth label of the source domain, is a predicted probability distribution, is a number of samples in the batch, is a cross-entropy loss; 6.2) using consistency loss as unsupervised loss of the target domain; 6.3) Use joint optimization loss as total loss function : ； wherein, is a consistency loss for the th augmented version of the target domain, is a hyper-parameter; 6.4) updating the model parameters using the stochastic gradient descent method to minimize the joint optimization loss.

6. An adaptive image semantic segmentation system based on pixel dual consistency and energy score, used to implement the adaptive image semantic segmentation method based on pixel dual consistency and energy score in claim 1, characterized in that, It comprises: An image acquisition module for obtaining labeled and unlabeled street view image data from source and target domains respectively; A semantic segmentation model training module for training a semantic segmentation model using labeled data from the source domain; An image enhancement module for performing three different image enhancement operations on the unlabeled data from the target domain, and using the semantic segmentation model to predict the three enhanced images to obtain corresponding probability distribution and logit value; An energy score calculation module for calculating the energy score of each pixel, screening out pixels meeting the threshold condition according to the energy score threshold, and generating pseudo-labels for the pixels; A consistency loss calculation module for using the pseudo-labels of the image-level weakly enhanced image and the prediction results of the other two enhanced images to perform consistency constraint, and calculating consistency loss; A semantic segmentation model updating module for jointly optimizing the semantic segmentation model using labeled data from the source domain and pseudo-label data from the target domain, and updating model parameters.

7. An adaptive image semantic segmentation device based on pixel dual consistency and energy score, characterized in that, The application discloses a kind of based on pixel double consistency and energy score's self-adapting image semantic segmentation method, including memory and processor;The memory is used to store computer program;The processor is used to realize the method as claimed in any one of claims 1-5 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The storage medium has a computer program stored thereon, and the computer program, when executed by a processor, implements the self-adapting image semantic segmentation method based on pixel double consistency and energy score as claimed in any one of claims 1-5.

Images