Domain-adaptive semantic segmentation method based on dual-pseudo-label fusion and adaptive region pasting

Abstract

The invention provides a domain adaptive semantic segmentation method based on double pseudo label fusion and adaptive region pasting. According to the method, a pre-trained encoder-decoder network is used for carrying out style migration on a target domain image; respectively inputting the original target domain image and the style migrated target domain image into an EMA teacher network to generate a multi-view pseudo tag; designing a multi-dimensional adaptive fusion strategy based on confidence, entropy and consistency, and generating a fusion pseudo tag by calculating the confidence, the prediction entropy and the cosine similarity of prediction distribution; and selecting a region with low confidence in the target domain image through an adaptive region pasting method, and adaptively pasting the scaled source domain object to the region with low confidence to enhance semantic information of the target region, thereby improving the attention capability of the model to a difficult region. According to the method, the performance of domain adaptive semantic segmentation can be effectively improved, and the generalization ability and robustness of the model are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and deep learning, in particular, to a domain adaptation semantic segmentation method based on pseudo label denoising and data augmentation. BACKGROUND

[0002] Semantic segmentation is a fundamental task in computer vision, which aims to assign a semantic label to each pixel in a given image. This task helps to achieve precise understanding and expression of objects and scenes in images and is widely applied in various fields, such as autonomous driving, human-computer interaction, and medicine. The combination of large-scale labeled datasets and complex network architectures promotes the development of deep neural networks, which has significantly improved the accuracy and robustness of supervised semantic segmentation in diverse scenarios. However, the main challenge of supervised semantic segmentation is the high dependence on pixel-level labeled data. The acquisition of such data usually requires a large amount of manpower and cost, especially in tasks with high labeling granularity requirements. To alleviate the labeling burden, unsupervised domain adaptation (UDA) semantic segmentation methods are proposed.

[0003] Unsupervised domain adaptation aims to utilize the labeled data in the source domain to improve the performance of the model on the unlabeled data in the target domain through transfer learning methods. This technology mainly involves two domains: source domain and target domain. The source domain contains rich labeled data, which is used for model training to learn features and patterns; the target domain is data with scarce or no labeled data. Existing unsupervised domain adaptation semantic segmentation methods mainly include GAN-based UDA and ST-based UDA. GAN-based UDA models mainly use the generator and discriminator in the GAN model to align the source domain and target domain images to achieve segmentation of the target domain data. ST-based UDA is inspired by semi-supervised learning, which generates pseudo labels for unlabeled target domain data by using a teacher model trained on labeled data, and iteratively fine-tunes the target model using these pseudo labels to improve the model's performance on the target domain.

[0004] In recent years, the emergence of synthetic datasets such as GTA5 and SYNTHIA provides important support for unsupervised domain adaptation semantic segmentation. Such datasets can be automatically generated by game engines, with characteristics such as large scale, diversity and pixel-level accurate annotation, significantly reducing the cost and difficulty of manual annotation. Combined with these resources and transfer learning technology, unsupervised domain adaptation has achieved good results in multiple computer vision tasks while reducing the dependence on annotation. However, compared with supervised training, the existing unsupervised domain adaptation semantic segmentation method still has a performance gap. For example, the self-training method, the pseudo-label generated by the teacher network is often derived from a model that has not been fully trained, which is easily affected by prediction errors and introduces noise, limiting the improvement of model performance. In addition, when there is a large distribution difference between the source domain and the target domain, the model is difficult to effectively generalize, especially in complex structures or significantly different feature areas, the segmentation performance decreases more obviously. Therefore, how to ensure the generalization ability while reducing noise interference has become one of the key challenges in the current research of unsupervised domain adaptation semantic segmentation. SUMMARY

[0005] The technical problem to be solved by the present application is how to ensure the quality of the fused pseudo-label and how to determine the best position of the pasted area to improve the overall segmentation effect in the task of unsupervised domain adaptation semantic segmentation.

[0006] To solve the above technical problems, the technical scheme adopted by the present application is: a domain adaptation semantic segmentation method based on double pseudo-label fusion and adaptive region pasting, characterized by comprising the following steps:

[0007] S1, based on the strategy of double pseudo-label fusion, the pseudo-labels generated by multiple perspectives are integrated to generate complementary labels, and a multi-dimensional adaptive fusion strategy is used to weight and fuse the multi-perspective pseudo-labels to generate a fused pseudo-label;

[0008] S2, an adaptive region pasting method is used to select a region with low confidence in the target domain image, and the scaled source domain object is adaptively pasted to the selected region to enhance the semantic information of the region.

[0009] According to a preferred embodiment of the present application, the strategy of double pseudo-label fusion in S1 comprises the following specific steps:

[0010] S101, use a pre-trained network based on CycleGAN to extract the content features and style features of the input source domain image and target image, and perform style transfer on the target image through a cycle consistency loss.

[0011] S102, input the original target domain image and the target domain image after style transfer into the EMA teacher network respectively, and generate multi-perspective pseudo-labels from the teacher network;

[0012] S103, combine the multi-view pseudo labels based on the confidence, consistency and uncertainty to generate a fused pseudo label;

[0013] According to a preferred embodiment of the present application, in S2, an adaptive region pasting method is used to select a region with low confidence in the target domain image, and the specific steps are as follows:

[0014] The adaptive region pasting method comprises: using a sliding window to scan on the target domain confidence map to calculate the average confidence of the pixels in each window, and taking the window with the lowest confidence as the region to be enhanced; and pasting the scaled source domain object to the region to be enhanced.

[0015] The beneficial effects of the present application are as follows:

[0016] By using cycle consistency loss to perform style conversion on the target domain image, the domain gap between the source domain and the target domain is narrowed. The teacher network generates pseudo labels for the original target domain image and the target domain image after style transfer, solving the problem that the pseudo labels generated under a single view are limited by the model's adaptation ability to the target domain distribution. In addition, through a multi-dimensional adaptive fusion strategy based on confidence, entropy and consistency between predicted distributions, the multi-view pseudo labels are optimally fused, improving the overall quality and accuracy of the fused pseudo labels. The adaptive region pasting method is used to select a region with low confidence in the target domain image using a sliding window, and the scaled source domain object is pasted to the selected region, maintaining context consistency through position and scale matching, and further enhancing the semantic information of the low-confidence region in the target domain. In order to improve the stability and generalization ability of the domain adaptation semantic segmentation task, the method of the present application effectively guarantees the quality of the fused pseudo labels and optimizes the selection of the pasted region, thereby significantly improving the overall segmentation effect. By comprehensively utilizing the double pseudo label fusion and adaptive region pasting mechanism, the method of the present application effectively alleviates the challenges of unsupervised domain adaptation semantic segmentation in cross-domain scenarios, and further improves the segmentation accuracy and robustness of the model. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 The figure is a schematic diagram of the overall network framework of the present application; DETAILED DESCRIPTION

[0018] The present application will be described below in conjunction with the drawings and specific embodiments:

[0019] As Figure 1 shown, the domain adaptation semantic segmentation method based on double pseudo label fusion and adaptive region pasting of the present application comprises the following steps:

[0020] S1. Based on the dual pseudo-label fusion strategy, the pseudo-labels generated from multiple perspectives are integrated to generate complementary labels, and combined with a multi-dimensional adaptive fusion strategy, the pseudo-labels from multiple perspectives are weighted and fused to generate fused pseudo-labels.

[0021] S101. Employ a pre-trained and fixed target-to-source image converter. This converter, trained on the CycleGAN framework, can transform a target domain image into a source domain style. Specifically, we convert the target image x... t Input to Generate source domain style target image:

[0022]

[0023] S102. In the self-training method of unsupervised adaptive semantic segmentation, the neural network f θ Using source domain datasets and its corresponding tags By supervising the source domain loss L S Training is performed on the target domain dataset. Achieving good performance. Among them, Indicates the i-th s One source domain image, Here, H and W represent the image height and width, respectively, and C is the number of categories in the source domain. Since only source domain labels are available, the supervised classification cross-entropy loss is only applied to source domain predictions.

[0024]

[0025] Where j∈{1,2,…,H·W} represents the pixel index, and c∈{1,2,…,C} represents the category index. Since the network f is trained using only source domain data... θ The difficulty in overcoming inter-domain differences limits its generalization ability in the target domain. Self-training methods typically assign semantic pseudo-labels to the target domain images and use the pseudo-labeled images to train f. θ Training is performed to improve performance. A common practice is to increase f θ The category with the highest probability in the prediction results is used as the semantic pseudo-label:

[0026]

[0027] in, Represents pixels Belongs to the cth ′ The softmax probability of the class.

[0028] The original target domain image and the style-transferred target domain image are respectively input into the EMA teacher network to generate pseudo-labels for the original target image and the style-transferred image:

[0029]

[0030] Among them, g φ These pseudo-labels represent the teacher network. They reflect the semantic predictions of the model from different perspectives, providing a foundation for subsequent fusion and optimization.

[0031] S103. To optimize noise reduction for pseudo-labels, we not only extract the probability distributions of the teacher network's predictions of the original and style-transferred images, but also aim to leverage the differences and complementarities in their semantic outputs. Therefore, we introduce three levels of metrics: confidence, entropy, and consistency, which are used to evaluate and optimize the reliability of the prediction results, the uncertainty of the distribution, and the similarity between the outputs, respectively.

[0032] Confidence level, specifically, is the maximum probability value in the softmax output of each branch, denoted as [the confidence level is not specified in the original text]. and Based on this, normalized confidence weights are constructed:

[0033]

[0034] Here, ε is a very small positive number. If a branch has a higher prediction confidence at a certain pixel location, it should be given a higher weight when fusing pseudo-labels. This dynamic adjustment mechanism can enhance the expressive power of fused pseudo-labels in uncertain regions of the model, thereby improving the overall reliability of pseudo-supervision.

[0035] Entropy is calculated by taking the predicted probabilities of the original branch and the style transfer branch at each pixel and using their information entropy to quantify their uncertainty.

[0036]

[0037] in, and These represent the model at the pixel level. and The prediction entropy at a given point reflects the degree of dispersion of the output distribution. A higher entropy value indicates less certainty in the model's prediction of the pixel category; a lower entropy value indicates a more robust prediction. Weights and Normalization using the form exp(-H) allows branches with lower entropy to receive greater weight in the fusion process, thus favoring the selection of predictions with a more concentrated distribution. The introduction of the smoothing term ε ensures numerical stability and avoids computational problems in extreme cases.

[0038] A consistency metric is introduced to further measure the consistency of predictions made by the model under different input styles. Specifically, it first measures the predicted probability vector for each pixel. and Perform L2 normalization on each vector to obtain the unit vector:

[0039]

[0040] in, and Let represent the predicted vector of a pixel across all category dimensions c∈{1,2,…,C}. Next, the two vectors are computed at the pixel level. and The cosine similarity at each point is used as the consistency score:

[0041]

[0042] The score ranges from [0,1]. A higher value indicates greater consistency in the predictions of the two branches for that pixel, reflecting higher stability in the model's semantic judgment in that region. Through consistency measurement, the model can adapt to changes in predictions caused by input style perturbations, further improving the discriminativeness and robustness of fused pseudo-labels.

[0043] Combining the above three metrics, we calculate the final weights through linear fusion:

[0044] ω t =α·ω+β·v+γ·η ω t2s =1-ω t

[0045] Here, α, β, and γ are hyperparameters used to balance the contributions of each metric. Based on the final weights, the probability distributions of the two branches are weighted and fused:

[0046]

[0047] From fusion probability Extracting hard pseudo-labels l fus and confidence level π fus The final fused pseudo-labels are generated. To control the quality of the pseudo-labels, a quality threshold is set; only fused pseudo-labels exceeding this threshold are used for training supervision, thus providing the model with more reliable supervision signals.

[0048] S2. An adaptive region pasting method is used to select a region with low confidence in the target domain image, and the scaled source domain object is adaptively pasted into the selected region to enhance the semantic information of the region.

[0049] Building upon ClassMix, an image scaling step is first introduced. Specifically, a scaling factor α is introduced, and its size is adjusted through bilinear interpolation to obtain a scaled image. and tags Then, from the scaled label We randomly select a category c and extract its corresponding object mask M. The mask M is a binary tensor representing the spatial distribution of category c in the scaled image.

[0050] To accurately locate regions in the target image lacking semantic information, the confidence scores P obtained from the initial mixing are first... t Perform downsampling, where P t∈R H×W This represents the class confidence score for each pixel, reducing computational complexity and memory usage. Specifically, average pooling is used to adaptively downsample P by the downsampling factor d. t Convert to low-resolution confidence map Subsequently, a sliding window scan was applied to the downsampled confidence plot, and the average confidence score for each window was calculated as follows:

[0051]

[0052] in Let (i,j) represent the window region, and (i,j) be the coordinates of the window. By traversing all window positions, we identify the region with the lowest confidence and center it at (i,j). min ,j min Mapping back to the original resolution gives the paste position (x). c ,y c )=(i min ×d,j min ×d). After determining the paste location, we paste the scaled source image object into the low-confidence region of the initial blended image to obtain the enhanced image x. m and tags The introduction of the adaptive region pasting method significantly enhances the spatial structure diversity and semantic expressiveness of the target domain, effectively alleviating challenges such as inter-class imbalance, object scale variation, and complex geometry.

Claims

1. A domain-adaptive semantic segmentation method based on dual pseudo-label fusion and adaptive region pasting, characterized in that: S1. Based on the dual pseudo-label fusion strategy, the pseudo-labels generated from multiple perspectives are integrated to generate complementary labels, and combined with a multi-dimensional adaptive fusion strategy, the pseudo-labels from multiple perspectives are weighted and fused to generate fused pseudo-labels. S2. An adaptive region pasting method is used to select a region with low confidence in the target domain image, and the scaled source domain object is adaptively pasted into the selected region to enhance the semantic information of the region.

2. The domain-adaptive semantic segmentation method based on dual pseudo-label fusion and adaptive region pasting according to claim 1, characterized in that, In S1, the strategy based on dual pseudo-label fusion involves the following steps: S101. Using a CycleGAN-based pre-trained network, extract the content and style features of the input source image and the target image, and perform style transfer on the target image through cycle consistency loss. S102. Input the original target domain image and the style-transferred target domain image into the EMA teacher network, and the teacher network generates multi-view pseudo-labels. S103. Combining a multi-dimensional adaptive fusion strategy based on confidence, consistency, and uncertainty, the multi-view pseudo-labels are weighted and fused to generate fused pseudo-labels.

3. The domain-adaptive semantic segmentation method based on dual pseudo-label fusion and adaptive region pasting according to claim 1, characterized in that, In S2, an adaptive region pasting method is used to select regions with low confidence in the target domain image. The specific steps are as follows: The adaptive region pasting method includes: scanning the target domain confidence map using a sliding window to calculate the average confidence of pixels within each window, taking the window with the lowest confidence as the region to be enhanced, and adaptively pasting the scaled source domain object into the region to be enhanced.

Images