A cross-pseudo-supervised based domain adaptation semantic segmentation method
By employing a dual-model training approach with cross-pseudo-supervision and a complementary loss mechanism, the adaptability and accuracy issues of semantic segmentation models across different domains are addressed, achieving efficient semantic segmentation in complex environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA UNIV OF MINING & TECH
- Filing Date
- 2024-07-26
- Publication Date
- 2026-07-03
AI Technical Summary
Existing semantic segmentation methods suffer from domain bias and domain drift when dealing with different domains. They rely on a large amount of labeled data, which is difficult to obtain. Furthermore, pseudo-label noise and low-confidence predictions affect model performance, especially when dealing with complex input data.
A dual-model training strategy based on cross-pseudo-supervision is adopted, which combines attention modulation loss and entropy consistency loss. The model is optimized through dual-model cross-pseudo-supervision and complementary loss mechanism to improve the model's adaptability and accuracy in different environments.
It significantly improves the model's generalization ability and robustness, and enhances the accuracy and computational efficiency of semantic segmentation in different environments, especially when labeled data is scarce.
Smart Images
Figure CN118968062B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of semi-supervised domain-adaptive semantic segmentation, specifically involving a domain-adaptive semantic segmentation method based on cross-pseudo-supervision. Background Technology
[0002] In computer vision, semantic segmentation involves assigning pixels in an image to predefined category labels, a process crucial for understanding image content. Accurate semantic segmentation is fundamental to reliable perception in fields such as autonomous driving, robot navigation, and medical image analysis. However, despite significant progress in this task using deep learning techniques, models often experience a sharp performance drop when faced with new domains that are inconsistent with the distribution of their training data. This phenomenon, known as domain bias or domain drift, is a core issue in domain adaptation research.
[0003] Traditional semantic segmentation methods rely on large amounts of labeled data during training, which are often difficult to obtain, especially in specific application domains. Furthermore, even with abundant labeled data, models may fail to generalize to new domains due to over-reliance on features specific to the training domain. Therefore, researchers have proposed various semi-supervised learning methods for semantic segmentation to address this problem. These methods aim to learn a network by performing semantic segmentation using only a small subset of precisely labeled, pixel-by-pixel data and a large amount of unlabeled data.
[0004] Recently, self-supervised training has facilitated domain adaptation by using pseudo-labels generated from target domain predictions as supervised training for the network. Many methods directly enable the network to generate more accurate pseudo-labels from the output level perspective. For example, Tuan-Hung Vu et al. proposed two complementary approaches to address the unsupervised domain adaptation problem, both based on pixel-level prediction entropy. The first approach directly utilizes entropy loss, penalizing low-confidence predictions in the target domain. The second approach is an indirect entropy minimization achieved using adversarial loss. (Vu TH, Jain H, Bucher M, et al. Advent: Adversarial entropy minimization for domain adaptation in semanticsegmentation[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019: 2517-2526). Iqbal J et al. used a multi-level self-supervised learning strategy, including generating spatially independent and semantically consistent (SISC) pseudo-labels and image-level weak labels (PWL). This approach helps identify domain-invariant features at both the fine-grained pixel and image levels. (Iqbal J, Ali M. Mlsl: Multi-level self-supervised learning for domain adaptation with spatially independent and semantically consistent labeling[C] / / Proceedings of the IEEE / CVF WinterConference on Applications of Computer Vision. 2020: 1864-1873). Guo X et al. proposed an unsupervised domain adaptation method focused on solving the problem of noisy labeling in self-supervised learning, employing a domain-aware meta-learning strategy to correct the loss function. Specifically, it introduces a NoiseTransition Matrix (NTM) to model the noise distribution of pseudo-labels in the target domain, and estimates this matrix through domain-aware meta-learning.(Guo X, Yang C, Li B, et al. Metacorrection: Domain-aware metaloss correction for unsupervised domain adaptation in semantic segmentation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 3927-3936). Melas-Kyriazi L et al. proposed an unsupervised domain adaptation method based on pixel-wise consistency training. The core idea is that in order to perform well in the target domain, the model should produce stable and consistent outputs in response to small perturbations in the input. (Melas-Kyriazi L, Manrai A K. Pixmatch: Unsupervised domain adaptation via pixelwise consistency training[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 12435-12445).
[0005] Existing methods suffer from the following drawbacks in semantic segmentation tasks: First, model performance drops significantly when faced with new domains whose distributions differ from the training data. This is because they rely too heavily on specific features of the training domain, making it difficult to generalize to new domains (i.e., domain bias and domain drift). Second, traditional methods depend on large amounts of labeled data, which are difficult to obtain in specific application domains. Although semi-supervised learning methods alleviate this to some extent, they still face problems such as pseudo-label noise and low-confidence predictions. Furthermore, relying on training a single model can easily lead to model bias, especially when dealing with complex and variable input data. Finally, pseudo-labels in self-supervised learning introduce noise, affecting model training and leading to inaccurate predictions. Summary of the Invention
[0006] This invention proposes a domain-adaptive semantic segmentation method based on cross-pseudo-supervision, aiming to improve the adaptability and accuracy of the model in different environments. The method first uses the SYNTHIA dataset as the source domain for pre-training a deep convolutional neural network, and then uses the Cityscapes dataset as the target domain for further adaptation. The core innovation lies in implementing a dual-model cross-pseudo-supervision training strategy, which combines attention-modulation-based loss and entropy consistency loss to form a complementary loss mechanism. This mechanism not only optimizes the model's predictive ability in high-confidence regions but also ensures predictive consistency across different environments. By validating and saving the best-performing model parameters in the target domain, this invention significantly improves the model's generalization ability and robustness, making it more adaptable to semantic segmentation tasks in various domains or environments.
[0007] The technical solution for implementing this invention is: a domain-adaptive semantic segmentation method based on cross-pseudo-supervision, comprising the following steps:
[0008] Step 1: Select the SYNTHIA dataset to construct the source domain, select the Cityscapes dataset to construct the target domain, divide the images in the target domain into training set and validation set, and proceed to Step 2;
[0009] Step 2: Use real labels to perform dual-model cross-supervised training on the images in the source domain to obtain a pre-trained dual semantic segmentation model, and then proceed to Step 3.
[0010] Step 3: Load the pre-trained dual semantic segmentation model into the training set of the target domain, and use pseudo-labels to perform dual-model cross-pseudo-supervised training to construct a cross-pseudo-supervised dual semantic segmentation model, then proceed to Step 4.
[0011] Step 4: Extract target domain features using a cross-pseudo-supervised dual semantic segmentation model, and design an attention modulation mechanism to modulate the target domain features. At the same time, introduce attention modulation loss and entropy consistency loss to jointly construct a complementary loss mechanism to further optimize the cross-pseudo-supervised dual semantic segmentation model, and proceed to step 5.
[0012] Step 5: Use the optimized cross-supervised dual semantic segmentation model to verify the semantic segmentation performance on the validation set in the target domain, and save the parameters of the semantic segmentation model with the best performance in the model.
[0013] Compared with the prior art, the significant advantages of this invention are:
[0014] 1) In traditional single-model training, the single model is often sensitive to incorrect predictions, especially in domain adaptation scenarios, and is prone to overfitting to the characteristics of a specific dataset during training. The dual-model training strategy mitigates this problem by leveraging the complementary advantages of two models initialized differently. As the two models mutually correct each other during training, they help identify and avoid their respective overfitting tendencies. Compared to existing single-model methods, this invention significantly improves training efficiency and the accuracy of the final model. This dual-model strategy is particularly suitable for semantic segmentation tasks that exhibit significant differences across different domains.
[0015] 2) The dual-model complementary attention modulation mechanism of this invention, compared to traditional semantic segmentation methods, has the main advantage of allowing the branches to learn from and adjust each other in its cross-pseudo-supervised semantic segmentation model's dual-branch structure. This structure enables each branch to not only focus on regions it identifies as important but also to learn from the key regions of the other branch, thereby achieving a more comprehensive and in-depth understanding of features. This complementary and cross-attention strategy significantly improves the model's ability to capture details and the accuracy of classification, while optimizing overall computational efficiency.
[0016] 3) Unlike traditional methods that rely on a single loss function, this invention innovatively introduces attention-modulation-based loss and entropy consistency loss, which complement each other. Attention-modulation-based loss focuses on the "deterministic" difference between the two models, aiming to leverage the high confidence prediction of one model to guide the other model to make more accurate predictions at the same location. Entropy consistency loss, on the other hand, aims to ensure that the predictions of the two models are "deterministically" consistent on the same input. This complementary loss design not only improves the prediction accuracy of the model in high-confidence regions but also ensures the consistency and stability of predictions across the entire image. Attached Figure Description
[0017] Figure 1 This is a flowchart of the domain-adaptive semantic segmentation method based on cross-pseudo-supervision of the present invention.
[0018] Figure 2 This is an overall block diagram of the domain-adaptive semantic segmentation method based on cross-pseudo-supervision of the present invention. Detailed Implementation
[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0020] The technical solutions of the various embodiments of the present invention can be combined with each other, but only if they can be implemented by those skilled in the art. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.
[0021] The following section will further introduce the specific implementation method, as well as the technical difficulties and inventive points of this invention, using examples from this design.
[0022] Combination Figure 1 and Figure 2 A domain-adaptive semantic segmentation method based on cross-pseudo-supervision includes the following steps:
[0023] Step 1: Select the SYNTHIA dataset to construct the source domain, select the Cityscapes dataset to construct the target domain, and divide the images in the target domain into training and validation sets;
[0024] Step 2: Use the real labels to perform dual-model cross-self-supervised training on the images in the source domain to obtain a pre-trained dual semantic segmentation model, as follows:
[0025] Step 2.1: SYNTHIA, as the source domain data, is a dataset with real labels. Therefore, a self-supervised training strategy is constructed during the source domain training process. First, the images in the source domain are used as the input of the model. The feature vectors of the source domain images are extracted by the feature extractor and then input into the classifier to obtain the prediction probability matrix of each pixel in the source domain image belonging to 19 classes. The prediction probability matrix and the real ground labels are used to perform cross-entropy loss to optimize the segmentation performance of the semantic segmentation model. Finally, the pre-trained dual semantic segmentation model is obtained, as shown in the following equation (1):
[0026] (1),
[0027] in, Represents the source domain image Cross-entropy loss, Indicates the height of the source domain image, Indicates the width of the source domain image. C represents the total number of pixels in the source image, and C represents the total number of categories. Representing the true ground label The hot encoding of the i-th pixel, Represents the source domain image prediction probability matrix The predicted probability that the i-th pixel belongs to class c. .
[0028] Step 2.2: Train two semantic segmentation models with different initializations and pre-training. During training, two branches are run simultaneously, referred to as the first branch and the second branch. In the second branch, a dropout is added to randomly discard some neurons, aiming to improve the branch's ability to handle perturbations or generalize to unseen data. This approach is common in ensemble learning and deep learning and helps improve the model's performance in practical applications. Next, the source domain image is input into the first and second branches to extract features from the source domain image and generate the corresponding prediction probability matrix. The original ground truth labels of the source domain image are used to supervise these two branches, as shown in equations (2) and (3):
[0029] (2),
[0030] (3),
[0031] in, This represents the cross-entropy loss of the source domain image in the first branch. This represents the cross-entropy loss of the source domain image in the second branch. Represents the source domain image prediction probability matrix in the first branch. The predicted probability that the i-th pixel belongs to class c. Represents the source domain image prediction probability matrix in the second branch. The prediction probability matrix for the i-th pixel belonging to category c.
[0032] The total loss function of the source domain during training is shown in equation (4):
[0033] (4),
[0034] in, This represents the total loss function of the source domain image. The hyperparameters representing the source domain loss.
[0035] Step 2.3: Through the above dual-model cross-self-supervised training, a pre-trained dual semantic segmentation model is obtained.
[0036] Step 3: Load the pre-trained dual semantic segmentation model into the training set of the target domain, and use pseudo-labels to perform cross-model pseudo-supervised training to construct a cross-pseudo-supervised dual semantic segmentation model, as detailed below:
[0037] Step 3.1: Load the pre-trained dual semantic segmentation model into the training set of the target domain;
[0038] Step 3.2: Because the cross-supervised model allows each branch to rely not only on its own predictions during training but also on the predictions of the other branch, this multi-faceted learning approach helps improve the model's generalization ability to unseen data. Therefore, this invention applies the cross-supervised model to the target domain of domain-adaptive semantic segmentation, combined with... Figure 2 A dual-model cross-pseudo-supervision training strategy is constructed using pseudo-labels in the target domain. A dual semantic segmentation model is used to extract features from the target domain and generate corresponding prediction probability matrices and pseudo-labels, namely, the prediction probability matrix of the first branch and the prediction probability matrix of the second branch of the target domain image, and the pseudo-labels of the first and second branches of the target domain image. In the target domain, the pseudo-labels of the second branch supervise the training process of the first branch and optimize its parameters, while the pseudo-labels of the first branch supervise the training process of the second branch and optimize its parameters, thus forming a dual-model cross-pseudo-supervision training strategy. The loss functions of the two branches during training are shown in equations (5) and (6), respectively.
[0039] (5),
[0040] (6),
[0041] in, This represents the cross-entropy loss of the first branch of the target domain image. This represents the cross-entropy loss of the second branch of the target domain image. Pseudo-labels representing the first branch of the target domain image The hot encoding of the i-th pixel, Pseudo-labels representing the second branch of the target domain image The hot encoding of the i-th pixel, This represents the prediction probability matrix of the first branch of the target domain image. The predicted probability that the i-th pixel belongs to class c. This represents the prediction probability matrix of the second branch of the target domain image. The predicted probability that the i-th pixel belongs to category c.
[0042] Step 4, Combining Figure 2 A cross-pseudo-supervised dual semantic segmentation model is used to extract target domain features. An attention modulation mechanism is designed to modulate the target domain features extracted by the cross-pseudo-supervised semantic segmentation model. Similarly, a loss based on attention modulation is constructed using cross-pseudo-supervision. Entropy consistency loss and attention modulation-based loss are introduced to jointly construct a complementary loss mechanism. The cross-pseudo-supervised dual semantic segmentation model is further optimized as follows:
[0043] Step 4.1: Considering that the two branches of the cross-supervised semantic segmentation model may learn different semantic information for the same target image, in order to enable the two branches to fully interact and learn together, this invention proposes an attention modulation mechanism, the specific operation process of which is as follows:
[0044] First, an attention map is generated from the predicted probability matrix obtained from the target domain image (i.e., the highest predicted probability of each pixel in its corresponding channel is selected as the attention weight for that pixel). This allows the cross-supervised semantic segmentation model to obtain attention maps for both the first and second branches. Therefore, the attention modulation mechanism enables each branch to focus on regions more important to the final prediction, thereby improving the prediction accuracy for these regions. This is particularly crucial for complex semantic segmentation tasks, as it helps the semantic segmentation model distinguish and correctly label detailed parts of the image.
[0045] Secondly, the attention map of the first branch is multiplied with the features extracted from the target domain image by the second branch, and the attention map of the second branch is multiplied with the features extracted from the target domain image by the first branch, respectively obtaining the modulated features of the target domain image by the two branches. By exchanging the attention maps of the two branches and applying them to each other's features, this mechanism achieves effective cross-utilization of information. This bidirectional information flow enhances the model's understanding and utilization of the target domain image features, thereby improving the model's performance.
[0046] Finally, the modulated features of the target domain image are convolved through a 1x1 convolutional layer and then fed into the classifier for further classification. By processing the modulated features through a 1x1 convolutional layer and then classifying them again, this invention further optimizes the classification performance. This step ensures that the model can fully utilize the modulated features and avoids the feature distribution of the modulated branch from excessively tending towards the feature distribution of another branch.
[0047] During the target domain training process, the modulation process of the extracted target domain image features by the first branch is shown in equations (7), (8), and (9):
[0048] (7),
[0049] (8),
[0050] (9),
[0051] in, This represents the maximum predicted probability matrix for each pixel in the target domain image in the corresponding channel on the second branch. This indicates that the maximum value is taken in the channel dimension after normalization. This represents the prediction probability matrix for the second branch of the target domain image. This represents the target domain image features extracted by the first branch. This indicates that the first branch is modulated via an attention modulation mechanism. The modulation characteristics obtained afterwards Indicates the convolution operation; This represents the classifier for the first branch. This represents the enhanced prediction probability matrix obtained by reclassifying the target domain image on the first branch after modulation through the attention mechanism.
[0052] During the target domain training process, the modulation process of the extracted target domain image features by the second branch is shown in equations (10), (11), and (12):
[0053] (10),
[0054] (11),
[0055] (12),
[0056] in, This represents the maximum predicted probability matrix for each pixel in the target domain image within its corresponding channel on the first branch. This represents the prediction probability matrix of the first branch of the target domain image. This represents the target domain image features extracted by the second branch. This indicates that the second branch is modulated via an attention modulation mechanism. The modulation characteristics obtained afterwards This represents the classifier for the second branch. Let represent the enhanced prediction probability matrix obtained by reclassifying the target domain image on the second branch after modulation through the attention mechanism.
[0057] Step 4.2: Two enhanced prediction probability matrices for the target domain image are obtained in the first and second branches respectively through the attention modulation mechanism. and In order to make full use of the feature information after the interaction of the two branches, during the training of the target domain image, the pseudo-labels of the first and second branches are still used to carry out the training strategy of dual-model cross-pseudo-supervision to optimize the parameters of the two branches of the cross-pseudo-supervision semantic segmentation model respectively. The loss functions of the two branches based on attention modulation during training are shown in Equations (13) and (14) respectively:
[0058] (13),
[0059] (14),
[0060] in, This represents the attention-modulated loss of the target domain image on the first branch. This represents the attention-modulated loss of the target domain image in the second branch. Pseudo-labels representing the second branch of the target domain image The hot encoding of the i-th pixel, Pseudo-labels representing the first branch of the target domain image The hot encoding of the i-th pixel.
[0061] Step 4.3, Attention Modulation-Based Loss, focuses on the difference in predictive "determinism" between the two models. This method allows a high-confidence prediction from one model to guide the other model to make a more accurate prediction at the same location. Given that both branches process information from the same image, to prevent them from learning incorrect information and generating excessive "determinism" bias, this invention also introduces entropy consistency loss. This supplementary mechanism ensures that during training, the two models not only identify the differences in "determinism" between each other but also maintain consistency in predictive "determinism" on the same input data, thereby improving the overall accuracy and reliability of the predictions.
[0062] The prediction probability matrices of the first and second branches of the target domain image and the enhanced prediction probability matrices of the first and second branches of the target domain image are respectively regularized for consistency. That is, the entropy consistency loss and the attention modulation-based loss are introduced to jointly construct a complementary loss mechanism. For example, the entropy consistency loss of the target domain is shown in equations (15) and (16):
[0063] (15),
[0064] (16),
[0065] in, This represents the L2 distance loss function. The distance loss represents the entropy between the prediction probability matrices of the first and second branches of the target domain image. The distance loss between the entropies of the enhanced prediction probability matrices of the first and second branches of the target domain image is represented by equations (17), (18), (19), and (20):
[0066] (17),
[0067] (18),
[0068] (19),
[0069] (20),
[0070] in, The entropy is obtained by calculating the prediction probability matrix of the first branch of the target domain image. The entropy is obtained by calculating the prediction probability matrix of the second branch of the target domain image. The entropy is obtained by using the enhanced prediction probability matrix of the first branch of the target domain image. The entropy is obtained by using the enhanced prediction probability matrix of the second branch of the target domain image.
[0071] Step 4.4: During the training of the cross-supervised semantic segmentation model, source and target domain images are input in the same batch. The source domain optimizes its model parameters through supervised training, while the target domain optimizes its model parameters through self-training. Therefore, the overall loss function of the cross-supervised semantic segmentation model is... As shown in formula (21):
[0072] (twenty one),
[0073] in, Let represent the total loss function of the target domain during the training process, as shown in equation (22):
[0074] (twenty two),
[0075] in, , , These represent the hyperparameters for the cross-entropy loss, the attention-based modulation loss, and the entropy consistency loss during the target domain training process, respectively.
[0076] Step 5: Use the optimized cross-supervised dual semantic segmentation model to perform semantic segmentation performance verification on the validation set in the target domain, and save the parameters of the semantic segmentation model with the best performance in the model.
[0077] Example 1
[0078] Combination Figure 1 , Figure 2 A domain-adaptive semantic segmentation method based on cross-pseudo-supervision includes the following steps:
[0079] Step 1: Select GTA5 as the source domain dataset, containing 24,966 images and labels, each with a pixel size of 1914×1052. Select the SYNTHIA synthesized urban landscape dataset as the target domain dataset, which mainly contains 9,400 images with a pixel size of 1280×760. Due to the different resolutions of the source and target domains, normalization is required. This invention unifies the pixel size to 896×512. The source and target domains share 16 common classes, resulting in 16 and 13 final classifications. Let the source domain be represented as... ,in, Represents the image of the source domain. express Corresponding ground reality labels express The height, that is , express The width, i.e. , express The resolution size also indicates The total number of pixels, In this context, 3 represents the three color channels of RGB; let the target domain be represented as... ,in, If the target image in the target domain does not have a corresponding semantic label, proceed to step 2.
[0080] Step 2: Use the real labels to perform dual-model cross-self-supervised training on the images in the source domain to obtain a pre-trained dual semantic segmentation model, as follows:
[0081] Step 2.1: This invention uses ResNet101-Deeplabv2 joint encoding as the semantic segmentation model. ResNet101 is the backbone network used for feature extraction. This network has 101 convolutions divided into 5 convolutional layers, with the convolutional part serving as the feature extractor. Finally, there is a fully connected layer as the classifier. In this invention, the fully connected layer is discarded, and only the first 5 convolutional layers are retained as the encoder for feature extraction. The fully connected layer is replaced by Deeplabv2 as the classifier. Deeplabv2 has a trackless spatial pyramid (ASPP) scheme, which applies parallel dilation convolutions at different rates to the input feature map and then fuses them together. Since objects of the same category may have different sizes in the image, ASPP helps to account for different object sizes. In this invention, Deeplabv2 is used as the classifier to obtain the predicted probability matrix of pixels. This network has four branches, each consisting of 3 fully connected layers with different dilation rates: [6, 12, 28, 24].
[0082] GTA5, as the source domain data, is a dataset with real labels. Therefore, a self-supervised training strategy is constructed during the source domain training process. The source domain images are used as input and loaded into the ResNet101-Deeplabv2 semantic segmentation model. The feature vectors of the source domain images are extracted by the ResNet101 encoder and then input into the Deeplabv2 classifier to obtain the prediction probability matrix of each pixel in the source domain image belonging to 19 classes. The cross-entropy loss of the prediction probability matrix and the real ground labels is used to optimize the segmentation performance of the semantic segmentation model. Finally, the pre-trained dual semantic segmentation model is obtained, as shown in Equation (1) below:
[0083] (1),
[0084] in, Represents the source domain image Cross-entropy loss, Indicates the height of the source domain image, Indicates the width of the source domain image. C represents the total number of pixels in the source image, and C represents the total number of categories. Representing the true ground label The hot encoding of the i-th pixel, Represents the source domain image prediction probability matrix The predicted probability that the i-th pixel belongs to class c. .
[0085] Step 2.2: Train two semantic segmentation models with different initializations and pre-training. During training, two branches are run simultaneously, referred to as the first branch and the second branch. In the second branch, a dropout is added to randomly discard a portion of neurons. The source domain image is input into the first and second branches to extract features from the source domain image and generate the corresponding prediction probability matrix. The original ground truth labels of the source domain image are used to supervise these two branches, as shown in equations (2) and (3).
[0086] (2),
[0087] (3),
[0088] in, This represents the cross-entropy loss of the source domain image in the first branch. This represents the cross-entropy loss of the source domain image in the second branch. Represents the source domain image prediction probability matrix in the first branch. The predicted probability that the i-th pixel belongs to class c. Represents the source domain image prediction probability matrix in the second branch. The prediction probability matrix for the i-th pixel belonging to category c.
[0089] The total loss function of the source domain during training is shown in equation (4):
[0090] (4),
[0091] in, This represents the total loss function of the source domain image. Hyperparameters representing the source domain loss;
[0092] Step 2.3: Through the above dual-model cross-self-supervised training, a pre-trained dual semantic segmentation model is obtained.
[0093] Step 3: Simultaneously construct a supervised training strategy in the source domain and a dual-model cross-pseudo-supervised training strategy in the target domain, as detailed below:
[0094] Pseudo-labels are used for training with dual-model cross-pseudo-supervision. Dual semantic segmentation models are used to extract features from the target domain and generate corresponding prediction probability matrices and pseudo-labels, namely, the prediction probability matrices of the first branch and the second branch of the target domain image, and the pseudo-labels of the first and second branches of the target domain image. In the target domain, the pseudo-labels of the second branch supervise the training process of the first branch and optimize its parameters, while the pseudo-labels of the first branch supervise the training process of the second branch and optimize its parameters, thus forming a dual-model cross-pseudo-supervision training strategy. The loss functions of the two branches during training are shown in equations (5) and (6), respectively.
[0095] (5),
[0096] (6),
[0097] in, This represents the cross-entropy loss of the first branch of the target domain image. This represents the cross-entropy loss of the second branch of the target domain image. Pseudo-labels representing the first branch of the target domain image The hot encoding of the i-th pixel, Pseudo-labels representing the second branch of the target domain image The hot encoding of the i-th pixel, This represents the prediction probability matrix of the first branch of the target domain image. The predicted probability that the i-th pixel belongs to class c. This represents the prediction probability matrix of the second branch of the target domain image. The predicted probability that the i-th pixel belongs to category c.
[0098] Step 4: Extract target domain features using a cross-pseudo-supervised dual semantic segmentation model, and design an attention modulation mechanism to modulate these features. Simultaneously, introduce attention modulation loss and entropy consistency loss to construct a complementary loss mechanism, further optimizing the cross-pseudo-supervised dual semantic segmentation model, as detailed below:
[0099] Step 4.1: The operation of the attention modulation mechanism proposed in this invention is as follows: First, an attention map is generated from the prediction probability matrix obtained from the target domain image (i.e., the maximum prediction probability of each pixel in the corresponding channel is selected as the attention weight of that pixel). In this way, the cross-supervised semantic segmentation model will obtain the attention map of the first branch and the attention map of the second branch. Second, the attention map of the first branch is multiplied with the features extracted from the target domain image by the second branch of the pre-trained semantic segmentation model, and the attention map of the second branch is multiplied with the features extracted from the target domain image by the first branch of the pre-trained semantic segmentation model, respectively, to obtain the modulated features of the target domain image by the two branches. Finally, the modulated features of the target domain image are convolved through a 1*1 convolutional layer and then fed into the classifier for classification again.
[0100] During the target domain training process, the modulation process of the extracted target domain image features by the first branch is shown in equations (7), (8), and (9):
[0101] (7),
[0102] (8),
[0103] (9),
[0104] in, This represents the maximum predicted probability matrix for each pixel in the target domain image in the corresponding channel on the second branch. This represents the target domain image features extracted by the first branch. This indicates that the first branch is modulated via an attention modulation mechanism. The modulation characteristics obtained afterwards This represents the classifier for the first branch. This represents the enhanced prediction probability matrix obtained by reclassifying the target domain image on the first branch after modulation through the attention mechanism.
[0105] During the target domain training process, the modulation process of the extracted target domain image features by the second branch is shown in equations (10), (11), and (12):
[0106] (10),
[0107] (11),
[0108] (12),
[0109] in, This represents the maximum predicted probability matrix for each pixel in the target domain image within its corresponding channel on the first branch. This represents the target domain image features extracted by the second branch. This indicates that the second branch is modulated via an attention modulation mechanism. The modulation characteristics obtained afterwards This represents the classifier for the second branch. The representation represents the enhanced prediction probability matrix obtained by reclassifying the target domain image in the second branch after modulation through the attention mechanism;
[0110] Step 4.2: Two enhanced prediction probability matrices for the target domain image are obtained in the first and second branches respectively through the attention modulation mechanism. and During the training of the target domain image, the training strategy of dual-model cross-supervision is still used, which involves using pseudo-labels generated from the target domain image on the first and second branches. The loss functions based on attention modulation for the two branches during training are shown in equations (13) and (14), respectively:
[0111] (13),
[0112] (14),
[0113] in, This represents the attention-modulated loss of the target domain image on the first branch. This represents the attention-modulated loss of the target domain image in the second branch;
[0114] Step 4.3: Perform consistency regularization on the prediction probability matrices of the first and second branches of the target domain image and the enhanced prediction probability matrices of the first and second branches of the target domain image, respectively. That is, introduce entropy consistency loss and attention modulation-based loss to jointly construct a complementary loss mechanism. For example, the entropy consistency loss of the target domain is shown in equations (15) and (16):
[0115] (15),
[0116] (16),
[0117] in, This represents the L2 distance loss function. The distance loss represents the entropy between the prediction probability matrices of the first and second branches of the target domain image. The distance loss between the entropies of the enhanced prediction probability matrices of the first and second branches of the target domain image is represented by equations (17), (18), (19), and (20):
[0118] (17),
[0119] (18),
[0120] (19),
[0121] (20),
[0122] in, The entropy is obtained by calculating the prediction probability matrix of the first branch of the target domain image. The entropy is obtained by calculating the prediction probability matrix of the second branch of the target domain image. The entropy is obtained by using the enhanced prediction probability matrix of the first branch of the target domain image. The entropy is obtained by using the enhanced prediction probability matrix of the second branch of the target domain image;
[0123] Step 4.4: In the process of training the cross-pseudo-supervised semantic segmentation model, the batch size is set to 2, that is, two source domain images and two target domain images are input simultaneously. The source domain optimizes the model parameters through a supervised strategy, and the target domain optimizes the model parameters through self-training. Therefore, the overall loss function of the cross-pseudo-supervised semantic segmentation model is... As shown in formula (21):
[0124] (twenty one),
[0125] in, Let represent the total loss function of the target domain during the training process, as shown in equation (22):
[0126] (twenty two),
[0127] in, , , represents the hyperparameters set for cross-entropy loss, attention-based modulation loss, and entropy consistency loss during the target domain training process, respectively, and is set to 0.1, 0.05, and 0.4.
[0128] Step 5: In this invention, the total number of iterations is set to 15,000. Every 500 iterations, 500 target images from the target domain validation set are input into the trained cross-pseudo-supervised semantic segmentation model to generate pseudo-labels. Since the target images in the target domain validation set have real ground labels, the average intersection-union ratio (IUR) of the generated pseudo-labels and the real ground labels of the images is used to verify the segmentation performance of the cross-pseudo-supervised semantic segmentation model. The IUR of the two branches is compared, and the parameters of the semantic segmentation model with the best performance in the cross-pseudo-supervised semantic segmentation model are saved.
[0129] This invention utilizes a network framework built on an Nvidia 3090Ti GPU using Python and PyTorch for experimental setup. In the first stage, a ResNet101 network initialized on ImageNet was used to extract features. This network consists of 101 convolutions divided into 5 convolutional layers, which act as feature extractors. A fully connected layer is then used as the classifier, with Deeplabv2 replacing the convolutional layer. The dilation rates are [6, 12, 18, 24]. Deeplabv2 is trained using the SGD optimizer alongside ResNet101, with a batch size of 2, an input image size cropped to 896×512, an initial learning rate of 1e-4, a momentum of 0.9, and a weight decay coefficient of 0.0005. In the second stage, the network parameters saved from the first stage are used to initialize ResNet101+Deeplabv2, with an initial learning rate of 2.5e-4 and other parameters remaining unchanged.
[0130] To better demonstrate the semantic segmentation performance of the proposed algorithm on real images, this section presents a comparison with other popular algorithms. The domain-adaptive semantic segmentation method introduced in this invention starts from the output level, with the core objective of directly optimizing the accuracy of the pseudo-labels output by the model, thereby improving segmentation performance on the target domain dataset from the output level. Therefore, we compared it with some standard output-level-based domain-adaptive semantic segmentation methods, such as AdvEnt, SISC, DRSL, and ARAS, as shown in Table 1.
[0131] Table 1 SYNTHIA Cityscapes Domain-Adaptive Semantic Segmentation Results
[0132] Performance comparison of domain-adaptive semantic segmentation tasks on the SYNTHIA to Cityscapes dataset. In the first training phase, although the proposed domain-adaptive semantic segmentation method based on cross-pseudo-supervision achieved the best performance across 7 categories, it still achieved the highest mean Intersection over Union (mIoU) score. Specifically, when evaluating performance across 16 categories, the proposed algorithm achieved an mIoU of 49.4%; while in a simplified 13-category task, the performance improved to 53.4%. In the second training phase, the proposed algorithm achieved the highest mIoU score across 12 categories, with significant improvements compared to other methods in certain categories, such as walls, traffic lights, and motorcycles, increasing by 10.4%, 8.0%, and 20.8% respectively; and in the simplified 13-category task, the performance was significantly improved to a top score of 60.6%. This demonstrates the effectiveness of our cross-pseudo-supervision method.
[0133] In summary, compared to existing technologies, the method of this invention achieves significant progress in improving the accuracy of cross-domain semantic segmentation, especially under conditions of scarce labeled data. Through these experiments, we have verified the potential of the cross-pseudo-supervision method as an effective domain adaptation tool.
Claims
1. A domain-adaptive semantic segmentation method based on cross-pseudo-supervision, characterized in that, Includes the following steps: Step 1: Select the SYNTHIA dataset to construct the source domain, select the Cityscapes dataset to construct the target domain, divide the images in the target domain into training set and validation set, and proceed to Step 2; Step 2: Use real labels to perform dual-model cross-self-supervised training on the images in the source domain to obtain a pre-trained dual semantic segmentation model, as follows: Step 2.1: SYNTHIA, as the source domain data, is itself a dataset with real labels. Therefore, a self-supervised training strategy is constructed during the source domain training process, as shown in Equation (1): (1)， in, Represents the cross-entropy loss of the source domain image. Indicates the height of the source domain image, C represents the width of the source domain image, and C represents the total number of categories. This represents the hot code of the i-th pixel of the actual ground label. This represents the predicted probability that the i-th pixel in the source domain image prediction probability matrix belongs to category c; Step 2.2: Train two semantic segmentation models with different initializations. During training, the two branches are run simultaneously, referred to as the first branch and the second branch, respectively. A dropout is added to the second branch to randomly discard a portion of neurons. Input the source domain image into the first branch and the second branch to extract features from the source domain image and generate the corresponding prediction probability matrix. Use the original true labels of the images in the source domain to supervise the two branches, as shown in Equations (2) and (3): (2)， (3)， in, This represents the cross-entropy loss of the source domain image in the first branch. This represents the cross-entropy loss of the source domain image in the second branch. Represents the source domain image prediction probability matrix in the first branch. The predicted probability that the i-th pixel belongs to class c. , Represents the source domain image prediction probability matrix in the second branch. The prediction probability matrix of the i-th pixel belonging to class c ; The total loss function of the source domain during training is shown in equation (4): (4)， in, This represents the total loss function of the source domain image. Hyperparameters representing the source domain loss; Step 2.3: Through the above dual-model cross-self-supervised training, a pre-trained dual semantic segmentation model is obtained; Proceed to step 3; Step 3: Load the pre-trained dual semantic segmentation model into the training set of the target domain, and use pseudo-labels to perform dual-model cross-pseudo-supervised training to construct a cross-pseudo-supervised dual semantic segmentation model, and proceed to step 4. Step 4: Extract target domain features using a cross-pseudo-supervised dual semantic segmentation model, and design an attention modulation mechanism to modulate the above target domain features. At the same time, introduce attention modulation loss and entropy consistency loss to jointly construct a complementary loss mechanism to further optimize the cross-pseudo-supervised dual semantic segmentation model, and proceed to step 5. Step 5: Use the optimized cross-supervised dual semantic segmentation model to verify the semantic segmentation performance on the validation set in the target domain, and save the parameters of the semantic segmentation model with the best performance in the model.
2. The domain-adaptive semantic segmentation method based on cross-pseudo-supervision as described in claim 1, characterized in that, In step 3, a pre-trained dual semantic segmentation model is loaded into the training set of the target domain, and pseudo-labels are used for cross-model pseudo-supervision training to construct a cross-pseudo-supervised dual semantic segmentation model, as detailed below: Step 3.1: Load the pre-trained dual semantic segmentation model into the training set of the target domain; Step 3.2: Use pseudo-labels for training with dual-model cross-pseudo-supervision. Use the pre-trained dual semantic segmentation model to extract features from the target domain and generate corresponding prediction probability matrices and corresponding pseudo-labels, namely the prediction probability matrix of the first branch of the target domain image and the prediction probability matrix of the second branch of the target domain image, the pseudo-label of the first branch of the target domain image and the pseudo-label of the second branch of the target domain image; in the target domain, use the pseudo-label of the second branch to supervise the training process of the first branch and optimize the parameters of the first branch, and use the pseudo-label of the first branch to supervise the training process of the second branch and optimize the parameters of the second branch, thus forming a dual-model cross-pseudo-supervision training strategy. The loss functions of the two branches during training are shown in equations (5) and (6), respectively: (5)， (6)， in, This represents the cross-entropy loss of the first branch of the target domain image. This represents the cross-entropy loss of the second branch of the target domain image. The pseudo-label representing the first branch of the target domain image The hot encoding of the i-th pixel, Pseudo-labels representing the second branch of the target domain image The hot encoding of the i-th pixel, This represents the prediction probability matrix of the first branch of the target domain image. The predicted probability that the i-th pixel belongs to class c. This represents the prediction probability matrix of the second branch of the target domain image. The predicted probability that the i-th pixel belongs to category c.
3. The domain-adaptive semantic segmentation method based on cross-pseudo-supervision as described in claim 2, characterized in that, In step 4, a cross-pseudo-supervised dual semantic segmentation model is used to extract target domain features, and an attention modulation mechanism is designed to modulate these features. Simultaneously, attention modulation loss and entropy consistency loss are introduced to construct a complementary loss mechanism, further optimizing the cross-pseudo-supervised dual semantic segmentation model, as detailed below: Step 4.1: The operation of the attention modulation mechanism is as follows: First, the predicted probability matrix obtained from the target domain image is used to generate an attention map. That is, the maximum predicted probability of each pixel in the corresponding channel is selected as the attention weight of that pixel. In this way, the cross-supervised semantic segmentation model will obtain the attention map of the first branch and the attention map of the second branch. Secondly, the attention map of the first branch is multiplied with the features extracted from the target domain image by the second branch of the pre-trained semantic segmentation model, and the attention map of the second branch is multiplied with the features extracted from the target domain image by the first branch of the pre-trained semantic segmentation model, so as to obtain the modulated features of the target domain image by the two branches respectively. Finally, the modulated features of the target domain image are convolved by a 1*1 convolutional layer and then fed into the classifier for classification again. Step 4.2: Two enhanced prediction probability matrices for the target domain image are obtained in the first and second branches respectively through the attention modulation mechanism. and During the training of the target domain image, the training strategy of dual-model cross-supervision is still used, which involves using pseudo-labels generated from the target domain image on the first and second branches. The loss functions based on attention modulation for the two branches during training are shown in equations (13) and (14), respectively: (13)， (14)， in, This represents the attention-modulated loss of the target domain image on the first branch. This represents the attention-modulated loss of the target domain image in the second branch. Pseudo-labels representing the second branch of the target domain image The hot encoding of the i-th pixel, The pseudo-label representing the first branch of the target domain image The hot encoding of the i-th pixel; Step 4.3: Perform consistency regularization on the prediction probability matrices of the first and second branches of the target domain image and the enhanced prediction probability matrices of the first and second branches of the target domain image, respectively. That is, introduce entropy consistency loss and attention modulation-based loss to jointly construct a complementary loss mechanism. For example, the entropy consistency loss of the target domain is shown in equations (15) and (16): (15)， (16)， in, This represents the L2 distance loss function. The distance loss represents the entropy between the prediction probability matrices of the first and second branches of the target domain image. The distance loss between the entropies of the enhanced prediction probability matrices of the first and second branches of the target domain image is represented by equations (17), (18), (19), and (20): (17)， (18)， (19)， (20)， in, The entropy is obtained by calculating the prediction probability matrix of the first branch of the target domain image. The entropy is obtained by calculating the prediction probability matrix of the second branch of the target domain image. The entropy is obtained by using the enhanced prediction probability matrix of the first branch of the target domain image. The entropy is obtained by using the enhanced prediction probability matrix of the second branch of the target domain image; Step 4.4: During the training of the cross-supervised semantic segmentation model, source and target domain images are input in the same batch. The source domain optimizes the model parameters through self-supervision, while the target domain optimizes the model parameters through pseudo-supervision. Therefore, the overall loss function of the cross-supervised semantic segmentation model is... As shown in formula (21): (21)， in, Let represent the total loss function of the target domain during the training process, as shown in equation (22): (22)， in, , , These represent the hyperparameters for the cross-entropy loss, the attention-based modulation loss, and the entropy consistency loss during the target domain training process, respectively.
4. The domain-adaptive semantic segmentation method based on cross-pseudo-supervision according to claim 3, characterized in that, In step 4.1, the attention modulation mechanism operates as follows: During the target domain training process, the modulation process of the extracted target domain image features by the first branch is shown in Equations (7), (8), and (9): (7)， (8)， (9)， in, This represents the maximum predicted probability matrix for each pixel in the target domain image in the corresponding channel on the second branch. This indicates that the maximum value is taken in the channel dimension after normalization. This represents the prediction probability matrix for the second branch of the target domain image. This represents the target domain image features extracted by the first branch. This indicates that the first branch is modulated via an attention modulation mechanism. The modulation characteristics obtained afterwards Indicates the convolution operation; This represents the classifier for the first branch. This represents the enhanced prediction probability matrix obtained by reclassifying the target domain image in the first branch after modulation through the attention mechanism; During the target domain training process, the modulation process of the extracted target domain image features by the second branch is shown in Equations (10), (11), and (12): (10)， (11)， (12)， in, This represents the maximum predicted probability matrix for each pixel in the target domain image within its corresponding channel on the first branch. This represents the prediction probability matrix of the first branch of the target domain image. This represents the target domain image features extracted by the second branch. This indicates that the second branch is modulated via an attention modulation mechanism. The modulation characteristics obtained afterwards This represents the classifier for the second branch. Let represent the enhanced prediction probability matrix obtained by reclassifying the target domain image on the second branch after modulation through the attention mechanism.