Adaptive semantic segmentation method and device under harsh conditions
By correcting the pseudo-labels and performing hybrid training, the problems of pseudo-label noise and domain bias in semantic segmentation models under adverse conditions are solved, improving the performance of the model under adverse conditions and achieving better semantic segmentation results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- AEROSPACE SCI & IND SHENZHEN GROUP
- Filing Date
- 2023-06-01
- Publication Date
- 2026-05-19
AI Technical Summary
In existing technologies, deep learning-based semantic segmentation models perform well in clear scenes, but suffer from pseudo-label noise and domain bias under adverse conditions (such as rain, fog, snow, and darkness), leading to performance degradation.
By acquiring source and target domain datasets, pseudo-labels are generated and corrected using a teacher model. A student model is trained by combining dynamic thresholds and uncertainty weights. Visual enhancement is performed using an atmospheric scattering model, and the cross-entropy loss function is optimized to achieve high-quality correction of pseudo-labels and multiple iterations of model training.
It improves the robustness and robustness of the model under harsh conditions, significantly enhances semantic segmentation performance, and reduces pseudo-label noise and domain bias.
Smart Images
Figure CN116645512B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of semantic segmentation, and in particular relates to an adaptive semantic segmentation method and apparatus under adverse conditions. Background Technology
[0002] Semantic segmentation plays a crucial role in autonomous driving and intelligent monitoring. While deep learning-based semantic segmentation models have achieved good results in clear scenes, significant challenges remain in semantic segmentation under adverse conditions (such as rain, fog, snow, and darkness). This is because image collection is difficult in these conditions, annotation is labor-intensive, and the lack of high-quality labeled data leads to performance degradation. Existing methods, primarily self-training-based domain adaptation, generalize semantic segmentation models from clear scenes to adverse conditions. However, during self-training, pseudo-label noise caused by domain bias between the source domain dataset (a labeled dataset from clear scenes) and the target domain dataset (an unlabeled dataset from adverse conditions) negatively impacts model performance. Summary of the Invention
[0003] The technical problem to be solved by this invention is how to address the problem of pseudo-label noise under adverse conditions, and proposes an adaptive semantic segmentation method and apparatus under adverse conditions.
[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:
[0005] An adaptive semantic segmentation method under adverse conditions includes the following steps:
[0006] Step 1: Obtain the source domain dataset and the target domain dataset;
[0007] Step 2: Preprocess the image data in the source domain dataset and the target domain dataset;
[0008] Step 3: Use the teacher model to predict the preprocessed target domain data and generate pseudo-labels based on the prediction results;
[0009] Step 4: Correct the pseudo-labels;
[0010] Step 5: Mix the source domain data and the target domain data with corrected pseudo-labels according to a pre-set ratio, and input them into the teacher model and student model respectively for mixed training. After multiple iterations of training, the trained model is obtained.
[0011] Step 6: Evaluate the model. If the evaluation is successful, use the trained student model to perform semantic segmentation on the input image.
[0012] Furthermore, the method for generating pseudo-labels based on the prediction results in step 3 is as follows:
[0013] The teacher model is used to predict the target domain data, and the probability distribution of each pixel belonging to each category is obtained.
[0014] The category with the highest probability for each pixel is used as the pseudo-label for that pixel.
[0015] Furthermore, the method for correcting the pseudo-label is as follows:
[0016] The target domain data and corresponding pseudo-labels are input into the student model, and the new pseudo-labels predicted by the student model are used as the corrected pseudo-labels for the target domain data; the cross-entropy loss function of the student model includes:
[0017] The global weight parameter sets the global weight of the source domain to 1 and uses the proportion of the predicted value of the target domain that is greater than the threshold as the global weight of the target domain.
[0018] Local weight parameters are used to take the difference between each pixel in the target domain and the master-slave classifiers as local weights. The master-slave classifiers refer to:
[0019] The threshold parameter is initially set to 1 / C, where C is the number of categories, and is then dynamically adjusted based on the predicted value of each pixel's category.
[0020] Furthermore, the method for performing hybrid training in step 5 is as follows:
[0021] Step 5.1: Initialize the parameters of the student model and teacher model;
[0022] Step 5.2: Randomly sample a batch of images and labels from the source domain dataset, and randomly sample a batch of images from the target domain dataset;
[0023] Step 5.3: Train the cross-entropy loss function of the student model using a blended image and blended labels. The blended image refers to the object pixels corresponding to half of the class labels taken from an image in the source domain dataset, and the remaining pixels are filled with other pixels in the target domain. The blended labels are formed by mixing the labels of each class taken from the source domain image in the blended image with the pseudo labels of the pixels taken from the target domain image.
[0024] Step 5.4: Train the cross-entropy loss function of the student model using the enhanced blended image and the enhanced blended label. The enhanced blended image refers to the object pixels corresponding to half of the class labels in the source domain image, and the remaining pixels are filled with pixels in the image after visual enhancement of the target domain image. The enhanced blended label is formed by mixing the labels of each class taken from the source domain image in the enhanced blended image with the pseudo labels of the pixels taken from the enhanced target domain image.
[0025] Step 5.5: Update the student model and sum the weights of the student model at the current moment and the previous teacher model weights to obtain the new teacher model;
[0026] Step 5.6: Stop training after the preset number of training iterations and output the trained student model.
[0027] Furthermore, the method for visual enhancement in step 5.4 is as follows:
[0028] Image enhancement is performed using an atmospheric scattering model.
[0029] The following formula is used for rain, fog, and snow:
[0030] I(x)=J(x)t(x)+A(1-t(x))
[0031] Use the following formula for nighttime:
[0032] 1-I(x)=(1-J(x))t(x)+A(1-t(x)),
[0033] Where I(x) represents the original input image, J(x) represents the enhanced sharp image, A is atmospheric light, and t(x) is the transmission map of the original image.
[0034] Furthermore, the method for updating the teacher model in step 5.5 is as follows:
[0035] ψ t+1 ←αψ t +(1-α)θ t
[0036] θ t Let ψ represent the student model after the t-th training iteration. t ψ t+1 Let represent the teacher model after the t-th and t+1-th training iterations, respectively, and α be the momentum coefficient.
[0037] Furthermore, the method for dynamically adjusting the threshold parameter is as follows:
[0038] The threshold for each category is calculated by weighting and summing the threshold for that category at the previous time step with the average predicted value of each pixel for that category at the current time step.
[0039] The present invention also provides an adaptive semantic segmentation device under adverse conditions, comprising the following modules:
[0040] Input module: Used to obtain the source domain dataset and the target domain dataset;
[0041] Preprocessing module: Used to preprocess the image data in the source domain dataset and the target domain dataset, and to extract features from the preprocessed images to obtain the feature map of each training sample;
[0042] Pseudo-label generation module: Used to predict target domain data using the teacher model and generate pseudo-labels based on the prediction results;
[0043] Pseudo-label correction module: used to correct the pseudo-labels;
[0044] Hybrid training module: This module is used to mix source domain data and target domain data with corrected pseudo-labels according to a pre-set ratio, and then input them into the teacher model and student model respectively for hybrid training. After multiple iterations of training, a trained model is obtained.
[0045] Segmentation module: Used to evaluate the model. Once the evaluation is passed, the trained model is used to perform semantic segmentation on the input image.
[0046] By adopting the above technical solution, the present invention has the following beneficial effects:
[0047] This invention provides an adaptive semantic segmentation method and apparatus under harsh conditions. It corrects pseudo-labels and then mixes source domain data and target domain data with corrected pseudo-labels in a certain proportion. Training the model using this mixed source and target domain data improves its robustness and effectiveness under harsh conditions. The model trained using this method can improve semantic segmentation performance under adverse conditions. Attached Figure Description
[0048] Figure 1 This is a system flowchart of the present invention;
[0049] Figure 2 A schematic diagram of the model framework is provided. Detailed Implementation
[0050] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0051] Figure 1 A specific embodiment of the adaptive semantic segmentation method under adverse conditions of the present invention is given, including the following steps:
[0052] Step 1: Obtain the source domain dataset and the target domain dataset.
[0053] Step 2: Preprocess the image data in the source domain dataset and the target domain dataset.
[0054] In this embodiment, after obtaining the dataset, it is also necessary to preprocess the data in the source domain dataset and the target domain dataset, i.e., each image, including operations such as image resizing and data augmentation. The purpose of image resizing is to make the images have the same specifications, while data augmentation is to enhance the quality of the images under harsh conditions.
[0055] Step 3: Use the teacher model to predict the preprocessed target domain data and generate pseudo-labels based on the prediction results. Since the teacher model is a relatively mature model and does not have gradient backpropagation capabilities, using the teacher model to make preliminary predictions on the target domain data can improve the accuracy of the predictions.
[0056] In this embodiment, as Figure 2 As shown, the method for generating pseudo-labels based on the prediction results is:
[0057] The teacher model is used to predict the target domain data, and the probability distribution of each pixel belonging to each category is obtained.
[0058] The category with the highest probability for each pixel is used as the pseudo-label for that pixel.
[0059] Step 4: Correct the pseudo-label.
[0060] In this embodiment, the method for correcting the pseudo-label is as follows:
[0061] The target domain data and corresponding pseudo-labels are input into the student model, and the new pseudo-labels predicted by the student model are used as the corrected pseudo-labels for the target domain data; the cross-entropy loss function of the student model includes:
[0062] The global weight parameter sets the global weight of the source domain to 1 and uses the proportion of the predicted value of the target domain that is greater than the threshold as the global weight of the target domain.
[0063] The local weight parameter uses the difference between each pixel in the target domain and the main and auxiliary classifiers as the local weight. The local weight is determined using the KL divergence of the main classifier F_main and the auxiliary classifier F_auxiliary.
[0064] The threshold parameter is initially set to 1 / C, where C is the number of categories. The initial value is very small, and it is then dynamically adjusted based on the predicted value for each pixel's category. In this embodiment, the threshold is dynamically adjusted by weighting the threshold for each category by the average of the predicted values for each pixel's category at the current time step. As the number of training iterations increases, the predicted value for each category continuously adjusts and increases, thus increasing the threshold accordingly. This dynamic change in the threshold allows for greater consideration of pseudo-label diversity in the early stages of training and greater consideration of pseudo-label reliability in the later stages, further improving the quality of the pseudo-labels.
[0065] By using a student model to correct false labels, the key is how the parameters in the student model's cross-entropy loss function are valued. In this embodiment, the global and local weight parameters are uncertain and change continuously with the predicted value of the target domain. Moreover, the threshold parameter is also continuously adjusted according to the predicted value of the category of each pixel. This method of correcting false labels can reduce false label noise and neighborhood bias.
[0066] Step 5: Mix the source domain data and the target domain data with corrected pseudo-labels according to a pre-set ratio, and input them into the teacher model and student model respectively for mixed training. After multiple iterations of training, the trained model is obtained.
[0067] like Figure 2 The model framework shown in this embodiment employs a hybrid training method as follows:
[0068] Step 5.1: Initialize the parameters of the student model and teacher model;
[0069] Step 5.2: Randomly sample a batch of images and labels from the source domain dataset, and randomly sample a batch of images from the target domain dataset;
[0070] Step 5.3: Train the student model's cross-entropy loss function using a mixed image and mixed labels. The mixed image refers to the object pixels corresponding to half the number of class labels taken from an image in the source domain dataset, with the remaining pixels filled with other pixels from the target domain. The mixed labels are formed by mixing the class labels taken from the source domain image with the pseudo labels of the pixels taken from the target domain image. The source domain data and target domain data are mixed in a certain proportion, and the pseudo labels are added to the training data to further improve the model's performance.
[0071] Step 5.4: Train the student model's cross-entropy loss function using the enhanced blended image and enhanced blended labels. The enhanced blended image refers to taking half of the object pixels corresponding to the class labels from the source domain image, and filling the remaining pixels with pixels from the visually enhanced target domain image. The enhanced blended labels are formed by mixing the class labels taken from the source domain image with the pseudo-labels of the pixels taken from the enhanced target domain image. In this embodiment, the difference between the enhanced blended labels and the blended labels in Step 5.3 lies in the use of different pseudo-labels from the target domain. Step 5.3 uses pseudo-labels predicted by the teacher model for the original target domain, while Step 5.4 uses pseudo-labels predicted by the teacher model for the visually enhanced target data, thus improving the quality of the pseudo-labels.
[0072] In this embodiment, the method for visual enhancement is:
[0073] Image enhancement is performed using an atmospheric scattering model, with the following formula applied to rain, fog, and snow:
[0074] I(x)=J(x)t(x)+A(1-t(x))
[0075] Use the following formula for nighttime:
[0076] 1-I(x)=(1-J(x))t(x)+A(1-t(x)),
[0077] Where I(x) represents the original input image, J(x) represents the enhanced sharp image, A is atmospheric light, and t(x) is the transmission map of the original image. The atmospheric scattering model comes from the atmospheric visibility formula proposed by Koschmieder in his 1997 paper "Theorie derhorizontalen Sichtweite" (Theorie derhorizontalen Sichtweite). This formula can be used to describe the propagation and scattering of light in the atmosphere, thereby calculating atmospheric transparency and achieving the effect of removing haze from images or videos.
[0078] Step 5.5: Update the student model and use the EMA momentum to update the teacher model by weighting and summing the student model weights at the current time and the previous teacher model weights.
[0079] The method for updating the teacher model in this embodiment is:
[0080] ψ t+1 ←αψ t +(1-α)θ t
[0081] θ t Let ψ represent the student model after the t-th training iteration. t ψ t+1Let represent the teacher model after the t-th and t+1-th training iterations, respectively, and α be the momentum coefficient, which is 0.99 in this embodiment.
[0082] Step 5.6: Stop training after the preset number of training iterations and output the trained student model.
[0083] Step 6: Evaluate the model. If the evaluation is successful, use the trained student model to perform semantic segmentation on the input image.
[0084] In this embodiment, cross-validation is used to evaluate the model, including calculating the pixel accuracy and average intersection-over-union ratio (mIoU) for each category.
[0085] This embodiment further reduces pseudo-label noise and domain bias by employing dynamic thresholds and uncertain weights, and improves the model's robustness and robustness under harsh conditions by training with mixed source and target domain data. Experimental results show that the technical solution of this invention can significantly improve the semantic segmentation performance of the model under harsh conditions, and has better robustness and robustness. This makes the use of semantic segmentation models more robust in autonomous driving and intelligent monitoring.
[0086] The present invention also provides an adaptive semantic segmentation device under adverse conditions, comprising the following modules:
[0087] Input module: Used to obtain the source domain dataset and the target domain dataset;
[0088] Preprocessing module: Used to preprocess the image data in the source domain dataset and the target domain dataset, and to extract features from the preprocessed images to obtain the feature map of each training sample;
[0089] Pseudo-label generation module: Used to predict target domain data using the teacher model and generate pseudo-labels based on the prediction results;
[0090] Pseudo-label correction module: used to correct the pseudo-labels;
[0091] Hybrid training module: This module is used to mix source domain data and target domain data with corrected pseudo-labels according to a pre-set ratio, and then input them into the teacher model and student model respectively for hybrid training. After multiple iterations of training, a trained model is obtained.
[0092] Segmentation module: Used to evaluate the model. Once the evaluation is passed, the trained model is used to perform semantic segmentation on the input image.
[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. An adaptive semantic segmentation method under adverse conditions, characterized in that, Includes the following steps: Step 1: Obtain the source domain dataset and the target domain dataset; Step 2: Preprocess the image data in the source domain dataset and the target domain dataset; Step 3: Use the teacher model to predict the preprocessed target domain data and generate pseudo-labels based on the prediction results; Step 4: Correct the pseudo-labels; the method for correcting the pseudo-labels is as follows: input the target domain data and the corresponding pseudo-labels into the student model, and use the new pseudo-labels predicted by the student model as the corrected pseudo-labels for the target domain data; in the cross-entropy loss function of the student model: The global weight parameter sets the global weight of the source domain to 1 and uses the proportion of the predicted value of the target domain that is greater than the threshold as the global weight of the target domain. Local weight parameters are used to take the difference between each pixel in the target domain and the master-slave classifiers as local weights. The master-slave classifiers refer to: The threshold parameter is initially set to 1 / C, where C is the number of categories, and is then dynamically adjusted based on the predicted value of each pixel's category. Step 5: Mix the source domain data and the target domain data with corrected pseudo-labels according to a pre-set ratio, and input them into the student model for mixed training. After multiple iterations of training, a trained student model is obtained. Step 6: Evaluate the model. If the evaluation is successful, use the trained student model to perform semantic segmentation on the input image.
2. The adaptive semantic segmentation method under adverse conditions according to claim 1, characterized in that, The method for generating pseudo-labels based on the prediction results in step 3 is as follows: The teacher model is used to predict the target domain data, and the probability distribution of each pixel belonging to each category is obtained. The category with the highest probability for each pixel is used as the pseudo-label for that pixel.
3. The adaptive semantic segmentation method under adverse conditions according to claim 2, characterized in that, The method for performing hybrid training in step 5 is as follows: Step 5.1: Initialize the parameters of the student model and teacher model; Step 5.2: Randomly sample a batch of images and labels from the source domain dataset, and randomly sample a batch of images from the target domain dataset; Step 5.3: Train the cross-entropy loss function of the student model using a blended image and blended labels. The blended image refers to the object pixels corresponding to half of the class labels taken from an image in the source domain dataset, and the remaining pixels are filled with other pixels in the target domain. The blended labels are formed by mixing the labels of each class taken from the source domain image in the blended image with the pseudo labels of the pixels taken from the target domain image. Step 5.4: Train the cross-entropy loss function of the student model using the enhanced blended image and the enhanced blended label. The enhanced blended image refers to the object pixels corresponding to half of the class labels in the source domain image, and the remaining pixels are filled with pixels in the image after visual enhancement of the target domain image. The enhanced blended label is formed by mixing the labels of each class taken from the source domain image in the enhanced blended image with the pseudo labels of the pixels taken from the enhanced target domain image. Step 5.5: Update the student model and sum the weights of the student model at the current moment and the previous teacher model weights to obtain the new teacher model; Step 5.6: Stop training after the preset number of training iterations and output the trained student model.
4. The adaptive semantic segmentation method under adverse conditions according to claim 3, characterized in that, The method for visual enhancement in step 5.4 is as follows: Image enhancement is performed using an atmospheric scattering model. The following formula is used for rain, fog, and snow: Use the following formula for nighttime: ， in, This represents the original input image. This represents the enhanced, sharpened image, where A represents atmospheric light. This is a transmission map of the original image.
5. The adaptive semantic segmentation method under adverse conditions according to claim 3, characterized in that, The method for updating the teacher model in step 5.5 is as follows: Indicates the first The student model after the training iteration , They represent the first sequence The teacher model after the second training iteration. This is the momentum coefficient.
6. The adaptive semantic segmentation method under adverse conditions according to claim 3, characterized in that, The method for dynamically adjusting the threshold parameter is: The threshold for each category is calculated by weighting and summing the threshold for that category at the previous time step with the average predicted value of each pixel for that category at the current time step.
7. An adaptive semantic segmentation device under adverse conditions, characterized in that, Includes the following modules: Input module: Used to obtain the source domain dataset and the target domain dataset; Preprocessing module: Used to preprocess the image data in the source domain dataset and the target domain dataset, and to extract features from the preprocessed images to obtain the feature map of each training sample; Pseudo-label generation module: Used to predict target domain data using the teacher model and generate pseudo-labels based on the prediction results; Pseudo-label correction module: used to correct the pseudo-labels; the method for correcting the pseudo-labels is as follows: input the target domain data and the corresponding pseudo-labels into the student model, and use the new pseudo-labels predicted by the student model as the corrected pseudo-labels for the target domain data; in the cross-entropy loss function of the student model: The global weight parameter sets the global weight of the source domain to 1 and uses the proportion of the predicted value of the target domain that is greater than the threshold as the global weight of the target domain. Local weight parameters are used to take the difference between each pixel in the target domain and the master-slave classifiers as local weights. The master-slave classifiers refer to: The threshold parameter is initially set to 1 / C, where C is the number of categories, and is then dynamically adjusted based on the predicted value of each pixel's category. Hybrid training module: This module is used to mix source domain data and target domain data with corrected pseudo-labels according to a pre-set ratio, and then input them into the teacher model and student model respectively for hybrid training. After multiple iterations of training, a trained model is obtained. Segmentation module: Used to evaluate the model. Once the evaluation is passed, the trained model is used to perform semantic segmentation on the input image.