Semantic segmentation domain adaptation label correction method and device based on candidate label set
By pre-training a model on a set of clear-sky images and generating a set of candidate labels, and then expanding the pseudo-labels by ranking the credibility of foggy images, the problem of insufficient accuracy of semantic segmentation models in foggy scenes is solved, and efficient semantic segmentation in foggy weather is achieved.
Patent Information
- Application Number
- CN202211573989.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-08
- Publication Date
- 2025-12-05
- Estimated Expiration
- 2042-12-08
AI Technical Summary
Existing semantic segmentation models lack accuracy in foggy scenarios, fail to effectively utilize the characteristics of foggy data, and have sparse and inaccurate pseudo-labels, resulting in model performance that cannot meet the requirements of autonomous driving.
By pre-training a preset model using a set of sunny day images, a candidate label set is generated. The candidate label set is then used to perform semantic segmentation on foggy day images. Credible pseudo-labels are expanded according to their credibility. The pseudo-labels are then corrected using the candidate label set, and the model is trained to obtain a semantic segmentation model.
It improves the accuracy of semantic segmentation in foggy scenes, solves the problems of sparse and inaccurate pseudo-labels, and achieves efficient semantic segmentation under foggy conditions.
Smart Images
Figure CN115830324B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of autonomous driving technology, and in particular to a semantic segmentation domain adaptation label correction method and apparatus based on candidate label sets. Background Technology
[0002] In recent years, with the deepening application of deep learning in the field of autonomous driving, semantic segmentation, as an important research direction, has developed rapidly. However, semantic segmentation research is usually conducted only in clear weather, while inclement weather such as fog is unavoidable in the actual application scenarios of autonomous driving. Therefore, although the accuracy of current semantic segmentation technology in clear weather is sufficient to meet the needs of autonomous driving tasks, the accuracy of models used in clear weather is far from meeting the requirements of autonomous driving tasks when used directly in foggy conditions. Thus, it is necessary to train corresponding models for foggy scenarios. However, since foggy data is difficult to label, a self-training-based foggy semantic segmentation domain adaptation strategy that can be trained for this situation has become an important issue in autonomous driving.
[0003] In recent years, many excellent general semantic segmentation adaptation methods have emerged, such as CBST, CRST, and ProDA. Although these methods have achieved significant results on commonly used datasets in this field, such as GTA5 and Cityscapes, they still have many serious problems. First, these methods have not been tested on foggy datasets, so their effectiveness in foggy scenarios cannot be guaranteed. Second, these methods have not deeply explored the characteristics of foggy data and have not used these characteristics to assist in the semantic segmentation adaptation task. Finally, they have not optimized for pseudo-labels, which directly affect the final performance of the model, and have not eliminated or mitigated the contradiction between pseudo-label sparsity and accuracy.
[0004] In recent years, several works have focused on improving the performance of deep learning semantic segmentation domain adaptation models in foggy scenes. For example, the CMAda method proposes a stepwise training approach based on curriculum learning, taking into account the characteristics of fog concentration. This method has two steps: it learns from synthetic fog images and unlabeled real fog images, gradually transferring the model from clear scenes to light fog and dense fog scenes. This idea was subsequently extended by further classifying the fog concentration to accommodate multiple adaptive steps, gradually transferring the model from clear scenes to dense fog scenes.
[0005] While these works have achieved good results in adapting to the semantic segmentation domain in foggy conditions, several issues remain. They fail to explore the characteristics of foggy scenes or fully utilize relevant information. To ensure the credibility of pseudo-labels, the number of selected pseudo-labels is very sparse, insufficient for model training. Furthermore, they do not leverage the characteristics of foggy scenes to correct pseudo-label errors and further improve model performance. Summary of the Invention
[0006] To address at least one of the aforementioned technical problems, the main objective of this invention is to provide a semantic segmentation domain adaptation label correction method and apparatus based on a candidate label set.
[0007] In a first aspect, the present invention provides a semantic segmentation domain adaptation label correction method based on a candidate label set, the semantic segmentation domain adaptation label correction method based on a candidate label set comprising:
[0008] Using a set of sunny day images and the labels corresponding to each sunny day image in the set, a pre-trained model is pre-trained using a supervised semantic segmentation training strategy to obtain a pre-trained model. The pre-trained model includes a feature dimensionality reduction projection network and a semantic segmentation network, wherein the semantic segmentation network is a derived network based on U-Net.
[0009] The foggy images in the foggy image set are semantically segmented by a pre-trained model. The semantic segmentation results are sorted in descending order of credibility. The semantic segmentation results with the highest credibility are used as credible pseudo-labels. The credible pseudo-labels are expanded to obtain a candidate label set.
[0010] The expanded pseudo-labels are corrected in real time using the candidate label set, and the pre-trained model is trained to obtain a semantic segmentation model.
[0011] The semantic segmentation task is performed based on the semantic segmentation model.
[0012] Optionally, the foggy image set is and in, This represents the i-th foggy image. This is a set of foggy images, where ch represents the number of color channels in the image, h and w represent the length and width of the foggy image, respectively, and N... t Let R represent the size of the foggy image set, and let R represent the set of real numbers. The semantic segmentation results are sorted in descending order of confidence. A predetermined proportion of the semantic segmentation results at the top of the sorting are designated as reliable pseudo-labels. The steps to expand these reliable pseudo-labels to obtain a candidate label set include:
[0013] Based on the reliability of the semantic segmentation results, the semantic segmentation results of foggy images are divided into three types of regions, and the division method of the three types of regions is as follows:
[0014]
[0015] Where 1, 2, and 3 represent the first type of candidate label region, the second type of candidate label region, and the third type of candidate label region, respectively; m(p) indicates which type of candidate label region pixel p belongs to; c indicates that the semantic category of the pixel is the c-th class; and Φ is the model parameter. sp represents the probability that pixel p is predicted to be of class c. i This represents the i-th superpixel block. λ represents the component with the highest confidence in the prediction result of pixel p. c For the preset ratio, stpandq∈sp represents the component with the highest confidence in the prediction result of pixel q. i This indicates that pixels p and q both belong to sp. i ;
[0016] The candidate label set is the label corresponding to the top-ranked category in the predicted confidence of each pixel in the first type of candidate label region, the candidate label set is the label corresponding to the top-ranked 3 categories in the predicted confidence of each pixel in the second type of candidate label region, and the candidate label set is the label corresponding to the top-ranked 5 categories in the predicted confidence of each pixel in the third type of candidate label region.
[0017] Optionally, the steps of correcting the expanded pseudo-labels in real time using the candidate label set and training the pre-trained model to obtain the semantic segmentation model include:
[0018] Step 3.1, will The input to the pre-trained model is processed by an encoder, and then by a feature reduction and projection network to obtain the features. And the semantic segmentation results obtained from the classification network. in, This represents the process by which the i-th foggy image is processed by an encoder and a classification network to obtain the semantic segmentation result.
[0019] Step 3.2: Calculate the prototype for each category according to the first formula. The prototype for each category is represented as Ψ={ψ1,…,ψ c ,…,ψ c}, where ψ c This represents the prototype of the c-th category, where the category refers to the predicted category corresponding to the semantic segmentation, and the prototype refers to the average feature of the category. The first formula is:
[0020]
[0021] in, This represents the feature of the p-th pixel in the i-th foggy image. Let p represent the probability of pixel p belonging to class c in the soft label. Indicates an indicator function, The result within the parentheses is 1 if it is greater than 0, otherwise it is 0.
[0022] Step 3.3, based on prototype Ψ and z i Features for soft labels y t To correct this, use the formula shown below:
[0023]
[0024]
[0025] in, This represents the set of candidate labels for pixel p. This represents the weight of category c for pixel p.
[0026] ||·|| represents Euclidean distance;
[0027] Step 3.4: Calculate the cross-entropy loss and contrastive learning loss based on the updated soft labels. The two losses are as follows:
[0028]
[0029]
[0030] Where v represents the feature of the p-th pixel in the i-th foggy image, and p does not belong to the third type of candidate label region; v + V represents a positive sample with feature v. - This represents a negative sample of v; represents the negative sample set, which is the set of all pixels q that do not belong to the candidate label region of type 3, and the set of pixels whose most confident category is not in the candidate label set of pixel p; exp(·) represents the exponentiation operation of the base e of the natural logarithm.
[0031] Step 3.5: For the sunny day image set, calculate the corresponding standard cross-entropy loss, add it to the cross-entropy loss in step 5.4 and the contrastive learning loss, and then backpropagate using the gradient descent algorithm to optimize the pre-trained model.
[0032] Step 3.6: Perform the operations from steps 3.1 to 3.5 above for each foggy image to obtain the semantic segmentation model.
[0033] Optionally, the semantic segmentation network is a deep learning RefineNet semantic segmentation network based on the U-Net architecture.
[0034] Optionally, the feature reduction projection network includes a 1x1 two-dimensional convolutional layer and a ReLU layer.
[0035] Secondly, the present invention also provides a semantic segmentation domain adaptation label correction device based on a candidate label set, the semantic segmentation domain adaptation label correction device based on a candidate label set comprising:
[0036] The pre-training module is used to pre-train a preset model using a set of sunny day images and the label corresponding to each sunny day image in the set, and adopts a supervised semantic segmentation training strategy to obtain a pre-trained model. The preset model includes a feature dimensionality reduction projection network and a semantic segmentation network. The semantic segmentation network is a derived network based on U-Net.
[0037] An expansion module is used to perform semantic segmentation on foggy images in a foggy image set using a pre-trained model, sort the semantic segmentation results in descending order of credibility, use the semantic segmentation results with the highest-ranked results as credible pseudo-labels, and expand the credible pseudo-labels to obtain a candidate label set.
[0038] The training module is used to correct the expanded pseudo-labels in real time using the candidate label set and train the pre-trained model to obtain the semantic segmentation model.
[0039] The execution module is used to perform semantic segmentation tasks based on the semantic segmentation model.
[0040] Optionally, the foggy image set is and in, This represents the i-th foggy image. This is a set of foggy images, where ch represents the number of color channels in the image, h and w represent the length and width of the foggy image, respectively, and N... t The size of the foggy image set is represented by R, which represents the set of real numbers. The extension module is used for:
[0041] Based on the reliability of the semantic segmentation results, the semantic segmentation results of foggy images are divided into three types of regions, and the division method of the three types of regions is as follows:
[0042]
[0043] Where 1, 2, and 3 represent the first type of candidate label region, the second type of candidate label region, and the third type of candidate label region, respectively; m(p) indicates which type of candidate label region pixel p belongs to; c indicates that the semantic category of the pixel is the c-th class; and Φ is the model parameter. sp represents the probability that pixel p is predicted to be of class c. i This represents the i-th superpixel block. λ represents the component with the highest confidence in the prediction result of pixel p. c For the preset ratio, stpandq∈sp represents the component with the highest confidence in the prediction result of pixel q. i This indicates that pixels p and q both belong to sp. i ;
[0044] The candidate label set is the label corresponding to the top-ranked category in the predicted confidence of each pixel in the first type of candidate label region, the candidate label set is the label corresponding to the top-ranked 3 categories in the predicted confidence of each pixel in the second type of candidate label region, and the candidate label set is the label corresponding to the top-ranked 5 categories in the predicted confidence of each pixel in the third type of candidate label region.
[0045] Optional, training module, used for:
[0046] Step 3.1, will The input to the pre-trained model is processed by an encoder, and then by a feature reduction and projection network to obtain the features. And the semantic segmentation results obtained from the classification network. in, This represents the process by which the i-th foggy image is processed by an encoder and a classification network to obtain the semantic segmentation result.
[0047] Step 3.2: Calculate the prototype for each category according to the first formula. The prototype for each category is represented as Ψ={ψ1,…,ψ c ,…,ψ c}, where ψ c This represents the prototype of the c-th category, where the category refers to the predicted category corresponding to the semantic segmentation, and the prototype refers to the average feature of the category. The first formula is:
[0048]
[0049] in, This represents the feature of the p-th pixel in the i-th foggy image. Let p represent the probability of pixel p belonging to class c in the soft label. Indicates an indicator function, The result within the parentheses is 1 if it is greater than 0, otherwise it is 0.
[0050] Step 3.3, based on prototype Ψ and z i Features for soft labels y t The correction is performed as shown in the following formula:
[0051]
[0052]
[0053] in, This represents the set of candidate labels for pixel p. This represents the weight of category c for pixel p.
[0054] ||·|| represents Euclidean distance;
[0055] Step 3.4: Calculate the cross-entropy loss and contrastive learning loss based on the updated soft labels. The two losses are as follows:
[0056]
[0057]
[0058] Where v represents the feature of the p-th pixel in the i-th foggy image, and p does not belong to the third type of candidate label region; v + V represents a positive sample with feature v. - This represents a negative sample of v; represents the negative sample set, which is the set of all pixels q that do not belong to the candidate label region of type 3, and the set of pixels whose most confident category is not in the candidate label set of pixel p; exp(·) represents the exponentiation operation of the base e of the natural logarithm.
[0059] Step 3.5: For the sunny day image set, calculate the corresponding standard cross-entropy loss, add it to the cross-entropy loss in step 5.4 and the contrastive learning loss, and then backpropagate using the gradient descent algorithm to optimize the pre-trained model.
[0060] Step 3.6: Perform the operations from steps 3.1 to 3.5 above for each foggy image to obtain the semantic segmentation model.
[0061] Optionally, the semantic segmentation network is a deep learning RefineNet semantic segmentation network based on the U-Net architecture.
[0062] Optionally, the feature reduction projection network includes a 1x1 two-dimensional convolutional layer and a ReLU layer.
[0063] In this invention, a pre-trained model is pre-trained using a supervised semantic segmentation training strategy, utilizing a set of sunny day images and the corresponding labels for each image. This pre-trained model includes a feature reduction projection network and a semantic segmentation network, where the semantic segmentation network is a derivative network based on U-Net. The pre-trained model is then used to perform semantic segmentation on foggy day images in a foggy day image set. The semantic segmentation results are sorted in descending order of confidence, and the top-ranked semantic segmentation results are used as reliable pseudo-labels. These reliable pseudo-labels are then expanded to obtain a candidate label set. The expanded pseudo-labels are corrected in real-time using the candidate label set, and the pre-trained model is trained to obtain a semantic segmentation model. The semantic segmentation task is then performed based on this model. This invention solves the problem of semantic segmentation domain adaptation in foggy day scenarios. This solution is simple, effective, highly accurate, and easy to implement. Compared with existing technologies, this invention has the following advantages:
[0064] (1) The proposed foggy semantic segmentation domain adaptation label correction framework based on candidate label set can make good use of the characteristics of foggy images to solve the semantic segmentation domain adaptation problem.
[0065] (2) To address the problem of self-trained pseudo-label generation, the concept of candidate label set is proposed, which makes better use of the network's prediction information, improves the quality of pseudo-labels, and retains more pseudo-label information.
[0066] (3) It has strong practicality and versatility. The backbone of the framework can be replaced with any semantic segmentation network based on U-Net to improve the network performance. Attached Figure Description
[0067] Figure 1 This is a flowchart illustrating an embodiment of the semantic segmentation domain adaptation label correction method based on candidate label sets of the present invention.
[0068] Figure 2 This is a schematic diagram of the architecture of a preset model in one embodiment of the semantic segmentation domain adaptation label correction method based on candidate label set of the present invention;
[0069] Figure 3 for Figure 2 A schematic diagram of the architecture of the RNx module;
[0070] Figure 4 This is a schematic diagram of the training architecture in one embodiment of the semantic segmentation domain adaptation label correction method based on candidate label set of the present invention;
[0071] Figure 5 This is a schematic diagram of the functional modules of an embodiment of the semantic segmentation domain adaptation label correction device based on candidate label set of the present invention.
[0072] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0073] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0074] This invention addresses the problem of semantic segmentation domain adaptation in foggy scenes by providing a label correction method and apparatus based on a candidate label set. After detailed experimental analysis of a foggy image dataset, it was found that although the one-hot pseudo-label with the highest prediction confidence does not always match the true category, the true category has a high probability of appearing among the top-ranking categories in prediction confidence. Based on this finding, it can be assumed that the differences in the dataset will only confuse the true category of pixels with a few other categories, rather than confusing them all. Therefore, if the ambiguity of the true category can be eliminated from a few highly confusing categories, the network model can perform semantic segmentation of foggy images more accurately.
[0075] In a first aspect, embodiments of the present invention provide a semantic segmentation domain adaptation label correction method based on a candidate label set.
[0076] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the semantic segmentation domain adaptation label correction method based on candidate label sets according to the present invention. Figure 1 As shown, the semantic segmentation domain adaptation label correction method based on candidate label sets includes:
[0077] Step S10: Using the sunny day image set and the label corresponding to each sunny day image in the sunny day image set, a supervised semantic segmentation training strategy is used to pre-train the preset model to obtain the pre-trained model. The preset model includes a feature dimensionality reduction projection network and a semantic segmentation network. The semantic segmentation network is a derived network based on U-Net.
[0078] In this embodiment, the Cityscapes and FoggyZurich datasets were used. During training, the datasets were cropped to a size of 786×768, and random data augmentation was performed on each image. The Cityscapes dataset was collected from 50 cities in Germany and nearby countries, including autonomous driving scenes in urban areas during spring, summer, and autumn, and includes annotations required for semantic segmentation, with a resolution of 2048×1024. The FoggyZurich dataset was collected from autonomous driving scenes in foggy urban areas of Zurich, Switzerland, but lacks the annotations required for autonomous driving, with a resolution of 1920×1080.
[0079] In this embodiment, a portion of image data from Cityscapes is selected as a sunny day image set, and its labels are used as a training set. A supervised semantic segmentation training strategy is employed to pre-train a pre-defined model, resulting in a pre-trained model. For example, the supervised semantic segmentation training strategy is used to train the sunny day image set for 100 rounds to obtain the pre-trained model.
[0080] The preset model is obtained by adding a feature dimensionality reduction projection network to the semantic segmentation network. The semantic segmentation network uses a derivative network based on U-Net, such as RefineNet or DeepLab. In one embodiment, the semantic segmentation network is a deep learning RefineNet semantic segmentation network based on the U-Net architecture.
[0081] RefineNet consists of two parts: an encoder and a decoder. The encoder is based on the U-Net framework and combines downsampling and upsampling modules. The downsampling module is composed of multiple two-dimensional convolutional layers, batch normalization (BN) layers, and ReLU activation layers. The upsampling module consists of an RCU module, an MRF module, and a CRP module, all of which are essentially two-dimensional convolutional layers. The decoder, or classifier, is a simple 3×3 two-dimensional convolutional layer.
[0082] The feature reduction projection network consists of a 1x1 two-dimensional convolutional layer and a ReLU layer.
[0083] Reference Figure 2 , Figure 2 This is a schematic diagram of the architecture of a preset model in one embodiment of the semantic segmentation domain adaptation label correction method based on candidate label sets of the present invention. Figure 2 As shown, the encoder and classifier are completely identical to RefineNet, while the feature extraction network (i.e., the feature dimensionality reduction projection network) consists of a 1x1 two-dimensional convolutional layer and a ReLU layer at the end of the encoder. Here, Conv represents a two-dimensional convolutional layer, Max Pooling represents a max pooling layer, and ReLU and Softmax represent two different activation layers.
[0084] Reference Figure 3 , Figure 3 for Figure 2 A schematic diagram of the architecture of the RNx module. (See diagram below.) Figure 3 As shown, the RNx module is structured the same as the corresponding module in RefineNet. Its function is to upsample the features and make the most of the information from the output features in the encoder to improve the upsampling effect.
[0085] Step S20: Semantic segmentation of foggy images in the foggy image set is performed using a pre-trained model. The semantic segmentation results are sorted in descending order of credibility. The semantic segmentation results with the highest credibility are used as credible pseudo-labels. The credible pseudo-labels are expanded to obtain a candidate label set.
[0086] In this embodiment, a subset of images from FoggyZurich is selected as the foggy image set. A pre-trained model is used to generate reliable pseudo-labels for the foggy image set, specifically selecting the top preset proportion of pixels with the highest reliability for each category as reliable pseudo-labels. Specifically, the image data of the foggy image set is input into the pre-trained model, and the prediction results are obtained. For each category, its predicted probability value is calculated, sorted from highest to lowest, and the top preset proportion of pixels is selected; this subset of pixels constitutes the reliable pseudo-labels.
[0087] Then, the set of candidate labels is expanded using trusted pseudo-labels, and the true category of each pixel can be selected from the candidate label set. The true category is the actual category corresponding to that pixel, i.e., the category of the label, which is manually labeled.
[0088] Specifically, the SLIC superpixel partitioning method is first used to partition the foggy image into superpixels. Then, a candidate label set is constructed for each pixel using a relevant formula based on trusted pseudo-labels. Afterward, for each pixel, the position of the category not in the candidate label set is set to 0 and saved as a soft label. The expansion method is based on the fact that fog has a relatively small impact on superpixel partitioning, allowing the foggy image to be divided into multiple semantic spaces. Since each semantic space has the same semantic meaning, the pseudo-labels can be expanded using trusted labels within the same semantic space to obtain pseudo-labels with slightly lower confidence. These labels are designated as the second type of candidate label region (C.II). The remaining labels are designated as the third type of candidate label region (C.III).
[0089] Furthermore, in one embodiment, the foggy image set is: and in, This represents the i-th foggy image. This is a set of foggy images, where ch represents the number of color channels in the image, h and w represent the length and width of the foggy image, respectively, and N... t Let R represent the size of the foggy image set, and let R represent the set of real numbers. The semantic segmentation results are sorted in descending order of confidence. A predetermined proportion of the semantic segmentation results at the top of the sorting are designated as reliable pseudo-labels. The steps to expand these reliable pseudo-labels to obtain a candidate label set include:
[0090] Based on the reliability of the semantic segmentation results, the semantic segmentation results of foggy images are divided into three types of regions, and the division method of the three types of regions is as follows:
[0091]
[0092] Where 1, 2, and 3 represent the first type of candidate label region, the second type of candidate label region, and the third type of candidate label region, respectively; m(p) indicates which type of candidate label region pixel p belongs to; c indicates that the semantic category of the pixel is the c-th class; and Φ is the model parameter. sp represents the probability that pixel p is predicted to be of class c. i This represents the i-th superpixel block. λ represents the component with the highest confidence in the prediction result of pixel p. c For the preset ratio, Let stp and q∈sp be the component with the highest confidence in the prediction result of pixel q. i This indicates that pixels p and q both belong to sp. i ;
[0093] The candidate label set is the label corresponding to the top-ranked category in the predicted confidence of each pixel in the first type of candidate label region, the candidate label set is the label corresponding to the top-ranked 3 categories in the predicted confidence of each pixel in the second type of candidate label region, and the candidate label set is the label corresponding to the top-ranked 5 categories in the predicted confidence of each pixel in the third type of candidate label region.
[0094] For example, if pixel p belongs to the second type of candidate label region, and its prediction confidence ranking for each category is 3, 6, 1, 7, 10, 18, 0, 4… then the candidate label set for p is 3, 6, 1. In summary, each pixel has a candidate label set, which represents a range of guesses about the pixel's true category; that is, the true category is highly likely to be within the candidate label set.
[0095] Step S30: Correct the expanded pseudo-labels in real time using the candidate label set and train the pre-trained model to obtain the semantic segmentation model;
[0096] In this embodiment, the expanded pseudo-labels are corrected in real time using a candidate label set. Based on this, the pre-trained model is trained using a set of clear-day images and a set of foggy images to obtain a semantic segmentation model. Specifically, the clear-day image set and its annotations, and the foggy image set and its soft labels are sequentially input into the pre-trained model for training, and prototype updates and loss calculations are performed. Backpropagation is then conducted, and finally, the SGD optimizer is used for optimization.
[0097] Further, in one embodiment, step S30 includes:
[0098] Step 3.1, will The input to the pre-trained model is processed by an encoder, and then by a feature reduction and projection network to obtain the features. And the semantic segmentation results obtained from the classification network. in, This represents the process by which the i-th foggy image is processed by an encoder and a classification network to obtain the semantic segmentation result.
[0099] Step 3.2: Calculate the prototype for each category according to the first formula. The prototype for each category is represented as Ψ={ψ1,…,ψ c ,…,ψ c}, where ψ c This represents the prototype of the c-th category, where the category refers to the predicted category corresponding to the semantic segmentation, and the prototype refers to the average feature of the category. The first formula is:
[0100]
[0101] in, This represents the feature of the p-th pixel in the i-th foggy image. Let p represent the probability of pixel p belonging to class c in the soft label. Indicates an indicator function, The result within the parentheses is 1 if it is greater than 0, otherwise it is 0.
[0102] Step 3.3, based on prototype Ψ and z i Features for soft labels y t The correction is performed as shown in the following formula:
[0103]
[0104]
[0105] in, This represents the set of candidate labels for pixel p. This represents the weight of category c for pixel p.
[0106] ||·|| represents Euclidean distance;
[0107] Step 3.4: Calculate the cross-entropy loss and contrastive learning loss based on the updated soft labels. The two losses are as follows:
[0108]
[0109]
[0110] Where v represents the feature of the p-th pixel in the i-th foggy image, and p does not belong to the third type of candidate label region; v +V represents a positive sample with feature v. - This represents a negative sample of v; represents the negative sample set, which is the set of all pixels q that do not belong to the candidate label region of type 3, and the set of pixels whose most confident category is not in the candidate label set of pixel p; exp(·) represents the exponentiation operation of the base e of the natural logarithm.
[0111] Step 3.5: For the sunny day image set, calculate the corresponding standard cross-entropy loss, add it to the cross-entropy loss in step 5.4 and the contrastive learning loss, and then backpropagate using the gradient descent algorithm to optimize the pre-trained model.
[0112] Step 3.6: Perform the operations from steps 3.1 to 3.5 above for each foggy image to obtain the semantic segmentation model.
[0113] Reference Figure 4 , Figure 4 This is a schematic diagram of the training architecture in one embodiment of the semantic segmentation domain adaptation label correction method based on candidate label sets of the present invention. Figure 4 As shown, including Figure 2 The network structure consists of an encoder, a classifier, and a feature reduction projection network (i.e.,...). Figure 4 The feature-based dimensionality reduction projection network generates features Z from the foggy image data. After passing through the encoder and classifier, the foggy image can generate reliable pseudo-labels in step S20, which are used to calculate the cross-entropy loss in subsequent step S30 and to generate a candidate label set.
[0114] Step S40: Perform a semantic segmentation task based on the semantic segmentation model.
[0115] In this embodiment, the obtained semantic segmentation model is then used to perform a semantic segmentation task, that is, to perform semantic segmentation on the foggy image to be segmented and obtain the semantic segmentation.
[0116] The final semantic segmentation model is evaluated using the classic MIOU metric from the semantic segmentation field. The specific calculation method is as follows: Assuming the number of categories is N, and C is an N×N confusion matrix, C... ij Let represent the element value in the i-th row and j-th column of the matrix Confusin, and also represent the number of points whose true class is i and which are classified into class j, where i, j ∈ {1, ..., C}. Then, for class i, let:
[0117] TP=C ij
[0118] FP=∑ k≠j C kj
[0119] FN=∑ k≠i C ik
[0120] TN = C*C - FN - FP - TP
[0121] Then we have:
[0122]
[0123] In practice, the NumPy library in Python can be used to quickly calculate the above formulas.
[0124] To facilitate understanding of the beneficial effects of this embodiment, a comparison between the present invention and the most advanced methods in the same field is provided below:
[0125]
[0126] The table above records the scheme used in this embodiment and the results using only the backbone network on the Jilin-1 satellite video test set. To ensure a fair comparison, all models except AdSegNet are based on RefineNet, using the same training strategy and the same hyperparameter settings, while our method modifies the CRST training strategy. It can be seen that our proposed fog-day semantic segmentation domain adaptive labeling method based on candidate label sets has a significant improvement over current state-of-the-art methods.
[0127] In this embodiment, a pre-trained model is pre-trained using a supervised semantic segmentation training strategy, utilizing a set of sunny day images and the corresponding labels for each image. This pre-trained model includes a feature reduction projection network and a semantic segmentation network, where the semantic segmentation network is a derivative network based on U-Net. The pre-trained model is then used to perform semantic segmentation on foggy images in a foggy day image set. The semantic segmentation results are sorted in descending order of confidence, and the top-ranked semantic segmentation results are used as reliable pseudo-labels. These reliable pseudo-labels are then expanded to obtain a candidate label set. The expanded pseudo-labels are corrected in real-time using the candidate label set, and the pre-trained model is trained to obtain a semantic segmentation model. Finally, a semantic segmentation task is performed based on this model. This embodiment solves the problem of semantic segmentation domain adaptation in foggy scenes. The solution is simple, effective, highly accurate, and easy to implement. Compared with existing technologies, this embodiment has the following advantages:
[0128] (1) The proposed foggy semantic segmentation domain adaptation label correction framework based on candidate label set can make good use of the characteristics of foggy images to solve the semantic segmentation domain adaptation problem.
[0129] (2) To address the problem of self-trained pseudo-label generation, the concept of candidate label set is proposed, which makes better use of the network's prediction information, improves the quality of pseudo-labels, and retains more pseudo-label information.
[0130] (3) It has strong practicality and versatility. The backbone of the framework can be replaced with any semantic segmentation network based on U-Net to improve the network performance.
[0131] Secondly, embodiments of the present invention also provide a semantic segmentation domain adaptation label correction device based on a candidate label set.
[0132] In one embodiment, reference is made to Figure 5 , Figure 5 This is a functional module diagram of an embodiment of the semantic segmentation domain adaptation label correction device based on candidate label sets of the present invention. Figure 5 As shown, the semantic segmentation domain adaptation label correction device based on the candidate label set includes:
[0133] The pre-training module 10 is used to pre-train a preset model using a set of sunny day images and the label corresponding to each sunny day image in the set of sunny day images, and adopts a supervised semantic segmentation training strategy to obtain a pre-trained model. The preset model includes a feature dimensionality reduction projection network and a semantic segmentation network. The semantic segmentation network is a derived network based on U-Net.
[0134] The expansion module 20 is used to perform semantic segmentation on foggy images in the foggy image set using a pre-trained model, sort the semantic segmentation results in descending order of credibility, use the semantic segmentation results with the highest pre-ordered proportion as credible pseudo-labels, and expand the credible pseudo-labels to obtain a candidate label set.
[0135] Training module 30 is used to correct the expanded pseudo-labels in real time using the candidate label set and train the pre-trained model to obtain a semantic segmentation model.
[0136] The execution module 40 is used to perform semantic segmentation tasks based on the semantic segmentation model.
[0137] Furthermore, in one embodiment, the foggy image set is: and in, This represents the i-th foggy image. This is a set of foggy images, where ch represents the number of color channels in the image, h and w represent the length and width of the foggy image, respectively, and N... t The size of the foggy image set is represented by R, which represents the set of real numbers. Extension module 20 is used for:
[0138] Based on the reliability of the semantic segmentation results, the semantic segmentation results of foggy images are divided into three types of regions, and the division method of the three types of regions is as follows:
[0139]
[0140] Where 1, 2, and 3 represent the first type of candidate label region, the second type of candidate label region, and the third type of candidate label region, respectively; m(p) indicates which type of candidate label region pixel p belongs to; c indicates that the semantic category of the pixel is the c-th class; and Φ is the model parameter. sp represents the probability that pixel p is predicted to be of class c. i This represents the i-th superpixel block. λ represents the component with the highest confidence in the prediction result of pixel p. c For the preset ratio, stpandq∈sp represents the component with the highest confidence in the prediction result of pixel q. i This indicates that pixels p and q both belong to sp. i ;
[0141] The candidate label set is the label corresponding to the top-ranked category in the predicted confidence of each pixel in the first type of candidate label region, the candidate label set is the label corresponding to the top-ranked 3 categories in the predicted confidence of each pixel in the second type of candidate label region, and the candidate label set is the label corresponding to the top-ranked 5 categories in the predicted confidence of each pixel in the third type of candidate label region.
[0142] Furthermore, in one embodiment, the training module 30 is used for:
[0143] Step 3.1, will The input to the pre-trained model is processed by an encoder, and then by a feature reduction and projection network to obtain the features. And the semantic segmentation results obtained from the classification network. in, This represents the process by which the i-th foggy image is processed by an encoder and a classification network to obtain the semantic segmentation result.
[0144] Step 3.2: Calculate the prototype for each category according to the first formula. The prototype for each category is represented as Ψ={ψ1,…,ψ c ,…,ψ c}, where ψ c This represents the prototype of the c-th category, where the category refers to the predicted category corresponding to the semantic segmentation, and the prototype refers to the average feature of the category. The first formula is:
[0145]
[0146] in, This represents the feature of the p-th pixel in the i-th foggy image. Let p represent the probability of pixel p belonging to class c in the soft label. Indicates an indicator function, The result within the parentheses is 1 if it is greater than 0, otherwise it is 0.
[0147] Step 3.3, based on prototype Ψ and z i Features for soft labels y t To correct this, use the formula shown below:
[0148]
[0149]
[0150] in, This represents the set of candidate labels for pixel p. This represents the weight of category c for pixel p.
[0151] ||·|| represents Euclidean distance;
[0152] Step 3.4: Calculate the cross-entropy loss and contrastive learning loss based on the updated soft labels. The two losses are as follows:
[0153]
[0154]
[0155] Where v represents the feature of the p-th pixel in the i-th foggy image, and p does not belong to the third type of candidate label region; v + V represents a positive sample with feature v. - This represents a negative sample of v; represents the negative sample set, which is the set of all pixels q that do not belong to the candidate label region of type 3, and the set of pixels whose most confident category is not in the candidate label set of pixel p; exp(·) represents the exponentiation operation of the base e of the natural logarithm.
[0156] Step 3.5: For the sunny day image set, calculate the corresponding standard cross-entropy loss, add it to the cross-entropy loss in step 5.4 and the contrastive learning loss, and then backpropagate using the gradient descent algorithm to optimize the pre-trained model.
[0157] Step 3.6: Perform the operations from steps 3.1 to 3.5 above for each foggy image to obtain the semantic segmentation model.
[0158] Furthermore, in one embodiment, the semantic segmentation network is a deep learning RefineNet semantic segmentation network based on the U-Net architecture.
[0159] Furthermore, in one embodiment, the feature reduction projection network comprises a 1x1 two-dimensional convolutional layer and a ReLU layer.
[0160] The functions of each module in the above-mentioned semantic segmentation domain adaptation label correction device based on candidate label set correspond to the steps in the above-mentioned semantic segmentation domain adaptation label correction method embodiment based on candidate label set. Their functions and implementation processes will not be described in detail here.
[0161] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.
[0162] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0163] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of the present invention.
[0164] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.
Claims
1. A semantic segmentation domain adaptation label correction method based on a candidate label set, characterized in that, The semantic segmentation domain adaptation label correction method based on the candidate label set comprises the following steps: A pre-training model is obtained by pre-training a preset model using a supervised semantic segmentation training strategy and a set of sunny images and labels corresponding to each sunny image in the set of sunny images, wherein the preset model comprises a feature dimension reduction projection network and a semantic segmentation network, and the semantic segmentation network is a derivative network based on U-Net. The foggy images in the set of foggy images are subjected to semantic segmentation by the pre-training model, the semantic segmentation results are sorted in descending order of credibility, and the semantic segmentation results in the front of a preset proportion are regarded as credible pseudo labels, the credible pseudo labels are expanded, and a candidate label set is obtained. The semantic segmentation model is obtained by real-time correction of the expanded pseudo labels and training of the pre-training model based on the candidate label set. The semantic segmentation task is performed based on the semantic segmentation model. The foggy image set is , and wherein, represents the i-th foggy image, is the foggy image set, represents the number of image color channels, h and w represent the length and width size of the foggy image respectively, represents the size of the foggy image set, R represents a real number set, the semantic segmentation results are sorted in descending order of credibility, the semantic segmentation results with a preset proportion in the front of the sorting are regarded as credible pseudo labels, and the step of expanding the credible pseudo labels to obtain a candidate label set comprises: The semantic segmentation results of the foggy images are divided into three regions according to the credibility of the semantic segmentation results, and the division of the three regions is as follows: wherein 1, 2, 3 represent a first type of candidate label region, a second type of candidate label region, and a third type of candidate label region respectively, indicates which type of candidate label region the pixel point p belongs to, indicates that the semantic category of the pixel is the class, is a model parameter, indicates the probability that the pixel p is predicted to be the class, indicates the i-th superpixel block, indicates the component with the highest confidence in the prediction result of the pixel point p, is a preset ratio, indicates the component with the highest confidence in the prediction result of the pixel point q, indicates that the pixel points p and q both belong to ; The top one category corresponding to the label in the prediction confidence of each pixel in the first type of candidate label region is taken as the candidate label set, the top three categories corresponding to the label in the prediction confidence of each pixel in the second type of candidate label region is taken as the candidate label set, and the top five categories corresponding to the label in the prediction confidence of each pixel in the third type of candidate label region is taken as the candidate label set. The steps of obtaining the semantic segmentation model by real-time correction of the expanded pseudo labels and training of the pre-training model based on the candidate label set comprise the following steps: Step 3.1, will The input to the pre-trained model is processed by an encoder, and then by a feature reduction and projection network to obtain the features. And the semantic segmentation results obtained by the classification network. ,in, This represents the process by which the i-th foggy image is processed by an encoder and a classification network to obtain the semantic segmentation result. Step 3.2, calculate the prototype of each class according to the first formula, each class prototype is represented as wherein represents the prototype of the c-th class, the class refers to the predicted class corresponding to the semantic segmentation, the prototype refers to the average feature of the class, and the first formula is: wherein, represents a feature of the pth pixel of the ith hazy image, represents a probability of the cth class in the soft label of the pixel p, represents an indicator function, 1 when the result in the bracket is greater than 0, otherwise 0; Step 3.3, according to the prototype With characteristics of the soft label is corrected as shown in the following equation: wherein, represents a set of candidate labels for pixel p, represents a weight of class c for pixel p, represents the Euclidean distance; Step 3.4, the cross-entropy loss and the contrastive learning loss are calculated based on the updated soft labels, and the two losses are as follows: wherein, represents a feature of a pth pixel of an ith hazy image, and p does not belong to the third type of candidate label region; represents a feature of a positive sample, represents a negative sample; represents a set of negative samples, i.e., all pixels q not belonging to the third type of candidate label region, and the set of pixels whose most confident class is not in the candidate label set of the pixel p; represents an exponential operation of the base number e of the natural logarithm; Step 3.5, for the set of sunny images, the standard cross-entropy loss is calculated, and the cross-entropy loss and the contrastive learning loss obtained in step 5.4 are added, and then the gradient descent algorithm is used for back propagation to optimize the pre-training model. Step 3.6, the above steps 3.1 to 3.5 are performed for each foggy image to obtain the semantic segmentation model.
2. The semantic segmentation domain adaptation label correction method based on candidate label set according to claim 1, wherein, The semantic segmentation network is a deep learning RefineNet semantic segmentation network based on the U-Net architecture. 3.The method of claim 1, wherein, The feature dimension reduction projection network comprises a 1x1 two-dimensional convolution layer and a Relu layer.
4. A device for semantic segmentation domain adaptation label correction based on a candidate label set, characterized in that, The semantic segmentation domain adaptation label correction device based on the candidate label set comprises: A pre-training module is configured to pre-train a preset model using a supervised semantic segmentation training strategy and a set of sunny images and labels corresponding to each sunny image in the set of sunny images, wherein the preset model comprises a feature dimension reduction projection network and a semantic segmentation network, and the semantic segmentation network is a derivative network based on U-Net. An expansion module is configured to perform semantic segmentation on the foggy images in the set of foggy images by the pre-training model, sort the semantic segmentation results in descending order of credibility, take the semantic segmentation results in the front of a preset proportion as credible pseudo labels, expand the credible pseudo labels, and obtain a candidate label set. The training module is configured to correct the expanded pseudo labels in real time through the candidate label set and train the pre-training model to obtain a semantic segmentation model; The execution module is configured to execute a semantic segmentation task based on the semantic segmentation model; A foggy image set is denoted as , and wherein, denotes the ith foggy image, is a foggy image set, denotes the number of color channels of the image, and h and w respectively denote the length and width size of the foggy image, denotes the size of the foggy image set, and R denotes a real number set, and the expansion module is configured to: According to the reliability of the semantic segmentation result, the semantic segmentation result of the foggy image is divided into three regions, and the division mode of the three regions is as follows: wherein 1, 2, 3 represent a first type of candidate label region, a second type of candidate label region, and a third type of candidate label region respectively, represents which type of candidate label region the pixel point p belongs to, represents that the semantic category of the pixel is the class, is a model parameter, represents the probability that the pixel p is predicted to be the class, represents an i-th superpixel block, represents a component with the highest confidence in the prediction result of the pixel point p, is a preset ratio, represents a component with the highest confidence in the prediction result of the pixel point q, represents that the pixel points p and q both belong to ; The label corresponding to the first category in the prediction confidence of each pixel in the first type candidate label region is taken as the candidate label set of the first type candidate label region, the label corresponding to the top three categories in the prediction confidence of each pixel in the second type candidate label region is taken as the candidate label set of the second type candidate label region, and the label corresponding to the top five categories in the prediction confidence of each pixel in the third type candidate label region is taken as the candidate label set of the third type candidate label region; The training module is configured to: Step 3.1, the input pre-training model is encoded by an encoder, and then is respectively projected by a feature dimension reduction projection network to obtain features and a classification network to obtain a semantic segmentation result , , denotes a process of obtaining a semantic segmentation result of the i-th foggy image by the encoder and the classification network. Step 3.2, calculate the prototype of each class according to the first formula, each class prototype is represented as wherein represents the prototype of the c-th class, the class refers to the predicted class corresponding to the semantic segmentation, the prototype refers to the average feature of the class, and the first formula is: wherein, represents a feature of the pth pixel of the ith hazy image, represents a probability of the cth class in the soft label of the pixel p, represents an indicator function, 1 when the result in the bracket is greater than 0, otherwise 0; Step 3.3, according to the prototype With characteristics of the soft label is corrected as shown in the following equation: wherein, represents a set of candidate labels for pixel p, represents a weight of class c for pixel p, represents the Euclidean distance; Step 3.4, based on the updated soft label, calculate the cross-entropy loss and the contrastive learning loss, and the two losses are as follows: wherein, represents a feature of a p-th pixel of an i-th hazy image, and p does not belong to the third type of candidate label region; represents a feature of a positive sample, represents of a negative sample; represents a set of negative samples, i.e., a set of pixels q that do not belong to the third type of candidate label region, and the class with the highest confidence of the pixel q is not in the set of pixels whose candidate labels are in the candidate label set of the pixel p; represents an exponential operation of the base e of the natural logarithm; Step 3.5, for the sunny image set, calculate the corresponding standard cross-entropy loss, add the cross-entropy loss and the contrastive learning loss of step 5.4, and then perform back propagation through the gradient descent algorithm to optimize the pre-training model; Step 3.6, for each foggy picture, perform the above steps 3.1 to 3.5 to obtain a semantic segmentation model.
5. The semantic segmentation domain adaptation label correction apparatus based on a candidate label set according to claim 4, wherein, The semantic segmentation network is a deep learning RefineNet semantic segmentation network based on a U-Net architecture.
6. The semantic segmentation domain adaptation label correction apparatus based on a candidate label set according to claim 4, wherein, The feature dimension reduction projection network comprises a 1x1 two-dimensional convolution layer and a Relu layer.