Semi-supervised semantic segmentation method based on prototype cross-image consistency
By using a combination of prototype and attention modules in semi-supervised semantic segmentation, the problems of inconsistent data distribution and insufficient correlation between pixels are solved, and the pseudo-label quality and segmentation effect of unlabeled images are improved.
Patent Information
- Application Number
- CN202410946270.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-15
AI Technical Summary
There are problems in the existing semi-supervised semantic segmentation technology, which leads to performance degradation.
Using a prototype-based cross-image pixel consistency approach, combining labeled and unlabeled data through confidence, using channel and spatial attention modules to enhance feature representation capabilities, and improving inter-pixel correlation through cross-pseudo-supervised loss and similarity calculations .
It effectively reduces the performance degradation caused by data distribution, improves the quality of pseudo-labels generated by unlabeled images, and enhances the segmentation effect.
Smart Images

Figure CN118968049A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision and deep learning, and mainly to the improvement of the attention mechanism in deep learning semi-supervised semantic segmentation and related research based on prototype similarity learning. Background Art
[0002] Semantic segmentation plays an important role in the field of computer vision. Its goal is to assign semantic-level labels to each pixel in an image. It is also widely used in real life, such as autonomous driving, medical image analysis, remote sensing image processing, etc. However, traditional supervised learning methods usually require a large amount of data with pixel-level labels for training, and these annotations are expensive and time-consuming. Therefore, exploring how to use unlabeled data to improve the performance of semantic segmentation models, semi-supervised semantic segmentation has become a mainstream research direction in fields such as medicine.
[0003] Semi-supervised learning is to learn representations from labeled and unlabeled data using a small number of labeled samples. Adding consistency regularization is a common method in semi-supervised learning, which encourages the network to generate similar predictions for the same unlabeled images with different enhancements by calculating the difference between the outputs as the loss function. This type of method only promotes image consistency within the unlabeled dataset. Once there is a distribution difference between the labeled and unlabeled data, it will lead to misalignment in the feature space, further damaging the performance. To improve performance, the extended copy-paste strategy is selected by estimating the confidence score ρi to reduce the distribution mismatch problem.
[0004] In order to better model the relationship between labeled data and unlabeled data, attention modules are added from both channel and spatial aspects to enhance feature representation capabilities.
[0005] Following the trend of contrastive learning, which learns representations or models by comparing similarities and differences between samples, we perform a high-dimensional projection before outputting the data to sample a set of prototypes from the labeled data and calculate a similarity matrix to approximate the same pixel features in the unlabeled data. During training, we explicitly constrain the full correlation computed using the heterogeneous unlabeled features from these two separate branches to remain unchanged. Summary of the invention
[0006] In order to solve the problems of inconsistent data distribution and insufficient correlation between pixels in the prior art, the present invention provides a semi-supervised semantic segmentation method based on prototype-based pixel consistency across images. The method combines labeled data and unlabeled data by using confidence to reduce the performance degradation caused by data distribution, and generates prototypes based on labeled data in an auxiliary network to promote the correlation between pixels, i.e., intra-class correlation, to further improve the segmentation effect.
[0007] The technical solution adopted by the present invention comprises the following steps:
[0008] Step 1: Select Pascal Voc2012 and Cityscapes as datasets, preprocess the data, and obtain additional training images through SBD.
[0009] Step 2: Use a weak enhancement transformation on the unlabeled image data to estimate the confidence score ρi, which represents the prediction confidence of the current model for the i-th labeled data. Use it as the trigger probability to randomly mix the labeled data and the unlabeled data.
[0010] Step 3: Input the labeled data into the network and obtain the supervised loss through the cross entropy function; input the mixed data into the two networks respectively, and use their prediction results as the pseudo labels of the other network to obtain the cross-pseudo label loss.
[0011] Step 4: Add channel and spatial attention modules (CSM) to the auxiliary network to make the network pay more attention to those insignificant feature categories.
[0012] Step 5: Sample the prototype in the marked part of the image, calculate the similarity with the unlabeled pixels, and use memorybank to ensure the diversity of features.
[0013] Optionally, step 1 is implemented by converting the boundary information in the SBD data into pixel-level labels and using them together with the unlabeled images. The same data augmentation operations are performed on the labeled and unlabeled data, such as random cropping, flipping, scaling, etc., to increase the diversity and generalization ability of the data.
[0014] Optionally, step 2 of obtaining the confidence score ρ and using it as the trigger probability of the mixed data includes the following steps:
[0015] Step 2.1: Perform a weak augmentation strategy on the unlabeled data and estimate the confidence score ρi using the weighted average of the normalized prediction entropy on ui.
[0016]
[0017] Step 2.2: Use ρi as the trigger probability to randomly apply the mixture between labeled instances and unlabeled instances to obtain the mixture candidates {u'n}.
[0018]
[0019] A final blending step is applied between the unlabeled instances {um} and the permutation blending candidates
[0020]
[0021] Where Mn and Mm are randomly generated masks.
[0022] The core idea of mixed data is that unlabeled samples with lower confidence, having lower ρi values, are more likely to be helped by these confident labeled samples.
[0023] Optionally, adding a channel and spatial attention CMS structure to the auxiliary network in step 3 includes the following steps:
[0024] Step 3.1: Perform average pooling on the input feature x; then input it into MLP for mapping, where the MLP contains two linear layers with a Relu function in the middle to learn the channel attention weights; then accumulate the obtained features, that is, the weighted sum; perform Gaussian normalization on it, and calculate the mean and standard deviation of the channel attention; multiply the normalized channel attention weights by the original features element by element to obtain the weighted features, which are used as the output of the module.
[0025]
[0026] Step 3.2: The input tensor x is max-pooled and average-pooled along the channel dimension, and the two pooling results are concatenated together to obtain a richer feature representation; then it passes through the convolution module, the initialization parameters specify the input channel as 2, the output channel as 1 (a spatial attention map), and the convolution kernel size, stride, and padding parameters; the spatial attention map is Gaussian normalized and multiplied element-by-element with the original feature to obtain the weighted feature to enhance the model's attention to important spatial locations; finally, the result of multiplying the original input feature by the spatial attention weight is returned.
[0027] Optionally, the steps of step 4 to obtain the supervised loss and the cross pseudo-supervised loss are as follows:
[0028] Step 4.1: The pixel-level image is input into the original network and the correct supervision loss is obtained using the cross entropy function.
[0029] Step 4.2: The mixed image is input into the original network and the auxiliary network respectively, and a parallel training process is used. The predictions after the network output are used as the pseudo labels of each other, and the average is taken to obtain the cross-pseudo-supervision loss, as shown in the figure below. The cross-pseudo-supervision loss is bidirectional, which encourages the two perturbation networks to make highly similar predictions for the same input image.
[0030]
[0031] Optionally, step 5 samples prototypes at the network output and expands for intra-class compactness. The specific steps are as follows:
[0032] Step 5.1: Use a memory bank repository T to iteratively update the prototype. Specifically, initialize N slots for each labeled data, and select embeddings by projection at the positions correctly predicted by both the original network F(θb) and the auxiliary network F(θc) for the reliability of the features in T. T updates the slots corresponding to the labeled samples in the current mini-batch in a similar way to queries.
[0033] Step 5.2: First select the pixels for which F(θb) and F(θc) have the same prediction. For each class, sort the confidence scores of these pixels and then select the features of the first i pixels as the sampled unlabeled features. Therefore, m = i × C, where C represents the number of classes. For the prototype embedding, randomly sample j embeddings from each class among all the embeddings contained in T to increase its diversity.
[0034] Step 5.3: Use pb and pc to connect to the projection heads of the two networks respectively. zb∈Rm×c′ and zc∈Rm×c′ represent two sets of embeddings sampled from the projection features extracted from unlabeled samples, where m is the number of sampled features, c' is the dimension of the projection features, and za and zb are sampled from the corresponding embeddings at the same position on the unlabeled data. Assuming zp represents a set of prototype embeddings sampled from labeled instances, the correlation matrix calculation between zb and zp can be expressed as:
[0035]
[0036] Similarly, replacing zb with zc can calculate the similarity distribution between zc and zp. In order to constrain the consistency of full correlation between two branches, cross entropy loss can be used for correlation consistency regularization.
[0037] The beneficial effects of the present invention are: effectively improving the generation of low-quality pseudo labels for unlabeled images caused by problems such as data distribution, data long tail, cognitive bias, etc. Adding effective semantic information in labeled data to unlabeled data greatly improves the quality of labels. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 This is a framework diagram of the cross-image semantic correlation of the present invention.
[0039] Figure 2 This is the structural diagram of the attention module in the present invention.
[0040] Figure 3 It is the similarity calculation module diagram in the present invention DETAILED DESCRIPTION
[0041] This paper invents a method for semi-supervised semantic segmentation of cross-image semantic correlation, which is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0042] Step 1: Following the currently dominant consistency regularization method in semi-supervised learning, the segmentation model is trained on both labeled and unlabeled data. Figure 1 As shown in the figure, a parallel semi-supervised training framework is adopted, which mainly consists of two parts: the original network and the auxiliary network.
[0043] Step 2: Unlike most works on how to use unlabeled data efficiently, this paper uses mixed two parts of data to prevent cognitive bias caused by different data distribution between labeled data and unlabeled data. The core idea of mixed data is that unlabeled samples with lower confidence have lower ρi values and are more likely to be helped by labeled samples. The unlabeled data is input into the teacher network through a weak enhancement strategy, and the confidence score ρi is estimated using the weighted average of the normalized prediction entropy on ui. The score ρ is specific to the instance and is closely related to the generalization probability of the current model.
[0044]
[0045] Then, use it as the trigger probability to randomly apply a mixture between labeled and unlabeled instances to obtain the mixed candidates:
[0046]
[0047] A final blending step is applied between the unlabeled instances {um} and the permutation blending candidates
[0048]
[0049] Step 3: Use an auxiliary network, whose network structure is compared with the original network, with additional channel and spatial attention CMS structures, and the initialization parameters are the same. The purpose of the channel attention module is to adjust the channel dimension of the feature map to emphasize those features that are helpful but not very significant for the segmentation task. This module first captures global information through global average pooling (GAP), and then learns the dependencies between channels, namely the attention weights, through the convolutional layer. It is Gaussian normalized, and the mean and standard deviation of the channel attention are calculated; the normalized channel attention weights are multiplied element by element with the original features to obtain the weighted features, which are used as the output of the module.
[0050]
[0051] The spatial attention module focuses on the spatial dimension of the feature map, aiming to enhance those areas that are easily overlooked in space. Similar to the channel attention module, maximum pooling and average pooling are performed along the channel dimension, and the two pooling results are spliced together to obtain a richer feature representation; then it passes through the convolution module, the initialization parameters specify the input channel as 2, the output channel as 1 (a spatial attention map), and specify the convolution kernel size, stride and padding parameters; the spatial attention map is Gaussian normalized and multiplied element-by-element with the original feature to obtain the weighted feature to enhance the model's attention to important spatial positions; finally, the result of multiplying the original input feature by the spatial attention weight is returned.
[0052] Step 4: Input the pixel-level image into the original network and use the cross entropy function to get the correct supervision loss. After the above two steps, the mixed data is input into the original network and the auxiliary network respectively. Using cross pseudo supervision, the auxiliary network's prediction Lc2 is used as the pseudo label of the student network, and vice versa. The cross loss is as follows:
[0053]
[0054] Step 5: According to the principle of intra-class consistency, relevant consistency is introduced to formulate additional model regularization, which is a similarity matrix calculated between the features of unlabeled pixels and a set of prototype features sampled from labeled instance features, which reflects the similarity relationship between unlabeled pixels and a set of labeled reference pixels. In the feature space, features are projected into a high-dimensional space to better distinguish categories and features, and prototype sampling is performed at the location of the image labeled data, which is specifically expressed as:
[0055]
[0056] where i and {pm}m are the embeddings of data samples and prototypes in the feature space, and <.,.> represents the distance metric, which is usually set to 12 distance.
[0057] Sampling is performed on the remaining parts to calculate the similarity with the marked parts, but the data in a batch may not be fully utilized. In order to ensure sufficient comparison, a semantic repository T is designed to iteratively update the prototype and initialize N slots for each marked data. In order to ensure the reliability of the repository, projection embedding is selected at the same predicted position of the original network and the auxiliary network respectively. The update method is similar to MoCo.
[0058] At the same time, for the sake of computational efficiency, similarity calculations are not performed on all labeled and unlabeled pixels, and pixel features are sampled from unlabeled data based on the confidence mechanism. First, pixels with the same prediction output by two different networks are selected, and for each category, they are sorted according to the confidence level, and the features of the first i pixels are selected as the sampled unlabeled features. For the embedding of the prototype, j embeddings are randomly sampled from each class of all embeddings contained in the semantic library to increase its diversity. The similarity calculation formula is as follows:
[0059]
[0060] In summary, the present invention discloses a semi-supervised semantic segmentation method based on semantic consistency of prototypes across images. First, prediction is performed through the original network, and the semantic information in the labeled data is added to the unlabeled data by filtering the copy-and-paste method based on confidence; then the mixed data is input into two different networks, in which the auxiliary network adds channels and spatial attention to enhance the semantic expression ability; finally, different networks generate cross-pseudo-labels, project the features on the labeled part for prototype sampling, and the rest will be randomly sampled at the same position in the feature space, to add a semantic library to ensure data diversity, and perform similarity calculations.
[0061] The above embodiments do not limit the protection scope of the present invention in any form, and all technical solutions obtained by equivalent replacement and the like fall within the protection scope of the present invention.
Claims
1. A semi-supervised semantic segmentation method based on prototype cross-image consistency, characterized by The following steps are involved: Step 1: Preprocess the dataset and obtain additional training images through SBD. Step 2: Feed the unlabeled data into the network, estimate the confidence score, and use it as a trigger probability to randomly mix the labeled and unlabeled data. Step 3: Input the labeled data into the network and obtain the supervised loss through the cross entropy function; input the mixed data into the two networks respectively, and use their prediction results as the pseudo labels of the other network to obtain the cross-pseudo label loss. Step 4: Add channel and spatial attention modules (CSM) to the auxiliary network to make the network pay more attention to those insignificant feature categories. Step 5: Sample the prototype in the marked part of the image, calculate the similarity with the unlabeled pixels, and use the memory bank to ensure the diversity of features.
2. According to claim 1, a semi-supervised semantic segmentation method based on prototype cross-image consistency is characterized in that: The specific steps of step 1 are as follows: Step 1: Convert the boundary information in the SBD data into pixel-level labels and use them together with unlabeled images to achieve the same data augmentation operations on labeled and unlabeled data, such as random cropping, flipping, scaling, etc., to increase the diversity and generalization ability of the data.
3. The semi-supervised semantic segmentation method based on prototype cross-image consistency according to claim 1, characterized in that: The specific steps of step 2 are as follows: Step 2.1: Perform a weak augmentation strategy on the unlabeled data and estimate the confidence score ρi using the weighted average of the normalized prediction entropy on ui. Step 2.2: Use ρi as the trigger probability to randomly apply the mixture between labeled instances and unlabeled instances to obtain the mixture candidates {u'n}. A final blending step is applied between the unlabeled instances {um} and the permutation blending candidates Where Mn and Mm are randomly generated masks. The core idea of mixed data is that unlabeled samples with lower confidence, having lower ρi values, are more likely to be helped by these confident labeled samples.
4. The semi-supervised semantic segmentation method based on prototype cross-image consistency according to claim 1, characterized in that: The specific steps of step 3 are as follows: Step 3.1: Perform average pooling on the input feature x; then input it into MLP for mapping, where the MLP contains two linear layers with a Relu function in the middle to learn the channel attention weights; then accumulate the obtained features, that is, the weighted sum; perform Gaussian normalization on it, and calculate the mean and standard deviation of the channel attention; multiply the normalized channel attention weights by the original features element by element to obtain the weighted features, which are used as the output of the module. Step 3.2: The input tensor x is max-pooled and average-pooled along the channel dimension, and the two pooling results are concatenated together to obtain a richer feature representation; then it passes through the convolution module, the initialization parameters specify the input channel as 2, the output channel as 1 (a spatial attention map), and the convolution kernel size, stride, and padding parameters; the spatial attention map is Gaussian normalized and multiplied element-by-element with the original feature to obtain the weighted feature to enhance the model's attention to important spatial locations; finally, the result of multiplying the original input feature by the spatial attention weight is returned.
5. The semi-supervised semantic segmentation method based on prototype cross-image consistency according to claim 1, characterized in that: The specific steps of step 4 are as follows: Step 4.1: The pixel-level image is input into the original network and the correct supervision loss is obtained using the cross entropy function. Step 4.2: The mixed image is input into the original network and the auxiliary network respectively, and a parallel training process is used. The predictions after the network output are used as the pseudo labels of each other, and the average is taken to obtain the cross-pseudo-supervision loss, as shown in the figure below. The cross-pseudo-supervision loss is bidirectional, which encourages the two perturbation networks to make highly similar predictions for the same input image.
6. The semi-supervised semantic segmentation method based on prototype cross-image consistency according to claim 1, characterized in that: The specific steps of step 5 are as follows: Step 5.1: Use a semantic repository T to iteratively update the prototype. Specifically, N slots are initialized for each labeled data, and for the reliability of the features in T, embeddings are selected by projection at the positions correctly predicted by both the original network F(θb) and the auxiliary network F(θc). Step 5.2: First select the pixels for which F(θb) and F(θc) have the same prediction. For each class, sort the confidence scores of these pixels and then select the features of the first i pixels as the sampled unlabeled features. Therefore, m = i × C, where C represents the number of classes. For the prototype embedding, randomly sample j embeddings from each class among all the embeddings contained in T to increase its diversity. Step 5.3: Use pb and pc to connect to the projection heads of the two networks respectively. zb and zc represent two sets of embeddings sampled from the projection features extracted from the unlabeled samples, where m is the number of sampled features, c' is the dimension of the projection features, and za and zb are sampled from the embeddings corresponding to the same position on the unlabeled data.
Citation Information
Cited By
Unmanned aerial vehicle real-time ground object segmentation method based on mixed images
CN119762778A
Neural network driven electric vehicle temperature control strategy optimization method and system
CN119806244A
SAR image road segmentation method based on semi-supervised prototype representation learning and pseudo label balance
CN120032130A
Semi-supervised landslide image semantic segmentation method and system
CN120833489A
A semi-supervised landslide image semantic segmentation method and system
CN120833489B
