A confidence-guided self-supervised semantic segmentation method and system

Abstract

The application discloses a confidence-based self-supervised semantic segmentation method and system, wherein the method steps comprise: acquiring an image and performing global cropping and local cropping to obtain different scale image views; a dense feature is extracted by using a visual Transformer encoder, and a plurality of semantic prototypes are learned by using a decoder; a similarity matrix of the dense feature and the semantic prototype is calculated to obtain a probability distribution of the feature and the prototype; a feature confidence is calculated based on the probability distribution, and the original feature is weighted; the semantic prototype is combined by using the probability distribution, and is fused with the original feature to obtain a reconstructed feature; a lightweight segmentation head is constructed, the reconstructed feature is input to generate a semantic prediction result; the similarity is recalculated by using the weighted feature and the semantic prototype to generate a pseudo label, the pseudo label consistency loss training network is constructed by combining the semantic prediction of the segmentation head, and the semantic segmentation of the image is completed.

Description

Technical Field

[0001] This application relates to the fields of image processing and computer vision technology, specifically to a self-supervised semantic segmentation method and system based on confidence guidance. Background Technology

[0002] Semantic segmentation aims to classify different semantic regions in an image at the pixel level, and is a fundamental technology in tasks such as scene understanding, intelligent perception, medical image analysis, remote sensing image interpretation, and visual perception for autonomous driving. Existing deep semantic segmentation methods typically rely on a large amount of manually labeled pixel-level data for training, but pixel-level labeling is costly, time-consuming, and difficult to scale continuously in complex scenes. Therefore, self-supervised semantic segmentation has gradually become an important technical direction for reducing labeling dependence.

[0003] Existing self-supervised visual representation learning methods can learn visual features from unlabeled images through contrastive learning, masked image modeling, or self-distillation. However, these methods are mainly geared towards image-level or local block-level representation learning, and their training objectives are not entirely aligned with the pixel-level consistency, region boundary preservation, and fine-grained class discrimination requirements of semantic segmentation. Directly applying these self-supervised features to semantic segmentation often requires additional linear probing, clustering, or post-processing, resulting in the final segmentation result being significantly affected by the quality of intermediate features, clustering stability, and class alignment processes.

[0004] Prototype learning methods generate semantic regions in an end-to-end framework by setting several learnable semantic prototypes and assigning dense features to different prototypes. However, without pixel-level supervision, image boundaries, occluded regions, complex backgrounds, and visually similar categories can lead to unstable assignment relationships between dense features and semantic prototypes. If all features are treated equally during training, unreliable and ambiguous regions will continuously interfere with prototype updates and segmentation head learning, resulting in problems such as semantic drift, region fragmentation, or excessive merging.

[0005] Therefore, the key issue in improving the performance of self-supervised semantic segmentation is how to fully utilize the probability distribution from features to semantic prototypes to determine the reliability of different features without relying on manual pixel-level annotation and external clustering post-processing, and how to transform reliable semantic information into stable pseudo-supervision signals. Summary of the Invention

[0006] To address the aforementioned technical issues, this application provides a self-supervised semantic segmentation method and system based on confidence guidance. The method aims to estimate the reliability of semantic assignment through the probability distribution from features to prototypes, enhance reliable semantic features using confidence weighting, learn semantic prototypes through a decoder, generate semantic predictions by receiving reconstructed features through an independent segmentation head, and then complete the training of the segmentation network under unlabeled conditions by combining pseudo-label consistency loss.

[0007] To achieve the above objectives, this application provides a self-supervised semantic segmentation method based on confidence guidance, the steps of which include:

[0008] Acquire images and perform global and local cropping to obtain image views at different scales;

[0009] Dense features are extracted using a visual Transformer encoder, and several semantic prototypes are learned through a decoder.

[0010] Calculate the similarity matrix between features and semantic prototypes to obtain the probability distribution of features and prototypes;

[0011] The confidence level of a feature is calculated based on a probability distribution, and the original features are weighted to obtain confidence-weighted features.

[0012] The semantic prototypes are weighted and combined using probability distributions, and then fused with the original features to obtain reconstructed features;

[0013] A lightweight segmentation head is constructed, and the reconstructed features are input into the lightweight segmentation head to generate semantic prediction results;

[0014] Similarity is recalculated using confidence-weighted features and semantic prototypes to generate pseudo-labels. A pseudo-label consistency loss network is then constructed by combining the semantic prediction of the segmentation head to complete the semantic segmentation of the image.

[0015] Preferably, the visual Transformer encoder is used to extract dense visual features from the image view; the decoder is used to learn several semantic prototypes based on the dense visual features, so that the semantic prototypes serve as the representation center of the potential semantic region.

[0016] Preferably, the probability distribution is obtained by temperature normalization of the cosine similarity matrix. The temperature parameter is used to adjust the sharpness of the probability distribution, so that the network can form a clearer allocation of the correspondence between dense features and semantic prototypes.

[0017] Preferably, the confidence level is calculated from the information entropy of the probability distribution. The smaller the information entropy, the more likely the dense feature is to be assigned to a certain semantic prototype, and the higher the confidence level; the larger the information entropy, the more uncertain the dense feature is among multiple semantic prototypes, and the lower the confidence level.

[0018] Preferably, the reconstructed features are composed of a weighted combination of the original features and semantic prototypes, which allows dense features to move closer to the corresponding semantic prototypes while maintaining local visual information, thereby improving the semantic consistency of the segmentation head.

[0019] Preferably, the lightweight segmentation head is an independent prediction branch distinct from the decoder. The lightweight segmentation head receives reconstructed features and generates semantic prediction results. The pseudo-labels are obtained by rematching the confidence-weighted features with the semantic prototype and are used to constrain the semantic prediction results.

[0020] This application also provides a confidence-guided self-supervised semantic segmentation system, including: an image preprocessing module, an encoder feature extraction module, a decoder prototype learning module, a probability distribution construction module, a confidence weighting module, a prototype reconstruction module, a pseudo-label supervision module, and a segmentation output module; each module works together to complete multi-view input of unlabeled images, dense feature extraction, semantic prototype learning, confidence estimation, feature reconstruction, pseudo-label supervised training, and semantic segmentation output.

[0021] Compared with the prior art, the beneficial effects of this application are as follows:

[0022] This application extracts dense features through a visual Transformer encoder and learns semantic prototypes through a decoder, completing self-supervised semantic modeling within a unified framework. Simultaneously, it estimates the reliability of semantic assignment using the probability distribution of features and prototypes, and enhances semantically clear regions and suppresses ambiguous regions and complex background interference through confidence weighting, obtaining a more stable dense semantic representation. Furthermore, it uses probability distribution to weight and combine semantic prototypes, fusing them with the original features to aggregate features towards potential semantic categories, improving semantic consistency between features and prototypes. In addition, it constructs a lightweight segmentation head and provides explicit supervision signals through pseudo-label consistency learning, enabling the model to directly output semantic segmentation results after training without additional linear probing, external clustering, or complex post-processing. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of this application, the accompanying drawings used in the embodiments are briefly described below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0024] Figure 1 This is a schematic diagram of the method flow of an embodiment of this application;

[0025] Figure 2 This is a schematic diagram of the overall network flow in an embodiment of this application;

[0026] Figure 3 This is a schematic diagram of the confidence weighting, prototype reconstruction, and pseudo-label consistency module in an embodiment of this application.

[0027] Figure 4 This is a schematic diagram of the system structure according to an embodiment of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are all within the protection scope of this application.

[0029] To make the objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] Example 1

[0031] like Figure 1 The diagram shown is a flowchart of the self-supervised semantic segmentation method based on confidence guidance in this embodiment. The steps include:

[0032] S1. Acquire the image and perform multi-view cropping.

[0033] In this embodiment, the input data is image I. To enable the network to learn semantic correspondences from different spatial ranges of the same image, image I is globally and locally cropped to obtain a global image view Ig and a local image view IIl. The global image view preserves a more complete scene structure, while the local image view highlights fine-grained semantic information in local regions. The cropped image views are scaled to a uniform size before being input into the network.

[0034] S2. Extract dense features using a visual Transformer encoder.

[0035] The global and local image views are input into the visual Transformer encoder to extract dense features X={x} corresponding to N spatial locations. i | i=1, 2, ..., N}, where each x i This represents a D-dimensional feature vector. The visual Transformer encoder can use either a frozen pre-trained encoder or a trainable encoder. Using a frozen encoder reduces training instability and preserves the structural information in the pre-trained visual representation.

[0036] S3. Learn semantic prototypes through decoders.

[0037] The dense features output by the encoder are input into the decoder, and K semantic prototypes P={p} are learned through the decoder. j | j=1,2, ..., K}. Each semantic prototype p jK is used to represent potential semantic categories. K can be pre-set according to the semantic granularity required by the target task, or it can be set according to the number of semantic categories in the training set.

[0038] S4. Construct the probability distribution from features to semantic prototypes.

[0039] To characterize the relationship between each dense feature and each semantic prototype, the dense feature x is calculated. i With semantic prototype p j The cosine similarity between the two pairs is used to obtain the similarity matrix S. Then, the similarity matrix is ​​normalized using the temperature parameter τ to obtain the probability distribution Q.

[0040]

[0041]

[0042] Among them, s ij q represents the cosine similarity between the i-th feature and the j-th semantic prototype; ij Let represent the probability that the i-th feature belongs to the j-th semantic prototype; τ is the temperature parameter. By normalizing the temperature, the network can form a more discriminative assignment relationship among multiple semantic prototypes.

[0043] S5. Calculate the confidence level and obtain the confidence-weighted features.

[0044] The probability distribution of features and prototypes reflects the degree of certainty with which the model assigns semantic meaning to each feature. If the probability of a feature on one semantic prototype is significantly higher than on other prototypes, the assignment of that feature is relatively reliable; if its probability is dispersed among multiple semantic prototypes, the feature has high uncertainty. Therefore, this embodiment uses Shannon entropy to calculate the uncertainty of each dense feature and further obtains a confidence score.

[0045]

[0046]

[0047] Among them, h i The information entropy of the i-th feature is represented by c; logK represents the maximum entropy given the K semantic prototypes; i x represents the confidence score of the i-th feature; i ′ represents the confidence-weighted feature. When h i When it is small, c i A larger h indicates that the semantic assignment of the feature is more reliable; when h i When it is large, c iA smaller value indicates that the contribution of this feature to network training should be appropriately suppressed. In this way, the network pays more attention to semantically clear regions during training, reducing the impact of low-confidence regions such as boundaries, occlusions, and complex backgrounds on semantic prototype learning.

[0048] S6. Perform prototype-driven feature reconstruction.

[0049] After obtaining the probability distribution Q, this embodiment uses the probability vector corresponding to each dense feature as the weight of the semantic prototype combination, and performs a weighted summation on the K semantic prototypes to obtain the corresponding semantic prototype combination. Subsequently, the semantic prototype combination is fused with the original features to obtain the prototype-driven reconstructed features.

[0050]

[0051] Where, x i "" represents the reconstructed feature corresponding to the i-th dense feature; λ is the fusion weight parameter. When λ is large, the reconstructed feature retains more of the original visual information; when λ is small, the reconstructed feature absorbs more category structure information of the semantic prototype. This design allows dense features to move closer to the corresponding latent semantic region while preserving local image details.

[0052] S7. Construct a lightweight segmentation head and generate semantic prediction results.

[0053] The lightweight segmentation head is a semantic prediction branch independent of the decoder. Prototype-driven reconstructed features are input into the lightweight segmentation head to obtain the semantic prediction distribution X. out The lightweight segmentation head can be implemented using a multilayer perceptron, a lightweight convolutional structure, or a combination of both, and is used to map reconstructed features to a prediction space corresponding to K semantic prototypes.

[0054]

[0055] Where MLP represents the multilayer perceptron mapping function; X out This represents the semantic prediction distribution.

[0056] S8. Generate pseudo-labels and perform consistency-supervised training.

[0057] To provide explicit supervision for a lightweight segmentation head without pixel-level manual annotation, this embodiment recalculates the similarity and probability distribution using confidence-weighted features X′ and semantic prototypes P, obtaining a new probability distribution Q′. Subsequently, a maximum probability indexing operation is performed on Q′ to obtain the pseudo-label ŷ. i .

[0058]

[0059] Among them, ŷ iThis represents the pseudo-label corresponding to the i-th dense feature. This pseudo-label is induced by the semantic prototype and can reflect the current network's judgment on the potential semantic attribution.

[0060] Based on pseudo-labels i and lightweight splitter output X out We construct a cross-entropy loss function and use this loss function to optimize the segmentation head and related trainable modules.

[0061]

[0062] Where N represents the number of dense features; x out i,j This indicates that the i-th dense feature is predicted as the j-th semantic prototype; L ce This represents the pseudo-label consistency loss function. Through this training method, the lightweight segmentation head can learn semantic prediction results consistent with the prototype assignment.

[0063] S9. Output the semantic segmentation results.

[0064] After training, the image to be segmented is input into the trained network model to obtain the dense features of the image and their corresponding semantic prototypes. Then, the most similar semantic prototype is matched for each feature to obtain the prediction result, and the prediction result is restored to the input image space to obtain the final semantic segmentation mask. This process does not require manual pixel-level labeling, linear probe classifiers, or external clustering post-processing.

[0065] Figure 2 A schematic diagram of the confidence-weighted feature process in this embodiment is provided. Figure 3 A schematic diagram of the process for constructing the pseudo-label consistency loss function is further provided.

[0066] Example 2

[0067] like Figure 4 The diagram shown is a schematic of the structure of the self-supervised semantic segmentation system based on confidence guidance in this embodiment, including: an image preprocessing module, an encoder feature extraction module, a decoder prototype learning module, a probability distribution construction module, a confidence weighting module, a prototype reconstruction module, a segmentation prediction module, a pseudo-label supervision module, and a segmentation output module.

[0068] The system comprises the following modules: an image preprocessing module for acquiring images and performing global and local cropping to obtain image views at different scales; an encoder feature extraction module for extracting dense features using a visual Transformer encoder; a decoder prototype learning module for learning several semantic prototypes based on the dense features; a probability distribution construction module for calculating the similarity between dense features and semantic prototypes and constructing a probability distribution from features to prototypes; a confidence weighting module for calculating feature confidence based on the probability distribution and weighting the original features; a prototype reconstruction module for using the probability distribution to weight and combine the semantic prototypes and fuse them with the original features to obtain reconstructed features; a segmentation prediction module for inputting the reconstructed features into a lightweight segmentation head to generate semantic prediction results; a pseudo-label supervision module for generating pseudo-labels based on the confidence weighted features and semantic prototypes, and constructing a pseudo-label consistency loss based on the semantic prediction results; and a segmentation output module for outputting a semantic segmentation mask for the image to be segmented.

[0069] Through the above system structure, the modules form a complete process from unlabeled image input, dense feature extraction, decoder prototype learning, probability distribution construction, confidence weighting, prototype feature reconstruction, segmentation head prediction, pseudo-label supervision to semantic segmentation output, which can improve the training stability of the self-supervised semantic segmentation model in fuzzy regions and complex backgrounds.

[0070] The above embodiments are merely descriptions of preferred embodiments of this application and are not intended to limit the scope of this application. Various modifications and improvements made to the technical solutions of this application by those skilled in the art without departing from the spirit of this application should fall within the protection scope defined by the claims of this application.

Claims

1. A self-supervised semantic segmentation method based on confidence guidance, characterized in that the steps are as follows: include: Images are acquired and subjected to global and local cropping to obtain image views at different scales. Dense features of the image views are extracted using a visual Transformer encoder, and several semantic prototypes are learned through a decoder. Calculate the similarity matrix between dense features and semantic prototypes to obtain the probability distribution of features and prototypes. Calculate the feature confidence based on the probability distribution and use the feature confidence to weight the original features to obtain confidence-weighted features. The semantic prototypes are weighted and combined using probability distributions, and the weighted combination results are fused with the original features to obtain reconstructed features. A lightweight segmentation head is then constructed, and the reconstructed features are input into the lightweight segmentation head to generate semantic prediction results. By recalculating similarity using confidence-weighted features and semantic prototypes and generating pseudo-labels, a pseudo-label consistency loss network is constructed in combination with semantic prediction results to complete the semantic segmentation of images.

2. The self-supervised semantic segmentation method based on confidence guidance as described in claim 1, characterized in that, Global cropping is used to preserve the main scene structure of the image, while local cropping is used to preserve the fine-grained semantic information of local regions. Global and local cropping together form a multi-scale self-supervised training signal. The visual Transformer encoder is used to extract dense features from the image view, and the decoder is used to learn K semantic prototypes based on the dense features. Each semantic prototype corresponds to a potential semantic category.

3. The self-supervised semantic segmentation method based on confidence guidance as described in claim 1, characterized in that, The method for calculating the similarity matrix, probability distribution, and confidence-weighted features includes: calculating the cosine similarity between dense features and semantic prototypes, and normalizing it using a temperature parameter to obtain the probability distribution; calculating the information entropy of each feature based on the probability distribution, and normalizing it using the maximum entropy to obtain the feature confidence; and applying the feature confidence as a weight to the original features to obtain the confidence-weighted features.

4. The self-supervised semantic segmentation method based on confidence guidance as described in claim 1, characterized in that, The methods for obtaining reconstructed features include: using the probability distribution as the combined weight of semantic prototypes, performing a weighted summation on the semantic prototypes, and fusing the weighted summation result with the original features according to the fusion weights to obtain the reconstructed features.

5. The self-supervised semantic segmentation method based on confidence guidance as described in claim 4, characterized in that, The lightweight segmentation head is a semantic prediction branch independent of the decoder. The lightweight segmentation head receives reconstructed features and outputs pixel-level semantic prediction distribution.

6. The self-supervised semantic segmentation method based on confidence guidance as described in claims 3 and 5, characterized in that, The method for generating pseudo-labels and constructing consistency loss includes: recalculating the similarity matrix and new probability distribution using confidence-weighted features and semantic prototypes, and performing a maximum probability indexing operation on the new probability distribution to obtain pseudo-labels; constructing a cross-entropy loss function using the pseudo-labels and the semantic prediction results output by the lightweight segmentation head, and updating the network parameters based on the cross-entropy loss function.

7. The self-supervised semantic segmentation method based on confidence guidance as described in claim 6, characterized in that, After training, the image to be segmented is input into the trained network model, and the semantic segmentation mask is obtained directly by judging the feature category through semantic prototype. The semantic segmentation mask does not need to rely on manual pixel-level annotation, linear probe classifier or external clustering post-processing.

Images