A small sample semantic segmentation method based on feature uncertainty quantification

Abstract

This invention introduces a few-sample semantic segmentation method based on feature uncertainty quantification, belonging to the field of natural image analysis. The method includes acquiring raw natural images and performing preprocessing and feature extraction; constructing an uncertainty quantification method to perform statistical modeling using spatial distribution entropy combined with channel variance in both spatial and channel dimensions, explicitly quantifying feature uncertainty and obtaining a pixel-level feature uncertainty map; further constructing a differential feature matching module guided by cross-level uncertainty, which adaptively weights and deeply aggregates query set features by calculating the cross-set uncertainty deviation between the query set and the support set; finally, constructing an uncertainty contrastive loss function to merge the contrastive constraints of hierarchical features. This invention effectively solves the challenges of feature uncertainty and domain offset in few-sample semantic segmentation.

Description

Technical Field

[0001] This invention relates to image data processing methods, specifically to a few-sample semantic segmentation method based on feature uncertainty quantification, which addresses the challenges of feature uncertainty and domain offset in few-sample semantic segmentation and belongs to the field of natural image analysis. Background Technology

[0002] Image semantic segmentation is a crucial technique in computer vision, aiming to achieve a fine-grained understanding of scenes by assigning semantic labels to each pixel in an image. It has played a key role in advancements in fields such as autonomous driving and medical image analysis. With the rapid development of deep learning, semantic segmentation models have achieved breakthrough performance, a success heavily reliant on dense, pixel-by-pixel annotations. However, the high cost of annotation significantly limits the application of semantic segmentation techniques in real-world scenarios, making it impractical in some situations. Recently, researchers have attempted to alleviate the high cost of pixel-by-pixel annotation using bounding boxes, doodles, and image-level category labeling. However, segmentation models trained with these weak labels exhibit limited generalization to new categories not seen during training, making such methods difficult to apply in real-world scenarios. Furthermore, while recent large-scale visual models have shown high generalization performance in semantic segmentation, their transfer or fine-tuning processes typically require a large number of labeled samples.

[0003] Deep learning's learning mechanism contrasts sharply with human learning, which leverages prior knowledge to quickly learn new things. Specifically, after learning image segmentation rules for certain categories, humans can quickly locate and identify new targets when encountering previously unseen images of a new category, using only one or a few examples. Therefore, few-shot semantic segmentation was proposed to address this challenge. To enable the segmentation model to generalize to new categories, it learns pixel-level dense predictions for unseen categories (query images) using only a small amount of labeled data (supporting images).

[0004] Current research on few-shot semantic segmentation primarily employs meta-learning paradigms. This involves simulating few-shot scenarios on known categories, allowing the model to adapt to unseen categories. Specifically, it utilizes information from support images and query images to predict the segmentation of target categories. The most classic approach is prototype-based learning. Typically, features from both support and query images are extracted, and masked average pooling is used to generate one or more prototype vectors representing the target category features. Subsequently, metrics such as cosine similarity are used to densely match the prototypes with the query category features, thereby predicting the target category in the query image. However, few-shot semantic segmentation faces numerous challenges. In real-world scenarios, issues such as blurred target boundaries and background interference hinder the model's ability to accurately capture target boundaries and semantic information, thus limiting segmentation accuracy. Furthermore, features such as appearance, texture, and shape within the same category may differ significantly from the training data, while features from different categories may have high similarity to the training data, leading to a lack of generalization ability in the model. Summary of the Invention

[0005] The present invention aims to provide a few-sample semantic segmentation method based on feature uncertainty quantization to solve the problems of feature uncertainty and domain offset in the processing of natural semantic images by existing methods.

[0006] This scheme includes a few-sample semantic segmentation method based on feature uncertainty quantification, comprising the following steps:

[0007] S1: Acquire the original natural image and perform preprocessing and feature extraction;

[0008] S2: Constructing an uncertainty quantification method;

[0009] S3: Construct a differential feature matching module guided by cross-level uncertainty;

[0010] S4: Construct the uncertainty contrastive loss function;

[0011] S5: Based on the above-mentioned uncertainty quantification method, the differential feature matching module guided by cross-level uncertainty, and the uncertainty comparison loss function, perform small sample semantic segmentation.

[0012] 2. The few-sample semantic segmentation method based on feature uncertainty quantization according to claim 1, characterized in that: in step S1, the experimental dataset uses PASCAL-5i, constructed from the PASCAL VOC 2012 dataset and SDS additional annotations, containing images of 20 categories; data augmentation is performed using methods such as random rotation, horizontal flipping, and random cropping; features of arbitrary dimensions are extracted using ResNet-50 features pre-trained on the ImageNet classification task. .

[0013] 3. The small-sample semantic segmentation method based on feature uncertainty quantification according to claim 1, characterized in that: in step S2, an uncertainty quantification method is constructed, which utilizes the spatial distribution entropy combined with the channel variance in both spatial and channel dimensions to statistically model and directly mine the reliability prior within the features to achieve explicit quantification of feature uncertainty, specifically as follows:

[0014] S21. Spatial distribution entropy essentially models the spatial response value of a single-channel feature as a spatial probability distribution, quantifying the degree of disorder in the spatial distribution of that channel feature through the entropy value. The higher the entropy value, the more dispersed the spatial response of that channel feature, and the lower the certainty of the spatial feature. The formula can be expressed as:

[0015]

[0016]

[0017]

[0018] in, Channel normalization for input features. It is the spatial probability distribution of each channel. Calculate the entropy value to obtain the spatial entropy scalar for each channel. Then, the spatial entropy of the channel dimension is averaged across channels and aggregated into a pixel-level spatial uncertainty feature map. ;

[0019] S22. Channel variance characterizes the dispersion of feature responses from different channels at the same spatial location. A larger variance indicates greater inconsistency in the feature representation of that pixel location across channels, and lower determinism of the channel features. A smaller variance indicates more consistent channel feature responses for that pixel, and more stable feature representation. The formula can be expressed as:

[0020]

[0021] Among them, variance is calculated in the channel dimension. This directly yields pixel-level channel variance feature maps. For variance calculation, Normalization of features yields ;

[0022] S23. The degree of feature disorder in the spatial dimension and the degree of feature dispersion in the channel dimension are complementary uncertainty representations; in order to comprehensively utilize these two complementary representations, learnable adaptive weight parameters are introduced. To achieve the two dimensions of S22 and S23 and The weighted fusion is performed; furthermore, to ensure the stability of the uncertainty scores, the original uncertainty scores of the fusion are trimmed to... The interval is used to obtain the final pixel-level feature uncertainty map. The formula can be expressed as:

[0023]

[0024] in, For the cropping operation, Weights for spatial dimensions, The weights are for the channel dimension.

[0025] 4. The few-sample semantic segmentation method based on feature uncertainty quantization according to claim 1, characterized in that: in step S3, a differential feature matching module guided by cross-level uncertainty is constructed to address the significant heterogeneity in the reliability of shallow-level focused edge and texture detail features and deep-level dominant category discrimination semantic features during the matching process, specifically as follows:

[0026] S31. Using the uncertainty quantization method in S2, pixel-level uncertainty maps are established for both shallow detail features and deep semantic features. Then, by calculating and normalizing the cross-set uncertainty deviation between the query set and the support set, an uncertainty deviation map is obtained. This is used to quantify the matching reliability of the two types of features in cross-set scenarios. Furthermore, to integrate the uncertainty information of k-shot support samples, the hierarchical uncertainty features of the k support samples are stacked and averaged to obtain the average uncertainty features of the support set, eliminating multi-sample redundancy and enhancing effective information. The formula can be expressed as:

[0027]

[0028]

[0029] in, To support the uncertainty maps of shallow and deep layers obtained from the samples, Let i be the uncertainty graph corresponding to the i-th supporting sample; Normalized bias maps obtained from shallow and deep layers. To query the uncertainty graph corresponding to the sample;

[0030] S32, obtained using S31 Features of the query set Adaptive weighting adjustment is performed, and finally, the weighted hierarchical features and uncertainty deviation map are merged to achieve differentiated feature matching and aggregation that strengthens high-reliability features and suppresses low-reliability features. The formula can be expressed as:

[0031]

[0032] in, These are the different query set features obtained after weighting for the shallow and deep layers. For Hadamard product operation;

[0033] S33. Combine the support set prototype generated from the features with the weighted hierarchical features. , Deviation Chart Channel splicing is performed, and the convolutional layer consists of Conv+BN+ReLU. Deep aggregation is performed to enhance highly reliable features. The formula can be expressed as:

[0034]

[0035] in, The final spliced ​​features, For splicing operations, Generate a support set prototype for the features.

[0036] 5. A few-sample semantic segmentation method based on feature uncertainty quantization according to claim 1, characterized in that: in step S4, an uncertainty contrastive loss function is constructed to address the problem of loose feature distribution and insufficient inter-class discriminativeness caused by limited labeled samples, specifically as follows:

[0037] S41. Adopting the InfoNCE paradigm, using high-determinism features of the query set as anchors, high-determinism features of the support set as positive examples, and low-determinism features as negative examples, the comparison constraints of shallow and deep features are combined. The formula can be expressed as:

[0038]

[0039] in For uncertainty, contrastive loss function, and This is a set of highly reliable positive examples for the query set and support set. They belong to the query set respectively and support set The characteristics of a single query on a set It is a comprehensive set of negative characteristics of low reliability, belonging to A single negative feature of a set, This indicates the calculation of cosine similarity.

[0040] 6. A few-sample semantic segmentation method based on feature uncertainty quantization according to claim 1, characterized in that: the image is preprocessed and features are extracted in step S1, and then uncertainty quantization is performed on the features in step S2 to obtain a pixel-level feature uncertainty map. Then, according to step S3, the difference between the support set and the query set is considered, and the quantized uncertainty map and features are spliced ​​together. Finally, according to step S4, an uncertainty contrastive loss function is constructed to constrain feature learning, thereby realizing a few-sample semantic segmentation method based on feature uncertainty quantization. Attached Figure Description

[0041] Figure 1 This is a schematic diagram illustrating the uncertainty quantization of a few-sample semantic segmentation method based on feature uncertainty quantization according to an embodiment of the present invention.

[0042] Figure 2 This is a schematic diagram of a differential feature matching module guided by cross-level uncertainty, according to an embodiment of a few-sample semantic segmentation method based on feature uncertainty quantification of the present invention.

[0043] Figure 3 This is a schematic diagram of the uncertainty comparison loss function in an embodiment of a few-sample semantic segmentation method based on feature uncertainty quantification according to the present invention. Detailed Implementation

[0044] S1: Acquire the original natural image and perform preprocessing and feature extraction;

[0045] S11. Preprocessing and Feature Extraction Stage:

[0046] In the experiments of this invention, the dataset used is PASCAL-5i, constructed from the PASCAL VOC 2012 dataset with SDS additional annotations, containing images of 20 categories. Data augmentation was performed using methods such as random rotation, horizontal flipping, etc. Each input image was randomly cropped to 473×473 pixels, and these images included natural images with occlusion, complex backgrounds, and small objects. Features of arbitrary dimensions were extracted using ResNet-50 features pre-trained on the ImageNet classification task. .

[0047] S2: Constructing an uncertainty quantification method;

[0048] S21. Spatial distribution entropy essentially models the spatial response value of a single-channel feature as a spatial probability distribution, quantifying the degree of disorder in the spatial distribution of that channel feature through the entropy value. The higher the entropy value, the more dispersed the spatial response of that channel feature, and the lower the certainty of the spatial feature. The formula can be expressed as:

[0049]

[0050]

[0051]

[0052] in, Channel normalization of input features, It is the spatial probability distribution of each channel. Calculate the entropy value to obtain the spatial entropy scalar for each channel. Then, the spatial entropy of the channel dimension is averaged across channels and aggregated into a pixel-level spatial uncertainty feature map. ;

[0053] S22. Channel variance characterizes the dispersion of feature responses from different channels at the same spatial location. A larger variance indicates greater inconsistency in the feature representation of that pixel location across channels, and lower determinism of the channel features. A smaller variance indicates more consistent channel feature responses for that pixel, and more stable feature representation. The formula can be expressed as:

[0054]

[0055] Among them, variance is calculated in the channel dimension. This directly yields pixel-level channel variance feature maps. For variance calculation, Normalization of features yields ;

[0056] S23. The degree of feature disorder in the spatial dimension and the degree of feature dispersion in the channel dimension are complementary uncertainty representations; in order to comprehensively utilize these two complementary representations, learnable adaptive weight parameters are introduced. To achieve the two dimensions of S22 and S23 and The weighted fusion is performed; furthermore, to ensure the stability of the uncertainty scores, the original uncertainty scores of the fusion are trimmed to... The interval is used to obtain the final pixel-level feature uncertainty map. The formula can be expressed as:

[0057]

[0058] in, For the cropping operation, Weights for spatial dimensions, The weights are for the channel dimension.

[0059] S3: Construct a differential feature matching module guided by cross-level uncertainty;

[0060] S31. Using the uncertainty quantization method in S2, pixel-level uncertainty maps are established for both shallow detail features and deep semantic features. Then, by calculating and normalizing the cross-set uncertainty deviation between the query set and the support set, an uncertainty deviation map is obtained. This is used to quantify the matching reliability of the two types of features in cross-set scenarios. Furthermore, to integrate the uncertainty information of k-shot support samples, the hierarchical uncertainty features of the k support samples are stacked and averaged to obtain the average uncertainty features of the support set, eliminating multi-sample redundancy and enhancing effective information. The formula can be expressed as:

[0061]

[0062]

[0063] in, To support the uncertainty maps of shallow and deep layers obtained from the samples, Let i be the uncertainty graph corresponding to the i-th supporting sample; Normalized bias maps obtained from shallow and deep layers. To query the uncertainty graph corresponding to the sample;

[0064] S32, obtained using S31 Features of the query set Adaptive weighting adjustment is performed, and finally, the weighted hierarchical features and uncertainty deviation map are merged to achieve differentiated feature matching and aggregation that strengthens high-reliability features and suppresses low-reliability features. The formula can be expressed as:

[0065]

[0066] in, These are the different query set features obtained after weighting for the shallow and deep layers. For Hadamard product operation;

[0067] S33. Combine the support set prototype generated from the features with the weighted hierarchical features. , Deviation Chart Channel splicing is performed, and the convolutional layer consists of Conv+BN+ReLU. Deep aggregation is performed to enhance highly reliable features. The formula can be expressed as:

[0068]

[0069] in, The final spliced ​​features, For splicing operations, Generate a support set prototype for the features.

[0070] S4: Construct the uncertainty contrastive loss function;

[0071] S41. Adopting the InfoNCE paradigm, using high-determinism features of the query set as anchors, high-determinism features of the support set as positive examples, and low-determinism features as negative examples, the comparison constraints of shallow and deep features are combined. The formula can be expressed as:

[0072]

[0073] in For uncertainty, contrastive loss function, and This is a set of highly reliable positive examples for the query set and support set. They belong to the query set respectively and support set The characteristics of a single query on a set It is a comprehensive set of negative characteristics of low reliability, belonging to A single negative feature of a set, This indicates the calculation of cosine similarity.

Claims

1. A few-sample semantic segmentation method based on feature uncertainty quantification, characterized in that, Includes the following steps: S1: Acquire the original natural image and perform preprocessing and feature extraction; S2: Constructing an uncertainty quantification method; S3: Construct a differential feature matching module guided by cross-level uncertainty; S4: Construct the uncertainty contrastive loss function; S5: Based on the above-mentioned uncertainty quantification method, the differential feature matching module guided by cross-level uncertainty, and the uncertainty comparison loss function, perform small sample semantic segmentation.

2. The few-sample semantic segmentation method based on feature uncertainty quantization according to claim 1, characterized in that: In step S1, the experimental dataset used is PASCAL-5i, constructed from the PASCAL VOC 2012 dataset with SDS annotations, containing images of 20 categories; data augmentation was performed using methods such as random rotation, horizontal flipping, and random cropping; features of arbitrary dimensions were extracted using ResNet-50 features pre-trained on the ImageNet classification task. .

3. The few-sample semantic segmentation method based on feature uncertainty quantization according to claim 1, characterized in that: In step S2, an uncertainty quantification method is constructed. This method utilizes statistical modeling combining spatial distribution entropy and channel variance in both spatial and channel dimensions to directly mine reliability priors within the features, thereby achieving explicit quantification of feature uncertainty. Specifically: S21. Spatial distribution entropy essentially models the spatial response value of a single-channel feature as a spatial probability distribution, quantifying the degree of disorder in the spatial distribution of that channel feature through the entropy value. The higher the entropy value, the more dispersed the spatial response of that channel feature, and the lower the certainty of the spatial feature. The formula can be expressed as: in, Channel normalization for input features. It is the spatial probability distribution of each channel. Calculate the entropy value to obtain the spatial entropy scalar for each channel. Then, the spatial entropy of the channel dimension is averaged across channels and aggregated into a pixel-level spatial uncertainty feature map. ; S22. Channel variance characterizes the dispersion of feature responses from different channels at the same spatial location. A larger variance indicates greater inconsistency in the feature representation of that pixel location across channels, and lower determinism of the channel features. A smaller variance indicates more consistent channel feature responses for that pixel, and more stable feature representation. The formula can be expressed as: Among them, variance is calculated in the channel dimension. This directly yields pixel-level channel variance feature maps. For variance calculation, Normalization of features yields ; S23. The degree of feature disorder in the spatial dimension and the degree of feature dispersion in the channel dimension are complementary uncertainty representations; in order to comprehensively utilize these two complementary representations, learnable adaptive weight parameters are introduced. To achieve the two dimensions of S22 and S23 and The weighted fusion is performed; furthermore, to ensure the stability of the uncertainty scores, the original uncertainty scores of the fusion are trimmed to... The interval is used to obtain the final pixel-level feature uncertainty map. The formula can be expressed as: in, For the cropping operation, Weights for spatial dimensions, The weights are for the channel dimension.

4. The few-sample semantic segmentation method based on feature uncertainty quantization according to claim 1, characterized in that: In step S3, a differential feature matching module guided by cross-level uncertainty is constructed to address the significant heterogeneity in reliability between shallow-level focused edge and texture detail features and deep-level dominant category semantic features during the matching process. Specifically: S31. Using the uncertainty quantization method in S2, pixel-level uncertainty maps are established for both shallow detail features and deep semantic features. ; Then, by calculating and normalizing the cross-set uncertainty deviation between the query set and the support set, an uncertainty deviation map is obtained. This is used to quantify the matching reliability of the two types of features in cross-set scenarios. Furthermore, to integrate the uncertainty information of k-shot support samples, the hierarchical uncertainty features of the k support samples are stacked and averaged to obtain the average uncertainty features of the support set, eliminating multi-sample redundancy and enhancing effective information. The formula can be expressed as: in, To support the uncertainty maps of shallow and deep layers obtained from the samples, Let i be the uncertainty graph corresponding to the i-th supporting sample; Normalized bias maps obtained from shallow and deep layers. To query the uncertainty graph corresponding to the sample; S32, obtained using S31 Features of the query set Adaptive weighting adjustment is performed, and finally, the weighted hierarchical features and uncertainty deviation map are merged to achieve differentiated feature matching and aggregation that strengthens high-reliability features and suppresses low-reliability features. The formula can be expressed as: in, These are the different query set features obtained after weighting for the shallow and deep layers. For Hadamard product operation; S33. Combine the support set prototype generated from the features with the weighted hierarchical features. , Deviation Chart Channel splicing is performed, and the convolutional layer consists of Conv+BN+ReLU. Deep aggregation is performed to enhance highly reliable features. The formula can be expressed as: in, The final spliced ​​features, For splicing operations, Generate a support set prototype for the features.

5. The few-sample semantic segmentation method based on feature uncertainty quantization according to claim 1, characterized in that: In step S4, an uncertainty contrastive loss function is constructed to address the problem of loose feature distribution and insufficient inter-class discriminative power learned by the model due to limited labeled samples. Specifically: S41. Adopting the InfoNCE paradigm, using high-determinism features of the query set as anchors, high-determinism features of the support set as positive examples, and low-determinism features as negative examples, the comparison constraints of shallow and deep features are combined. The formula can be expressed as: in For uncertainty, contrastive loss function, and This is a set of highly reliable positive examples for the query set and support set. They belong to the query set respectively and support set The characteristics of a single query on a set It is a comprehensive set of negative characteristics of low reliability, belonging to A single negative feature of a set, This indicates the calculation of cosine similarity.

6. The few-sample semantic segmentation method based on feature uncertainty quantization according to claim 1, characterized in that: In step S1, the image is preprocessed and features are extracted. Then, in step S2, the uncertainty of the features is quantized to obtain a pixel-level feature uncertainty map. Then, according to step S3, the difference between the support set and the query set is considered, and the quantized uncertainty map and features are concatenated. Finally, according to step S4, an uncertainty contrastive loss function is constructed to constrain feature learning, thereby realizing a few-sample semantic segmentation method with feature uncertainty quantization.

Images