Weakly supervised semantic segmentation method based on background prior

Through a weakly supervised semantic segmentation method based on background priors, combined with background clustering and pre-training models, the problem of difficult to obtain segmentation labels and poor weak supervised segmentation effects are solved, and the segmentation effect and model of medical, infrared and remote sensing images are improved. Generalization ability reduces dependence on large-scale annotation data.

CN118015282BActive Publication Date: 2024-11-05HARBIN INST OF TECH AT WEIHAI +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410311121.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-19
Publication Date
2024-11-05
Estimated Expiration
2044-03-19

AI Technical Summary

Technical Problem

In the existing semantic segmentation technology, there is a problem of difficult to obtain segmentation labels and poor weak supervision segmentation effects, especially in the fields of medical images, infrared images and remote sensing images, the model generalization ability is poor and large-scale annotation data cannot be effectively utilized.

Method used

Using a weakly supervised semantic segmentation method based on background priors, the background clustering algorithm is used to input the data set with only image-level labels to the background clustering algorithm, combined with the pre-trained SAM model for inference, and the background mask is generated using the segmentation result fusion module. The code diagram is used to extract features through multi-scale splicing convolution blocks, and finally the foreground input classifier is classified and visualized to generate a semantic segmentation diagram.

Benefits of technology

It effectively reduces the difficulty of weak supervision tasks, improves the effectiveness and information quality of the prospect segmentation area, improves the segmentation effect and generalization ability of the weak supervision semantic segmentation model, and reduces the dependence on large-scale annotation data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118015282B_ABST
    Figure CN118015282B_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of data processing, and in particular to a weakly supervised semantic segmentation method based on background prior, comprising: inputting a specific data set with only image-level labels into a background clustering algorithm to perform background clustering according to pixel values ​​to obtain a clustering mask map; inputting the data set into a SAM model pre-trained on a natural image for inference to obtain a pre-trained mask map; performing mask fusion based on IOU on the clustering mask map and the pre-trained mask map through a segmentation result fusion module to generate a background mask map; utilizing a plurality of serial multi-scale splicing convolution blocks to realize tower-like extraction of data set features to generate a classification feature map; utilizing the background mask map to distinguish foreground and background in the classification feature map; inputting the foreground into a classifier for classification operation, and performing visual visualization; the invention solves the problems in the existing semantic segmentation technology that segmentation labels are difficult to obtain and the weakly supervised segmentation effect is poor.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a weakly supervised semantic segmentation method based on background prior. Background Art

[0002] Semantic segmentation is one of the classic problems in computer vision and can be widely used in vision-based road scene segmentation, remote sensing image segmentation and other fine segmentation scenarios. The semantic segmentation task has achieved high segmentation accuracy and effect with pixel-level label supervision. The large segmentation model trained with large-scale annotated data can even perform relatively accurate semantic segmentation on all targets contained in natural images. However, the effect of the large segmentation model in the fields of medical images, infrared images, remote sensing images, etc. is still poor, and the cost of obtaining pixel-level labels in special fields is high, making it impossible to use the fully supervised semantic segmentation method to realize the training of deep learning models.

[0003] With the rapid development of image semantic segmentation technology, weakly supervised semantic segmentation technology has emerged. It aims to build a learning process for prediction models without pixel-level label supervision, and only uses some existing image-level labels, object detection boxes or fuzzy labels to achieve semantic segmentation. Compared with supervised learning semantic annotation, weakly supervised learning semantic annotation is easier to obtain.

[0004] However, traditional weak supervision technology has the limitations of small amount of labeled data and poor network generalization ability, and there are problems such as difficulty in obtaining segmentation labels and poor weak supervision segmentation effect. Summary of the invention

[0005] The purpose of the present invention is to provide a weakly supervised semantic segmentation method based on background prior, aiming to solve the problems in existing semantic segmentation technology that segmentation labels are difficult to obtain and the weakly supervised segmentation effect is poor.

[0006] To achieve the above object, the present invention provides a weakly supervised semantic segmentation method based on background prior, comprising:

[0007] Input a specific data set with only image-level labels into the background clustering algorithm to perform background clustering based on pixel values ​​and obtain a clustering mask map;

[0008] Inputting the data set into a SAM model pre-trained on natural images for inference to obtain a pre-trained mask map;

[0009] The clustering mask map and the pre-trained mask map are subjected to IOU-based mask fusion through a segmentation result fusion module to generate a background mask map;

[0010] Using multiple serial multi-scale convolution blocks, the tower-like extraction of the features of the data set is realized to generate a classification feature map;

[0011] Using the background mask image to distinguish the foreground and background in the classification feature image;

[0012] The foreground is input into a classifier for classification operation, and visually visualized to generate a semantic segmentation map.

[0013] The specific steps of inputting a specific data set with only image-level labels into a background clustering algorithm to perform background clustering according to pixel values ​​to obtain a clustering mask map include:

[0014] Convert the input image from RGB to grayscale and delete the transparency channel of the image.

[0015] By arranging pixels from large to small, the pixels contained in the image are organized into a pixel sequence, and the distance between the first pixel value and the last pixel value of a fixed value of adjacent pixel sequence is calculated to find the pixel sequence with the largest distance. The median value of the pixel sequence is used as the threshold for division to achieve color segmentation and generate a cluster mask map.

[0016] The specific steps of inputting the data set into a SAM model pre-trained on natural images for inference to obtain a pre-trained mask map include:

[0017] Encode the dataset through a weight-frozen image token encoder and save the feature vector in numpy format;

[0018] Multiple instance segmentation maps are obtained by decoding with numpy vectors and filtered. Noise-level instances are deleted and segmentation map masks are accumulated for the remaining instances. When masks overlap, a larger area mask map is used to prevent the loss of target information.

[0019] The specific steps of performing IOU-based mask fusion on the clustering mask map and the pre-trained mask map through the segmentation result fusion module to generate a background mask map include:

[0020] Use the clustered mask map and the pre-trained mask map to find the area where the masks intersect;

[0021] When the IOU between the intersection area and the corresponding pre-trained mask area exceeds 0.5, the corresponding area on the pre-trained mask map is selected, and when the IOU is less than 0.5, it is shielded to generate a fused background mask map.

[0022] The specific steps of using multiple serial multi-scale convolution blocks include:

[0023] Use square convolution kernels to perform preliminary feature extraction, and use horizontal and vertical group convolution to achieve horizontal and vertical mixed extraction of feature maps in spatial position;

[0024] The activation features accumulated from the horizontal and vertical convolutions at multiple scales are concatenated, and point convolution is used to perform channel dimensionality reduction.

[0025] The specific steps of implementing the tower extraction of the data set features include:

[0026] At each layer, the input features are initially processed by batch normalization and convolution kernels from small to large;

[0027] The preliminary processed features are skipped to realize the accumulation of horizontal and vertical extracted features and preliminary processed features, and the accumulated features are output using LP pooling. The above operation is repeated many times to realize the tower extraction of features.

[0028] The specific step of using the background mask image to distinguish the foreground and background in the classification feature image includes:

[0029] According to the foreground points in the background mask image, the shortest distance d between each background point and the foreground area is calculated, and the pixel weakening value W = 1-d / N is obtained, where N is the side length of the mask image;

[0030] The background feature is multiplied by the weakening value, and the foreground feature is multiplied by 1 to generate a foreground feature map after the background is weakened.

[0031] The specific steps of inputting the foreground into a classifier for classification and visually visualizing to generate a semantic segmentation map include:

[0032] Provide a visual explanation of the last convolutional layer of the classification operation, and use the register_forward_hook and register_backward_hook hook functions to record the forward and backward propagation gradient weights of the last convolution layer;

[0033] The Grad-CAM is generated using the gradient weights and normalized to obtain an interpretable weight map. The threshold Seg_threshold is used to select pixels. Pixels exceeding the threshold are set to white, otherwise they are set to black to obtain a weakly supervised segmentation map.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] Use background prior knowledge to perform weakly supervised semantic segmentation of images, use clustering algorithms and SAM pre-trained models to pre-segment images, reduce the difficulty of weakly supervised tasks, and use the segmentation result fusion module to screen pre-segmented areas, thereby improving the effectiveness and information quality of foreground segmentation areas. Fully combine background prior knowledge and large model pre-training prior knowledge to achieve weakly supervised semantic segmentation of images, effectively solve the problems of difficulty in obtaining labeled samples, poor weakly supervised segmentation effects, and poor model generalization capabilities in weakly supervised semantic segmentation in medical imaging, infrared imaging, and remote sensing imaging, effectively improve the segmentation effect and generalization capabilities of weakly supervised semantic segmentation models in image segmentation tasks, and reduce the model's dependence on large-scale labeled data. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0037] Figure 1 It is a flow chart of a weakly supervised semantic segmentation method based on background prior of the present invention.

[0038] Figure 2 It is a flowchart of the present invention for inputting a specific data set with only image-level labels into a background clustering algorithm, performing background clustering according to pixel values, and obtaining a clustering mask graph.

[0039] Figure 3 It is a flowchart of the present invention for inputting the data set into a SAM model pre-trained on a natural image for inference to obtain a pre-trained mask map.

[0040] Figure 4 It is a flowchart of the present invention for performing IOU-based mask fusion on the clustering mask map and the pre-training mask map through the segmentation result fusion module to generate a background mask map.

[0041] Figure 5 It is a flowchart of the present invention using multiple serial multi-scale splicing convolution blocks.

[0042] Figure 6 It is a flow chart of the present invention for realizing the tower extraction of the data set features.

[0043] Figure 7 It is a flow chart of the present invention for distinguishing the foreground and background in the classification feature map using the background mask map.

[0044] Figure 8 It is a flowchart of the present invention that inputs the foreground into a classifier for classification operation, performs visual visualization, and generates a semantic segmentation map.

[0045] Fig. 9 This is a rendering of another embodiment of a weakly supervised semantic segmentation method based on background prior of the present invention.

[0046] Fig.10 It is a schematic diagram of another embodiment of a weakly supervised semantic segmentation method based on background prior of the present invention. DETAILED DESCRIPTION

[0047] See also Figure 1-Figure 8 ,in, Figure 1 It is a flow chart of a weakly supervised semantic segmentation method based on background prior of the present invention. Figure 2 It is a flowchart of the present invention for inputting a specific data set with only image-level labels into a background clustering algorithm, performing background clustering according to pixel values, and obtaining a clustering mask graph. Figure 3 It is a flowchart of the present invention for inputting the data set into a SAM model pre-trained on a natural image for inference to obtain a pre-trained mask map. Figure 4 It is a flowchart of the present invention for performing IOU-based mask fusion on the clustering mask map and the pre-training mask map through the segmentation result fusion module to generate a background mask map. Figure 5 It is a flowchart of the present invention using multiple serial multi-scale splicing convolution blocks. Figure 6 It is a flow chart of the present invention for realizing the tower extraction of the data set features. Figure 7 It is a flow chart of the present invention for distinguishing the foreground and background in the classification feature map using the background mask map. Figure 8 It is a flowchart of the present invention that inputs the foreground into a classifier for classification operation, performs visual visualization, and generates a semantic segmentation map.

[0048] The present invention provides a weakly supervised semantic segmentation method based on background prior, comprising:

[0049] S1 inputs a specific data set with only image-level labels into the background clustering algorithm to perform background clustering based on pixel values ​​to obtain a clustering mask map;

[0050] The specific steps include:

[0051] S11 converts the input image from RGB to grayscale, and deletes the transparency channel of the image;

[0052] S12 arranges pixels from large to small, organizes the pixels contained in the image into a pixel sequence, calculates the distance between the first pixel value and the last pixel value of a fixed value of adjacent pixel sequence, finds the pixel sequence with the largest distance, uses the median of the pixel sequence as the threshold for division, realizes color segmentation, and generates a cluster mask map.

[0053] S2 inputs the data set into a SAM model pre-trained on natural images for inference to obtain a pre-trained mask map;

[0054] The specific steps include:

[0055] S21 encodes the dataset through a weight-frozen image token encoder and saves the feature vector in numpy format;

[0056] S22 uses numpy vector decoding to obtain multiple instance segmentation maps and filters them, deletes instances at the noise level and accumulates segmentation map masks for the remaining instances. When masks overlap, a larger area mask map is used to prevent the loss of target information.

[0057] S3 performs IOU-based mask fusion on the clustering mask map and the pre-trained mask map through a segmentation result fusion module to generate a background mask map;

[0058] The specific steps include:

[0059] S31 uses the clustered mask map and the pre-trained mask map to find the area where the masks intersect;

[0060] S32: When the IOU between the intersection area and the corresponding pre-trained mask area exceeds 0.5, the corresponding area on the pre-trained mask image is selected, and when the IOU is less than 0.5, it is shielded to generate a fused background mask image.

[0061] S4 utilizes multiple serial multi-scale convolution blocks to realize the tower extraction of the features of the data set and generate a classification feature map;

[0062] The specific steps of using multiple serial multi-scale convolution blocks include:

[0063] S41 uses square convolution kernels to perform preliminary feature extraction, and uses horizontal and then vertical group convolution to achieve horizontal and vertical mixed extraction of feature maps in spatial position;

[0064] S42 concatenates the activation features accumulated by horizontal and vertical convolutions at multiple scales, uses point convolution to perform channel dimensionality reduction, and the extracted features are expressed as: P = Pointconv(concat(GELU(Vconv(Hconv(F))))), where F represents the preliminary extracted features, Pointconv represents point convolution, GELU is the activation function, Vconv represents vertical convolution, and Hconv represents horizontal convolution.

[0065] The specific steps of implementing the tower extraction of the data set features include:

[0066] S43 performs batch normalization and convolution kernels from small to large on the input features at each layer;

[0067] S44 performs jump connection on the preliminary processed features to realize the accumulation of horizontal and vertical extracted features and preliminary processed features, outputs the accumulated features using LP pooling, and repeats the above operation multiple times to realize tower-like extraction of features.

[0068] S5: using the background mask image to distinguish the foreground and background in the classification feature image;

[0069] The specific steps include:

[0070] S51 calculates the shortest distance d between each background point and the foreground area according to the foreground point in the background mask image, and obtains a pixel weakening value W=1-d / N, where N is the side length of the mask image;

[0071] S52 multiplies the background feature by the weakening value, and multiplies the foreground feature by 1, to generate a foreground feature map after the background is weakened.

[0072] S6 inputs the foreground into a classifier for classification, performs visual visualization, and generates a semantic segmentation map.

[0073] The specific steps include:

[0074] S61 performs a visual interpretation on the last convolution layer of the classification operation, and uses two hook functions, register_forward_hook and register_backward_hook, to record the forward and backward propagation gradient weights of the last convolution layer;

[0075] S62 uses the gradient weight to generate Grad-CAM and normalizes it to obtain an interpretable weight map. The threshold Seg_threshold is used to select pixels. Pixels exceeding the threshold are set to white, otherwise they are set to black, to obtain a weakly supervised segmentation map.

[0076] like Fig. 9 As shown, in another embodiment of the present invention, a weakly supervised semantic segmentation method based on background prior provided by the present invention also includes: arranging the pixels from large to small to form a pixel sequence of the pixel points contained in the image, finding the pixel sequence with the maximum distance, realizing color segmentation, and generating a clustering mask map; encoding through a weight-frozen image token encoder, saving the feature vector in numpy format, and decoding using numpy vectors to generate a pre-trained mask map.

[0077] like Fig.10As shown, in another embodiment of the present invention, a weakly supervised semantic segmentation method based on background prior provided by the present invention also includes: combining the clustering algorithm, the background prior generated by SAM and the feature map extracted by the backbone network to generate a weakly supervised segmentation image, and the specific process includes: inputting the data set into the background clustering algorithm to perform background clustering according to the pixel value to obtain a clustering mask map; inputting the data set into the SAM model that has been pre-trained on a large scale on natural images for inference to obtain a pre-trained mask map; performing IOU-based mask fusion on the clustering mask map and the pre-trained mask map through the segmentation result fusion module to generate a background mask map; using multiple serial multi-scale splicing convolution blocks to realize the tower extraction of the data set features to generate a classification feature map; using the background mask map to distinguish the foreground and background in the classification feature map; inputting the foreground into the classifier for classification operation, and performing visual visualization to generate a semantic segmentation map.

[0078] The invention discloses a weakly supervised semantic segmentation method based on background prior, which uses background prior knowledge to perform weakly supervised semantic segmentation of images, uses a clustering algorithm and a SAM pre-training model to pre-segment images, reduces the difficulty of weakly supervised tasks, and uses a segmentation result fusion module to implement screening of pre-segmented areas, thereby improving the effectiveness and information quality of foreground segmented areas. The invention fully combines background prior knowledge and large model pre-training prior knowledge to achieve weakly supervised semantic segmentation of images, effectively solves the problems of difficult acquisition of labeled samples, poor weakly supervised segmentation effect, and poor model generalization ability in weakly supervised semantic segmentation in medical imaging, infrared imaging, and remote sensing imaging, effectively improves the segmentation effect and generalization ability of a weakly supervised semantic segmentation model in image segmentation tasks, and reduces the model's dependence on large-scale labeled data.

[0079] What is disclosed above is only one or more preferred embodiments of the present application, and cannot be used to limit the scope of rights of the present application. Ordinary technicians in this field can understand that all or part of the processes of implementing the above embodiments and equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A weakly supervised semantic segmentation method based on background prior, characterized in that: include: Input a specific data set with only image-level labels into the background clustering algorithm to perform background clustering based on pixel values ​​and obtain a clustering mask map; Inputting the data set into a SAM model pre-trained on natural images for inference to obtain a pre-trained mask map; The clustering mask map and the pre-trained mask map are subjected to IOU-based mask fusion through a segmentation result fusion module to generate a background mask map; Using multiple serial multi-scale convolution blocks, the tower-like extraction of the features of the data set is realized to generate a classification feature map; Using the background mask image to distinguish the foreground and background in the classification feature image; Inputting the foreground into a classifier for classification, and performing visual visualization to generate a semantic segmentation map; The specific steps of inputting a specific data set with only image-level labels into a background clustering algorithm to perform background clustering according to pixel values ​​to obtain a clustering mask map include: Convert the input image from RGB to grayscale and delete the transparency channel of the image. By arranging pixels from large to small, the pixels contained in the image are formed into a pixel sequence, the distance between the first pixel value and the last pixel value of a fixed value of adjacent pixel sequence is calculated, the pixel sequence with the largest distance is found, the median value of the pixel sequence is used as the threshold for division, color segmentation is achieved, and a clustering mask image is generated; The specific steps of inputting the data set into the SAM model pre-trained on natural images for inference to obtain the pre-trained mask map include: Encode the dataset through a weight-frozen image token encoder and save the feature vector in numpy format; Use numpy vector decoding to obtain multiple instance segmentation maps and filter them, delete the noise level instances and accumulate the segmentation map masks for the remaining instances. When the masks overlap, use the mask map to prevent the loss of target information.

2. The weakly supervised semantic segmentation method based on background prior according to claim 1, characterized in that: The specific steps of performing mask fusion based on IOU on the clustering mask map and the pre-trained mask map through the segmentation result fusion module to generate a background mask map include: Use the clustered mask map and the pre-trained mask map to find the area where the masks intersect; When the IOU between the intersection area and the corresponding pre-trained mask area exceeds 0.5, the corresponding area on the pre-trained mask map is selected, and when the IOU is less than 0.5, it is shielded to generate a fused background mask map.

3. The weakly supervised semantic segmentation method based on background prior according to claim 1, characterized in that: The specific steps of using multiple serial multi-scale convolution blocks include: Use square convolution kernels to perform preliminary feature extraction, and use horizontal and vertical group convolution to achieve horizontal and vertical mixed extraction of feature maps in spatial position; The activation features accumulated from the horizontal and vertical convolutions at multiple scales are concatenated, and point convolution is used to perform channel dimensionality reduction.

4. The weakly supervised semantic segmentation method based on background prior according to claim 1, characterized in that: The specific steps of implementing the tower extraction of the data set features include: At each layer, the input features are initially processed by batch normalization and convolution kernels from small to large; The preliminary processed features are skipped to realize the accumulation of horizontal and vertical extracted features and preliminary processed features, and the accumulated features are output using LP pooling. The above operation is repeated many times to realize the tower extraction of features.

5. The weakly supervised semantic segmentation method based on background prior according to claim 1, characterized in that: The specific step of using the background mask image to distinguish the foreground and background in the classification feature image includes: According to the foreground points in the background mask image, the shortest distance d between each background point and the foreground area is calculated, and the pixel weakening value W = 1-d / N is obtained, where N is the side length of the mask image; The background feature is multiplied by the weakening value, and the foreground feature is multiplied by 1 to generate a foreground feature map after the background is weakened.

6. The weakly supervised semantic segmentation method based on background prior according to claim 1, characterized in that: The specific steps of inputting the foreground into the classifier for classification operation and performing visual visualization to generate a semantic segmentation map include: Provide a visual explanation of the last convolutional layer of the classification operation, and use the register_forward_hook and register_backward_hook hook functions to record the forward and backward propagation gradient weights of the last convolution layer; The Grad-CAM is generated using the gradient weights and normalized to obtain an interpretable weight map. The threshold Seg_threshold is used to select pixels. Pixels exceeding the threshold are set to white, otherwise they are set to black to obtain a weakly supervised segmentation map.