Weakly supervised image semantic segmentation method and system based on multi-scale category prototype
By fusing semantic information at different granular levels through a multi-scale category prototype mechanism, the problem of insufficient multi-class location and semantic mining in weakly supervised image semantic segmentation is solved, and more efficient target localization and segmentation results are achieved.
Patent Information
- Application Number
- CN202211215498.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-28
- Publication Date
- 2026-02-03
- Estimated Expiration
- 2042-09-28
AI Technical Summary
Existing weakly supervised image semantic segmentation methods have shortcomings in multi-class location and semantic mining, and cannot effectively integrate multi-scale semantic information, resulting in poor feature representation ability and difficulty in accurately locating targets.
A multi-scale category prototype mechanism is adopted to discover category location and semantics from image feature maps. Feature map fusion and classification learning are performed through three sets of prototype vectors. The model is optimized by cross-entropy loss to generate category segmentation maps.
It improves the ability of weakly supervised pre-classification tasks to locate targets, achieves more accurate semantic segmentation, and enhances the model's generalization ability in test images.
Smart Images
Figure CN115482387B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and in particular to a weakly supervised image semantic segmentation method and system based on multi-scale category prototypes. Background Technology
[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.
[0003] Semantic segmentation aims to assign a semantic category to each pixel in an image. It has consistently been one of the most important tasks in computer vision, with wide applications such as image editing and scene understanding. Recently, methods based on deep convolutional neural networks have been used for semantic segmentation and have achieved significant progress. However, this approach relies on learning supervised models that require pixel-by-pixel annotation, consuming considerable effort and time. To reduce the workload of annotating pixel-level ground truth labels, many weakly supervised methods using various types of labels have been proposed, such as image-level annotation, video-level annotation, bounding box annotation, point-level annotation, and graffiti-based annotation. Among these, image-level annotation is one of the most economical and efficient settings.
[0004] However, the location and size of image-level labeled objects are unknown, so it is necessary to infer the semantic information and pixel location of the objects. Existing methods have too simplistic a form of information category mining and cannot achieve multi-category location and semantic mining. In addition, after traditional methods determine the semantic information, they cannot achieve multi-scale semantic information fusion, resulting in large semantic gaps and poor feature representation capabilities, making it difficult to locate the target. Summary of the Invention
[0005] To address the shortcomings of existing technologies, the present invention aims to provide a weakly supervised image semantic segmentation method and system based on multi-scale category prototypes. This method uses a category prototype mechanism to discover the location and semantics of categories from image feature maps and uses multi-scale prototypes to fuse semantic information at different granularities, thereby improving the target localization capability of weakly supervised pre-classification tasks.
[0006] To achieve the above objectives, the present invention is implemented through the following technical solution:
[0007] The first aspect of this invention provides a weakly supervised image semantic segmentation method based on multi-scale category prototypes, comprising the following steps:
[0008] Collect images to be segmented, label the category of each image, and preprocess the images; input the images into a feature extraction network to obtain feature maps;
[0009] Set up three sets of prototype vectors and initialize the prototypes;
[0010] The target pixel is located using the prototype to generate a category feature map. The specific steps are as follows: the prototype is used to perform a dot product on the feature map pixel by pixel, and each prototype generates a one-channel feature map.
[0011] The class feature maps are fused and used for image classification learning. The specific process of class feature map fusion is as follows: three sets of prototypes generate three class feature maps; the three class feature maps are scaled and then harmonicly added, and then fed into a convolution to obtain the processed feature map.
[0012] The category activation map is extracted from the category feature map to obtain the image category mask, and then the category segmentation map is obtained.
[0013] Further, the specific steps for image preprocessing are as follows: scale the image size proportionally and regularize each pixel element.
[0014] Furthermore, the specific steps for initializing the prototypes are as follows: initialize three sets of prototype vectors using random numbers, each set containing the total number of category prototypes.
[0015] Furthermore, the specific steps of image classification learning are as follows: global average pooling is performed on the obtained convolutional feature map to obtain a feature vector, and the cross-entropy loss is directly calculated using the feature vector.
[0016] Furthermore, a class activation map algorithm is used to extract the class activation map, and an image class mask of the original image size is obtained by upsampling.
[0017] A second aspect of the present invention provides a weakly supervised image semantic segmentation system based on multi-scale category prototypes, comprising:
[0018] The image processing module is configured to collect images to be segmented, label the category of each image, preprocess the images, and input the images into the feature extraction network to obtain feature maps.
[0019] The prototype building module is configured to set up three sets of prototype vectors and initialize the prototypes;
[0020] The target pixel localization module is configured to use prototypes to locate target pixels in the feature map and generate category feature maps. The specific steps are: use prototypes to perform dot multiplication on the feature map pixel by pixel, and each prototype generates a one-channel feature map.
[0021] The classification learning module is configured to fuse category feature maps and perform image classification learning. The specific process of category feature map fusion is as follows: three sets of prototypes generate three category feature maps; the three category feature maps are scaled and then harmonicly added, and then fed into a convolution to obtain the processed feature map.
[0022] The category segmentation map generation module is configured to extract the category activation map from the category feature map to obtain the image category mask, and then obtain the category segmentation map.
[0023] Furthermore, the image processing module is configured to scale the image size proportionally and regularize each pixel element when preprocessing the image.
[0024] Furthermore, the prototype building module is configured to initialize three sets of prototype vectors using random numbers when initializing the prototype, with each set containing a total number of category prototypes.
[0025] Furthermore, when performing image classification learning, the classification learning module is configured to perform global average pooling on the convolutional feature map to obtain a feature vector, and then directly use the feature vector to calculate the cross-entropy loss.
[0026] Furthermore, the category segmentation map generation module is also configured to extract the category activation map using a class activation map algorithm, and obtain the image category mask of the original image size through upsampling.
[0027] The beneficial effects of the above embodiments of the present invention are as follows:
[0028] This invention discloses a weakly supervised image semantic segmentation method based on multi-scale category prototypes. By using a category prototype mechanism, the method discovers the location and semantics of categories from image feature maps, accurately inferring the speech information and pixel location of objects. After extracting semantic information, the use of multi-scale prototypes can better fuse semantic information at different granularities, greatly improving the target localization ability of the weakly supervised pre-classification task.
[0029] This invention learns a set of semantic prototypes to learn the ability to locate common features from similar images, thereby maintaining a high localization ability when generalizing to test images, and thus better achieving semantic segmentation of weakly supervised images. Attached Figure Description
[0030] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0031] Figure 1 This is a schematic diagram of the overall process of the weakly supervised image semantic segmentation method based on multi-scale category prototypes in Embodiment 1 of the present invention;
[0032] Figure 2 This is a schematic diagram illustrating the process of obtaining a category segmentation map in the weakly supervised image semantic segmentation method based on multi-scale category prototypes according to Embodiment 1 of the present invention. Detailed implementation method:
[0033] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.
[0034] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0035] Example 1:
[0036] Embodiment 1 of this invention provides a weakly supervised image semantic segmentation method based on multi-scale category prototypes, such as... Figure 1 As shown, the image to be processed is input into a convolutional neural network, resulting in three sets of feature map shapes f1, f2, and f3. f1, f2, and f3 are then multiplied pixel-by-pixel with a predefined vector length for localization, with corresponding relationships P. [1] Position f1, P [2] Position f2, P [3] The f3 is located, and the three prototypes generate three category feature maps M1, M2, and M3. The category feature maps are fused and fed into a 1*1 convolution to obtain feature map Z. Global average pooling is performed on Z to obtain the classification score, and cross-entropy loss is calculated.
[0037] Furthermore, the specific process for obtaining the category segmentation map is as follows: Figure 2 As shown, the category feature maps are fused and then fed into a 1*1 convolution to obtain the feature map Z. Global average pooling is performed on Z to obtain the classification score. Backpropagation is performed from the category with the highest score to obtain the gradient of the feature map before the input 1*1 convolution, thus obtaining the category segmentation map.
[0038] Specifically, the following steps are included:
[0039] S1: Collect the images to be segmented, label the category of each image, preprocess the images, and input the images into the feature extraction network to obtain feature maps.
[0040] S1.1: Convert the image to RGB three-channel, scale the long side to 512, shrink the short side proportionally, and fill any part smaller than 512 with 0.
[0041] S1.2: The RGB three channels are regularized using 0.485, 0.456, and 0.406 as the mean and 0.229, 0.224, and 0.225 as the variance, respectively.
[0042] S2: Set up three sets of prototype vectors and initialize the prototypes.
[0043] S2.1: Initialize three sets of prototype vectors using random numbers, each set containing the total number of class prototypes (class_num).
[0044] Preferably, the elements of each prototype group are configured to be updatable through backpropagation. The vector lengths are 512 (for the first group P). [1] ), 1024 (Group 2 P) [2] ), 2048 (Group 3 P) [3] The number of prototypes in each set is the same as the total number of categories. Each vector can be learned through gradient descent.
[0045] S3: Use the prototype to locate target pixels in the feature map and generate a category feature map.
[0046] S3.1: Remove the last three layers (which are fully connected layers) from the ResNet50 network, keeping only the convolutional layers.
[0047] S3.2: Input the image into the ResNet50 network and obtain the feature maps of the last three convolutional layers of ResNet50.
[0048] Preferably, the feature map shapes are 28*28*2048 (f3), 56*56*1024 (f2), and 112*112*512 (f1).
[0049] S3.3: Perform dot product on the feature map pixel by pixel using the prototype <P [1] f1>location, the corresponding relationships are P respectively [1] Position f1, P [2] Position f2, P [3] Localize f3. Each prototype generates a one-channel feature map. That is:
[0050] M i = <P [i] f i > i∈{1, 2, ..., n}
[0051] Where M is the localization map obtained by the interaction between the prototype and the feature map, P is the prototype, f is the feature map of the network, i represents the i-th feature map, and n is the number of feature maps, which is equal to 3 here.
[0052] S4: Multi-scale Prototype Feature Interaction: Features generated from the three prototypes interact in pairs. During the interaction, the smaller feature map is upsampled. The interaction method uses cross-attention. The smaller feature map is used as the query, and the larger feature map is used as the key and value. Then, three sets of interactive feature maps are obtained.
[0053] S4.1: The three prototypes generate three categorical feature maps M1, M2, and M3. M1 and M2 are first cross-attentioned twice, with M2 as query Q and M1 as key K and value V, resulting in Z1. M2 and M3 undergo the same operation, with M3 as query Q and M2 as key K and value V, resulting in Z2 after two cross-attention cycles. Similarly, M1 and M3 are cross-attentioned twice, with M3 as query Q and M1 as key K and value V, resulting in Z3.
[0054] S5: Fuse the category feature maps and perform image classification learning. Specifically, image classification learning involves global max pooling of the three interacting feature maps to obtain three feature vectors. The three feature maps are harmonicly summed to obtain a new fused feature map, which is then subjected to global max pooling to obtain the fused feature vector. The cross-entropy loss of the four feature maps is calculated separately to train the network.
[0055] S5.1: The three prototypes generated three categorical feature maps M1, M2, and M3. After scaling all three to 112x112 pixels, they were harmonic-matched and added together.
[0056]
[0057]
[0058] Where n is the number of feature maps, This represents the feature map after scaling, where i represents the i-th feature map. This is the feature map after harmonic addition.
[0059] S5.2: Will The feature map Z is obtained by processing the input through a 1x1 convolution:
[0060]
[0061] S5.3: Perform global max pooling on the processed feature map Z obtained in S5.2 and use softmax to obtain the classification score:
[0062]
[0063] Where max H, W() represents taking the maximum value in the spatial dimension, and H and W represent the width and height of the feature map Z, respectively. The classification score is obtained by average pooling the feature map Z, where i represents the i-th feature map.
[0064] S5.4: Calculate cross-entropy loss
[0065]
[0066] By reducing the predicted classification score The difference between the model and the true label y is used to provide a supervisory signal by employing cross-entropy as the loss function to optimize the model.
[0067] S6: Extract the category activation map from the category feature map, obtain the image category mask through upsampling, and then obtain the category segmentation map.
[0068] S6.1: Use the class activation map (grad-cam) algorithm to backpropagate from the class with the highest score to obtain the gradient of the feature map before the input 1*1 convolution in Z4, and then use the grad-cam algorithm to obtain the class segmentation map.
[0069] Example 2:
[0070] Embodiment 2 of the present invention provides a weakly supervised image semantic segmentation system based on multi-scale category prototypes, comprising:
[0071] The image processing module is configured to collect images to be segmented, label the category of each image, preprocess the images, and input the images into the feature extraction network to obtain feature maps.
[0072] The prototype building module is configured to set up three sets of prototype vectors and initialize the prototypes;
[0073] The target pixel localization module is configured to use prototypes to locate target pixels in the feature map and generate category feature maps. The specific steps are: use prototypes to perform dot multiplication on the feature map pixel by pixel, and each prototype generates a one-channel feature map.
[0074] The classification learning module is configured to fuse category feature maps and perform image classification learning. The specific process of category feature map fusion is as follows: three sets of prototypes generate three category feature maps; the three category feature maps are scaled and then harmonicly added, and then fed into a convolution to obtain the processed feature map.
[0075] The category segmentation map generation module is configured to extract the category activation map from the category feature map to obtain the image category mask, and then obtain the category segmentation map.
[0076] The image processing module is configured to scale the image size proportionally and regularize each pixel element when preprocessing the image.
[0077] When initializing the prototype, the prototype building module is configured to initialize three sets of prototype vectors using random numbers, each set containing a total number of category prototypes.
[0078] When performing image classification learning, the classification learning module is configured to perform global average pooling on the convolutional feature map to obtain a feature vector, and then directly use the feature vector to calculate the cross-entropy loss.
[0079] The category segmentation map generation module is also configured to extract the category activation map using a class activation map algorithm and obtain the image category mask of the original image size through upsampling.
[0080] The detailed steps are the same as those of the weakly supervised image semantic segmentation method based on multi-scale category prototypes provided in Example 1, and will not be repeated here.
[0081] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.
[0082] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A weakly supervised image semantic segmentation method based on multi-scale category prototypes, characterized in that, Includes the following steps: Collect images to be segmented, label the category of each image, and preprocess the images; input the images into a feature extraction network to obtain feature maps; Set up three sets of prototype vectors and initialize the prototypes. Use random numbers to initialize the three sets of prototype vectors. Each set contains the total number of categories and category prototypes. The prototype is used to locate target pixels in the feature map to generate a category feature map. The specific steps are as follows: the prototype is used to perform a dot product on the feature map pixel by pixel, and each prototype generates a one-channel feature map. The class feature maps are fused and used for image classification learning. The specific process of class feature map fusion is as follows: three sets of prototypes generate three class feature maps; the three class feature maps are scaled and then harmonicly added, and then fed into a convolution to obtain the processed feature map. In this process, the three category feature maps are scaled and then harmonicly summed, specifically as follows: ； ； Where n is the number of feature maps, This represents the feature map after scaling, where i represents the i-th feature map. The feature map after harmonic addition; The class activation map is extracted from the class feature map to obtain the image class mask, and then the class segmentation map is obtained. The class activation map algorithm is used to backpropagate from the class with the highest score to obtain the gradient of the input feature map before convolution. Then, the class activation map algorithm is used to extract the class activation map, and the image class mask of the original image size is obtained by upsampling.
2. The weakly supervised image semantic segmentation method based on multi-scale category prototypes as described in claim 1, characterized in that, The specific steps for image preprocessing are as follows: scale the image size proportionally and regularize each pixel element.
3. The weakly supervised image semantic segmentation method based on multi-scale category prototypes as described in claim 1, characterized in that, The specific steps of image classification learning are as follows: global average pooling is performed on the obtained convolutional feature map to obtain the feature vector, and the cross-entropy loss is directly calculated using the feature vector.
4. A weakly supervised image semantic segmentation system based on multi-scale category prototypes, characterized in that, include: The image processing module is configured to collect images to be segmented, label the category of each image, preprocess the images, and input the images into the feature extraction network to obtain feature maps. The prototype building module is configured to set up three sets of prototype vectors and initialize the prototypes. The three sets of prototype vectors are initialized with random numbers, and each set contains a total number of category prototypes. The target pixel localization module is configured to use prototypes to locate target pixels in the feature map and generate category feature maps. The specific steps are: use prototypes to perform dot multiplication on the feature map pixel by pixel, and each prototype generates a one-channel feature map. The classification learning module is configured to fuse category feature maps and perform image classification learning. The specific process of category feature map fusion is as follows: three sets of prototypes generate three category feature maps; the three category feature maps are scaled and then harmonicly added, and then fed into a convolution to obtain the processed feature map. In this process, the three category feature maps are scaled and then harmonicly summed, specifically as follows: ； ； Where n is the number of feature maps, This represents the feature map after scaling, where i represents the i-th feature map. The feature map after harmonic addition; The category segmentation map generation module is configured to extract the category activation map from the category feature map to obtain the image category mask, and then obtain the category segmentation map. Specifically, the class activation map algorithm is used to backpropagate from the category with the highest score to obtain the gradient of the input feature map before convolution. Then, the class activation map algorithm is used to extract the category activation map, and the image category mask of the original image size is obtained by upsampling.
5. The weakly supervised image semantic segmentation system based on multi-scale category prototypes as described in claim 4, characterized in that, The image processing module is configured to scale the image size proportionally and regularize each pixel element when preprocessing the image.
6. The weakly supervised image semantic segmentation system based on multi-scale category prototypes as described in claim 4, characterized in that, When performing image classification learning, the classification learning module is configured to perform global average pooling on the convolutional feature map to obtain a feature vector, and then directly use the feature vector to calculate the cross-entropy loss.