Weak supervision semantic segmentation method based on category representation comparison and aggregation

Abstract

The invention discloses a weak supervision semantic segmentation method based on category representation comparison and aggregation, and the method comprises the steps: firstly carrying out the image enhancement preprocessing of N images containing C target categories; secondly, introducing a category characterization concept, constructing a category characterization extraction module, obtaining category characterization based on the preprocessed image, constructing a category characterization aggregation module, aggregating the category characterization, and obtaining category characterization aggregation features; then, carrying out category characterization on the aggregated features, and generating a pseudo tag and a category score through normalized global weighted pooling; and finally, constructing a segmentation model, and outputting a semantic segmentation result based on the pseudo tag. According to the method, the error activation of the error position is effectively inhibited, the generation quality of the seed region is effectively improved, and the semantic segmentation correctness is further improved.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology and relates to a weakly supervised semantic segmentation method based on image-level annotation, which is commonly used in scene understanding, autonomous driving, medical image segmentation, etc. Background Art

[0002] With its widespread application in fields such as autonomous driving, robotics, human-computer interaction, and medical imaging analysis, semantic segmentation is becoming increasingly important in computer vision. Although fully supervised semantic segmentation has made significant progress, its training requires pixel-level labels. Due to the difficulty in obtaining pixel-level labels, the use cases of fully supervised semantic segmentation are limited. However, weakly supervised semantic segmentation can address this shortcoming. Weakly supervised semantic segmentation can utilize simple annotation information, such as image-level annotations and point annotations, to complete the semantic segmentation task, greatly reducing the difficulty of label acquisition for semantic segmentation tasks. There have been many successful studies based on weakly supervised semantic segmentation, but the current segmentation accuracy is still insufficient.

[0003] Currently, weakly supervised semantic segmentation based on deep learning generally follows a two-stage process: In the first stage, class activation mapping is used to obtain an initial seed region to preliminarily locate the segmentation target. This seed region is then optimized to obtain higher-quality pseudo-labels for the next stage of training. In the second stage, the pseudo-labels generated in the first stage and the images are used to train the segmentation model.

[0004] Since class activation mapping often only gives the most discriminative areas, it often fails to cover the segmentation target, leading to under-activation problems. Methods such as adversarial erasure and CAM integration proposed by Wei Yunchao are intended to solve this type of problem. However, these methods often introduce other problems, such as incorrect activation of non-target categories. This is called over-activation. At present, the main methods to solve the overfitting problem are based on the co-occurrence relationship of images, training additional inference frameworks, or performing corresponding enhancements based on the characteristics of the training set. Over-activation and under-activation problems go hand in hand. Past research often focused on only one of the problems, but failed to suppress the other problem or even caused the other problem to arise. How to coordinate the over-activation and under-activation problems to achieve an optimal state to improve the accuracy of weakly supervised semantic segmentation remains a key issue that needs to be solved in the field of weakly supervised semantic segmentation. Summary of the Invention

[0005] This paper provides a weakly supervised semantic segmentation method based on category representation attention to address the under-activation and over-activation issues in related technologies. While introducing cross-image pixel affinity based on category representation, it also optimizes the CAP function to adapt to objects of different sizes, thereby achieving the goal of improving segmentation accuracy. The technical solution is as follows.

[0006] A weakly supervised semantic segmentation method based on category representation attention is a two-stage weakly supervised semantic segmentation method. The first stage is to generate seed regions. To solve the problems of underfitting and overfitting, it includes a category representation extraction module, a category representation aggregation module, and a pseudo-label generation module. The second stage is the semantic segmentation pseudo-label training stage, which contains a segmentation model, where:

[0007] The category representation extraction module includes:

[0008] Step 1: Obtain training and test sample sets: Label the target categories in N images containing a total of C target categories and divide them into training set, test set, and cross-validation set.

[0009] Step 2: Image preprocessing: Perform a series of image enhancements on the image from step 1, including adjusting the image size, randomly changing the image brightness, contrast, saturation, etc. Based on the prior knowledge that weak enhancement does not change the distribution of image features, the robustness of the model is enhanced and the number of training sets is expanded.

[0010] Step 3: By introducing the concept of category representation, a category representation extraction module is constructed to obtain category representation based on the preprocessed image: the preprocessed image is input into a multi-scale fully-convolutional network (MSFCN). The network mainly realizes the extraction of multi-scale features through void convolution. The image is input into the network to obtain an image feature mask. The image feature mask is then input into the CAM module to generate an attention map. The attention map is then input into c category feature extractors to obtain c category features, each category representation representing the category feature of the corresponding category of the training image in this round of training. Further, the c categories are input into the mask average pooling to obtain c category representations of the image. Furthermore, based on the semantic information of the category representation, a non-parametric dynamic memory library M is established for storage. The category representation in the memory library is gradually updated by the momentum formula. The present invention collects the category semantic information of the entire data set, i.e., the category representation, in the form of a non-parametric dynamic memory library, and integrates the category representation of the entire data set into the training process of each image in each batch. Furthermore, by comparing the category representations in the memory library and the category representations obtained in the training process, the distance with the same category is shortened, that is, the similarity is increased, and the distance with the representations of different categories is expanded, that is, the similarity is reduced. The loss function is the NCE loss. On this basis, in order to enhance the robustness of the training model, the present invention also incorporates the negative samples of the image, that is, the category representations that do not appear in the true label of the image, into the training. The NCE loss is calculated for the positive sample of the image and the positive sample in the non-parametric dynamic library, and the NCE loss is calculated for the negative sample of the image and the negative sample in the non-parametric dynamic memory library. The mixed loss is called CRM-NCE loss.

[0011] Step four, construct a category representation aggregation module, aggregate the category representation, and obtain the category representation aggregation feature: In step three, the present invention obtains a non-parametric dynamic memory library that stores the category representation of the entire data set, and passes it through the K-Mean clustering algorithm to obtain several category representation centroids, which are spliced ​​into a matrix shape. Then all the centroids are spliced ​​together to obtain its overall category representation Q. Q contains the category representation aggregation information of the entire data set, and integrating it into the training process will bring rich cross-image semantics. The specific fusion process is as follows: first, the features of each image I in this training batch are matrix multiplied with the overall category representation Q, and then a softmax function is performed to obtain the category representation attention matrix S. Further, the category representation affinity matrix F is calculated by matrix multiplication through the category representation attention matrix S and the overall category representation Q. ′ Finally, concatenate the image features F and the category representation affinity matrix F ′ Get category representation aggregation features

[0012] Step 5: Generate pseudo-labels and category scores through normalized global weighted pooling: The aggregated feature map generated in step 4 is input into the normalized global weighted pooling module to generate pseudo-labels and category scores. The normalized global weighted pooling function optimizes the large discrepancy in the global average pooling function's recognition accuracy for objects of different sizes. Finally, a mixed contrast loss and category comparison loss are applied, where the category comparison loss is the cross-entropy loss between the generated category scores and the true labels. This loss function is then trained through backpropagation. The localization model M1, consisting of a category representation extraction module, a category representation extraction module, a category representation aggregation module, and a segmentation module, is trained through repeated backpropagation until the localization model M1 converges.

[0013] Step 6: Build a segmentation model and output semantic segmentation results based on pseudo-labels. A pre-trained ResNet38 is used as the segmentation model. First, a batch of images I is fed into the localization model M1 trained in Step 5 to generate pseudo-labels. These pseudo-labels are fed into the segmentation model along with images I as the true labels. Backpropagation is performed using the cross-entropy loss. This process is repeated until the segmentation model converges. Finally, segmentation performance is evaluated using the mIoU metric using samples from the validation set.

[0014] The main advantages of this invention compared to other methods are:

[0015] 1. A low-complexity multi-scale fully connected network is designed to obtain image features that aggregate receptive fields of various sizes at a lower cost.

[0016] 2. A category representation extraction module and mask average pooling were designed, and a non-parametric dynamic memory library of category representation based on the entire dataset was constructed based on contrastive learning.

[0017] 3. We designed a category representation aggregation module based on the K-Mean clustering algorithm. The category representation provides category supervision for the entire dataset during training, effectively suppressing incorrect activations at the wrong locations, effectively improving the quality of seed region generation and thus enhancing semantic segmentation accuracy.

[0018] 4. A normalized global pooling function was designed to optimize the accuracy of small target positioning, avoid the huge degradation effect caused by a small amount of erroneous pseudo-labels in the training process, and improve the final segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly present the embodiments and technical solutions of the present application, the services required in the embodiments and technical solutions will be briefly introduced below.

[0020] Figure 1 This is a flowchart of a weakly supervised semantic segmentation method based on category representation comparison and aggregation proposed in the present invention;

[0021] Figure 2 Schematic diagram of a framework of a weakly supervised semantic segmentation method based on category representation comparison and aggregation proposed in the present invention;

[0022] Figure 3 : This is a network structure diagram of the multi-scale fully connected network proposed in the present invention;

[0023] Figure 4 This is a detailed explanation of the normalized global weighted pooling module proposed in this invention;

[0024] Figure 5 This is a comparison of the class activation effects of the multi-scale fully connected network of the present invention and a similar method (FCN);

[0025] Figure 6 This is a comparison of the segmentation effect of the present invention with other methods in the PASCAL VOC2012 dataset. DETAILED DESCRIPTION

[0026] This chapter will provide a more detailed and complete description of the technical solutions, objectives, embodiments, and advantages of the present invention in conjunction with the accompanying drawings. The embodiments described are only a portion of the embodiments of this specification. The method of the present invention is described in detail below.

[0027] Figure 1 This is a flowchart of a weakly supervised semantic segmentation method based on category representation attention provided in this manual. The overall architecture is as follows Figure 2 As shown, the specific steps include:

[0028] Step 1: Obtain sample images and label them with image-level classification labels. Finally, the images are divided into training set, test set, and sample set in a ratio of 8:1:1.

[0029] Step 2: Preprocess the training set, including resizing the images, randomly changing their brightness, contrast, and saturation, and finally converting them into the tensor format required by PyTorch. This step aims to resize the images to fit the model input and enhance the robustness of the model based on the prior knowledge that weak enhancement does not change the distribution of image features.

[0030] Step 3: By introducing the concept of category representation, construct a category representation extraction module corresponding to Figure 2 Category representation extraction module. The preprocessed image I obtained in step 2 is input into the multi-scale fully-convolutional network (MSFCN), whose formula is denoted as f msfcn Experiments have shown that compared with FCN, the class activation map obtained from MSFCN is more complete, and the over-activation phenomenon is significantly suppressed for small targets, such as Figure 5 The formula is as follows.

[0031]

[0032] It is then input into CAM to generate the attention map, which is expressed as follows:

[0033]

[0034] The multi-scale fully connected network consists of convolution kernels, average pooling and dilated convolution kernels. Figure 3 As shown, it usually corresponds to the convolution part of the classifier of other methods (such as VGG, ResNet). The output result F is a dense embedding representation of I. CAM It is a class-aware convolutional layer that generates P = [P1, P2, ..., P C ], where each mapping represents the network activation of the cth class. Next, a score vector is obtained from P through global average pooling (GAP) where p c =GAP(P c ) is the unnormalized score of class c.

[0035] Then input P into the category representation extraction module. Further, the feature F is input into c category feature extractors, where the category feature extractor consists of 3 groups of 3*3 convolution and ReLU functions with input of 512 and output of 512. C category features F are obtained. c , and its aggregation is expressed as Furthermore, each category feature and P are input into masked average pooling, and the formula is as follows:

[0037]

[0038] Among them, M c ={1(·)}(P c >μ)∈{0,1} {W×H} is a binary mask that represents the class c in p c 1(·) is an indicator function whose threshold μ is set to P c Here f c It is a category representation that contains a large amount of category semantic information.

[0039] Furthermore, a non-parametric dynamic memory library M is established to store these category representations. The memory library M consists of C dictionaries, namely M = [M1, M2, ..., M c ], each dictionary corresponds to a category, M c Each entry of M represents the representation of the i-th category in the image observed during the entire learning phase. i ∈R D (Entire dataset). Image L i The category representation will be gradually updated through the momentum formula

[0040] m i ←γm i +(1-γ)f i (4)

[0041] Where γ represents the moving value of memory momentum, which is usually set to 0.99. i is the category representation of category i recorded in the non-parametric dynamic memory library. i is the category representation obtained by extracting the category representation of image I. Only when the corresponding i-th category appears in this image and the predicted category score is higher than the threshold v, m is predicted. i Perform a momentum update, v is usually set to 0.75.

[0042] Then perform contrastive learning. For each image I, f cThe purpose of contrastive learning is to increase the similarity of representations of the same category in the memory bank, while reducing the similarity with representations of different categories. The contrastive loss is as follows:

[0043]

[0044] τ is a hyperparameter used to control the rate of change of similarity. sim(i, j) is the L2 regularized product of i and j e is an exponential function. is the category representation of category c stored in the non-parametric dynamic memory library, i.e., the positive sample. The category representation stored in the non-parametric dynamic memory library is not c, that is, the negative sample. is the category representation f c The similarity between it and the positive sample, is the category representation f c The similarity between and the positive sample. The optimization goal is to make As small as possible, it can be seen that as the distance from the negative sample increases, the distance from the positive sample gets closer, and the loss function can meet the training objectives.

[0045] It can be observed that this loss function still belongs to the supervised range, but since the labels here are weak labels, it brings great robustness challenges to the learning of category representation. The present invention mixes the category representations f of other images in the same training batch. c- ,use Perform contrastive learning, the formula is as follows

[0046]

[0047] Where w is a hyperparameter, set to 0.99. c Represents the category representation of this training image, f c- is the category representation of other images in the same batch. Then the category representation mixed contrast loss is defined as follows:

[0048]

[0049] Step 4: Construct a category representation aggregation module to aggregate category representations, corresponding to Figure 2 Category representation aggregation module. In the present invention, a category representation memory is obtained through the S3 step. The category representation memory can bring rich cross-image semantics to the training process, but it contains overly complete or redundant category representations. Directly aggregating large-scale category representations often brings some side effects in practice, greatly reducing the learning and reasoning process. c Perform K-mean clustering on all features in the dataset to obtain the centroid of K categories and organize them into a matrix form. Concatenate the centroids of the category representations of all categories to get the overall representation Then, the features of each image I in this training batch Calculate its category representation attention matrix S, and its calculation formula is as follows:

[0050]

[0051] in and For the convenience of calculation, it will be compressed from three dimensions to two dimensions. Represents matrix multiplication. Each of the formula calculation results S reflects each row of feature F and Q T The normalized similarity between each column (i.e., category representation) in S can be further calculated based on the representation affinity matrix F. ′ :

[0052]

[0053] Where F' is an enhanced expression of F that aggregates the category representation. Then F ′ Reconvert to a three-dimensional W×H×D structure. Then F ′ Combine it with F to get

[0054]

[0055] It not only has the image features of image I, but also integrates the category representation of the entire dataset, thus enriching the semantics. Pass it into CAM to get the final class activation map O:

[0056]

[0057] Step 5: Generate pseudo labels and category scores through normalized global weighted pooling. Figure 4 As shown, first get the The score map S is obtained through a 1*1 convolution score , and then S score Generate confidence mask M through softmax normalization mask , its formula is as follows

[0058]

[0059] Where conv represents a 1*1 convolution operation. softmax represents the softmax function. Then, the confidence mask M is obtained. mask , which is a pseudo label. Input into the normalized global adaptive weighted pooling, the formula is as follows

[0060]

[0061] Where ε is a hyperparameter set to 1, Represents the probability that image I may contain category c. c，w，h It's S score The probability that the pixel at position (w, h) is of category c. c，w，h It's M mask The normalized probability of the pixel at position (w, h) being of category c. The GAP-CAM method studied by other researchers incorporates all pixels equally into the calculation of category probabilities, resulting in under-activation of small targets and over-activation of large targets. nGWP solves this problem well, allowing both large and small targets to obtain an appropriate category probability score. Its overall process is as follows: Figure 5 As shown. Based on the above, the loss function L of the first stage training is constructed:

[0062] L=∑ I α1L CRM-NCE +α2L CE (Y nGWP ,y)+L CE (GAP(O), y) (14)

[0063] Where y is the real category one-hot real label given by the training set, One-hot vector of shape. L CE is the cross entropy loss. α1 and α2 are hyperparameters, which in this example are 0.3 and 0.7, respectively. Backpropagate the loss function L and iterate the training. Repeat the above process until the network converges.

[0065] Step six, construct a segmentation module. The backbone network used in the present invention is ResNet38. ResNet38 is composed of several residual blocks. The residual network is composed of countless residual blocks. The residual block is composed of multiple cascaded convolutional layers and a jump connection. After accumulating the outputs of the two, the output of the residual block is obtained through the ReLU activation layer. Multiple residual blocks can be connected in series to achieve a deeper network. In order to meet the needs of the segmentation network, a convolutional layer with a dimension of c+1 is finally connected to the ResNet38 network. The model composed of resnet38 and the last layer of convolution is denoted as M2. The image I is input into the model M1 trained by S5 to obtain a pseudo label Y. Y is a pixel-level pseudo label, that is, all its pixels are assigned a corresponding category. Then, the image I is input into M2 to obtain a prediction, and the prediction is restored to the original size of the image I through the bilinear interpolation algorithm and is denoted as It is called the category probability map. Then calculate Y, The cross entropy loss between is as follows:

[0066]

[0067] where y c is the one-hot vector from the pseudo-label Y, is the prediction obtained by M2, and its value means the probability that the pixel belongs to class C. The loss function is used to backpropagate the training model M2. Then the above process is repeated until the model converges to obtain the segmentation model M2. Then, the test set image is input to the model M2 to obtain its class probability map Each pixel is then assigned the category with the highest probability, yielding the final semantic segmentation result. Finally, the semantic segmentation performance is evaluated using the mIoU (mean Intersection Over Union) comparison between this final semantic segmentation result and the true label. In an embodiment using the PASCAL VOC2012 dataset, the present invention achieved an mIoU of 68.7%, which is superior to other similar methods. As shown in Table 1, when both methods use ResNet38 as the backbone network, the segmentation performance (mIoU) of the present invention is 6.5% higher than that of the conventional method SS-WSSS and 3.3% higher than that of CONTA. Compared to the ICD method using VGG16 as the backbone network, the present invention outperforms it by 6.5%. Figure 6 Comparison of semantic segmentation effects for a certain image in the test set.

[0068] Table 1

[0069] method backbone network mIoU (%) SS-WSSS ResNet38 62.2 ICD VGG16 62.2 CONTA ResNet38 65.4 The present invention ResNet38 68.7

Claims

1. A weakly supervised semantic segmentation method based on category representation comparison and aggregation, characterized in that: The following steps are involved: Step 1: Perform image enhancement preprocessing on N images containing C target categories; Step 2: Introduce the concept of category representation, build a category representation extraction module, and obtain category representation based on the preprocessed image; Step 3: construct a category representation aggregation module to aggregate category representations and obtain category representation aggregation features; Step 4: Aggregate features of category representation and generate pseudo labels and category scores through normalized global weighted pooling; Step 5: Build a segmentation model and output semantic segmentation results based on pseudo labels.

2. The weakly supervised semantic segmentation method based on category representation comparison and aggregation according to claim 1 is characterized in that: The specific process of constructing the category representation extraction module and obtaining the category representation is as follows: The preprocessed image is input into the multi-scale convolutional network MSFCN, which extracts multi-scale features through dilated convolution to obtain the image feature mask; The image feature mask is input into the CAM module to generate an attention map, and the attention map is input into c category feature extractors to obtain c category features. Each category representation represents the category feature of the corresponding category of this training image in this round of training. Input c categories into mask average pooling to obtain c category representations of the image.

3. The weakly supervised semantic segmentation method based on category representation comparison and aggregation according to claim 2 is characterized in that: The step 2 also includes establishing a non-parametric dynamic memory bank M for storage based on the semantic information of the category representation, and the category representation in the memory bank is gradually updated through the momentum formula; The category semantic information of the entire data set, i.e., the category representation, is collected in the form of a non-parametric dynamic memory library, and the category representation of the entire data set is integrated into the training process of each batch of each image; By comparing the category representations in the memory bank with the category representations obtained in the training process, the distance between them and the same category is shortened, the similarity is increased, and the distance between them and the representations of different categories is increased, that is, the similarity is reduced; The loss function is NCE loss, which includes negative samples of the image, that is, the category representations that do not appear in the true label of the image, into the training; The NCE loss is calculated by comparing the positive samples of the image with the positive samples in the non-parametric dynamic library, and the NCE loss is calculated by comparing the negative samples of the image with the negative samples in the non-parametric dynamic memory library. The mixed loss is called CRM-NCE loss.

4. The weakly supervised semantic segmentation method based on category representation comparison and aggregation according to claim 3 is characterized in that: The specific implementation process of step three is as follows: The non-parametric dynamic memory bank that stores the category representation of the entire data set is clustered by the K-Mean clustering algorithm to obtain several category representation centroids, which are spliced ​​into a matrix shape; then all the centroids are spliced ​​to obtain its overall category representation Q, which contains the category representation aggregation information of the entire data set, and is fused into the cross-image semantics obtained during the training process. The specific fusion process is as follows: First, the features of each image I in this training batch are matrix multiplied with the overall category representation Q and then passed through a softmax function to obtain the category representation attention matrix S; The category representation affinity matrix F is calculated by matrix multiplication through the category representation attention matrix S and the overall category representation Q ′ ; Finally, concatenate the image features F and the category representation affinity matrix F ′ Get category representation aggregation features 5. The weakly supervised semantic segmentation method based on category representation comparison and aggregation according to claim 4 is characterized in that: The specific implementation process of step 4 is as follows: The category representation aggregate feature map is input into the normalized global weighted pooling module to generate pseudo labels and category scores; then the contrast loss and category comparison loss are mixed, where the category comparison loss is the cross entropy loss between the generated category score and the true label; finally, the loss function is back-propagated for training; The positioning model M1 is composed of a category representation extraction module, a category representation extraction module, a category representation aggregation module, and a segmentation module. The back propagation training is repeated until the positioning model M1 converges.

6. The weakly supervised semantic segmentation method based on category representation comparison and aggregation according to claim 5 is characterized in that: The specific implementation process of the normalized global weighted pooling module is as follows: Aggregate features into categories The score map S is obtained through a 1*1 convolution score , and then S score Generate confidence mask M through softmax normalization mask ; Then, the obtained confidence mask M mask , that is, the pseudo label, is input into the normalized global adaptive weighted pooling, and its formula is expressed as follows: where ε is a hyperparameter, represents the probability that image I contains category c, s c，w，h YesS score The probability that the pixel at position (w, h) is of category c; m c，w，h It is M mask The normalized probability of the pixel at position (w, h) being of category c.

7. The weakly supervised semantic segmentation method based on category representation comparison and aggregation according to claim 6 is characterized in that: The specific implementation process of step 5 is as follows: A batch of images I is input into the trained localization model M1 to generate pseudo labels, and the pseudo labels are input into the segmentation model together with the images I as the true labels. Back propagation calculation is performed using the cross entropy loss until the segmentation model converges.

8. The weakly supervised semantic segmentation method based on category representation comparison and aggregation according to claim 7, characterized in that: The specific implementation process of the segmentation model is as follows: At the end of the ResNet38 network, a convolutional layer with a dimension of c+1 is connected; the model composed of ResNet38 and the last convolution layer is denoted as M2; the image I is input into the trained model M1 to obtain the pseudo label Y, which is a pixel-level pseudo label, that is, all its pixels are assigned a corresponding category; then, the image I is input into M2 to obtain the prediction, and the prediction is restored to the original size of the image I through the bilinear interpolation algorithm, denoted as It is called the category probability map; calculate Y, The cross entropy loss between them is used to back propagate the training model M2, and the trained segmentation model M2 is obtained; then, the test set image is input to the model M2 to obtain its category probability mapping For each pixel, assign it to the category with the highest probability to obtain the final semantic segmentation result.

Images