A Weakly Supervised Semantic Segmentation Method Based on Category Representation Comparison and Aggregation

Abstract

This invention discloses a weakly supervised semantic segmentation method based on category representation comparison and aggregation. The method first preprocesses N images containing C target categories using image enhancement. Next, it introduces the concept of category representation and constructs a category representation extraction module to obtain category representations from the preprocessed images. Then, it constructs a category representation aggregation module to aggregate the category representations, obtaining aggregated category representation features. Finally, the aggregated category representation features are used to generate pseudo-labels and category scores through normalized global weighted pooling. Finally, a segmentation model is constructed, outputting semantic segmentation results based on the pseudo-labels. This invention effectively suppresses erroneous activations at incorrect locations, effectively improves the generation quality of seed regions, and thus improves the accuracy of semantic segmentation.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology and relates to a weakly supervised semantic segmentation method based on image-level annotation, which is commonly used in scene understanding, autonomous driving, medical image segmentation, etc. Background Technology

[0002] With the widespread application of semantic segmentation in fields such as autonomous driving, robotics, human-computer interaction, and medical imaging analysis, its importance in computer vision is increasing. While fully supervised semantic segmentation has made significant progress, its training requires pixel-level labels. The difficulty in obtaining pixel-level labels limits the application scenarios of fully supervised semantic segmentation. However, weakly supervised semantic segmentation can overcome this limitation. Weakly supervised semantic segmentation can utilize simple annotation information, such as image-level annotations and point annotations, to complete semantic segmentation tasks, greatly reducing the difficulty of label acquisition. While there have been many successful studies based on weakly supervised semantic segmentation, the current segmentation accuracy remains insufficient.

[0003] Currently, weakly supervised semantic segmentation based on deep learning generally follows a two-stage process: First, class activation mapping is used to obtain initial seed regions for preliminary localization of the segmentation target. These seed regions are then optimized to obtain higher-quality pseudo-labels for training in the next stage. Second, the segmentation model is trained using the pseudo-labels and images generated in the first stage.

[0004] Because class activation mapping often only provides the most discriminative region, it frequently fails to cover the segmentation target, leading to underactivation. Adversarial erasure and CAM ensemble methods proposed by Wei Yunchao aim to address this issue. However, these methods often introduce other problems, such as incorrect activations in non-target categories, known as overactivation. Current solutions to overfitting primarily rely on image co-occurrence relationships, training additional inference frameworks, or performing corresponding enhancements based on training set characteristics. Overactivation and underactivation are intertwined; past research often focused on one problem while failing to suppress or even exacerbating the other. How to coordinate overactivation and underactivation to achieve an optimal state and thus improve the accuracy of weakly supervised semantic segmentation remains a crucial problem to be solved in the field of weakly supervised semantic segmentation. Summary of the Invention

[0005] This invention provides a weakly supervised semantic segmentation method based on category representation attention to address the underactivation and overactivation problems in related technologies. By introducing cross-pixel affinity based on category representation and optimizing the CAP function, it adapts to targets of different sizes, thereby improving segmentation accuracy. The technical solution is as follows.

[0006] A weakly supervised semantic segmentation method based on category representation attention is proposed. This method is a two-stage weakly supervised semantic segmentation approach. The first stage generates seed regions and, to address underfitting and overfitting issues, includes a category representation extraction module, a category representation aggregation module, and a pseudo-label generation module. The second stage is the semantic segmentation pseudo-label training stage, which contains a segmentation model, wherein:

[0007] The category representation extraction module includes:

[0008] Step 1: Obtain training and test sample sets: Label the N images containing C target categories and divide them into training, test, and cross-validation sets.

[0009] Step 2, Image Preprocessing: Perform a series of image enhancements on the image from Step 1, including adjusting the image size, randomly changing the image brightness, contrast, saturation, etc. Based on the prior knowledge that weak enhancement does not change the image feature distribution, enhance the robustness of the model and expand the training set.

[0010] Step 3: By introducing the concept of category representation, a category representation extraction module is constructed to obtain category representations based on the preprocessed image: The preprocessed image is input into a multi-scale fully-convolutional network (MSFCN). This network mainly extracts multi-scale features through dilated convolution. The image is input into this network to obtain an image feature mask. Then, the image feature mask is input into the CAM module to generate an attention map. The attention map is then input into c category feature extractors to obtain c category features, each category representation representing the category feature of the corresponding category of the training image in this round of training. Further, the c categories are input into mask average pooling to obtain c category representations of the image. Further, based on the semantic information of the category representations, a nonparametric dynamic memory M is established to store them. The category representations in the memory are updated step by step using the momentum formula. This invention collects the category semantic information, i.e., category representations, of the entire dataset in the form of a nonparametric dynamic memory, and integrates the category representations of the entire dataset into the training process of each batch and each image. Furthermore, by comparing the category representations in the memory bank with those obtained during the training process, the distance to the same category is shortened (i.e., similarity is increased), while the distance to different category representations is widened (i.e., similarity is reduced). The loss function is the NCE loss. To enhance the robustness of the training model, this invention also incorporates negative samples of the image—category representations not appearing in the true label of the image—into the training. The NCE loss is calculated between positive samples of the image and positive samples in the non-parametric dynamic memory bank, and the NCE loss is calculated between negative samples of the image and negative samples in the non-parametric dynamic memory bank. The resulting loss is called the CRM-NCE loss.

[0011] Step four: Construct a category representation aggregation module to aggregate category representations and obtain aggregated category representation features. In step three, this invention obtained a nonparametric dynamic memory containing category representations of the entire dataset. This memory was then processed using the K-Mean clustering algorithm to obtain several category representation centroids, which were then concatenated into a matrix shape. Finally, all centroids were concatenated to obtain the overall category representation Q. Q contains the aggregated category representation information of the entire dataset, and its integration into the training process will bring rich cross-image semantics. The specific fusion process is as follows: First, the features of each image I in this training batch are multiplied with the overall category representation Q, and then subjected to a softmax function to obtain the category representation attention matrix S. Further, the category representation affinity matrix F is calculated by matrix multiplication between the category representation attention matrix S and the overall category representation Q. ′ Finally, the image features F and the category representation affinity matrix F are concatenated. ′ Obtain category representation aggregate features

[0012] Step 5: Generate pseudo-labels and class scores through normalized global weighted pooling: Input the aggregated feature map generated in Step 4 into the normalized global weighted pooling module to generate pseudo-labels and class scores. The normalized global weighted pooling function optimizes the problem of excessively large accuracy differences in the global average pooling function for recognizing targets of different sizes. Finally, a combination of contrastive loss and class comparison loss is used, where the class comparison loss is the cross-entropy loss between the generated class scores and the true labels. This loss function is then used for backpropagation training. The localization model M1, consisting of a class representation extraction module, a class representation aggregation module, and a segmentation module, is repeatedly trained through backpropagation until the localization model M1 converges.

[0013] Step Six: Construct a segmentation model and output semantic segmentation results based on pseudo-labels: Use a pre-trained ResNet38 as the segmentation model. First, input a batch of images I into the localization model M1 trained in Step Five to generate pseudo-labels. Then, input these pseudo-labels as ground truth labels along with images I into the segmentation model, and perform backpropagation using cross-entropy loss. Repeat this process until the segmentation model converges. Finally, evaluate the segmentation performance using samples from the validation set based on the mIoU metric.

[0014] The main advantages of this invention compared to other methods are:

[0015] 1. A low-complexity, multi-scale fully connected network was designed to acquire image features that aggregate receptive fields of various sizes at a lower cost.

[0016] 2. A category representation extraction module and masked average pooling were designed, and a nonparametric dynamic memory library for category representation based on the entire dataset was constructed based on contrastive learning.

[0017] 3. A category representation aggregation module was designed based on the K-Mean clustering algorithm. The category representation provides category supervision for the entire dataset during the training process, effectively suppressing erroneous activations at incorrect locations, and effectively improving the generation quality of seed regions, thereby improving the accuracy of semantic segmentation.

[0018] 4. A normalized global pooling function was designed to optimize the accuracy of small target localization, avoid the huge degradation effect caused by a small number of erroneous pseudo-labels in the training process, and improve the final segmentation accuracy. Attached Figure Description

[0019] To more clearly illustrate the embodiments and technical solutions of this application, the services required in the embodiments and technical solutions will be briefly introduced below.

[0020] Figure 1 This is a flowchart of a weakly supervised semantic segmentation method based on category representation comparison and aggregation proposed in this invention;

[0021] Figure 2 This is a schematic diagram of the framework of a weakly supervised semantic segmentation method based on category representation comparison and aggregation proposed in this invention;

[0022] Figure 3 This is a network structure diagram of the multi-scale fully connected network proposed in this invention;

[0023] Figure 4 This is a detailed explanation of the normalized global weighted pooling module proposed in this invention;

[0024] Figure 5 This is a comparison of the activation effects of the multi-scale fully connected network of the present invention with similar methods (FCN);

[0025] Figure 6 This is a comparison of the segmentation performance of this invention on the PASCAL VOC2012 dataset with other methods. Detailed Implementation

[0026] This chapter will provide a more detailed and complete description of the technical solutions, objectives, embodiments, and advantages of the present invention in conjunction with the accompanying drawings. The described embodiments are only a part of the embodiments of the present invention. The method of the present invention will be specifically described below.

[0027] Figure 1 This is a flowchart illustrating a weakly supervised semantic segmentation method based on category representation attention provided in this specification. The overall architecture is as follows: Figure 2 As shown, the specific steps include:

[0028] Step 1: Acquire sample images and label them with image-level classification labels. Finally, divide the samples into training, test, and sample sets in an 8:1:1 ratio.

[0029] Step 2: Preprocess the training set, including resizing images, randomly changing image brightness, contrast, saturation, etc., and finally converting them into the tensor format required by PyTorch. The main purpose of this step is to adjust the image size to fit the model input and, based on the prior knowledge that weak enhancement does not change the image feature distribution, enhance the robustness of the model.

[0030] Step 3: By introducing the concept of category representation, construct a category representation extraction module, corresponding to... Figure 2 Category representation extraction module. The preprocessed image I obtained in step two is input into a multi-scale fully-convolutional network (MSFCN), whose formula is denoted as f. msfcn Experiments have shown that the class activation mapping obtained from MSFCN is more complete than that from FCN, and it significantly suppresses overactivation for small targets, such as... Figure 5 As shown. Its formula is expressed as follows:

[0031]

[0032] Then, it is input into CAM to generate an attention map, the formula of which is as follows:

[0033]

[0034] The multi-scale fully connected network consists of convolutional kernels, average pooling kernels, and dilated convolutional kernels, as shown below. Figure 3 As shown, this typically corresponds to the convolutional part of classifiers used in other methods (e.g., VGG, ResNet). The output F is a dense embedding representation of I. CAM It is a class-aware convolutional layer that generates P = [P1, P2, ..., P...]. C ], where each mapping Let P represent the network activation of class c. Next, a score vector is obtained from P using global average pooling (GAP). Where p c =GAP(P c ) represents the unnormalized score of class c.

[0035] Then, the feature P is input into the category representation extraction module. Further, the feature F is input into c category feature extractors, each consisting of three sets of 3x3 convolutions with 512 inputs and 512 outputs, and a ReLU function. This yields c category features F. c Represent its aggregation as Furthermore, each category feature and P are input into masked average pooling, the formula of which is as follows:

[0036]

[0037] Where M c ={1(·)}(P c >μ)∈{0,1} {W×H} It is a binary mask that represents category c in p. c The highly activated region in the image. 1(·) is an index function whose threshold μ is set to P. c The mean of f. c It is a category representation that contains a large amount of category semantic information.

[0038] Furthermore, a nonparametric dynamic memory M is established to store these category representations. Memory M consists of C dictionaries, i.e., M = [M1, M2, ..., M]. c Each dictionary corresponds to a category, M c Each entry represents the representation of the i-th class in the images observed throughout the learning phase, M i ∈R D (The entire dataset). Image L i The category representation will be updated gradually through the momentum formula.

[0039] m i ←γm i +(1-γ)f i (4)

[0040] Where γ represents the shift value of memory momentum, usually set to 0.99. i f represents the category representation of category i that has been recorded in the nonparametric dynamic memory. i The category representation is obtained by extracting the category representation from image I. m is only considered correct if the image contains the corresponding i-th category and the predicted category score is higher than the threshold v. i When performing momentum updates, v is typically set to 0.75.

[0041] Then, comparative learning is performed. For each image I, f... cThe goal of contrastive learning is to increase the similarity of representations of the same category in the memory bank while decreasing the similarity with representations of different categories. The contrastive loss is calculated as follows:

[0042]

[0043] τ is a hyperparameter used to control the rate of change in similarity. sim(i,j) is the L2 regularized product of i and j. e is an exponential function. The positive samples are the class representations of class c stored in the nonparametric dynamic memory. Negative samples are class representations of classes other than c stored in a nonparametric dynamic memory. It is a category representation f c Similarity with positive samples It is a category representation f c The similarity between the sample and the positive sample. The optimization objective is to make... By minimizing the value as much as possible, it can be seen that as the distance to negative samples increases and the distance to positive samples becomes closer, the loss function can satisfy the training objective.

[0044] It can be observed that the loss function still falls within the supervised learning range, but the weak labels present a significant challenge to the robustness of the class representation learning. This invention combines the class representations f from other images in the same training batch. c- ,use Comparative learning is performed, and the formula is as follows:

[0045]

[0046] Where w is a hyperparameter, set to 0.99. f c f represents the category representation of the training image. c- We define the category representation for other images in the same batch. Then, we define the category representation mixture contrast loss as follows:

[0047]

[0048] Step 4: Construct a category representation aggregation module to aggregate category representations, corresponding to... Figure 2 Category representation aggregation module. In this invention, a category representation memory is obtained through step S3. While a category representation memory can provide rich cross-image semantics for the training process, it contains overly complete or redundant category representations. Directly aggregating large-scale category representations often leads to side effects in practice, significantly reducing the learning and inference process. This invention will... c All features are clustered using K-means to obtain the centroids representing the class representations of K, and then organized into a matrix. By concatenating the centroids of the category representations of all categories, we obtain the overall representation. Then, for each image I in this training batch, the features... The category representation attention matrix S is calculated using the following formula:

[0049]

[0050] in and For ease of calculation, it will be compressed from three dimensions to two dimensions. This represents matrix multiplication. Each line in the formula's result S reflects the sum of each row in feature F and Q. T The normalized similarity between each column (i.e., category representation) in S. Based on S, the representation affinity matrix F can be further calculated. ′ :

[0051]

[0052] Where F' is the expression of reinforcement of the category representation aggregated by F. Then F ′ It is then reconstructed into a three-dimensional W×H×D structure. Then F... ′ By splicing it with F, we get

[0053]

[0054] It not only possesses the image features of image I but also incorporates the category representations of the entire dataset, thus enriching its semantics. Ultimately, it will... The final class activation map O is obtained by passing it into CAM:

[0055]

[0056] Step 5: Generate pseudo-labels and category scores through normalized global weighted pooling. For example... Figure 4 As shown, first obtain the result in step four. The fractional mapping S is obtained through a 1x1 convolution. score Then S score The confidence mask M is generated by softmax normalization. mask Its formula is expressed as follows:

[0057]

[0058] Where conv represents a 1x1 convolution operation, and softmax represents the softmax function. Then, the resulting confidence mask M... mask These are pseudo-labels. They are input into the normalized global adaptive weighted pooling process, and the formula is as follows:

[0059]

[0060] Where ε is a hyperparameter, set to 1. This represents the probability that image I may contain class c. c，w，h It is S score The probability that the pixel located at position (w, h) belongs to category c. c，w，h It is M mask The normalized probability of the pixel at position (w, h) belonging to class c. Other GAP-CAM methods, which equally include all pixels in the class probability calculation, result in underactivation for small targets and overactivation for large targets. nGWP effectively solves this problem, allowing targets of both sizes to obtain an appropriate class probability score. Its overall process is as follows: Figure 5 As shown above. Based on the above, the loss function L for the first stage of training is constructed as follows:

[0061] L=∑ I α1L CRM-NCE +α2L CE (Y nGWP ,y)+L CE (GAP(O), y) (14)

[0062] Where y represents the one-hot true label of the true class given in the training set. The unique heat vector of the shape. L CE The loss function is cross-entropy. α1 and α2 are hyperparameters, which in this embodiment are 0.3 and 0.7, respectively. Iterative training is performed using backpropagation through the loss function L. This process is repeated until the network converges.

[0063] Step Six: Constructing the Segmentation Module. The backbone network used in this invention is ResNet38. ResNet38 consists of several residual blocks. A residual network consists of countless residual blocks. A residual block consists of multiple cascaded convolutional layers and a skip connection. The outputs of the two are summed, and the output of the residual block is obtained through a ReLU activation layer. Multiple residual blocks can be chained together to achieve a deeper network. To meet the requirements of the segmentation network, a convolutional layer of dimension c+1 is added to the end of the ResNet38 network. The model composed of ResNet38 and the last convolutional layer is denoted as M2. Image I is input into the model M1 trained by S5 to obtain the pseudo-label Y. Y is a pixel-level pseudo-label, meaning that all its pixels are assigned a corresponding category. Then, image I is input into M2 to obtain a prediction. The prediction is restored to the original size of image I using a bilinear interpolation algorithm, denoted as . This is called the class probability mapping. Then, Y is calculated. The cross-entropy loss between them is calculated as follows:

[0064]

[0065] Where y c It is the one-hot vector from the pseudo-label Y. The prediction obtained through M2 represents the probability that the pixel belongs to class c. Model M2 is trained using backpropagation with this loss function. This process is repeated until the model converges, resulting in the segmentation model M2. Then, test set images are input into model M2 to obtain their class probability mappings. Then, for each pixel, it is assigned the category with the highest probability to obtain the final semantic segmentation result. Finally, the semantic segmentation performance is evaluated by comparing the final semantic segmentation result with the ground truth label using mIoU. In the embodiment using PASCAL VOC2012 as the dataset, the mIoU of this invention is 68.7%, which is relatively advanced compared to other similar methods. As shown in Table 1, when using ResNet38 as the backbone network, the segmentation performance (mIoU) of this invention is 6.5% higher than the previous method SS-WSSS and 3.3% higher than CONTA. Compared with the ICD method using VGG16 as the backbone network, the performance of this invention is 6.5% higher. Figure 6 This is a comparison of semantic segmentation performance for a specific image in the test set.

[0066] Table 1

[0067] method backbone network mIoU (%) SS-WSSS ResNet38 62.2 ICD VGG16 62.2 CONTA ResNet38 65.4 This invention ResNet38 68.7

Claims

1. A weakly supervised semantic segmentation method based on category representation comparison and aggregation, characterized in that, Includes the following steps: Step 1: Perform image enhancement preprocessing on N images containing C target categories; Step 2: Introduce the concept of category representation and construct a category representation extraction module to obtain category representations based on the preprocessed image. The specific process is as follows: The preprocessed image is input into the multi-scale convolutional network MSFCN. MSFCN extracts multi-scale features through dilated convolution to obtain the image feature mask. The image feature mask is input into the CAM module to generate an attention map. The attention map is then input into c category feature extractors to obtain c category features. Each category representation represents the category feature of the corresponding category of the training image in this round of training. The c categories are input into masked average pooling to obtain the c category representations of the image; Based on the semantic information of the category representation, a nonparametric dynamic memory M is established for storage, and the category representation in the memory is updated step by step through the momentum formula; The category semantic information, i.e. category representation, of the entire dataset was collected in the form of a nonparametric dynamic memory, and the category representation of the entire dataset was integrated into the training process of each batch and each image. By comparing the category representations in the memory bank with the category representations obtained in the training process, the distance between them and the same category is shortened, and the similarity is increased. At the same time, the distance between them and the representations of different categories is increased, that is, the similarity is reduced. The loss function is NCE loss, which includes negative samples of the image, i.e., category representations that do not appear in the true label of the image, in the training. The NCE loss is calculated by comparing positive samples of the image with positive samples in the nonparametric dynamic memory library, and the NCE loss is calculated by comparing negative samples of the image with negative samples in the nonparametric dynamic memory library. The loss after mixing is called CRM-NCE loss. Step 3: Construct a category representation aggregation module to aggregate category representations and obtain aggregated category representation features. The specific implementation process is as follows: The nonparametric dynamic memory containing the category representations of the entire dataset is used to obtain several category representation centroids through the K-Mean clustering algorithm, which are then concatenated into a matrix shape. Then, all centroids are concatenated to obtain the overall category representation Q, which contains the aggregated category representation information of the entire dataset. This Q is then fused into the cross-image semantics obtained during training. The specific fusion process is as follows: First, the features of each image I in this training batch are analyzed, and then a matrix multiplication operation is performed with the overall category representation Q. After passing through a softmax function, the category representation attention matrix S is obtained. The category representation affinity matrix F is calculated by matrix multiplication using the category representation attention matrix S and the overall category representation Q. ′ Finally, the image features F and the category representation affinity matrix F are concatenated. ′ Obtain category representation aggregate features Step four: Aggregate features to represent categories, and generate pseudo-labels and category scores through normalized global weighted pooling; The specific implementation process of the normalized global weighted pooling module is as follows: Aggregate features by category representation The fractional mapping S is obtained through a 1x1 convolution. score Then S score The confidence mask M is generated by softmax normalization. mask ; Then, the obtained confidence mask M mask The pseudo-labels are input into the normalized global adaptive weighted pooling, and their formula is as follows: Where ε is a hyperparameter. s represents the probability that image I contains class c. c，w，h It is S score The probability that the pixel located at position (w, h) belongs to category c; m c，w，h It is M mask The normalized probability of the pixel of class c located at position (w, h); Step 5: Construct a segmentation model and output semantic segmentation results based on pseudo-labels.

2. The weakly supervised semantic segmentation method based on category representation comparison and aggregation according to claim 1, characterized in that, The specific implementation process of step four is as follows: The category representation aggregated feature map is input into the normalized global weighted pooling module to generate pseudo-labels and category scores; then, the contrastive loss and the category comparison loss are mixed, where the category comparison loss is the cross-entropy loss between the generated category scores and the true labels; finally, this loss function is trained by backpropagation. The localization model M1 consists of a category representation extraction module, a category representation aggregation module, and a segmentation module. Backpropagation training is repeated until the localization model M1 converges.

3. The weakly supervised semantic segmentation method based on category representation comparison and aggregation according to claim 2, characterized in that, The specific implementation process of step five is as follows: A batch of images I is input into the trained localization model M1 to generate pseudo-labels. These pseudo-labels are then used as real labels and input together with images I into the segmentation model. Backpropagation is performed using cross-entropy loss until the segmentation model converges.

4. The weakly supervised semantic segmentation method based on category representation comparison and aggregation according to claim 3, characterized in that, The specific implementation process of the segmentation model is as follows: A convolutional layer of dimension c+1 is added to the end of the ResNet38 network; the model composed of ResNet38 and the last convolutional layer is denoted as M2; image I is input into the trained model M1 to obtain pseudo-label Y, which is a pixel-level pseudo-label, meaning that all its pixels are assigned a corresponding category; then, image I is input into M2 to obtain a prediction, which is then restored to the original size of image I using a bilinear interpolation algorithm, denoted as . This is called the class probability mapping; calculate Y, The cross-entropy loss between the images is used to backpropagate and train model M2, resulting in the trained segmentation model M2. Then, the test set images are input into model M2 to obtain their class probability mappings. For each pixel, assign it the category with the highest probability to obtain the final semantic segmentation result.

Images