A CAM optimization method based on patch semantic affinity enhancement

Abstract

The present invention discloses a CAM optimization method based on patch semantic affinity enhancement, including: extracting multi-head self-attention weights; using head average operation to remove redundant information, and then aggregating the global features of each attention map through global average pooling, and inputting the aggregated feature vector into a multi-layer perceptron for interaction, to obtain an enhanced attention map that takes into account the importance of attention at different layers; removing the corresponding dimensions of the class tokens in the attention after fusion by average operation, and using the remaining weights as the enhanced patch semantic affinity matrix; the output patch token generates CAM, and the ability of semantic affinity to capture more spatial relationships and fine-grained information is used to perform matrix multiplication of the patch semantic affinity matrix and CAM to complete the optimization. The present invention obtains the attention weight matrix containing semantic affinity information without additional calculation, and fuses and enhances the attention weight matrix through the head average attention fusion module.

Description

Technical Field

[0001] The invention belongs to the field of image-level weakly supervised semantic segmentation, and in particular relates to a CAM optimization method based on patch semantic affinity enhancement. Background Art

[0002] Weakly supervised semantic segmentation is an important branch of image segmentation, which aims to reduce the dependence on large-scale annotated data while achieving high segmentation accuracy. In recent years, deep learning-based methods, especially convolutional neural networks (CNNs) and Transformer architectures, have been widely used in weakly supervised semantic segmentation. These methods can learn effective feature representations from limited annotated data and gradually improve segmentation performance. CAM stands for Class Activation Map, which is used to visualize the decision-making process of CNN in image classification tasks. CAM technology is crucial to weakly supervised semantic segmentation, and subsequent weakly supervised semantic segmentation methods are all based on CAM technology. However, since there are usually a lot of errors and noise in CAM, the pseudo-labels generated by CAM are of poor quality and it is impossible to train a segmentation model with good performance. CAM usually needs to be optimized.

[0003] At present, the mainstream in the field of weakly supervised semantic segmentation is based on methods based on CNN and Transformer architecture. Zhou [Zhou B, Khosla A, Lapedriza A, et al. Learning deep features for discriminative localization [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 2921-2929.] proposed the CAM technology to help us understand which image regions play a key role in the classification of specific categories in the image classification process of CNN. After post-processing, CAM can obtain pixel-level pseudo labels. Using pseudo labels and pictures, an image segmentation model can be fully supervised, which has made a major breakthrough in the field of weakly supervised semantic segmentation. It should be pointed out that the spatial accuracy of CAM is relatively low, and there are limitations such as the need to use the global average pooling (GAP) layer and the ability to only use the output of the last convolution layer. To this end, Ram[Selvaraju RR, Cogswell M, Das A, et al. Grad-cam: Visual explanationsfrom deep networks via gradient-based localization[C] / / Proceedings of theIEEE international conference on computer vision. 2017: 618-626.] proposed GradCAM, which does not require any modification or retraining of the model. It calculates the gradient to obtain the weights between feature channels and categories, which can provide more accurate positioning and can be used for tasks other than image classification, greatly improving flexibility.Later, Ahn [Ahn J, Kwak S. Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 4981-4990.] proposed the AffinityNet method, which aims to improve the quality of pseudo-labels. It uses an end-to-end network called AffinityNet to learn the semantic affinity between pixels and propagate the affinity through random walks. Because previous methods usually use multi-stage solutions, that is, first use pictures and category labels to fully supervise the training of an image classification model, then use CAM technology to obtain the class activation map CAM from the image classification model and generate pixel-level pseudo-labels, and finally use pseudo-labels and pictures to fully supervise the training of an image segmentation model. This multi-stage framework requires training multiple models for different tasks, which will complicate the training process and reduce efficiency. To this end, Zhang[Zhang B, Xiao J, Wei Y, et al. End-to-end weakly supervised semanticsegmentation with reliable region mining[J]. Pattern Recognition, 2022, 128:108663.] proposed the RRM method (Reliable Region Mining), which is a fully end-to-end network divided into two parallel branches: the classification branch and the semantic segmentation branch. The two branches share the same backbone network and update the entire network simultaneously during training.

[0004] However, the above methods are mainly based on convolutional neural networks and cannot correctly explore global information, while the multi-head self-attention of Transformer can effectively capture global semantic information. To this end, Gao [Gao W, Wan F, PanX, et al. Ts-cam: Token semantic coupled attention map for weakly supervised object localization [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2021: 2886-2895.] introduced Transformer and proposed the TS-CAM method (Token Semantic Coupled Attention Map). By rearranging and convolving the tokens output by ViT, a semantically aware positioning map can be obtained, while the attention map obtained from ViT's self-attention is semantically agnostic. Combining the two can obtain a semantically aware positioning map. In order to learn richer and more comprehensive semantic affinity information from ViT's attention, Ru et al. [Ru L, Zhan Y, Yu B, et al. Learning affinity from attention: End-to-end weakly-supervised semantic segmentation with transformers[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022: 16846-16855.] designed an end-to-end Transformer-based weakly supervised semantic segmentation method. They proposed the AFA method (Affinity From Attention) to learn semantic affinity from multi-head self-attention in Transformer, and designed a pixel-adaptive refinement module PAR (Pixel-AdaptiveRefinement) to combine the RGB and position information of local pixels for label refinement. However, previous Transformer-based methods have a common problem, that is, only a single class token is used, which makes it challenging to accurately locate different objects on a single image.To this end, Xu [Xu L, Ouyang W, Bennamoun M, et al. Multi-class token transformer for weakly supervised semantic segmentation [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022: 4310-4319.] proposed the MCTFormer method (Multi-class Token Transformer). In ViT, they creatively used the multi-class token method to establish a one-to-one strong connection between each class tag and the corresponding class label, which can ensure that each Class Token can learn the corresponding category semantic information. The class-to-patch attention of different classes learned in this way can be directly used as a class-specific localization map. Although ViT can make up for the defect of traditional CAM in recognizing the entire object area that is limited to CNN local perception, it also brings the problem of over-smoothing, that is, the final patch labeling tends to be homogenized. Therefore, Ru[Ru L, Zheng H, Zhan Y, et al. Token contrast for weakly-supervised semantic segmentation[C] / / Proceedings of the IEEE / CVF Conferenceon Computer Vision and Pattern Recognition. 2023: 3093-3102.] proposed an end-to-end ToCo method (Token Contrast), which mainly consists of two key parts: Patch Token Contrast (PTC) and Class Token Contrast (CTC). PTC uses the pseudo-labeled relationship of the intermediate layer of ViT to supervise the final patch labeling to solve the over-smoothing problem of ViT, while CTC promotes the integrity of object activation in CAM by comparing the representation of the global foreground and the local uncertain area (background).

[0005] Although these weakly supervised semantic segmentation methods have improved the image-level weakly supervised semantic segmentation problem, there are still certain problems with the generated CAM: the CAM still has a lot of noise and positioning errors; the Transformer-based models are usually large and cannot fully obtain the attention maps of all layers in the Transformer; the importance of attention from different layers of the Transformer is not considered comprehensively and fully. Summary of the invention

[0006] Purpose of the invention: The purpose of the present invention is to provide a CAM optimization method based on patch semantic affinity enhancement. The attention weight matrix containing semantic affinity information is obtained without additional calculation, and the attention weight matrix is ​​fused and enhanced through the head average attention fusion module.

[0007] Technical solution: A CAM optimization method based on patch semantic affinity enhancement of the present invention comprises the following steps:

[0008] Step 1: Extract the multi-head self-attention weights from the Vision Transformer backbone network. This operation does not consume any additional computing resources because the multi-head self-attention weights are a by-product of the Transformer calculation.

[0009] Step 2. For the multi-head self-attention weights extracted in step 1, given that the multi-head self-attention of layers of different depths in the Transformer may focus on different parts, such as the shallow layer pays more attention to local structure, texture color, etc., and the deep layer can capture broader and more abstract visual semantic information, try to use the head average operation to remove redundant information and reduce the amount of calculation, and then aggregate the global features of each attention map through global average pooling, and input the aggregated feature vector into the multi-layer perceptron for interaction, extract more complex feature relationships, and finally obtain an enhanced attention map that fully considers the importance of attention at different layers.

[0010] Step 3: Based on the result of enhanced attention in step 2, the enhanced attention maps from different layers are averaged to fuse, and the dimensions corresponding to the class tokens in the self-attention are removed. The remaining weights can be used as the enhanced patch-level semantic affinity.

[0011] Step 4: Based on the patch-level semantic affinity matrix obtained in step 3, the ability of semantic affinity to capture more spatial relationships and fine-grained information is used to generate a class activation map CAM from the patch tokens output by the last layer of Transformer encoder block, and the patch semantic affinity matrix is ​​used to perform matrix multiplication with CAM to complete the optimization of CAM, making CAM more refined and accurate, thereby obtaining better pseudo-labels for supervising the training of the segmentation model.

[0012] Further, the step 1 is specifically as follows:

[0013] Step 101: Split the input image into N×N patches, and linearly transform them into a patch token sequence of dimension D, linking the previous class token of dimension D to form the input token sequence of the ViT encoder. .

[0014] Step 102, the ViT encoder consists of K encoding layers, each of which contains a multi-head self-attention and a multi-layer perceptron, as well as two layer normalizations before two sub-layers. The ViT encoder receives the input token sequence And output the token sequence .

[0015] Step 103: Extract the multi-head self-attention weight tensor from the multi-head self-attention module of the K-layer encoding layer in ViT , where H is the number of multi-head self-attention heads.

[0016] Further, the step 2 is specifically as follows:

[0017] Step 201: For the multi-head self-attention weight tensor obtained in step 103 , using the head average operation to remove the dimension H helps to remove redundant information and reduce the amount of calculation, and obtains .

[0018] Step 202: For the head averaged attention map obtained in step 201 , perform a concatenate operation on the 0th dimension to connect the K layers of attention maps in series to obtain a three-dimensional attention tensor .

[0019] Step 203: For the attention tensor obtained in step 202 , the global features of the K-layer attention map are aggregated by global average pooling to obtain the aggregated feature vector of length K , and input the feature vector into the multi-layer perceptron to interact with each other, extract more complex feature relationships, and the multi-layer perceptron outputs feature vectors of the same shape .

[0020] Step 204: Obtain the feature vector output by the multi-layer perceptron Then, the head average attention tensor obtained in step 201 is With the eigenvector Combined, the formula is as follows:

[0021]

[0022] in Represents the element-by-element multiplication symbol. The above formula can be used to obtain an enhanced attention map that fully considers the importance of attention at different layers .

[0023] Further, the step 3 is specifically as follows:

[0024] Step 301: Based on the result of step 2, the enhanced attention map Perform an average operation on the 0th dimension K to fuse the enhanced attention maps from different layers to obtain the fused enhanced attention .

[0025] Step 302: Enhance the attention of the fusion obtained in step 301 , removing the dimension corresponding to the class token, the remaining enhanced attention weights can be used as the enhanced patch-level semantic affinity .

[0026] Further, the step 4 is specifically as follows:

[0027] Step 401: From step 102, we know that the output token sequence of the last Transformer encoding layer is , remove the dimension corresponding to the class token and rearrange the patch token sequence , and perform a 1×1 convolution operation to convert the token dimension into the number of object categories. The formula is as follows:

[0028]

[0029] in, The input channel is D, the output channel is the number of object categories C, and the convolution kernel size is 1×1. The initial class activation map from the patch token can be obtained by the above formula .

[0030] Step 402: Since the initial class activation map is rough and may contain a lot of noise and errors, it is necessary to use the patch semantic affinity in step 302 to optimize the initial class activation map. The formula is as follows:

[0031]

[0032] The optimized class activation graph can be obtained through the above formula , which is more refined and accurate than the initial CAM, thus obtaining better pseudo labels for supervising the training of the segmentation model.

[0033] The present invention also discloses a computer device, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method of the present invention.

[0034] The present invention also discloses a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the method of the present invention are implemented.

[0035] The present invention also discloses a computer program product, comprising a computer program / instruction, which implements the steps of the method of the present invention when executed by a processor.

[0036] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0037] 1. In terms of obtaining affinity information, by extracting the multi-head self-attention weights that are a byproduct of training from ViT, the attention weight matrix containing the semantic affinity information of patches can be obtained without consuming additional computing resources;

[0038] 2. In terms of patch semantic affinity enhancement, the extracted multi-head self-attention weights are fused and enhanced through the head average attention fusion module. Performing head average operations on different heads of multi-head self-attention helps to: remove redundancy and extract more concise and effective information; reduce the amount of calculation and video memory usage, and improve computing efficiency; it can reduce the sensitivity of a single head to specific noise or anomalies and improve the robustness of the model. The fused enhanced attention map fully considers the importance of attention at different layers;

[0039] 3. In terms of optimizing CAM, obtaining, enhancing the affinity matrix and multiplying it with CAM is an end-to-end CAM optimization method, which can be integrated into various multi-stage or single-stage weakly supervised semantic segmentation methods that use patch tokens to generate CAM. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flow chart of the present invention;

[0041] Figure 2 This is a flowchart of multi-head self-attention weight extraction of the present invention;

[0042] Figure 3 It is a flowchart of head average attention enhancement of the present invention;

[0043] Figure 4 It is a flowchart of patch semantic affinity extraction of the present invention;

[0044] Figure 5 This is the CAM optimization flow chart of the present invention. DETAILED DESCRIPTION

[0045] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0046] The CAM optimization method based on patch semantic affinity enhancement described in this embodiment of the present invention is shown in the flowchart as follows: Figure 1 As shown, the following steps are included:

[0047] Step 1: Extract the multi-head self-attention weights from the Vision Transformer backbone network. This operation does not consume any additional computing resources because the multi-head self-attention weights are a byproduct of the Transformer calculation.

[0048] Step 101, split the input image into N×N patches, and convert them into a patch token sequence of dimension D by linear conversion, linking the previous class token of dimension D to form the input token sequence of ViT encoder .

[0049] Step 102, the ViT encoder consists of K encoding layers, each of which contains a multi-head self-attention and a multi-layer perceptron, as well as two layer normalizations before two sub-layers. The ViT encoder receives the input token sequence And output the token sequence .

[0050] Step 103: extract the multi-head self-attention weight tensor from the multi-head self-attention module of the K-layer encoding layer in ViT , where H is the number of multi-head self-attention heads, see the flowchart Figure 2 .

[0051] Step 2: For the multi-head self-attention weights extracted in step 1, considering that the multi-head self-attention of layers of different depths in the Transformer may focus on different parts, such as shallow layers pay more attention to local structures, texture colors, etc., and deep layers can capture more extensive and abstract visual semantic information, try to use head average operation to remove redundant information and reduce the amount of calculation, and then aggregate the global features of each attention map through global average pooling, and input the aggregated feature vector into the multi-layer perceptron for interaction, extract more complex feature relationships, and finally obtain an enhanced attention map that fully considers the importance of attention at different layers;

[0052] Step 201, for the multi-head self-attention weight tensor obtained in step 103 , using the head average operation to remove the dimension H helps to remove redundant information and reduce the amount of calculation, and obtains .

[0053] Step 202: for the head averaged attention map obtained in step 201 , perform a concatenate operation on the 0th dimension to connect the K layers of attention maps in series to obtain a three-dimensional attention tensor .

[0054] Step 203, for the attention tensor obtained in step 202 , the global features of the K-layer attention map are aggregated by global average pooling to obtain the aggregated feature vector of length K , and input the feature vector into the multi-layer perceptron to interact with each other, extract more complex feature relationships, and the multi-layer perceptron outputs feature vectors of the same shape .

[0055] Step 204, obtain the feature vector output by the multi-layer perceptron Then, the head average attention tensor obtained in step 201 is With the eigenvector Combined, the formula is as follows:

[0056]

[0057] in Represents the element-by-element multiplication symbol. The above formula can be used to obtain an enhanced attention map that fully considers the importance of attention at different layers , the flow chart is shown in Figure 3 .

[0058] Step 3: Based on the result of enhanced attention in step 2, the enhanced attention maps from different layers are averaged to fuse, and the dimensions corresponding to the class tokens in the self-attention are removed. The remaining weights can be used as the enhanced patch-level semantic affinity;

[0059] Step 301: Based on the result of step 2, the enhanced attention map Perform an average operation on the 0th dimension K to fuse the enhanced attention maps from different layers to obtain the fused enhanced attention .

[0060] Step 302, for the fusion enhanced attention obtained in step 301 , removing the dimension corresponding to the class token, the remaining enhanced attention weights can be used as the enhanced patch-level semantic affinity , the flow chart is shown in Figure 4 .

[0061] In step 4, based on the patch-level semantic affinity matrix obtained in step 3, the ability of semantic affinity to capture more spatial relationships and fine-grained information is used to generate a class activation map CAM from the patch tokens output by the last layer of Transformer encoder block, and the patch semantic affinity matrix is ​​used to perform matrix multiplication with CAM to complete the optimization of CAM, making CAM more refined and accurate, thereby obtaining better pseudo-labels for supervising the training of the segmentation model.

[0062] Step 401, from step 102, the output token sequence of the last layer of Transformer encoding layer , remove the dimension corresponding to the class token and rearrange the patch token sequence , and perform a 1×1 convolution operation to convert the token dimension into the number of object categories. The formula is as follows:

[0063]

[0064] in, The input channel is D, the output channel is the number of object categories C, and the convolution kernel size is 1×1. The initial class activation map from the patch token can be obtained by the above formula .

[0065] Step 402: Since the initial class activation map is rough and may contain a lot of noise and errors, it is necessary to use the patch semantic affinity in step 302 to optimize the initial class activation map. The formula is as follows:

[0066]

[0067] The optimized class activation graph can be obtained through the above formula , which is more refined and accurate than the initial CAM, thus obtaining better pseudo labels for supervising the training of the segmentation model. The optimization flow chart is as follows Figure 5 As shown:

[0068] To further verify the effect of the CAM optimization method, the effectiveness and accuracy of the CAM optimization method are simulated, and the results are shown in Tables 1, 2, and 3. The Pascal VOC 2012 dataset is used as the training set for this experiment, and 20,000 iterations are performed. The results are evaluated on the PASCALVOC online evaluation system. The ToCo (CVPR2023) weakly supervised semantic segmentation method is also trained and evaluated on the Pascal VOC 2012 dataset, and the results are compared with the present method. Where cam is the cam generated from the patch token output by the last layer of the Transformer encoder block, and aux_cam is the auxiliary cam generated from the middle layer of the Transformer. mIoU is mean intersection over union, which is one of the most commonly used evaluation indicators for semantic segmentation. train, val, and test are the training set, validation set, and test set of the Pascal VOC 2012 dataset, respectively. It can be concluded from Tables 1, 2, and 3 that the quality of the generated CAM and pseudo-labels, as well as the performance of the final segmentation model, of the present invention exceeds the ToCo weakly supervised semantic segmentation method.

[0069]

Claims

1. A CAM optimization method based on patch semantic affinity enhancement, characterized in that: The steps include: Step 1: Extract multi-head self-attention weights from the Vision Transformer backbone network; Step 2: For the multi-head self-attention weights extracted in step 1, use the head average operation to remove redundant information and reduce the amount of calculation, then aggregate the global features of each attention map through global average pooling, and input the aggregated feature vector into the multi-layer perceptron for interaction. The multi-layer perceptron outputs feature vectors of the same shape, and finally obtains an enhanced attention map from the importance of attention at different layers; Step 3: The enhanced attention maps from different layers of attention importance in step 2 are averaged to fuse, and the dimensions corresponding to the class tokens in the self-attention are removed. The remaining weights are used as the enhanced patch-level semantic affinity matrix; Step 4: Based on the patch-level semantic affinity matrix obtained in step 3, the ability of semantic affinity to capture more spatial relationships and fine-grained information is used to generate a class activation map CAM from the patch tokens output by the last layer of Transformer encoder block, and the patch-level semantic affinity matrix is ​​used to perform matrix multiplication with CAM to complete the optimization of CAM, making CAM more refined and accurate, thereby obtaining better pseudo-labels for supervising the training of the segmentation model.

2. A CAM optimization method based on patch semantic affinity enhancement according to claim 1, characterized in that: Step 1 specifically includes the following steps: Step 101: Split the input image into N×N patches, and convert them into a patch token sequence of dimension D by linear conversion, linking the previous class token of dimension D to form the input token sequence of ViT encoder ; Step 102, the ViT encoder consists of K encoding layers, each of which contains a multi-head self-attention and a multi-layer perceptron, as well as two layer normalizations before two sub-layers. The ViT encoder receives the input token sequence And output the token sequence ; Step 103: Extract the multi-head self-attention weight tensor from the multi-head self-attention module of the K-layer encoding layer in ViT , where H is the number of multi-head self-attention heads.

3. The CAM optimization method based on patch semantic affinity enhancement according to claim 1, characterized in that: Step 2 specifically includes the following steps: Step 201: For the obtained multi-head self-attention weight tensor , use the head average operation to remove the dimension H, remove redundant information and reduce the amount of calculation, and get ; Step 202: For the head averaged attention map obtained in step 201 , perform a concatenate operation on the 0th dimension to connect the K layers of attention maps in series to obtain a three-dimensional attention tensor ; Step 203: For the attention tensor obtained in step 202 , the global features of the K-layer attention map are aggregated by global average pooling to obtain the aggregated feature vector of length K , and the feature vector is input into the multi-layer perceptron to interact with each other, and the multi-layer perceptron outputs a feature vector of the same shape ; Step 204: Obtain the feature vector output by the multi-layer perceptron Then, the head average attention tensor obtained in step 201 is With the eigenvector Combined, the formula is as follows: in Represents the element-by-element multiplication symbol. The above formula can be used to obtain an enhanced attention map that fully considers the importance of attention at different layers. .

4. The CAM optimization method based on patch semantic affinity enhancement according to claim 1, characterized in that: Step 3 specifically includes the following steps: Step 301: Based on the result of step 2, the enhanced attention map Perform an average operation on the 0th dimension K to fuse the enhanced attention maps from different layers to obtain the fused enhanced attention ; Step 302: Enhance the attention of the fusion obtained in step 301 , removing the dimension corresponding to the class token, the remaining enhanced attention weights can be used as the enhanced patch-level semantic affinity .

5. The CAM optimization method based on patch semantic affinity enhancement according to claim 1, characterized in that: Step 4 specifically includes the following steps: Step 401: From step 102, we know that the output token sequence of the last Transformer encoding layer is , remove the dimension corresponding to the class token and rearrange the patch token sequence , and perform a 1×1 convolution operation to convert the token dimension into the number of object categories. The formula is as follows: in, The input channel is D, the output channel is the number of object categories C, and the convolution kernel size is 1×1. The initial class activation map from the patch token can be obtained by the above formula ; Step 402: Optimize the initial class activation map using the patch semantic affinity in step 302. The formula is as follows: The optimized class activation map can be obtained through the above formula , which is more refined and accurate than the initial CAM, thus obtaining better pseudo labels for supervising the training of segmentation models.

6. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method of claim 1.

7. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to claim 1 are implemented.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to claim 1 are implemented.

Images