An equivariant weakly supervised semantic segmentation method based on attention mechanism

Abstract

This paper discloses an equivariant weakly supervised semantic segmentation method based on an attention mechanism, comprising the following steps: Step 1: generating a class activation map; Step 2: inputting the generated class activation map into an AffinityNet model for further refinement to obtain pseudo-labels; Step 3: training the generated pseudo-labels using a fully supervised DeepLab v1 model, wherein the VGGNet network in the DeepLab v1 model is replaced with a ResNet network, and the prediction results obtained by the DeepLab v1 model are used as the final segmentation results. This method reduces the significant human and material resources consumed by manual labeling and achieves highly accurate segmentation results.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an equivariant weakly supervised semantic segmentation method based on an attention mechanism. Background Art

[0002] Semantic segmentation is a fundamental and critical task in computer vision, aiming to perform pixel-level classification predictions on images. In recent years, with the rapid development of deep learning, semantic segmentation models have been continuously optimized, achieving significant performance improvements, spurring the development of intelligent driving, smart security, satellite remote sensing imagery, medical image analysis, and other fields. However, semantic segmentation requires extensive manual labeling of each pixel, which is both expensive and time-consuming. Consequently, many works have begun to adopt weakly supervised semantic segmentation (WSSS), which addresses the problem of manual pixel-by-pixel labeling required for fully supervised semantic segmentation through weak supervision methods such as bounding boxes, graffiti, and image-level classification labels. Our research focuses on weakly supervised semantic segmentation with image-level classification labels.

[0003] According to previous research, most weakly supervised semantic segmentation with image-level classification labels uses class activation maps to generate initial pseudo labels. However, class activation maps typically only highlight the most discriminative regions of the object and incorrectly activate background regions, which can be summarized as under-activation and over-activation, respectively. Furthermore, when augmenting images with affine transformations, the generated class activation maps are inconsistent. Summary of the Invention

[0004] The purpose of the present invention is to address the deficiencies in the existing technology and propose an equivariant weakly supervised semantic segmentation method based on the attention mechanism. This method reduces the large amount of manpower and material resources consumed by manual labeling, and the segmentation results are highly accurate.

[0005] The technical solution for achieving the purpose of the present invention is:

[0006] An equivariant weakly supervised semantic segmentation method based on an attention mechanism includes the following steps:

[0007] Step 1 Generate class activation map:

[0008] 1.1 Input the image into the Resnet38 classification network with the ReLU layer removed to extract features;

[0009] 1.2 Then use the CSM module to extract important but easily overlooked features, namely secondary features;

[0010] 1.3 Then the extracted features are input into CAM to obtain the class activation map, which is recorded as output 1;

[0011] 1.4 The output features of the fourth and fifth layers of the Resnet38 classification network are fused, and then the fused features are input into the pixel correlation module PCM to further capture and correct the class activation map, obtaining output 2;

[0012] 1.5 Scale the image as input and perform the same operations as steps 1.1-1.4 as compensation supervision to obtain output 3 and output 4;

[0013] 1.6 Finally, the three loss functions are used to adjust the obtained outputs 1, 2, 3, and 4 to obtain the optimized class activation map. Steps 1.1 to 1.6 are the proposed DSEAM network. Step 2 inputs the generated class activation map into the AffinityNet model for further refinement to obtain pseudo labels.

[0014] In step 3, the pseudo labels generated in step 2 are trained using the fully supervised DeepLab v1 model, where the VGGNet network in the DeepLab v1 model is replaced by the Resnet network, and the prediction result obtained by the DeepLab v1 model is used as the final segmentation result.

[0015] The CSM module in step 1.2 includes a channel attention module and a spatial attention module;

[0016] First, the input features are sent to the channel attention module to enhance the secondary features in the channel dimension and suppress the most sensitive and least sensitive features. Then, the features before the input channel attention module are multiplied by the features after the channel attention module to achieve the secondary feature activation result in the channel dimension; recorded as output feature 1; then the output feature 1 is input to the spatial attention module to adjust the secondary features enhanced by the channel dimension in the spatial dimension, and further activate the secondary features; then the obtained output is multiplied by the output feature 1 to obtain the enhanced secondary features; then, the relu activation function is used to increase the sensitivity of the features, thereby improving the accuracy of the extraction; finally, the final result after the CSM module is obtained, and the process is expressed as:

[0017]

[0018] Among them, F represents the output of the classification network, Ca represents the channel attention module, Sa represents the spatial attention module, δ represents the relu function, represents element-wise multiplication;

[0019] The channel attention module can help the backbone network extract the area required for the semantic segmentation task in the channel dimension. The features are first extracted and enhanced through the maximum pooling layer and the average pooling layer. The output obtained by the fully connected network is then supplemented and adjusted. Finally, the two outputs are fused with sigmoid activation to obtain the enhanced secondary features. The specific operation is expressed as follows:

[0020]

[0021] Among them, F is the output of the classification network, Avg is the average pooling, Max represents the maximum pooling, MLP is the fully connected layer, Represents the sigmoid function;

[0022] The spatial attention module can help the backbone network extract the areas required for the semantic segmentation task in the spatial dimension. The input features are first activated by the maximum pooling layer and the average pooling layer for the secondary feature area, and then fused convolution is performed to re-extract features in the spatial dimension. Finally, the output is obtained through sigmoid activation. The specific operation is expressed as follows:

[0023]

[0024] Among them, F is the output of the classification network, Avg is the average pooling, Max is the maximum pooling, represents the sigmoid function, C 7×7 Indicates that the filter is a 7×7 convolution.

[0025] The pixel correlation module is the core part of the self-attention mechanism. By adding the pixel correlation module at the end of the network, training and correction are performed under the supervision of equivariant regularization. The cosine distance is used to evaluate the feature similarity between pixels. At the same time, the affinity relationship between the current pixel and other pixels is calculated by the inner product in the normalized feature space. ReLU activates the similarity to suppress negative values. The final CAM is the weighted sum of the original CAM with normalized similarity. The process can be expressed as:

[0026]

[0027] The three loss functions in step 1.6 include classification loss, equivariant regularization, and cross-equivariant regularization. The network uses image-level classification labels as weakly supervised label information. According to the CAM network architecture, a global average pooling layer is added at the end of the network to obtain the classification prediction vector z for the image. and Represent the original CAM output after the two branches of the affine transformation network, y 0 and y t Represent the output of the two branches modified by PCM, the classification loss L cls It is expressed as follows:

[0028]

[0029]

[0030] Where c-1 represents the number of categories excluding background, l cls represents the multi-label classification loss of a single branch, z 0 and z t Represents the prediction vectors aggregated by the global average pooling layer of the two branches of the affine transformation network;

[0031] The equivariant regularization loss means that the CAM generated after the image is affine transformed maintains the same transformation rules as the CAM of the original image, ensuring equivariance. The equivariant loss L er It is expressed as follows:

[0032]

[0033] Where A represents the affine transformation function;

[0034] Cross-equivariant regularization loss means that the output of PCM is supervised by the CAM output of the other branch of the twin network. It can not only prevent the output of PCM from falling into the local optimal solution, but also avoid the degradation of CAM during the PCM correction process. Cross-equivariant regularization loss L ecr It is expressed as follows:

[0035]

[0036] The loss function L of the network is expressed as:

[0037] L=L cls +L er +L ecr (9).

[0038] Compared with the existing technology, this technical solution has the following beneficial effects:

[0039] The method of this technical solution can complete the semantic segmentation task using only image-level labels, greatly reducing the large amount of manpower and material resources consumed by manual labeling. Compared with the existing weakly supervised semantic segmentation method, the segmentation results of this method are more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flow chart of an embodiment;

[0041] Figure 2 This is a DSEAM network model diagram in the embodiment;

[0042] Figure 3 A comparison chart of the segmentation effect of the embodiment and the existing advanced methods.

[0043] Figure 3 In the figure, (a) is the original image, (b) is the true label, (c) is the segmentation result of other methods, and (d) is the segmentation result of this method. DETAILED DESCRIPTION

[0044] The invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, but the invention is not limited thereto.

[0045] Example:

[0046] In this example, only image-level classification labels are available during training. The epoch and batch size are set to 8. The DSEAM network model is trained on three TITAN XP GPUs using the stochastic gradient descent algorithm. The learning rate is updated using the "Poly" strategy with polynomial decay, as shown below:

[0047]

[0048] Among them, irl is the initial value of the learning rate, which is set to 0.01, it represents the number of iterations, max_it is the maximum number of iterations, and time is used for decay, which is set to 0.9.

[0049] Reference Figure 1 , an equivariant weakly supervised semantic segmentation method based on attention mechanism, comprising the following steps:

[0050] Step 1 Generate class activation map:

[0051] 1.1 Input the image into the Resnet38 classification network with the ReLU layer removed;

[0052] 1.2 Then use the CSM module to extract important but easily overlooked features, namely secondary features;

[0053] 1.3 Then, the extracted features are input into CAM to obtain the class activation map, which is recorded as output 1;

[0054] 1.4 The output features of the fourth and fifth layers of the Resnet38 classification network are fused, and then the fused features are input into the pixel correlation module together with output 1 to further capture and correct the class activation map, obtaining output 2;

[0055] 1.5 Scale the image as input and perform the same operations as steps 1.1-1.4 as compensation supervision to obtain output 3 and output 4;

[0056] 1.6 Finally, the three loss functions are used to adjust the obtained output 1, output 2, output 3 and output 4 to obtain the optimized class activation map; Steps 1.1 to 1.6 are the proposed DSEAM network, as shown in Figure 2 As shown;

[0057] Step 2: Input the generated class activation map into the AffinityNet model for further refinement to obtain pseudo labels;

[0058] In step 3, the pseudo labels generated in step 2 are trained using the fully supervised DeepLab v1 model, where the VGGNet in the DeepLab v1 model is replaced with the ResNet network, and the prediction results obtained by the DeepLab v1 model are used as the final segmentation results. The CSM module in step 1.2 includes a channel attention module and a spatial attention module;

[0059] First, the input features are sent to the channel attention module to enhance the secondary features and suppress the most sensitive and least sensitive features in the channel dimension. Then, the features before the input channel attention module are multiplied by the features after the channel attention module to achieve the secondary feature activation result in the channel dimension; recorded as output feature 1; then the output feature 1 is input to the spatial attention module to adjust the secondary features enhanced by the channel dimension in the spatial dimension, and further activate the secondary features; then the obtained output is multiplied by the output feature 1 to obtain the enhanced secondary features; then, the relu activation function is used to increase the sensitivity of the features, thereby improving the accuracy of the extraction; finally, the final result after the CSM module is expressed as follows:

[0060]

[0061] Among them, F represents the output of the classification network, Ca represents the channel attention module, Sa represents the spatial attention module, δ represents the relu function, represents element-wise multiplication;

[0062] The channel attention module can help the backbone network extract the area required for the semantic segmentation task in the channel dimension. The features are first extracted and enhanced through the maximum pooling layer and the average pooling layer. The output obtained by the fully connected network is then supplemented and adjusted. Finally, the two outputs are fused with sigmoid activation to obtain the enhanced secondary features. The specific operation is expressed as follows:

[0063]

[0064] Among them, F is the output of the classification network, Avg is the average pooling, Max represents the maximum pooling, MLP is the fully connected layer, Represents the sigmoid function;

[0065] The spatial attention module can help the backbone network extract the areas required for the semantic segmentation task in the spatial dimension. The input features are first activated by the maximum pooling layer and the average pooling layer for the secondary feature area, and then fused convolution is performed to re-extract features in the spatial dimension. Finally, the output is obtained through sigmoid activation. The specific operation is expressed as follows:

[0066]

[0067] Among them, F is the output of the classification network, Avg is the average pooling, Max is the maximum pooling, represents the sigmoid function, C 7×7 Indicates that the filter is a 7×7 convolution.

[0068] The pixel correlation module is the core part of the self-attention mechanism. By adding the pixel correlation module at the end of the network, training and correction are performed under the supervision of equivariant regularization. The cosine distance is used to evaluate the feature similarity between pixels. At the same time, the affinity relationship between the current pixel and other pixels is calculated by the inner product in the normalized feature space. ReLU activates the similarity to suppress negative values. The final CAM is the weighted sum of the original CAM with normalized similarity. The process can be expressed as:

[0069]

[0070] The three loss functions in step 1.6 include classification loss, equivariant regularization, and cross-equivariant regularization. The network uses image-level classification labels as weakly supervised label information. According to the CAM network architecture, a global average pooling layer is added at the end of the network to obtain the classification prediction vector z for the image. and Represent the original CAM output after the two branches of the affine transformation network, y 0 and y t Represent the output of the two branches modified by PCM, the classification loss L cls It is expressed as follows:

[0071]

[0072]

[0073] Where c-1 represents the number of categories excluding background, l cls represents the multi-label classification loss of a single branch, z 0 and z t Represents the prediction vectors aggregated by the global average pooling layer of the two branches of the affine transformation network;

[0074] The equivariant regularization loss means that the CAM generated after the image is affine transformed maintains the same transformation rules as the CAM of the original image, ensuring equivariance. The equivariant loss L er It is expressed as follows:

[0075]

[0076] Where A represents the affine transformation function;

[0077] Cross-equivariant regularization loss means that the output of PCM is supervised by the CAM output of the other branch of the twin network. It can not only prevent the output of PCM from falling into the local optimal solution, but also avoid the degradation of CAM during the PCM correction process. Cross-equivariant regularization loss L ecr It is expressed as follows:

[0078]

[0079] The loss function L of the network is expressed as:

[0080] L=L cls +L er +L ecr (9).

[0081] like Figure 3 As shown in the figure, the experimental results show that (a) is the original picture, (b) is the real label, (c) is the segmentation result of other methods, and (d) is the segmentation result of this method. This method is more accurate than the existing weakly supervised semantic segmentation method. The performance improvement of this method mainly comes from the capture of secondary features that are easily inactivated due to complex backgrounds. This produces a better class activation map for the segmentation task, thereby obtaining more accurate pseudo labels, making the final segmentation effect better than the existing technology.

Claims

1. A method for equivariant weakly supervised semantic segmentation based on an attention mechanism, comprising the following steps: Step 1 Generate class activation map: 1.1 Input the image into the Resnet38 classification network with the ReLU layer removed to extract features; 1.2 Then use the CSM module to extract important but easily overlooked features, namely secondary features; 1.3 Then input the extracted features into CAM to obtain the class activation map, which is recorded as output 1; 1.4 The output features of the fourth and fifth layers of the Resnet38 classification network are fused, and then the fused features are input together with output 1 into the pixel correlation module PCM to further capture and correct the class activation map in step 1.3 to obtain output 2; 1.5 Scale the image as input and perform the same operations as steps 1.1-1.4 as compensation supervision to obtain output 3 and output 4; 1.6 Finally, use the three loss functions to adjust output 1, output 2, output 3, and output 4 to obtain the optimized class activation map; steps 1.1 to 1.6 are the proposed DSEAM network; The three loss functions include classification loss, equivariant regularization, and cross-equivariant regularization. The network uses image-level classification labels as weakly supervised label information. According to the CAM network architecture, a global average pooling layer is added at the end of the network to obtain the classification prediction vector z for the image. and Represent the original CAM output after the two branches of the affine transformation network, and Represent the output of the two branches corrected by the pixel correlation module, the classification loss It is expressed as follows: （5） （6） in Indicates the number of categories excluding background, represents the multi-label classification loss of a single branch, and Represents the prediction vectors aggregated by the global average pooling layer of the two branches of the affine transformation network; Equivariant regularization loss means that the CAM generated after the image is affine transformed maintains the same transformation rules as the CAM of the original image, ensuring equivariance and equivariant loss. It is expressed as follows: （7） in represents the affine transformation function; Cross-equivariant regularization loss means that the output of PCM is supervised by the CAM output of the other branch of the twin network. It can not only prevent the output of PCM from falling into the local optimal solution, but also avoid the degradation of CAM during the PCM correction process. Cross-equivariant regularization loss It is expressed as follows: （8） The network's loss function express: （9）； Step 2: Input the class activation map generated in step 1.6 into the AffinityNet model for further refinement to obtain pseudo labels. In step 3, the pseudo labels generated in step 2 are trained using the fully supervised DeepLab v1 model, where the VGGNet network in the DeepLab v1 model is replaced with the ResNet network, and the prediction result obtained by the DeepLab v1 model is used as the final segmentation result.

2. The equivariant weakly supervised semantic segmentation method based on the attention mechanism according to claim 1 is characterized in that: The CSM module in step 1.2 includes a channel attention module and a spatial attention module; First, the input features are sent to the channel attention module to enhance the secondary features in the channel dimension and suppress the most sensitive and least sensitive features. Then, the features before the input channel attention module are multiplied by the features after the channel attention module to achieve the secondary feature activation result in the channel dimension; recorded as output feature 1; then the output feature 1 is input to the spatial attention module to adjust the secondary features enhanced by the channel dimension in the spatial dimension, and further activate the secondary features; then the obtained output is multiplied by the output feature 1 to obtain the enhanced secondary features; then, the relu activation function is used to increase the sensitivity of the features, thereby improving the accuracy of the extraction; finally, the final result after the CSM module is obtained, and the process is expressed as: （1） in, represents the output of the classification network, represents the channel attention module, represents the spatial attention module, represents the relu function, represents element-wise multiplication; The channel attention module can help the backbone network extract the area required for the semantic segmentation task in the channel dimension. The features are first extracted and enhanced through the maximum pooling layer and the average pooling layer. The output obtained by the fully connected network is then supplemented and adjusted. Finally, the two outputs are fused with sigmoid activation to obtain the enhanced secondary features. The specific operation is expressed as follows: in, is the output of the classification network, is average pooling, represents maximum pooling, is the fully connected layer, Represents the sigmoid function; The spatial attention module can help the backbone network extract the areas required for the semantic segmentation task in the spatial dimension. The input features are first activated by the maximum pooling layer and the average pooling layer for the secondary feature area, and then fused convolution is performed to re-extract features in the spatial dimension. Finally, the output is obtained through sigmoid activation. The specific operation is expressed as follows: in, is the output of the classification network, is average pooling, is the maximum pooling, represents the sigmoid function, Indicates that the filter is a 7×7 convolution.

3. The equivariant weakly supervised semantic segmentation method based on the attention mechanism according to claim 1 is characterized in that: The pixel correlation module PCM is the core part of the self-attention mechanism. By adding the pixel correlation module at the end of the network, training and correction are performed under the supervision of equivariant regularization. The cosine distance is used to evaluate the feature similarity between pixels. At the same time, the affinity relationship between the current pixel and other pixels is calculated by the inner product in the normalized feature space. ReLU activates the similarity to suppress negative values. The final CAM is the weighted sum of the original CAM with normalized similarity. The process is expressed as: （4）。

Images