Weakly Supervised Semantic Segmentation Method and Device Based on Naive Vision Transformer

Abstract

The present invention discloses a weakly supervised semantic segmentation method based on a naive Vision Transformer, comprising: inputting patch labels and learnable class labels obtained based on an input image into a transformer encoder to obtain feature output; then obtaining a coarse CAM based on the output patch label portion through permutation and convolution operations, and extracting a self-attention map from the encoder model parameters; then fusing the self-attention map through an adaptive attention map fusion module to obtain a cross-attention map and a patch self-attention map, and sequentially optimizing the coarse CAM to obtain a final fine CAM; finally, obtaining a category prediction based on the class label output by the encoder, the coarse CAM, and the fine CAM through global pooling, and calculating a cross-entropy loss function with the category label to optimize the network; and online retraining based on a gradient truncation decoder. The present invention also provides a corresponding weakly supervised semantic segmentation device based on a naive Vision Transformer.

Description

Technical Field

[0001] The present invention belongs to the technical field of weakly supervised semantic segmentation in computer vision, and more specifically, relates to a weakly supervised semantic segmentation method and device based on a naive Vision Transformer. Background Art

[0002] Semantic segmentation is a fundamental task in computer vision, which aims to predict the category label for each pixel in an image. With the rapid development of deep neural networks, the performance of semantic segmentation has been greatly improved. However, strongly supervised learning methods based on pixel-level annotations are very expensive and time-consuming. To reduce the burden of manual labeling, more and more researchers are turning to the study of weakly supervised semantic segmentation (WSSS). Weakly supervised semantic segmentation can be achieved using only bounding box annotations, scribbled annotations, point annotations, or image-level classification annotations. Compared with pixel-by-pixel annotation, weakly supervised annotation consumes less time and is less expensive. This paper focuses on the most challenging WSSS method based on image-level classification annotation.

[0003] Due to the lack of pixel-level annotations, WSSS methods based on image-level classification annotations usually need to first generate coarse location annotations through Class Activation Maps (CAM). CAM is a technique based on deep classification networks that is used to generate feature maps with the same number of channels as the total number of categories. The feature map contains the approximate location of objects in each category, and we can obtain the initial CAM seeds from the feature map. However, CAM can only identify the most discriminative areas of the object, and other methods are usually required to optimize the CAM to obtain pseudo labels.

[0004] In order to identify more complete objects, many methods focus on how to optimize CAM. The SEC method expands sparse CAM labels through seed expansion methods. DSRG combines seed region growing methods to expand CAM clues. A similar method is DGCN, which assigns labels to areas around seeds using traditional graph cutting algorithms. AffinityNet and IRNet use random walk methods to expand labels. AuxSegNet propagates labels by learning cross-task affinity. Some methods use reverse erasing to help CAM focus more attention on non-discriminative areas. SEAM explores the consistency of CAM under different affine transformations. In addition, some methods choose to introduce network data, such as Co-segmentation and STC. However, this method of using methods such as random walks to enhance annotations and then retraining on Deeplab is relatively complicated.

[0005] In recent years, many researchers have applied Transformers to computer vision. The VisionTransformer (ViT) first demonstrated the superior performance of Transformers in image classification tasks. The key idea of the ViT model is to convert an image into non-overlapping patch labels, which are then fed into a Transformer architecture along with class labels. Finally, a fully connected layer is used to map the class labels to category predictions. With the advent of ViT, several Transformer-based methods for obtaining Camera (CAM) seeds have emerged. TS-CAM first introduced the Transformer to the task of weakly supervised object localization. This work used a cross-attention map between class labels and patch labels to obtain localization cues. This cross-attention map is obtained by averaging the attention heads of the same layer and then summing them across different layers. The cross-attention map is then combined with the Camera (CAM) generated by convolution with the patch labels. Following this approach, MCTformer introduced the Transformer to the task of weakly supervised semantic segmentation, proposing to take multiple class labels as input to learn cross-attention maps for different classes. During the CAM generation stage, the CAM is further optimized by using patch self-attention maps. In addition, TransCAM uses a hybrid transformer and convolutional conformer as the backbone network, and also uses patch self-attention maps to optimize CAM.

[0006] The above Transformer-based WSSS method mainly uses the Transformer's self-attention map to enhance the CAM seed in the post-processing stage. When we get the self-attention map, the above method directly averages the self-attention maps of different heads under the same layer, and then sums them up at different layers. However, we should point out that the method of averaging and summing the Transformer self-attention maps in the post-processing stage is rough. Since different attention heads of the Transformer focus on different targets, the corresponding details and semantic information are also different. If the mean summation method is used, the attention heads that are not focused on the foreground object will be given the same weight as the attention heads that focus on the foreground object. Summary of the Invention

[0007] To address the above-mentioned shortcomings or improvement needs of the existing technology, this paper provides a weakly supervised semantic segmentation method based on the naive Vision Transformer, WeakTr. It proposes the use of an adaptive attention fusion module to measure the importance of different attention heads to the CAM and the use of a gradient truncated decoder based on ViT for online retraining of pseudo-labels. Its purpose is to improve the performance of weakly supervised semantic segmentation based on the Vision Transformer, solving the problems of irrelevant attention heads in the Transformer misleading the CAM generation results and the time-consuming and complex CAM refinement and retraining of the traditional WSSS method.

[0008] To achieve the above objectives, according to one aspect of the present invention, a weakly supervised semantic segmentation method based on a naive VisionTransformer is provided, comprising the following steps:

[0009] (1) The classification network framework used to generate CAM uses naive ViT as the backbone network, and inputs the patch labels and learnable class labels obtained based on the input image into the transformer encoder to obtain feature output; then the coarse CAM is obtained by permutation and convolution operations based on the output patch label part, and the self-attention map is extracted from the model parameters of the encoder; then the self-attention map is fused through the adaptive attention map fusion module to obtain the cross attention map and patch self-attention map, which are then optimized in turn to obtain the final fine CAM; finally, the class label, coarse CAM and fine CAM are respectively obtained by global pooling, and the cross entropy loss function is calculated with the class label to optimize the network;

[0010] (2) Online retraining based on gradient truncation decoder, where the implementation of transformer encoder is the same as that in the previous step. In order to process the noise in CAM seed, gradient clipping is added to transformer decoder to truncate the gradient return of noise during back propagation. The retrained model is used to realize semantic segmentation of the image.

[0011] In one embodiment of the present invention, the Transformer encoder is composed of multiple layers of multi-head self-attention MSA layers. After the patch labels and learnable class labels obtained based on the input image are input into the encoder, the corresponding feature output will be obtained.

[0012] In one embodiment of the present invention, the implementation of the Transformer encoder is specifically as follows:

[0013] Divide the input image into N 2 The block is then transformed linearly to obtain N 2patch tags, and set C learnable category tags, which are combined with the patch tags as the input of the transformer encoder D is the dimension of the input token; the Transformer encoder consists of K encoding layers, and at each encoding layer, the input token T in First, it needs to be normalized through the LayerNorm layer and then input into the MSA layer:

[0014] T attn =T in +MSA(LN(T in ))#

[0015] Then, the output of the MSA layer also needs to be normalized by LN and then input into the multi-layer perceptron MLP layer to obtain the output of the encoding layer:

[0016] T out =T attn +MLP(LN(T attn ))#

[0017] Then for the next coding layer, T out As T in Input, repeat K times to get the final output

[0018] In one embodiment of the present invention, the rough CAM generation specifically includes:

[0019] After obtaining the feature output of the encoder, the patch marking part of the output is passed through the convolution layer to obtain a rough CAM, and the self-attention map is extracted from the model parameters of the encoder for the subsequent adaptive attention map module.

[0020] In one embodiment of the present invention, extracting the self-attention map specifically includes:

[0021] Generate a class activation map CAM based on the output of the encoder, which is divided into class label parts And the patch mark part Use patch marking parts to generate CAM; first rearrange the patch parts and pass them through a convolutional layer to obtain the initial rough class activation map Then extract the self-attention map from the encoder model parameters To optimize CAM coarse , where H is the number of attention heads in each encoding layer; considering class labels and patch labels, the self-attention map is divided into a cross attention map and patch attention map

[0022] In one embodiment of the present invention, adaptive attention graph fusion specifically includes:

[0023] Get dynamic weights from self-attention map A through global pooling Then input it into the feedforward network FFN to get the interaction weight As the weight of the merged attention map, it is finally multiplied back to the self-attention map to obtain the weighted merged self-attention map result:

[0024]

[0025]

[0026] In one embodiment of the present invention, optimizing the coarse CAM specifically includes:

[0027] use and CAM coarse Optimize to get the final output CAM:

[0028]

[0029] in is to transform the matrix shape to N×N×C, Is to transform the matrix shape to N 2 ×C, ⊙ refers to the Hadamard product;

[0030] After obtaining the final CAM output, the argmax operation is performed on the CAM to obtain the segmentation mask, and then the density conditional random field CRF is used for post-processing to obtain the CAM seed, which will be retrained online in the second step.

[0031] In one embodiment of the present invention, the class labels output by the encoder, the coarse CAM, and the fine CAM are used to obtain class predictions through global pooling, and the cross entropy loss function is calculated with the class labels to optimize the network.

[0032] First, the output class labeling part T of the encoder final-classes Get category prediction through the global pooling layer Then calculate the cross entropy loss function with the category label y:

[0033]

[0034]

[0035] Then for CAM coarse and CAM fine Similarly, the category prediction is obtained through the global pooling layer, and the loss function is calculated with the label y to obtain L Coarse-CAM and L Fine-CAM;

[0036] Finally, the above loss functions are summed to get the total loss:

[0037]

[0038] In one embodiment of the present invention, step (2) specifically includes:

[0039] First, the class label Q and patch label T obtained by the transformer encoder are input into the transformer decoder to obtain and Normalize and multiply these two items and then upsample to get the segmentation prediction result

[0040]

[0041] Then the gradient matrix is obtained by calculating the cross entropy loss function with the prediction result and the CAM seed Gradient truncation is achieved by setting a threshold on the gradient matrix, that is, truncating the back propagation of a larger part of the gradient; in order to achieve local gradient truncation, the matrix is divided into L 2 piece Calculate the local average gradient value {λ i} and the global average gradient value λ global ;

[0042]

[0043] λ global =mean(λ i ), i∈[1,...,L 2}

[0044] We will select {λ i} and λ global The larger value is used as the threshold to obtain the truncation mask

[0045]

[0046] Set the initial truncation threshold τ, only when the global average gradient λ global Gradient truncation begins only when it is less than τ, and the gradient matrix obtained for back propagation is as follows:

[0047]

[0048] By multiplying the truncation mask to the original gradient matrix, some holes are dynamically created in the gradient matrix before backpropagation during training. The network prediction results in this part will be retained, thereby achieving self-correction of the erroneous areas in the CAM seed.

[0049] According to another aspect of the present invention, a weakly supervised semantic segmentation device based on a naive Vision Transformer is also provided, comprising at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to complete the weakly supervised semantic segmentation method based on the naive Vision Transformer.

[0050] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:

[0051] (1) We exploit the inherent properties of multi-layer multi-head self-attention maps in naive ViT and design an effective adaptive attention fusion strategy to generate high-quality class activation maps. We also reveal for the first time the importance of different attention heads for CAM and WSSS.

[0052] (2) We propose an online retraining method with a gradient truncation decoder to exploit the contextual output tags of ViT, which allows us to retrain and self-correct CAM seeds through online retraining;

[0053] (3) The proposed WeakTr fully exploits the potential of naive ViT in the field of WSSS and achieves state-of-the-art results on two challenging WSSS benchmarks, namely, 78.4% and 46.9% mIoU on the PASCAL VOC 2012 and COCO 2014 validation sets, respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is an overview of the principles of the WeakTr method in an embodiment of the present invention. WeakTr first divides the input image into multiple blocks and then obtains image patch labels through linear transformation. It then merges them with multiple class labels and inputs them into the transformer encoder. Next, a coarse CAM is generated by applying a convolutional layer to the patch labels. Then, an adaptive attention fusion module is used to generate dynamic weights from all self-attention maps, and the dynamic weights are made to interact through a feedforward network (FFN). Finally, the coarse CAM is optimized into a fine CAM using a weighted cross attention map and a weighted patch attention map. The class labels, coarse CAM, and fine CAM are finally predicted by global average pooling to calculate the corresponding prediction loss.

[0055] Figure 2 This is an overview of the principle of the gradient truncation decoder in an embodiment of the present invention. In the present invention, the gradient truncation decoder first inputs the patch labels and class labels corresponding to the categories in the transformer encoder into the transformer decoder layer; after operating on the decoder layer, the corresponding patch predictions are obtained by combining the class labels and patch labels; the obtained patch predictions are then upsampled and the gradient map is calculated using the CAM seed; finally, a gradient clipping mask is dynamically generated and the gradient truncation mask is multiplied with the original gradient map. DETAILED DESCRIPTION

[0056] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0057] The WeakTr we proposed consists of two steps. First, CAM is generated based on a classification model and CAM seeds are obtained. Then, the CAM seeds are retrained online based on the gradient truncated decoder segmentation model. Next, I will explain the specific implementation methods of these two steps in turn.

[0058] 1.CAM generation

[0059] like Figure 1 As shown in the figure, the classification network framework we use to generate CAM uses naive ViT as the backbone network. First, the patch labels and learnable class labels obtained based on the input image are input into the transformer encoder to obtain feature output. Then, a coarse CAM is obtained by permutation and convolution operations based on the output patch label part, and the self-attention map is extracted from the encoder model parameters. The self-attention map is then fused through the adaptive attention map fusion module to obtain the cross attention map and patch self-attention map, which are then optimized in turn to obtain the final fine CAM. Finally, based on the class label output by the encoder, the coarse CAM and the fine CAM, the category prediction is obtained through global pooling, and the cross entropy loss function is calculated with the category label to optimize the network.

[0060] (1) Transformer Encoder

[0061] The Transformer encoder consists of multiple layers of Multi-head Self-Attention (MSA) layers. After the patch labels and learnable class labels obtained based on the input image are input into the encoder, the corresponding feature output will be obtained.

[0062] First, we divide the input image into N 2 The block is then transformed linearly to obtain N 2 patch tags, and we set C learnable category tags, which are merged with the patch tags as the input of the transformer encoder D is the dimension of the input token. The Transformer encoder consists of K encoding layers. At each encoding layer, the input token T in First, it needs to be normalized through the LayerNorm (LN) layer and then input into the MSA layer:

[0063] T attn =T in +MSA(LN(T in ))

[0064] Then, the output of the MSA layer also needs to be normalized by LN and then input into the Multi-Layer Perceptron (MLP) layer to obtain the output of the encoding layer:

[0065] T out =T attn +MLP(LN(T attn ))#

[0066] Then for the next coding layer, T out As T in Input, repeat K times to get the final output

[0067] (2) Rough CAM generation and extraction of self-attention maps

[0068] After obtaining the feature output of the encoder, we first obtain a coarse CAM based on the patch mark part of the output through the convolution layer, and extract the self-attention map from the model parameters of the encoder for the subsequent adaptive attention map module.

[0069] The next step is to generate the class activation map CAM based on the output of the encoder. The output of the encoder can be divided into class labeling parts. And the patch mark part We mainly use the patch marking part to generate CAM. First, we rearrange the patch part and pass it through a convolutional layer to obtain the initial rough class activation map. Then extract the self-attention map from the encoder model parameters To optimize CAM coarse , where H is the number of attention heads in each encoding layer. Considering class labels and patch labels, the self-attention map can be divided into cross-attention maps and patch attention map

[0070] (3) Adaptive Attention Graph Fusion and Optimized Coarse CAM

[0071] To merge the self-attention maps of all layers and all heads, unlike previous methods that simply average or sum them, we propose an adaptive attention map fusion module to evaluate the importance of different attention maps and merge all self-attention maps through a weighted summation. We then use the cross attention map and patch self-attention map in the self-attention map to optimize the coarse CAM to obtain the fine CAM.

[0072] First, we obtain dynamic weights from the self-attention map A through global pooling Then input it into the feed-forward network (FFN) to get the interaction weight As the weight of the merged attention map, it is finally multiplied back to the self-attention map to obtain the weighted merged self-attention map result:

[0073]

[0074]

[0075] use and CAM coarse Optimize to get the final output CAM:

[0076]

[0077] in is to transform the matrix shape to N×N×C, Is to transform the matrix shape to N 2 ×C, ⊙ refers to the Hadamard product.

[0078] After obtaining the final CAM output, the argmax operation is performed on the CAM to obtain the segmentation mask, and then the density conditional random field (CRF) is used for post-processing to obtain the CAM seed, which will be retrained online in the second step.

[0079] (4) Loss function calculation

[0080] Unlike previous methods that optimize the CAM in post-processing using transformer self-attention maps, our adaptive attention fusion module can be optimized during training. It optimizes the network by globally pooling the class labels output by the encoder, the coarse CAM, and the fine CAM to obtain class predictions. The cross-entropy loss function is then calculated with the class labels.

[0081] First, we label the output class of the encoder T final-classes Get category prediction through the global pooling layer Then calculate the cross entropy loss function with the category label y:

[0082]

[0083]

[0084] Then for CAM coarse and CAM fine Similarly, the category prediction is obtained through the global pooling layer, and the loss function is calculated with the label y to obtain L Coarse-CAM and L Fine-CAM .

[0085] Finally, the above loss functions are summed to get the total loss:

[0086]

[0087] 2. Online Retraining of Gradient-Truncation Decoder

[0088] like Figure 2 As shown in Figure 3, our online retraining model consists of a transformer encoder and a decoder based on gradient truncation. The implementation of the transformer encoder is the same as in the previous step. However, to deal with the noise in the CAM seed, we add a gradient clipping part to the transformer decoder to truncate the gradient return of the noise during backpropagation. The retrained model is used to perform semantic segmentation on the image to be segmented.

[0089] We first input the class label Q and patch label T obtained by the transformer encoder into the transformer decoder to obtain and Normalize and multiply these two items and then upsample to get the segmentation prediction result

[0090]

[0091] Then the gradient matrix can be obtained by calculating the cross entropy loss function with the prediction result and the CAM seed We implement gradient truncation by setting a threshold on the gradient matrix, that is, truncating the back propagation of a larger part of the gradient. To achieve local gradient truncation, we divide the matrix into L 2 piece The local average gradient value {λ i} and the global average gradient value λ global .

[0092]

[0093] λ global =mean(λ i ), i∈[1,...,L 2}

[0094] We will select {λ i} and λ global The larger value is used as the threshold to obtain the truncation mask

[0095]

[0096] During the experiment, we found that starting gradient truncation from the beginning during training has a greater impact on the results. Therefore, we set the initial truncation threshold τ. global Gradient truncation begins only when it is less than τ, so the gradient matrix we finally get for back propagation is as follows:

[0097]

[0098] By multiplying the truncation mask by the original gradient matrix, some holes (gradient values of 0) are dynamically created in the gradient matrix before backpropagation during training. The network prediction results in this part will be retained, thereby achieving self-correction of the erroneous areas in the CAM seed. During the inference process, we added CRF to improve the quality of the segmentation results.

[0099] After our online retraining, the network can already generate high-quality pseudo labels. From a practical point of view, we also get a segmentation network to output segmentation results without the need for another retraining step.

[0100] Furthermore, the present invention also provides a weakly supervised semantic segmentation device based on the naive Vision Transformer, comprising at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to complete the weakly supervised semantic segmentation method based on the naive Vision Transformer.

[0101] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A weakly supervised semantic segmentation method based on a naive Vision Transformer, characterized by: The steps include: (1) The classification network framework used to generate CAM uses naive ViT as the backbone network, and inputs the patch mark and learnable class mark obtained based on the input image into the transformer encoder to obtain feature output; then the coarse CAM is obtained by permutation and convolution operations based on the output patch mark part, and the self-attention map is extracted from the model parameters of the encoder; then the self-attention map is fused through the adaptive attention map fusion module to obtain the cross attention map and the patch self-attention map, and the coarse CAM is optimized in turn to obtain the final fine CAM; finally, the class mark, coarse CAM and fine CAM output by the encoder are respectively obtained by global pooling, and the cross entropy loss function is calculated with the class mark to optimize the network; among them, extracting the self-attention map specifically includes: generating the class activation map CAM based on the output of the encoder, and the output of the encoder is divided into the class mark part And the patch mark part , use the patch marking part to generate CAM; first rearrange the patch part and pass it through a convolutional layer to obtain the initial rough class activation map ; Then extract the self-attention map from the encoder model parameters To optimize ,in is the number of attention heads for each encoding layer; considering class labels and patch labels, the self-attention map is divided into cross attention maps and patch attention map ; (2) Online retraining based on gradient truncation decoder, where the implementation of transformer encoder is the same as that in the previous step. In order to deal with the noise in the CAM seed, gradient clipping is added to the transformer decoder to truncate the gradient return of the noise during back propagation. The retrained model is used to achieve semantic segmentation of the image.

2. The weakly supervised semantic segmentation method based on the naive Vision Transformer according to claim 1, characterized in that The Transformer encoder consists of multiple layers of multi-head self-attention MSA layers. After the patch labels and learnable class labels obtained based on the input image are input into the encoder, the corresponding feature output will be obtained.

3. The weakly supervised semantic segmentation method based on the naive Vision Transformer according to claim 2, characterized in that The implementation of the Transformer encoder is as follows: Divide the input image into The block is then linearly transformed to obtain patch markers, and set A learnable category label, which is combined with the patch label as the input of the transformer encoder , is the dimension of the input token; The Transformer encoder consists of encoding layers, in each encoding layer, the input mark First, it needs to be normalized through the LayerNorm layer and then input into the MSA layer: Then, the output of the MSA layer also needs to be normalized by LN and then input into the multi-layer perceptron MLP layer to obtain the output of the encoding layer: Then for the next encoding layer, As a Enter, repeat Get the final output .

4. The weakly supervised semantic segmentation method based on the naive Vision Transformer according to claim 1 or 2, characterized in that Rough CAM generation specifically includes: After obtaining the feature output of the encoder, the patch marking part of the output is passed through the convolution layer to obtain a rough CAM, and the self-attention map is extracted from the model parameters of the encoder for the subsequent adaptive attention map module.

5. The weakly supervised semantic segmentation method based on naive Vision Transformer according to claim 1 or 2, characterized in that Adaptive attention map fusion specifically includes: By global pooling from the self-attention map Get dynamic weight , and then input it into the feedforward network FFN to get the interaction weight As the weight of the merged attention map, it is finally multiplied back to the self-attention map to obtain the weighted merged self-attention map result:

6. The weakly supervised semantic segmentation method based on naive Vision Transformer according to claim 1 or 2, characterized in that Optimizing the rough CAM specifically includes: use and right Optimize to get the final output CAM: in The matrix shape is transformed into , The matrix shape is transformed into , It refers to the Hadamard product; After obtaining the final CAM output, the argmax operation is performed on the CAM to obtain the segmentation mask, and then the density conditional random field CRF is used for post-processing to obtain the CAM seed, which will be retrained online in the second step.

7. The weakly supervised semantic segmentation method based on naive Vision Transformer according to claim 1 or 2, characterized in that Based on the class label output by the encoder, the coarse CAM and the fine CAM, the category prediction is obtained through global pooling, and the cross entropy loss function is calculated with the category label to optimize the network; First, the output class of the encoder is labeled Get category prediction through the global pooling layer , and then with the category label Calculate the cross entropy loss function: Then for and The category prediction is also obtained through the global pooling layer and compared with the label Calculate the loss function to get and ; Finally, the above loss functions are summed to get the total loss:

8. The weakly supervised semantic segmentation method based on naive Vision Transformer according to claim 1 or 2, characterized in that: The step (2) specifically includes: First, the class label obtained by the transformer encoder and patch markers Input into the transformer decoder to get and , normalize and multiply these two items and then upsample to get the segmentation prediction result : Then the gradient matrix is obtained by calculating the cross entropy loss function with the prediction result and the CAM seed , gradient truncation is achieved by setting a threshold on the gradient matrix, that is, truncating the back propagation of a larger part of the gradient; in order to achieve local gradient truncation, the matrix is divided into piece , calculate the local average gradient value of each block separately And the global average gradient value ; Select and The larger value is used as the threshold to obtain the truncation mask : Set the initial threshold for truncation , only when the global average gradient Less than Gradient truncation begins when , and the gradient matrix obtained for back propagation is as follows: By multiplying the truncation mask to the original gradient matrix, some holes are dynamically created in the gradient matrix before backpropagation during training. The network prediction results in this part will be retained, thereby achieving self-correction of the erroneous areas in the CAM seed.

9. A weakly supervised semantic segmentation device based on a naive Vision Transformer, characterized by: The invention comprises at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor, and after the instructions are executed by the processor, they are used to complete the weakly supervised semantic segmentation method based on the naive Vision Transformer according to any one of claims 1 to 8.

Images