Weakly supervised semantic segmentation method based on attention regulation

Abstract

The application discloses a weakly supervised semantic segmentation method based on attention regulation, and explores the application of a transformer in a weakly supervised semantic segmentation task. A method based on the transformer optimizes a class activation map using attention. However, due to errors in the attention between some classes and blocks, the class activation map obtained after optimization has an incomplete activation problem. To solve this problem, the application proposes a novel weakly supervised semantic segmentation framework, in which an attention regulation strategy is designed to regulate the attention between classes and blocks according to the attention between blocks, and the regulated attention can activate more target regions. Compared with the latest method on the PASCAL VOC 2012 dataset and the MS COCO 2014 dataset, the application method achieves the optimal result.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image segmentation, and particularly relates to a weakly supervised semantic segmentation method based on attention regulation. BACKGROUND

[0002] Semantic segmentation is one of the basic and challenging tasks in computer vision, and its research purpose is to classify each pixel point in an image and assign it to a specific semantic category. Semantic segmentation has a wide range of applications in many fields, such as image recognition, autonomous driving, medical image analysis, scene understanding, and video analysis, etc. It can help computers better understand the content of images, thereby realizing automated scene understanding and decision-making. In recent years, due to the vigorous development of deep learning methods, semantic segmentation has also made significant progress, among which fully supervised semantic segmentation models are widely used and have achieved excellent performance. However, training fully supervised semantic segmentation models often requires large-scale pixel-level labeled data, and the acquisition of pixel-level labeled data is often difficult and time-consuming. In order to solve this problem, many works have begun to use weakly supervised semantic segmentation technology. It aims to train semantic segmentation networks using bounding box annotations, point annotations, scribble annotations, or image-level annotations. Among them, image-level annotation is the most convenient to obtain, and has been widely studied in weakly supervised semantic segmentation.

[0003] Although image-level annotation is very convenient to obtain, it has a problem that it cannot provide sufficient location supervision information, because it only gives the object class information contained in an image, and does not indicate the specific location information of the object class in the image. The development of class activation map (CAM) provides an effective method to obtain location information using only image-level annotation. For weakly supervised semantic segmentation with image-level annotation, most existing methods usually use the following process to solve it: 1) train a convolutional neural network (CNN) using image-level annotation to generate a class activation map to obtain seed regions; 2) extend the seed regions with certain constraints to obtain pseudo labels; 3) use the pseudo labels as real labels to train a fully supervised semantic segmentation network. However, the class activation map generated by the convolutional neural network has a problem that it tends to activate a local discriminative region, while ignoring the complete object range, resulting in incomplete activation problem. Current studies have shown that this is due to the inherent characteristics of convolutional neural networks, i.e. the convolution operation in convolutional neural networks can only capture small-range feature dependencies, and cannot explore global feature relationships, which leads to the object region being too small, thereby affecting the quality of the generated pseudo labels, and ultimately leading to difficulty in achieving ideal weakly supervised semantic segmentation results.

[0004] Currently, Transform has achieved great success in many computer vision tasks, which is mainly due to its attention mechanism. The attention mechanism of Transform can model global feature relationships and overcome the above-mentioned shortcomings of convolutional neural networks. Therefore, some researchers have begun to use Transform for weakly supervised semantic segmentation research. They usually use Transform structure to extract image features and generate class activation maps, and then use attention to optimize the class activation maps to obtain more complete class activation maps. Although existing weakly supervised semantic segmentation methods based on Transform usually use attention to optimize class activation maps, due to the errors in the attention between the sub-classes and blocks generated by Transform, the class activation maps cannot completely activate the object regions after attention optimization. SUMMARY

[0005] The purpose of the present application is to solve the problem that the target region cannot be completely activated in weakly supervised semantic segmentation, and to provide a weakly supervised semantic segmentation method based on attention fusion, which generates more complete and accurate class activation maps covering the target region. The method generates high-quality pixel-level pseudo-labels through the class activation maps to train the semantic segmentation network and improves the semantic segmentation performance of the network.

[0006] Technical scheme: In order to achieve the purpose of the present application, the technical scheme adopted by the present application is: a weakly supervised semantic segmentation method based on attention regulation, comprising the following steps:

[0007] Step 1, data preparation: obtaining labeled image dataset and dividing the dataset into training set, validation set and test set;

[0008] Step 2, data preprocessing: randomly horizontally flipping and color jittering the image, normalizing the image, and randomly cropping the image, and using the cropped image as the input of the weakly supervised semantic segmentation model;

[0009] Step 3, model building: using DeiT-S pre-trained on ImageNet as the backbone of the model to build a weakly supervised semantic segmentation model;

[0010] Step 4, model training: using Adam optimizer to optimize the weakly supervised semantic segmentation model, and using the training set to train the model for a set period of time, using multi-label cross-entropy loss as the loss function, and generating class activation maps from the trained model;

[0011] Step 5, assign a class to each pixel position according to the value of the class activation map to generate a pixel-level pseudo label, then train the semantic segmentation network DeeplabV2 using the pixel-level pseudo label; input the pictures in the validation set and the test set into the trained model to obtain the final segmentation map.

[0012] Further, the model building in step 3 comprises:

[0013] Step 3.1, build a weakly supervised semantic segmentation framework based on attention fusion, divide the preprocessed image into N non-overlapping blocks, then construct N block tokens through linear mapping, and splice C class tokens and N block tokens to obtain the input token of the framework;

[0014] Step 3.2, input the input token into the Transfomer encoding layer in the framework to obtain the output token; then extract the last N block tokens from the output token to form the output block token Tp_out, and reorganize and convolve it to obtain the initial class activation map Original-CAM;

[0015] Step 3.3, when the input token passes through the Transfomer encoding layer, the attention module calculates the attention of the input token Attention, and the calculation formula is as follows:

[0016]

[0017] Where Q and K represent the Quary matrix and Key matrix obtained by linear projection of the input token when passing through the Transformer encoding layer, T represents matrix transposition, d k represents a scaling factor;

[0018] Step 3.4, Attention is further divided into class-to-block attention A c2p and block-to-block attention A p2p , then adjust the attention A c2p between classes and blocks through the attention A p2p between blocks and blocks;

[0019] Step 3.5, use the class-to-block attention A c2p and the block-to-block attention A p2p to optimize the initial class activation map.

[0020] Further, the class-to-block attention A c2p and the block-to-block attention A p2p are as follows:

[0021] A c2p =Attention[1:C,C+1:C+N]

[0022] A p2p =Attention[C+1:C+N,C+1:C+N]

[0023] The process of adjusting the attention between class c and block i is as follows:

[0024] First, based on the attention between each block and block i, sort the blocks in descending order of attention value, and select the top p% of the sorted blocks.

[0025] Then, the attention between class c and the selected block is extracted and calculated to obtain the attention adjustment factor between class c and block i:

[0026]

[0027] Where r(c,i) represents A c2p The attention adjustment factor between class c and block i, where c∈{1,2,…,C} represents the total number of classes in the dataset, i,j represent blocks, i∈{1,2,…,N}, j∈U, U represents the set of the top p% blocks with the highest attention to block i, and S represents the number of blocks in U; ​​A c2p (c,j) represents the attention between class c and block j;

[0028] Next, the attention adjustment factor r(c,i) is added to the attention between class c and block i for adjustment:

[0029] A c2p (c,i)=A c2p (c,i)+α*r(c,i)

[0030] Among them, A c2p (c,i) represents the attention between class c and block i, and α represents the attention modulation factor coefficient.

[0031] Furthermore, in step 3.5, class-to-block attention A is used. c2p Block-to-block attention A p2p The initial class activation graph is optimized, including:

[0032] First, the initial class activation graph (Original-CAM) is multiplied by the class-to-block attention to obtain the preliminarily optimized adjustment class activation graph.

[0033] Then, further optimization is achieved by performing matrix multiplication between block-to-block attention and the regulation class activation graph to obtain the final class activation graph.

[0034] Furthermore, the model training process in step 4 is as follows:

[0035] Step 4.1, setting weakly supervised semantic segmentation model hyperparameters: model training times Epoch, initial learning rate and model training batch_size, the optimizer used in training is Adam optimizer, and the loss function is multilabel cross-entropy loss;

[0036] Step 4.2, the weakly supervised semantic segmentation model is trained for multiple rounds, and the parameters corresponding to the highest mIoU value in the training are saved;

[0037] Step 4.3, after the weakly supervised semantic segmentation model is trained, the saved best parameters are loaded into the model, and then the training set data is input into the model, and the trained model generates a complete class activation map.

[0038] Advantages: Compared with the prior art, the technical scheme of the present application has the following beneficial technical effects:

[0039] The present application mainly solves the incomplete activation problem of class activation map in weakly supervised semantic segmentation. A simple and effective weakly supervised semantic segmentation framework is proposed based on the basic network structure of Transformer. In the framework, first, an attention adjustment strategy is designed to adjust the attention between classes and blocks according to the attention between blocks, effectively reducing the probability of false association between classes and blocks, and then the adjusted attention is used to optimize the class activation map. At this time, the target area in the class activation map can be more completely and accurately activated, which can better solve the problem of incomplete activation of class activation map. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 It is a weakly supervised semantic segmentation overall framework based on attention fusion.

[0041] Figure 2 It is an example of segmentation result on PASCAL VOC 2012 validation set.

[0042] Figure 3 It is an example of segmentation result on MS COCO 2014 validation set. DETAILED DESCRIPTION

[0043] The technical scheme of the present application will be further described below in combination with the drawings and examples.

[0044] The weakly supervised semantic segmentation method based on attention adjustment described in the present application proposes a novel framework based on Transformer for weakly supervised semantic segmentation task under image-level annotation, and the overall structure of the framework is as follows: Figure 1As shown, it mainly consists of three parts: 1) feature extraction and initial class activation map generation using Transformer; 2) attention adjustment module, which adjusts the attention between classes and blocks according to the attention between blocks, effectively improving the accuracy of the attention between classes and blocks; 3) using attention to optimize the class activation map to obtain a more complete and accurate class activation map. The specific implementation of the method includes the following steps:

[0045] Step 1: data preparation.

[0046] In the present application, PASCAL VOC 2012 dataset and MS COCO 2014 dataset are used. Among them, the PASCAL VOC 2012 dataset has 21 categories, including 20 object categories and one background category; the MS COCO 2014 dataset has 81 categories, including 80 object categories and one background category. The PASCAL VOC 2012 dataset can be divided into three parts: training set (including 1464 images), validation set (including 1449 images) and test set (including 1456 images), wherein the training set usually uses 10582 images after data expansion; the MS COCO 2014 dataset can be divided into two parts: training set (including 82081 images) and validation set (including 40137 images).

[0047] Step 2: data preprocessing.

[0048] The image is randomly horizontally flipped and color jittered, and the brightness, contrast and saturation values of the image are set to 0.3. The image is normalized using transforms.Normalize, and its size is adjusted to 256x256, then the image is randomly cropped using transforms.RandomCrop, and the image size is adjusted to 224x224. The cropped image is input into the model.

[0049] Step 3: model building.

[0050] Step 3.1: build a weakly supervised semantic segmentation framework based on attention fusion, divide the image preprocessed in step 2 into N non-overlapping blocks, then construct N block tokens through linear mapping, and splice C class tokens and N block tokens to obtain input tokens of the framework.

[0051] Step 3.2: input the input token into the Transformer encoding layer in the framework to obtain the output token. Then extract the last N block tokens from the output token to form the output block token Tp_out, and perform reshaping (Reshape) and convolution (Conv) operations on it to obtain the initial class activation map Original-CAM.

[0052] Original-CAM = Conv(Reshape(Tp_out))

[0053] Step 3.3: When the input tokens pass through the Transformer encoding layer, the attention module will perform attention calculation on the input tokens to generate attention Attention, which has a shape of [C+N, C+N], and the calculation formula is as follows:

[0054]

[0055] Where Q, K represent the Query matrix and Key matrix obtained by linear projection of the input tokens when passing through the Transformer encoding layer, T represents matrix transposition, d k represents the scaling factor.

[0056] Step 3.4: Attention can be further divided into class-to-block attention A c2p and block-to-block attention A p2p , where A c2p = Attention[1:C, C+1:C+N], A p2p = Attention[C+1:C+N, C+1:C+N]. Then adjust the attention A p2p between classes and blocks through the attention A c2p between blocks. If you want to adjust the attention between class c and block i, first sort the blocks according to the attention value from large to small according to the attention between each block and block i, then select the top 30% of blocks after sorting, and then calculate the attention between these blocks to obtain the attention adjustment factor between class c and block i:

[0057]

[0058] Where r(c,i) represents the attention adjustment factor between class c and block i in A c2p , c∈{1,2,…,C} represents the total number of dataset classes, i,j represents the block, i∈{1,2,…,N}, j∈U, U represents the set of blocks with greater attention than block i, S represents the number of blocks in U. Next, add the attention adjustment factor r(c,i) to the attention between class c and block i for adjustment:

[0059] A c2p (c,i) = A c2p (c,i) + α*r(c,i)

[0060] Where A c2p (c,i) represents the attention between class c and block i, and α represents the attention adjustment factor coefficient.

[0061] Step 3.5: Use class-to-block attention A c2p and block-to-block attention A p2p to optimize the initial class activation map. First, multiply the initial class activation map Original-CAM with the class-to-block attention to obtain the preliminary optimized adjusted class activation map, and then further optimize by matrix multiplication between the block-to-block attention and the adjusted class activation map to obtain the final class activation map.

[0062] Step 4: Model training.

[0063] Step 4.1: Set the related hyperparameters of the weakly supervised semantic segmentation model, set the number of model training times Epoch to 60, set the model training batch size batch_size to 64, use the Adam optimizer as the optimizer during training, the loss function is the multi-label cross-entropy loss, and the initial learning rate is set to 5e-4.

[0064] Step 4.2: Train the weakly supervised semantic segmentation model for multiple rounds, and save the parameters corresponding to the best round result (the highest training mIoU value) by observing the training set results.

[0065] Step 4.3: After the weakly supervised semantic segmentation model is trained, load the saved best parameters into the model, and then input the training set data into the model. The trained model can generate a relatively complete class activation map.

[0066] Step 5: Assign a class to each pixel position according to the value of the class activation map to generate a pixel-level pseudo label, and then train the existing semantic segmentation network DeeplabV2 using the pixel-level pseudo label. Input the images in the validation set and test set into the trained model to obtain the final segmentation map, as shown in Figure 2 、 Figure 3 The second column is the real segmentation map, and the third column is the predicted segmentation map of the present application. It can be found that the segmentation map predicted by the model of the present application is very close to the real segmentation map.

Claims

1. A weakly supervised semantic segmentation method based on attention regulation, characterized in that, Comprising the following steps: Step 1, data preparation: obtain the labeled image dataset, and divide the dataset into training set, validation set and test set; Step 2, data preprocessing: randomly flip the image horizontally and perform color jittering, normalize the image, and randomly crop the image, and use the cropped image as the input of the weakly supervised semantic segmentation model; Step 3, model building: use DeiT-S pre-trained on ImageNet as the backbone of the model, and build a weakly supervised semantic segmentation model; Step 3.1, build a weakly supervised semantic segmentation framework based on attention fusion, divide the preprocessed image into N non-overlapping blocks, then construct N block tokens through linear mapping, and splice C class tokens and N block tokens to obtain input tokens of the framework; Step 3.2, input the input token into the Transfomer encoding layer in the framework to obtain the output token; then extract the last N block tokens from the output token to form the output block token Tp_out, and perform reorganization and convolution operation to obtain the initial class activation map Original-CAM; Step 3.3, when the input token passes through the Transfomer encoding layer, the attention module calculates the attention of the input token, and the calculation formula is as follows: wherein Q and K represent Query matrix and Key matrix respectively obtained by linear projection of input tokens when passing through the Transformer encoding layer, T represents matrix transposition, d k denotes a scaling factor; Step 3.4, Attention is further divided into class-to-block attention A c2p and block-to-block attention A p2p which is then conditioned on the attention A p2p between classes and blocks; and c2p blocks. Step 3.5, class-to-block attention A c2p and block-to-block attention A p2p Optimizing the initial class activation map; Step 4, model training: use Adam optimizer to optimize the weakly supervised semantic segmentation model, and use the training set to train the model for a certain period of time, the loss function uses multi-label cross entropy loss, and the trained model generates class activation map; Step 5, according to the value of the class activation map, assign a class to each pixel position to generate a pixel-level pseudo label, then train the semantic segmentation network DeeplabV2 using the pixel-level pseudo label; input the pictures in the validation set and test set into the trained model to obtain the final segmentation map.

2. The weakly supervised semantic segmentation method based on attention regulation according to claim 1, characterized in that, Class-to-block attention A c2p and block-to-block attention A p2p is represented as follows: A c2p = Attention[1:C,C+1:C+N] A p2p = Attention[C+1:C+N,C+1:C+N] Adjust the attention between class c and block i, the process is as follows: First, sort the blocks according to the attention value from large to small according to the attention between each block and block i, and select the top p% blocks after sorting; Then, take out and calculate the attention between class c and the selected block to obtain the attention adjustment factor r(c,i) between class c and block i: where r(c,i) represents the attention adjustment factor between class c and block i, c e {1,2,...,C} represents the total number of data set classes, i,j represents the block, i e {1,2,...,N}, j e U, U represents the set of the top p% blocks with the most attention to block i, S represents the number of blocks in U; A c2p where r(c,i) represents the attention adjustment factor between class c and block i, c e {1,2,...,C} represents the total number of data set classes, i,j represents the block, i e {1,2,...,N}, j e U, U represents the set of the top p% blocks with the most attention to block i, S represents the number of blocks in U; A c2p where r(c,i) represents the attention adjustment factor between class c and block i, c e {1,2,...,C} represents the total number of data set classes, i,j represents the block, i e {1,2,...,N}, j e U, U represents Next, add the attention adjustment factor r(c,i) to the attention between class c and block i for adjustment: A c2p (c,i) = A c2p (c,i) + a*r(c,i) where A c2p (c,i) denotes the attention between class c and bin i, and a represents the attention adjustment factor coefficient.

3. The weakly supervised semantic segmentation method based on attention regulation according to claim 1, characterized in that, Use class-to-block attention A in step 3.5 c2p and block-to-block attention A p2p Optimizing the initial class activation map, including: First, multiply the initial class activation map Original-CAM and the class-to-block attention to obtain the preliminary optimized adjusted class activation map; Then further optimize by performing matrix multiplication between the block-to-block attention and the adjusted class activation map to obtain the final class activation map.

4. The weakly supervised semantic segmentation method based on attention regulation according to any one of claims 1-3, characterized in that, The model training process in step 4 is as follows: Step 4.1, set the weakly supervised semantic segmentation model parameters: model training times Epoch, initial learning rate and model training batch size, the optimizer used during training is Adam optimizer, and the loss function is multi-label cross entropy loss; Step 4.2, train the weakly supervised semantic segmentation model for multiple rounds, and save the parameters corresponding to the round with the highest training mIoU value. Step 4.3, after the weakly supervised semantic segmentation model training is completed, the saved best parameters are loaded into the model, and then the training set data is input into the model, and the trained model generates a complete class activation map.

Images