Weakly supervised semantic segmentation method based on attention fusion
By optimizing the class activation graph through an adaptive attention fusion module and a modulation function, the problem of incomplete activation of the class activation graph in weakly supervised semantic segmentation is solved, thereby improving the accuracy and completeness of semantic segmentation.
Patent Information
- Application Number
- CN202310981553.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-04
- Publication Date
- 2025-12-16
- Estimated Expiration
- 2043-08-04
AI Technical Summary
Existing weakly supervised semantic segmentation methods, when generating class activation maps, focus on local image details with shallow attention and global information with deep attention. Direct fusion of these two approaches leads to misleading information, and insufficient activation of less important regions affects the segmentation results.
We adopt a weakly supervised semantic segmentation method based on attention fusion. We assign weights to attention in different layers through an adaptive attention fusion module, and use a modulation function to enhance the attention value of the less important region, thereby optimizing the coarse class activation map and generating a high-quality class activation map.
It effectively suppresses background noise, preserves target details, improves the integrity and accuracy of class activation maps, and enhances the performance of semantic segmentation.
Smart Images
Figure CN116912501B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of computer vision, and particularly relates to a weakly supervised semantic segmentation method based on attention fusion. BACKGROUND
[0002] Semantic segmentation is one of the basic and challenging tasks in the field of computer vision, which uses the feature expression of computers to simulate the human recognition process of images, and assigns a semantic class label to each pixel of a given image. In recent years, due to the vigorous development of deep learning methods, semantic segmentation has made significant progress. As a dense prediction task, the training of a semantic segmentation model cannot be separated from large-scale pixel-level labeled data. However, the acquisition of pixel-level labeled data of images is difficult and time-consuming.
[0003] Weakly supervised semantic segmentation technology can solve the dependence of existing semantic segmentation models on a large amount of pixel-level labeled data by only relying on weakly labeled data to train the segmentation model, and is thus becoming a hot topic of academic research. Common weak labels include bounding box labels, scribble labels, point labels, and image-level labels. Among the above weak labels, image-level labels are easier to obtain than other ways, and since only the specific target class information in the image is given, the location of the target class in the image is not indicated, weakly supervised semantic segmentation based on image-level labels is also the most challenging.
[0004] Due to the lack of specific location information of the target class in the image, most image-level weakly supervised semantic segmentation methods usually need rough location information generated by class activation maps. Class activation maps are a technique based on deep classification networks, which generate feature maps with the same number of channels as the total number of classes. The specific operation process of the method is as follows: 1) obtain a seed region through a class activation map; 2) expand the seed region to obtain pseudo-labels; 3) use the pseudo-labels to train a traditional fully supervised neural network to obtain the final segmentation result. Since class activation maps often only cover the most discriminative regions of objects and mistake the background as foreground, many works have been devoted to generating higher-quality class activation maps.
[0005] With the rapid development of Vision Transformer (ViT), researchers begin to introduce Vision Transformer into weakly supervised semantic segmentation task, some methods use Vision Transformer structure to extract image features and generate rough class activation map, and then use attention to optimize the rough class activation map to get higher quality class activation map. Usually these methods will directly add and fuse the attention of different layers. However, the shallow attention of Vision Transformer structure will pay more attention to the local detail features of the image, and the class activation map obtained by the shallow attention optimization often contains more detailed information; the deep attention pays more attention to the global information of the image, so directly adding and fusing the attention of different layers is not the best choice, which may lead to misleading information in the optimization stage. SUMMARY
[0006] In order to solve the above technical problems, the present application provides a weakly supervised semantic segmentation method based on attention fusion, comprising the following steps
[0007] S1, preparing a data set, the data set including a training set, a validation set and a test set;
[0008] S2, data preprocessing is performed on the images in the data set;
[0009] S3, a weakly supervised semantic segmentation model based on attention fusion is built, and a data image transformer DeiT pre-trained on an image recognition data set ImageNet is used as the backbone of the model; step S3 includes the following sub-steps:
[0010] S3.1, the image preprocessed in step S2 is divided into N non-overlapping blocks, then N block tokens are constructed by linear mapping, and C class tokens are spliced with N block tokens to obtain input tokens of the model;
[0011] S3.2, input the input tokens into the Transformer encoding layer of the weakly supervised semantic segmentation model based on attention fusion to obtain output tokens;
[0012] S3.3, extract the last N block tokens from the output tokens to form output block tokens, and perform reorganization operation and convolution operation on the output block tokens to obtain a rough class activation map,
[0013] Coarse-CAM=Conv(Reshape(Tp_out))
[0014] Wherein, Tp_out represents the output block token, Reshape represents the reorganization operation, Conv represents the convolution operation, and Coarse-CAM represents the rough class activation map;
[0015] S3.4, when the input token passes through the transformer encoding layer, the attention of the input token is calculated by the attention module to generate attention Attention, the calculation formula is as follows:
[0016]
[0017] Wherein, Q and K represent the Quary matrix and Key matrix obtained by linear projection of the input token when passing through the transformer encoding layer, d k The scaling factor is represented by T, and the matrix transposition operation is represented by T;
[0018] S3.5, each transformer encoding layer generates an attention, after L transformer encoding layers, all attentions are obtained, and the L attentions are named A; then, the A is subjected to a global average pooling operation, and then subjected to a full connection layer to generate a weight W, as follows:
[0019] W=FC(GAP(A))
[0020] Wherein, GAP represents the global average pooling operation, and FC represents the full connection layer;
[0021] S3.6, the weight W obtained is multiplied by A and fused to obtain the final attention W';
[0022] S3.7, the final attention W' is further divided into class-to-block attention A c2p And block-to-block attention A p2p The class-to-block attention A c2p And block-to-block attention A p2p Is multiplied by the modulation function G respectively;
[0023] S3.8, the rough class activation map is optimized in turn using the modulated class-to-block attention and block-to-block attention, to obtain the final class activation map;
[0024] S4, the weakly supervised semantic segmentation model based on attention fusion is trained for multiple rounds, and the best parameters corresponding to the best result of the training are saved;
[0025] S5, the saved best parameters are loaded into the weakly supervised semantic segmentation model based on attention fusion, and then the test set data is input into the model to generate a complete class activation map.
[0026] The further defined technical solutions of the application are:
[0027] Further, in step S1, the PASCAL VOC 2012 dataset and the MS COCO 2014 dataset are used as the dataset, the PASCAL VOC 2012 dataset has 21 categories, including 20 object categories and one background category; the MS COCO 2014 dataset has 81 categories, including 80 object categories and one background category.
[0028] The aforementioned weakly supervised semantic segmentation method based on attention fusion, the PASCAL VOC 2012 dataset includes a training set composed of 1464 images, a validation set composed of 1449 images and a test set composed of 1456 images, wherein the training set uses 10582 images after additional data expansion; the MS COCO 2014 dataset includes a training set composed of 82081 images and a validation set composed of 40137 images.
[0029] The aforementioned weakly supervised semantic segmentation method based on attention fusion, step S2 includes the following steps:
[0030] S2.1, randomly horizontally flipping and color jittering the image;
[0031] S2.2, then normalizing the image, adjusting the image size to 256x256;
[0032] S2.3, finally, randomly cropping the image, adjusting the image size to 224x224.
[0033] The aforementioned weakly supervised semantic segmentation method based on attention fusion, in step S2.1, the method of color jittering the image is specifically: setting the brightness, contrast and saturation values of the image to 0.3.
[0034] The aforementioned weakly supervised semantic segmentation method based on attention fusion, in step S3.7, the modulation function G is as follows:
[0035]
[0036] Wherein, x represents the input, μ represents the mean value of the input, σ represents the variance of the input, e represents the exponential function, and π represents the circular constant.
[0037] The method for optimizing the coarse class activation map in step S3.8 is specifically as follows: the coarse class activation map Coarse-CAM is first multiplied element by element with the modulated class-to-block attention to obtain a class activation map after preliminary optimization; and then the modulated block-to-block attention is used to further optimize the class activation map through matrix multiplication to obtain a final class activation map Final-CAM.
[0038] In step S4 of the weakly supervised semantic segmentation method based on attention fusion, the model-related hyperparameters are set, the number of model training times Epoch is set to 60, the model training batch size is set to 64, the optimizer used during training is an Adamw optimizer, the loss function is a multi-label cross-entropy loss, and the initial learning rate is set to 4e-4.
[0039] In step S5 of the weakly supervised semantic segmentation method based on attention fusion, the saved best parameters are loaded into the weakly supervised semantic segmentation model based on attention fusion, the images in the validation set and the test set are input into the model, and then the mean intersection over union MIoU is calculated, and the semantic segmentation performance of the weakly supervised semantic segmentation model based on attention fusion on the Pascal VOC 2012 and MS COCO 2014 data sets is measured according to the size of the obtained MIoU value.
[0040] The beneficial effects of the present application are:
[0041] (1) In the present application, the problem of incomplete activation of the class activation map in weakly supervised semantic segmentation is mainly solved, a simple and effective weakly supervised semantic segmentation framework is proposed based on a visual transformer as a basic network structure. In the framework, an adaptive attention fusion module is designed to assign different weights to different layers of attention, and the attention after fusion can effectively suppress background noise while retaining target details;
[0042] (2) In the present application, a modulation function is designed to increase the attention value of the less important region in the attention, effectively highlighting the target region, to solve the problem that the less important region in the attention cannot activate the target region well.
[0043] (3) In the present application, the modulated attention is used to optimize the coarse class activation map, and the target region in the obtained class activation map can be more completely and accurately activated, effectively solving the problem of incomplete activation of the class activation map. BRIEF DESCRIPTION OF DRAWINGS
[0044] Figure 1 FIG. 1 is a schematic diagram of a weakly supervised semantic segmentation model based on attention fusion in an embodiment of the present application;
[0045] Figure 2 An example of segmentation results on the PASCAL VOC 2012 validation set in the embodiment of the present application is shown in the following figure;
[0046] Figure 3 An example of segmentation results on the MS COCO 2014 validation set in the embodiment of the present application is shown in the following figure. DETAILED DESCRIPTION
[0047] The weakly supervised semantic segmentation method based on attention fusion provided in the embodiment is used for the weakly supervised semantic segmentation task under image-level annotation, and the overall structure of the framework is as shown in the following figure. Figure 1 The weakly supervised semantic segmentation method based on attention fusion provided in the embodiment is used for the weakly supervised semantic segmentation task under image-level annotation, and the overall structure of the framework is as shown in the following figure.
[0048] The weakly supervised semantic segmentation method based on attention fusion provided in the embodiment is used for the weakly supervised semantic segmentation task under image-level annotation, and the overall structure of the framework is as shown in the following figure. Figure 1 The weakly supervised semantic segmentation method based on attention fusion provided in the embodiment is used for the weakly supervised semantic segmentation task under image-level annotation, and the overall structure of the framework is as shown in the following figure.
[0049] The PASCAL VOC 2012 dataset and the MS COCO 2014 dataset are used as the dataset, the Pascal VOC 2012 dataset has 21 categories, including 20 object categories and one background category; and the MS COCO 2014 dataset has 81 categories, including 80 object categories and one background category.
[0050] The PASCAL VOC 2012 dataset includes a training set composed of 1464 images, a validation set composed of 1449 images, and a test set composed of 1456 images, wherein the training set uses 10582 images after data expansion; and the MS COCO 2014 dataset includes a training set composed of 82081 images and a validation set composed of 40137 images.
[0051] S2, data preprocessing is performed on the images in the dataset, specifically including the following steps:
[0052] S2.1, the image is subjected to random horizontal flipping and color jittering processing, and the method for color jittering processing of the image is specifically as follows: the brightness, contrast and saturation values of the image are all set to 0.3.
[0053] S2.2, then normalize the image, adjust the image size to 256x256;
[0054] S2.3, finally randomly crop the image, adjust the image size to 224x224.
[0055] S3, build a weakly supervised semantic segmentation model based on attention fusion, use a data image converter (Data-efficient image Transformers, DeiT) pre-trained on an image recognition dataset ImageNet as the backbone of the model; step S3 includes the following sub-steps:
[0056] S3.1, divide the image pre-processed in step S2 into N non-overlapping blocks, then construct N block tokens through linear mapping, and splice C class tokens with N block tokens to obtain input tokens of the model;
[0057] S3.2, input the input tokens into the Transfomer encoding layer of the weakly supervised semantic segmentation model based on attention fusion to obtain output tokens;
[0058] S3.3, extract the last N block tokens from the output tokens to form output block tokens, and perform reorganization and convolution operations on the output block tokens to obtain a coarse class activation map,
[0059] Coarse-CAM=Conv(Reshape(Tp_out))
[0060] wherein, Tp_out represents the output block token, Reshape represents the reorganization operation, Conv represents the convolution operation, and Coarse-CAM represents the coarse class activation map;
[0061] S3.4, when the input tokens pass through the Transfomer encoding layer, attention is calculated on the input tokens through the attention module to generate attention, and the calculation formula is as follows:
[0062]
[0063] wherein, Q and K represent the Quary matrix and the Key matrix obtained by linear projection of the input tokens when passing through the transformer encoding layer, d k represents a scaling factor, and T represents a matrix transposition operation;
[0064] S3.5, each transformer encoding layer produces an attention, and after L transformer encoding layers, all attentions are obtained, and the L attentions are named A; then the A is subjected to a global average pooling operation, and then a full connection layer is used to interact information to generate a weight W, as follows:
[0065] W = FC(GAP(A))
[0066] Wherein, GAP represents the global average pooling operation, and FC represents the full connection layer;
[0067] S3.6, the obtained weight W is multiplied by A and fused to obtain the final attention W';
[0068] S3.7, the final attention W' is further divided into class-to-block attention A c2p and block-to-block attention A p2p , and the class-to-block attention A c2p and the block-to-block attention A p2p are multiplied by the modulation function G respectively, and the modulation function G is as follows:
[0069]
[0070] Wherein, x represents the input, μ represents the average value of the input, σ represents the variance of the input, e represents the exponential function, and π represents the circular constant;
[0071] S3.8, the modulated class-to-block attention and block-to-block attention are used to optimize the coarse class activation map in turn, first, the coarse class activation map Coarse-CAM is multiplied by the modulated class-to-block attention element by element to obtain the preliminary optimized class activation map, and then the modulated block-to-block attention is used to further optimize the class activation map through matrix multiplication to obtain the final class activation map Final-CAM.
[0072] S4, the weakly supervised semantic segmentation model based on attention fusion is trained for multiple rounds, and the best parameters corresponding to the best result of the training are saved by observing the validation set results;
[0073] The model related hyperparameters are set, the model training times Epoch are set to 60, the model training batch size is set to 64, the optimizer used during training is Adamw optimizer, the loss function is multilabel cross entropy loss, and the initial learning rate is set to 4e-4.
[0074] S5, the saved best parameters are loaded into the weakly supervised semantic segmentation model based on attention fusion, and then the test set data is input into the model to generate a complete class activation map;
[0075] The saved best parameters are loaded into the weakly supervised semantic segmentation model based on attention fusion, the pictures in the validation set and the test set are input into the model, then the calculation of the mean intersection over union (MIoU) is performed, and the semantic segmentation performance of the weakly supervised semantic segmentation model based on attention fusion on the Pascal VOC 2012 and MS COCO 2014 data sets is measured according to the size of the obtained MIoU value, and the segmentation results are as shown in Figure 2 and Figure 3 as shown.
[0076] In summary, the purpose of the present application is to solve the problem that the target region cannot be completely activated in weakly supervised semantic segmentation. The present embodiment proposes an adaptive attention fusion module (AAF) to measure the importance of different layer attentions to the class activation map, which is used to assign weights to different layer attentions. In order to ensure that the adaptive attention fusion module can estimate accurate weights, the present embodiment proposes an end-to-end adaptive attention fusion module generation training strategy. In the training process, the adaptive attention fusion module estimates the weights applied to different layer attentions. The attention after using this method to fuse can better suppress background noise while retaining target details.
[0077] In addition, it is found through experiments that although the less important regions in the attention can activate the target region in the optimization process, some regions that should be activated are not well activated due to the small corresponding attention value, therefore a modulation function is designed to increase the attention value of the less important regions in the attention, so that more target regions can be activated. Finally, the class activation map that can cover more target regions is obtained by using the modulated attention to optimize the rough class activation map, while effectively suppressing the background noise, and finally the high-quality pseudo label is obtained for the training of the semantic segmentation network.
[0078] In addition to the above embodiments, the present application can also have other implementation manners. Any technical solutions formed by equivalent substitution or equivalent transformation shall fall within the protection scope required by the present application.
Claims
1. A weakly supervised semantic segmentation method based on attention fusion, characterized in that: The method comprises the following steps S1, preparing a data set, the data set comprising a training set, a validation set and a test set; S2, data preprocessing is performed on the images in the data set; S3, a weakly supervised semantic segmentation model based on attention fusion is built, and a data image converter DeiT pre-trained on an image recognition data set ImageNet is used as the backbone of the model; step S3 comprises the following sub-steps: S3.1, the images preprocessed in step S2 are segmented into N non-overlapping blocks, then N block tokens are constructed through linear mapping, and C class tokens are spliced with the N block tokens to obtain input tokens of the model; S3.2, the input tokens are input into a Transfomer encoding layer of the weakly supervised semantic segmentation model based on attention fusion to obtain output tokens; S3.3, the last N block tokens are extracted from the output tokens to form output block tokens, and the output block tokens are reorganized and convoluted to obtain a coarse class activation map, Coarse-CAM=Conv(Reshape(Tp_out)) Wherein, Tp_out represents the output block token, Reshape represents the reorganization operation, Conv represents the convolution operation, and Coarse-CAM represents the coarse class activation map; S3.4, when the input tokens pass through the Transfomer encoding layer, attention is calculated for the input tokens through an attention module to generate attention, and the calculation formula is as follows: Wherein, Q and K represent the Quary matrix and Key matrix obtained by linear projection of the input token when passing through the transformer encoding layer, d k denotes a scaling factor, and T denotes a matrix transposition operation. S3.5, each transformer encoding layer generates an attention, and after L transformer encoding layers, all attentions are obtained, and the L attentions are named A; then, A is subjected to a global average pooling operation, and then a full connection layer is used to interact information to generate a weight W, as shown below: W=FC(GAP(A)) Wherein, GAP represents the global average pooling operation, and FC represents the full connection layer; S3.6, the obtained weight W is multiplied by A and fused to obtain the final attention W′; S3.7, further partitioning the final attention W' into class-to-block attention A c2p and block-to-block attention A p2p and block-to-block attention A c2p and block-to-block attention A p2p are multiplied with a modulation function G, respectively; S3.8, the modulated class-to-block attention and block-to-block attention are used to optimize the coarse class activation map in turn to obtain a final class activation map; S4, the weakly supervised semantic segmentation model based on attention fusion is trained for multiple rounds, and the best parameters corresponding to the best result of the training are saved; S5, the saved best parameters are loaded into the weakly supervised semantic segmentation model based on attention fusion, and then the test set data is input into the model to generate a complete class activation map. 2.The weakly supervised semantic segmentation method based on attention fusion according to claim 1, characterized in that: In step S1, the PASCAL VOC 2012 data set and the MS COCO 2014 data set are used as the data set, the Pascal VOC 2012 data set has 21 categories, including 20 object categories and one background category; the MS COCO 2014 data set has 81 categories, including 80 object categories and one background category. 3.The weakly supervised semantic segmentation method based on attention fusion according to claim 2, characterized in that: The PASCAL VOC 2012 dataset includes a training set consisting of 1464 images, a validation set consisting of 1449 images, and a test set consisting of 1456 images, wherein the training set uses 10582 images expanded by additional data; the MSCOCO 2014 dataset includes a training set consisting of 82081 images and a validation set consisting of 40137 images.
4. The weakly supervised semantic segmentation method based on attention fusion according to claim 1, characterized in that: The step S2 includes the following sub-steps: S2.1, randomly horizontally flipping and color jittering the image; S2.2, then normalizing the image to adjust the image size to 256*256; S2.3, finally, randomly cropping the image to adjust the image size to 224*224.
5. The weakly supervised semantic segmentation method based on attention fusion according to claim 4, characterized in that: In the step S2.1, the method for color jittering the image is specifically: setting the brightness, contrast and saturation values of the image to 0.
3.
6. The weakly supervised semantic segmentation method based on attention fusion according to claim 1, characterized in that: In the step S3.7, the modulation function G is as follows: Wherein, x represents the input, μ represents the mean value of the input, σ represents the variance of the input, e represents the exponential function, and π represents the circular constant.
7. The weakly supervised semantic segmentation method based on attention fusion according to claim 1, characterized in that: In the step S3.8, the method for optimizing the coarse class activation map is specifically: first, multiply the coarse class activation map Coarse-CAM and the modulated class-to-block attention element by element to obtain a preliminary optimized class activation map; then, further optimize the class activation map by matrix multiplication using the modulated block-to-block attention to obtain the final class activation map Final-CAM. 8.The weakly supervised semantic segmentation method based on attention fusion of claim 1, characterized in that: In the step S4, the model-related hyperparameters are set, the model training times Epoch are set to 60, the model training batch size is set to 64, the optimizer used during training is the Adamw optimizer, the loss function is the multi-label cross-entropy loss, and the initial learning rate is set to 4e-4.
9. The weakly supervised semantic segmentation method based on attention fusion according to claim 1, characterized in that: In the step S5, the saved best parameters are loaded into the weakly supervised semantic segmentation model based on attention fusion, the pictures in the validation set and the test set are input into the model, then the mean intersection over union MIoU is calculated, and the semantic segmentation performance of the weakly supervised semantic segmentation model based on attention fusion on the Pascal VOC 2012 and MS COCO 2014 datasets is measured according to the size of the obtained MIoU value.