A text-driven efficient weakly supervised semantic segmentation method and device
By using vision-language models and multi-head self-attention mechanisms in weakly supervised semantic segmentation, improving and generating high-quality pseudo-labels, the problems of inefficient and high training cost of existing methods are solved, and efficient semantic segmentation effect is achieved.
Patent Information
- Application Number
- CN202211597202.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-12
- Publication Date
- 2025-04-25
- Estimated Expiration
- 2042-12-12
AI Technical Summary
The existing weakly supervised semantic segmentation methods are less efficient, and require training different models at different stages, which is expensive, and there is still a big gap in the effectiveness of single-stage methods compared with multi-stage methods.
The visual-language model based on the visual transformer architecture is adopted, and the class activation graph is improved and post-processed by introducing a multi-head self-attention mechanism and conditional random field to generate high-quality pseudo-labels, thereby improving the segmentation efficiency and effect of the semantic segmentation model.
This method greatly improves the efficiency of weakly supervised semantic segmentation, the generated pseudo-label quality is high, the segmentation effect is better than other similar methods, and does not require additional training models, reducing training costs.
Smart Images
Figure CN115937852B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of image semantic segmentation in computer vision, and in particular relates to a text-driven efficient weakly supervised semantic segmentation method and device. Background Art
[0002] Semantic segmentation is an important task in the field of computer vision, and its purpose is to assign corresponding semantic labels to each pixel in the image, such as cars, people, sky, etc. Traditional supervised semantic segmentation models require pixel-level labels as input, but this pixel-level annotation cost is very expensive, requiring a lot of manpower and annotation time. For this reason, people have proposed weakly supervised semantic segmentation to reduce the annotation cost. The weakly supervised semantic segmentation in the present invention refers to automatically generating pixel-level semantic labels (pseudo-labels) using only image-level annotations.
[0003] Most existing weakly supervised semantic segmentation methods follow a three-stage paradigm: (1) Use image-level annotations to train a classification model on a specific dataset, and generate some seed regions for each training image based on the classification model. This step is usually achieved by calculating the class activation map of the classification model; (2) Generate pseudo labels by propagating semantic information from the seed region to the entire image. This step usually requires training an additional semantic consistency network or using a saliency map as an aid. (3) Based on the obtained pixel-level pseudo labels, train the final semantic segmentation model. This step is mainly to verify the quality of the generated pseudo labels.
[0004] However, the above three-stage paradigm is inefficient, and it requires training different models at different stages, which has high training costs. Therefore, some works have proposed end-to-end models to improve efficiency. For example, the article "Single-Stage Semantic Segmentation From Image Labels" published in the Conference on Computer Vision and Pattern Recognition in 2020 and the article "Reliability does matter: An end-to-end weakly supervised semantic segmentation approach" published in the Proceedings of the AAAI Conference on Artificial Intelligence in the same year proposed a single-stage method, which greatly simplified the process of generating pseudo-labels. The article "Learning Affinity from Attention: End-to-End Weakly-Supervised Semantic Segmentation with Transformers" published in the Conference on Computer Vision and Pattern Recognition in 2022 uses visual transformers to generate and improve pseudo-labels end-to-end, further improving the effect of semantic segmentation while reducing training costs. Although this type of single-stage method is simple, its effect is still far behind that of the multi-stage method.
[0005] Therefore, how to improve the efficiency of weakly supervised semantic segmentation as much as possible without affecting the semantic segmentation effect is a problem worth studying. Summary of the invention
[0006] The present invention provides a text-driven efficient weakly supervised semantic segmentation method and device, which efficiently generates high-quality pseudo labels for semantic segmentation by introducing a visual-language model based on a visual transformer architecture, thereby improving the segmentation efficiency and effect of the semantic segmentation model.
[0007] An efficient text-driven weakly supervised semantic segmentation method includes the following steps:
[0008] (1) Input the image and the corresponding category name into the pre-trained visual-language model and use the gradient-based class activation map to obtain the initial class activation map;
[0009] (2) Improve the initial class activation map based on the multi-head self-attention mechanism in the visual transformer;
[0010] (3) Post-process the improved class activation map using conditional random fields to obtain pseudo labels for semantic segmentation;
[0011] (4) Use the generated pseudo-labels to train the semantic segmentation model, and use the trained semantic segmentation model to perform semantic segmentation on the images to be segmented.
[0012] Furthermore, in step (1), a visual-language model pre-trained on large-scale image-text pair data is used, which includes an image encoder and a text encoder. Before the class name is input into the text encoder, the class name is expanded by fusing prompt words and synonyms.
[0013] When using gradient-based class activation mapping, softmax is introduced into the class activation map, and the classification score Y of the current category c is c After softmax, we get the probability classification score s c , using the probability score s c Calculate the gradient and get the new weight as follows:
[0014]
[0015] The initial class activation map is obtained by weighted summation of the feature map, and the formula is:
[0016]
[0017] in, represents the weight connecting the kth feature map and the current category c, Z represents the number of pixels in the feature map, and Y c Represents the classification score of the current category c, Y c’ represents the classification score of the c'th category, Represents the activation value at position (i, j) on the feature map, s c ' represents Y c’ The probability score after being processed by the softmax function, and the ReLU activation function is used to remove features that have no positive impact on the current category.
[0018] After softmax processing, the probability score s is obtained c The formula is:
[0019]
[0020] In the formula, c' represents the c'th category, and C represents the total number of categories.
[0021] The specific process of step (2) is as follows:
[0022] (2-1) Step (1) obtains the class activation map M c At the same time, the self-attention weight matrix W is also obtained attn ; Use Sinkhorn regularization to normalize the self-attention weights so that the sum of its rows and columns is 1, recorded as matrix D, and then define the semantic consistency matrix A as follows:
[0023]
[0024] (2-2) is the class activation graph M c Set a threshold to obtain some connected areas, and take the bounding rectangles of these connected areas as category masks. The bounding rectangles contain the pixel positions of the current category. Then use these category masks to mask the consistency matrix A so that it only focuses on the pixels of the current category.
[0025] (2-3) Use the occluded consistency matrix to improve the class activation map. The process is as follows:
[0026]
[0027] Where M c and A represent the class activation map and consistency matrix respectively, Represents the improved class activation map, B c Represents the circumscribed rectangle mask, ⊙ represents the Hadamard product, t represents the number of iterations, and vec() represents matrix vectorization.
[0028] In step (3), when the conditional random field is used to post-process the improved class activation map, the pixel positions with low confidence in the pseudo-label are ignored and do not participate in the calculation of the loss function when the semantic segmentation model is subsequently trained.
[0029] An efficient text-driven weakly supervised semantic segmentation device, comprising:
[0030] The class activation map generation module is used to input the image and the corresponding class name into the pre-trained visual-language model and obtain the initial class activation map using the gradient-based class activation map;
[0031] The class activation map improvement module is used to improve the initial class activation map based on the multi-head self-attention mechanism in the visual transformer;
[0032] The class activation map post-processing module is used to post-process the improved class activation map using conditional random fields to obtain pseudo labels for semantic segmentation;
[0033] The semantic segmentation module is used to train the semantic segmentation model using the generated pseudo labels, and use the trained semantic segmentation model to perform semantic segmentation on the images to be segmented.
[0034] A text-driven efficient weakly supervised semantic segmentation device comprises a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement the above-mentioned efficient weakly supervised semantic segmentation method.
[0035] Compared with the prior art, the present invention has the following beneficial effects:
[0036] 1. The new weakly supervised semantic segmentation framework proposed in this paper makes full use of the multi-head self-attention mechanism in the vision-language pre-training model and the visual transformer, greatly improving the efficiency of weakly supervised semantic segmentation. The method is easy to implement and can generate pseudo labels for semantic segmentation without any training.
[0037] 2. The framework proposed in this invention is text-driven, and can improve the segmentation effect by specially designing text-side input for category names, and can be flexibly expanded to new categories. A large number of experiments have shown that the weakly supervised semantic segmentation model proposed in this invention is superior to other similar methods in pseudo-label generation efficiency and segmentation effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 This is a flow chart of an efficient text-driven weakly supervised semantic segmentation method of the present invention:
[0039] Figure 2 The relationship between the overall framework of the present invention and each module;
[0040] Figure 3 This is the pseudo label effect generated by the present invention on the PASCAL VOC 2012 training set. DETAILED DESCRIPTION
[0041] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.
[0042] like Figure 1 and Figure 2 As shown, an efficient text-driven weakly supervised semantic segmentation method includes the following steps:
[0043] S01, input the image and the corresponding category name into the pre-trained visual-language model, and use the gradient-based class activation map to obtain the initial class activation map (seed area).
[0044] (1) Text input design. Since the category names provided in the dataset are generally only one word, and the training text of the pre-trained visual-language model CLIP is a whole sentence, the present invention uses prompt words to expand the category name according to the method in CLIP, such as: "A photo of a{category name}". In addition, the category names given in the dataset may have limited meaning or be ambiguous. Therefore, the present invention uses synonym fusion to expand the category name to obtain a clearer semantic expression, such as using "person, people, human" instead of "person".
[0045] (2) Softmax-based gradient class activation mapping. In the original GradCAM, the score Y used to calculate the gradient c is input before softmax, which will lead to a lack of competition between categories and cause category confusion in weakly supervised semantic segmentation tasks. Therefore, the present invention introduces softmax into GradCAM, and the classification score Y of the current category c is c After softmax processing, the probability score s is obtained c :
[0046]
[0047] The probability score s obtained after softmax processing c Calculate the gradient and get the new weights as follows:
[0048]
[0049] The above formula shows that the weight of the feature map is not only related to the current category c, but also excludes non-target categories. Therefore, the class activation map of the target class finally generated only contains this class, which solves the category confusion problem to a certain extent.
[0050] However, the categories defined in a data set are limited, and the categories in the data set are often confused with some undefined background classes, such as mistaking the water surface for a ship. Therefore, the present invention defines a set of category-related background classes to distinguish the target class from the background class. Since the visual-language model CLIP used in the present invention has the ability of zero-sample classification, the above goal can be achieved by simply giving the class name of the background class without retraining the model.
[0051] S02, improves the initial class activation map based on the multi-head self-attention mechanism in the visual transformer.
[0052] In order to avoid additional training of the semantic consistency network, the present invention uses the inherent multi-head self-attention mechanism in the visual transformer as semantic consistency. However, the original multi-head self-attention mechanism is category-independent, while the obtained class activation map is category-aware. If it is used directly, the misdetected pixels will be propagated to semantically similar positions, thereby aggravating the influence of noise points. Therefore, the present invention proposes a category-aware consistency module. In the previous step, the class activation map M c Also obtained is the self-attention weight matrix W attn The present invention expects to expand the seed region based on the attention matrix. In order to generate a category-aware consistency matrix, the specific steps adopted are as follows:
[0053] (1) Through a class activation mapping, we can get the class activation map M of the target category c c ∈R h×w , and the self-attention weight W attn ∈R hw×hw In order to obtain a symmetric consistency matrix, we first use Sinkhorn regularization to normalize the self-attention weights so that the sum of its rows and columns is 1, recorded as matrix D, and then the semantic consistency matrix A is defined as follows:
[0054]
[0055] (2) In order to introduce category information, we activate the class map M c Set a threshold to get some connected regions, and take the bounding rectangles of these regions as category masks. The bounding rectangles obtained contain the pixel positions of the current category. Then use these masks to mask the consistency matrix A so that it only focuses on the pixels of the current category. In other words, only the pixels within the rectangular box will be used to expand the seed area of the current class activation map, thereby reducing the impact of noise.
[0056] (3) Use the masked consistency matrix to correct the seed region. The process is as follows:
[0057]
[0058] In the formula, Mc and A represent the class activation map and consistency matrix obtained in the above steps respectively. represents the improved class activation map, Bc represents the circumscribed rectangle mask, ⊙ represents the Hadamard product, t represents the number of iterations, and vec() represents matrix vectorization. Note that the above correction process S02 can be integrated into the seed region generation process S01 without additionally training a consistency network, which greatly improves the efficiency of weakly supervised semantic segmentation.
[0059] S03, use conditional random fields to post-process the improved class activation map to obtain pseudo labels for semantic segmentation.
[0060] The use of conditional random fields can clarify the boundaries of objects and further optimize the class activation map. The mainstream method to obtain pseudo labels from class activation maps is to set a threshold. The position above the threshold is the foreground, and vice versa. However, the generated pseudo labels are inaccurate, and the above approach will introduce noise. Based on the characteristic that each pixel position in the class activation map represents the confidence that it belongs to a certain category, the present invention ignores the position with lower confidence, that is, when training the subsequent segmentation network, it does not participate in the calculation of the loss function, which can reduce the impact of noise to a certain extent.
[0061] S04, use the generated pseudo labels to train the semantic segmentation model.
[0062] This stage is only to verify the quality of the generated semantic segmentation pseudo-labels. The pseudo-label generation process proposed in the present invention itself does not require training.
[0063] To verify the effectiveness of the present invention, the present invention is compared with similar methods on the PASCAL VOC 2012 and COCO 2014 datasets. The PASCAL VOC 2012 dataset contains 20 categories and 1 background class, and is divided into three parts: training set, validation set and test set, with 1464, 1449 and 1456 pictures respectively. The present invention uses an enhanced dataset of PASCAL VOC 2012 when training the final segmentation network, which contains 10582 pictures. The COCO 2014 dataset contains 80 categories and 1 background class, and the training set and validation set have 82081 and 40137 images respectively.
[0064] The evaluation index used in the present invention is the mean intersection-over-union (mIoU) score, which is the ratio of the intersection and union of objects in the predicted segmented image and the real segmented image. The present invention first verifies the effect of the generated pseudo labels on the PASCAL VOC 2012 training set, and the results are shown in Table 1.
[0065] Table 1
[0066]
[0067] In Table 1, +CRF and +Affinity network represent the use of conditional random fields and training of additional consistency networks to optimize class activation maps. The results show that the method proposed in the present invention can generate higher quality pseudo labels.
[0068] In order to verify the efficiency of the proposed method, the present invention compares the time and memory consumption of the proposed method and other methods on the PASCAL VOC 2012 enhanced training set, and the results are shown in Table 2. The unit of time is hours, and the unit of memory is gigabytes (GB). Since the present invention directly migrates the pre-trained visual-language model to the target dataset, there is no need to train the classification model additionally. In addition, the use of multi-head self-attention to improve the seed region avoids training an additional semantic consistency network. Therefore, the process of generating pseudo labels in the present invention does not require training, and the time and memory consumption are much better than the previous methods.
[0069] Table 2
[0070]
[0071] In order to further evaluate the quality of the generated pseudo labels, the present invention uses the generated pseudo labels to train the segmentation model and compares it with other methods on the PASCAL VOC 2012 validation set and test set (Table 3) and the COCO 2014 validation set (Table 4), where Backbone represents the backbone network used and Seg represents the segmentation model used.
[0072] The backbone network used in this experiment is ResNet101, and the semantic segmentation training is performed based on the DeepLabv2 network. As shown in Tables 3 and 4, the method proposed in the present invention performs better than similar methods on the PASCAL VOC 2012 and COCO 2014 datasets. This proves the versatility and superiority of the text-driven weakly supervised semantic segmentation method proposed in the present invention.
[0073] Table 3
[0074]
[0075] Table 4
[0076]
[0077] Figure 3 The effect of semantic segmentation pseudo labels generated by the algorithm of the present invention and another algorithm based on the language-vision model is demonstrated. It can be clearly seen that the pseudo labels generated by the algorithm of the present invention have more complete segmentation results, and the overall visual effect of the segmented image is also greatly improved. This qualitatively proves the effectiveness of the algorithm proposed by the present invention.
[0078] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.
Claims
1. An efficient text-driven weakly supervised semantic segmentation method, characterized in that: The steps include: (1) Input the image and the corresponding category name into the pre-trained visual-language model and use the gradient-based class activation map to obtain the initial class activation map; When using gradient-based class activation mapping, softmax is introduced into the class activation map, and the classification score Y of the current category c is c After softmax processing, the probability score s is obtained c , using the probability score s c Calculate the gradient and get the new weight as follows: The initial class activation map is obtained by weighted summation of the feature map, and the formula is: in, represents the weight connecting the kth feature map and the current category c, Z represents the number of pixels in the feature map, and Y c Represents the classification score of the current category c, Y c’ represents the classification score of the c'th category, Represents the activation value at position (i, j) on the feature map, s c ' represents Y c’ The probability score after the softmax function is processed, and the ReLU activation function is used to remove features that have no positive impact on the current category; After softmax processing, the probability score s is obtained c The formula is: In the formula, c' represents the c'th category, and C represents the total number of categories; (2) The initial class activation map is improved based on the multi-head self-attention mechanism in the visual transformer; the specific process is as follows: (2-1) Step (1) obtains the class activation map M c At the same time, the self-attention weight matrix W is also obtained attn ; Use Sinkhorn regularization to normalize the self-attention weights so that the sum of its rows and columns is 1, recorded as matrix D, and then define the semantic consistency matrix A as follows: (2-2) is the class activation graph M c Set a threshold to obtain some connected areas, and take the bounding rectangles of these connected areas as category masks. The bounding rectangles contain the pixel positions of the current category. Then use these category masks to mask the consistency matrix A so that it only focuses on the pixels of the current category. (2-3) Use the occluded consistency matrix to improve the class activation map. The process is as follows: Where M c and A represent the class activation map and consistency matrix respectively, Represents the improved class activation map, B c represents the circumscribed rectangular mask, ⊙ represents the Hadamard product, t represents the number of iterations, and vec() represents matrix vectorization; (3) Post-process the improved class activation map using conditional random fields to obtain pseudo labels for semantic segmentation; (4) Use the generated pseudo-labels to train the semantic segmentation model, and use the trained semantic segmentation model to perform semantic segmentation on the images to be segmented.
2. The text-driven efficient weakly supervised semantic segmentation method according to claim 1, characterized in that: In step (1), a visual-language model pre-trained on large-scale image-text pair data is used, which includes an image encoder and a text encoder. Before the class name is input into the text encoder, the class name is expanded by fusing prompt words and synonyms.
3. The text-driven efficient weakly supervised semantic segmentation method according to claim 1, characterized in that: In step (3), when the conditional random field is used to post-process the improved class activation map, the pixel positions with low confidence in the pseudo-label are ignored and do not participate in the calculation of the loss function when the semantic segmentation model is subsequently trained.
4. A text-driven efficient weakly supervised semantic segmentation device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement the efficient weakly supervised semantic segmentation method described in any one of claims 1 to 3.
Citation Information
Patent Citations
Instance segmentation method and device, electronic equipment and storage medium
CN114549833A
Model-Based Deep Learning for Globally Optimal Surface Segmentation
US20220012890A1