Weakly supervised semantic segmentation method and device based on prompt learning

Abstract

The application discloses a weakly supervised semantic segmentation method based on prompt learning, comprising the following steps: encoding an image and a category text through a learnable prompt adapter to generate feature embeddings, then querying the two, so that the image and the category label corresponding to the image level thereof are used as training data; querying by using attention, using the query and the image embedding result generated in the last step, adding a transformer to perform self-attention between the image embedding and the query; calculating a loss function during the training process to update the weight, and dividing the weakly supervised semantic segmentation (WSSS) loss into two parts: the first part is a seeding loss, and the second part is a boundary loss. The WeakCLIP method provided by the application is very simple and efficient. The application also provides a corresponding weakly supervised semantic segmentation device based on prompt learning.

Description

Technical Field

[0001] This technology primarily involves the field of weakly supervised semantic segmentation, specifically exploring methods for achieving semantic segmentation using simple annotations such as image-level labels, borders, points, and doodles. Background Technology

[0002] Semantic segmentation is a fundamental task in computer vision, aiming to infer image classification results pixel by pixel. However, unlike other tasks such as classification and detection, the pixel-level category labels required for semantic segmentation are time-consuming and expensive. Therefore, many studies in recent years have focused on weakly supervised semantic segmentation, exploring the use of simple annotations such as image-level labels, bounding boxes, dots, and doodles to reduce the burden of manual annotation.

[0003] Most weakly supervised localization methods first train a deep classification network using image-level labels, then use CAM (Classification Activation Maps) to select the most discriminative regions as localization maps, and finally construct a deterministic relationship between image-level labels and pixels. Despite some promising results, significant performance differences remain between weakly supervised and fully supervised methods. The core problem is that the noisy and error-prone CAM seed is the only source of information in a weakly supervised system, and the spurious indications it introduces severely impact the performance of the weakly supervised model.

[0004] Previous approaches to weakly supervised semantic segmentation have been relatively complex, with specialized designs for input, network, loss function, supervision information, and semantic expansion. First, some methods focus on mining semantic information between objects or incorporating network data into the input. Second, regarding the network, some methods use two backbone networks with shared weights for multi-task training, such as simultaneously performing weakly supervised semantic segmentation and saliency prediction. Third, in terms of loss function selection, in addition to the commonly used seeding loss and energy loss, contrastive loss has recently become popular. Regarding supervision information, some methods introduce saliency supervision or the uncertainty of semantic information. Finally, some methods additionally introduce positional saliency, AffinityNet, or other traditional region expansion methods for semantic expansion.

[0005] The main problems with existing technical solutions are as follows. First, the classification network in previous weakly supervised semantic segmentation methods only focuses on the most important components of the target, resulting in small and sparse discrimination regions. Higher semantic segmentation performance requires more precise and fine-grained supervision. Second, image-level labels cannot provide any information about the object's location. To detect the most representative elements of the target object and construct a localization map called a "seed," a method called CAM is used. However, because it ignores most of the object, the simple CAM seed cannot directly replace the labeled data. Summary of the Invention

[0006] To address the aforementioned shortcomings or improvement needs of existing technologies, this invention provides a weakly supervised semantic segmentation method based on a zero-shot CLIP (Contrastive Language and Image Pre-training) learner. This method boasts a concise and efficient architecture. The purpose of this invention is to mitigate the negative impact of incorrect CAM seeds in traditional methods by introducing a powerful zero-shot CLIP learner to provide additional image-text understanding information. This results in state-of-the-art weakly supervised semantic segmentation performance on both the PASCAL VOC 2012 and COCO 2014 datasets, and offers a promising new paradigm for ultimately solving the weakly supervised semantic segmentation problem.

[0007] To achieve the above objectives, according to one aspect of the present invention, a weakly supervised semantic segmentation method based on cue learning is provided, comprising the following steps:

[0008] (1) Encode the image and the category text through the learnable cue adapter to generate feature embeddings, and then query both to make the image and its corresponding category label at the image level the training data;

[0009] (2) Use attention for querying. Using the query and image embeddings generated in the previous step, add a transformer to perform self-attention between the image embeddings and the query. This implementation enhances the query's ability to find the most relevant visual cues. Where q represents query, This indicates the updated query, while It is a learnable proportional parameter that balances the effects of self-attention;

[0010] (3) During training, the loss function is calculated to update the weights. The weakly supervised semantic segmentation (WSSS) loss is divided into two parts: the first part is the seeding loss and the second part is the boundary loss. The seeding loss is the weighted cross-entropy loss between the CAM seed supervision and the WeakCLIP network segmentation graph. The boundary loss is to smooth the target boundary through the Conditional Random Field (CRF). The boundary loss algorithm calculates the KL divergence loss function between the smoothed result and the segmentation graph. The KL divergence is used to clarify the difference between the distribution of the segmentation graph and the CRF smoothed result.

[0011] In one embodiment of the present invention, in step (1), an image encoder is used to extract multi-layer feature maps, the size of the output feature map of each layer gradually decreases, and the feature map of the last layer is processed by global average pooling (GAP), and an attention pooling layer is added afterward, the formula of which is as follows:

[0012]

[0013]

[0014]

[0015] In the above formula, For image encoders, Let these be the image embeddings after passing through the encoder. Recorded as The representation after the GAP operation. This is recorded as a GAP operation. This is denoted as the attention pooling operation. They are respectively denoted as those obtained through attention pooling operations. .

[0016] In one embodiment of the present invention, the image encoder is a pre-trained contrastive image-text pre-trained CLIP network.

[0017] In one embodiment of the present invention, in step (1), the CLIP text encoder requires class text as input, uses the text to generate text embeddings, and the cue adapter consists of a class text vocabulary and several learnable embeddings, used as an aid to the text input, appending half of the learned embeddings to the beginning of the text and the other half to the end of the text. The formula of the cue adapter is defined as follows: In the above formula, This represents learnable embeddings that are consistent with the dimensions of the text query. Let N represent the text words of the k-th category, and N is a hyperparameter that determines the number of learnable parameters.

[0018] In one embodiment of the present invention, after obtaining image embeddings and text embeddings, the image embeddings can be queried by the text embeddings on the same number of channels. The query process is to calculate the cosine similarity between the two, and finally, the size of the queried image embeddings is scaled back to the original size.

[0019] In one embodiment of the present invention, after performing upsampling and slicing operations, the segmentation category result is obtained.

[0020] In one embodiment of the present invention, in order to compensate for the loss of detailed information, a feature pyramid network (FPN) is added to obtain a better segmentation result, and this result is used as the final result and to calculate the loss of the CAM seed.

[0021] In one embodiment of the present invention, in step (3), the seeing loss is calculated as follows:

[0022] Seeding loss is a weighted cross-entropy loss between CAM seed supervision and the WeakCLIP network segmentation graph, where C is defined as the set of classes in the image. Represents background, definition For the set of pixels classified as category c, Representing the segmentation prediction of class c at position u, the seeding loss is defined as follows: .

[0023] In one embodiment of the present invention, the boundary loss is calculated in step (3) as follows:

[0024] The target boundary is smoothed using a Conditional Random Field (CRF). A boundary loss algorithm is then used to calculate the KL divergence loss function between the smoothed result and the segmentation map. The KL divergence is used to clarify the difference between the distribution of the segmentation map and the CRF smoothing result. In the above formula, I represents the original image, and n is the total number of positions. It represents the paired potential energy independent of the image pixels.

[0025] According to another aspect of the present invention, a weakly supervised semantic segmentation apparatus based on cue learning is also provided, including at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor, wherein the instructions, after being executed by the processor, are used to complete the weakly supervised semantic segmentation method based on cue learning.

[0026] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0027] (1) This is the first work to introduce CLIP-based cue learning to alleviate the lack of supervision in weakly supervised image understanding; this new learning model is expected to eventually solve the problem of weakly supervised semantic segmentation that has emerged with the progress of large-scale vision and language pre-training.

[0028] (2) The WeakCLIP method proposed in this invention is very simple and efficient. Without using any manually labeled visual priors, our fully learning-based method has achieved the best weakly supervised semantic segmentation performance on both the PASCAL VOC 2012 and COCO 2014 datasets, reaching mIOU (Mean Intersection over Union) of 74.8% and 43.3%, respectively. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the WeakCLIP method of the present invention. Compared with the previous weakly supervised semantic segmentation methods, the WeakCLIP method proposed in this invention is more concise, containing only an encoder, a visual-language decoder, and a WSSS loss function.

[0030] Figure 2 This is a schematic diagram of the training architecture of the WeakCLIP segmentation network of this invention. First, the input image and text are converted into two identical embeddings using CLIP pre-trained image and text encoders. Implicit semantic relationships exist between the embeddings extracted by the cross-domain encoder, and cosine similarity is directly calculated between image features and text embeddings. Then, cross-attention is calculated using a simplified transformer. Next, the text embeddings are updated by the cross-attention, which is used in the decoder to estimate semantic distance using image features. In addition, this invention can also use image-level labels to train a classification network as CAM-based supervision to locate a subset of targets. This invention uses the WSSS (Weakly-supervised semantic segmentation) loss function and CAM seeds to train the segmentation network of this invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0032] This invention first introduces the paradigm shift of weakly supervised semantic segmentation to understand the core working principle of this invention.

[0033] The weakly supervised semantic segmentation paradigm first requires a classification network to generate a classification algorithm (CAM), and then uses the CAM to train a weakly supervised semantic segmentation model. This invention focuses primarily on the utilization of the CAM. We found that directly using discriminative regions as segmentation supervision produces poor results. This result is reasonable because the CAM typically loses most of the semantic information of the target region, resulting in a lack of supervision for the segmentation model.

[0034] Therefore, this invention aims to solve this imprecise supervision problem through a paradigm shift.

[0035] In our research on paradigm shifts, we needed to consider whether we could adaptively avoid the misleading effects of incomplete labels by exploring the semantic relationships between image pixels and categories. This led us to introduce CLIP, a classifier for extracting semantic features from images and text, into the WSSS domain. Our implemented WeakCLIP method is attached. Figure 1 As shown, we extract image features from the CLIP image encoder, deploy the CLIP text encoder, and convert text-like prompts into queries. The queries are then applied to the image features, transforming the WSSS model into a continuous visual-language matching problem. The CLIP model not only functions as a pure image feature extractor like a typical backbone network, but also provides more transferable knowledge for constructing implicit semantic relationships between images and text. This semantic relationship prevents our method from being misled by inaccurate supervision information at the beginning of training. Furthermore, this transformation can generate semantic cues for parts of the image that lack semantic supervision. Based on the above analysis, we transform the weakly supervised semantic segmentation problem into a visual-language matching problem.

[0036] The WeakCLIP build workflow is as follows. The overall structure and flow can be found in the attached document. Figure 2 :

[0037] (1) In weakly supervised semantic segmentation, the training data we have consists of images and their corresponding class labels at the image level. So the first step is to encode the images and the class text through a learnable cue adapter to generate feature embeddings, and then query both.

[0038] A multi-layer feature map is extracted using an image encoder (such as a CLIP network pre-trained on ResNet-101), with the output feature map size of each layer gradually decreasing. Furthermore, the feature map of the last layer is processed by GAP (Global Average Pooling), and an attention pooling layer is added afterward. The formula is shown below:

[0039]

[0040]

[0041]

[0042] In the above formula, For image encoders, Let these be the image embeddings after passing through the encoder. Recorded as The representation after GAP operation. This is recorded as a GAP operation. This is denoted as the attention pooling operation. They are respectively denoted as those obtained through attention pooling operations. .

[0043] As another part, the CLIP text encoder requires class-based text as input, which is then used to generate text embeddings. It's important to note that we find a small number of texts in image-text pairs that consist of only one word. In most cases, the text describing the corresponding image is a sentence. Therefore, if we use only class-based text as the text query, the distributional discrepancy will lead to poor results. A common solution for this is to use a general template called a cue. For different datasets, common cue suggestions include "a photo with {label}". For tasks with insufficient supervision, we deploy a lightweight set of learnable cue suggestions, called a cue adapter, as an aid to the text input. The entire cue adapter consists of a class-based text vocabulary and several learnable embeddings. We typically append half of the learned embeddings to the beginning of the text and the other half to the end. The formula for the cue adapter is defined as follows:

[0044]

[0045] In the above formula, This represents learnable embeddings that are consistent with the dimensions of the text query. Let N represent the text words of the k-th category, and N is a hyperparameter that determines the number of learnable parameters.

[0046] After completing the above two steps, image embeddings can be queried by text embeddings on the same number of channels. The query process involves calculating the cosine similarity between the two. Finally, the size of the queried image embeddings is scaled back to the original size. After a series of upsampling and slicing operations, we can obtain the segmentation category result through the argmax operation (a function defined in Python that finds the parameters when the function reaches its maximum value). However, this result is directly obtained from the downsampled feature map. To compensate for the loss of detailed information, we also added an FPN (Feature Pyramid Network) to obtain better segmentation results. This also uses the argmax operation, and the result is used as the final result and to calculate the loss of the CAM seed.

[0047] (2) The second step is to use attention to perform the query, utilizing the query and image embeddings results generated in the previous step. We added a simple transformer to perform self-attention between the image embeddings and the query. This implementation enhances the query's ability to discover the most relevant visual cues.

[0048]

[0049] Where q represents query. This indicates the updated query, while It is a learnable proportional parameter that balances the influence of self-attention.

[0050] (3) After completing the above operations, we know that we also need to calculate the loss function to update the weights during the training process. This paper divides the WSSS loss into two parts: the first part is the seeding loss, which is the weighted cross-entropy loss between the CAM seed supervision and the WeakCLIP network segmentation map. We define C as the set of categories in the image (excluding the background). Represents the background and defines For the set of pixels classified as category c, Representing the segmentation prediction of class c at position u, the seeding loss is defined as follows:

[0051]

[0052] The second part is the boundary loss. The target boundary is smoothed using a CRF (Conditional Random Field), and the boundary loss algorithm calculates the KL (Kullback-Leibler) divergence loss function between the smoothed result and the segmentation map. We use KL divergence to clarify the difference between the distribution of the segmentation map and the CRF smoothed result.

[0053]

[0054] In the above formula, I represents the original image, and n is the total number of positions. It represents the paired potential energy independent of the image pixels.

[0055] Following the workflow described above, we obtain high-quality proxy annotation data generated by the network, upon which we can perform a further retraining step. To avoid incorrect labels misleading the fully supervised network, we set the number of categories not present in the image-level labels to 255. Finally, we use the proxy annotation data generated by this WeakCLIP network to perform fully supervised segmentation.

[0056] Furthermore, the present invention also provides a weakly supervised semantic segmentation device based on cue learning, including at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to complete the weakly supervised semantic segmentation method based on cue learning.

[0057] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A weakly supervised semantic segmentation method based on cue learning, characterized in that, Includes the following steps: (1) Encode the image and the category text through the learnable cue adapter to generate feature embeddings, and then query both to make the image and its corresponding category label at the image level the training data; (2) Use attention for querying. Using the query and image embeddings generated in the previous step, add a transformer to perform self-attention between the image embeddings and the query. This implementation enhances the query's ability to find the most relevant visual cues. Where q represents query, This indicates the updated query, while It is a learnable proportionality parameter that balances the effects of self-attention. For attention pooling operations , Let these be the fourth layer feature image embeddings after passing through the encoder; (3) During training, the loss function is calculated to update the weights. The weakly supervised semantic segmentation (WSSS) loss is divided into two parts: the first part is the seeding loss and the second part is the boundary loss. The seeding loss is the weighted cross-entropy loss between the CAM seed supervision and the WeakCLIP network segmentation graph. The boundary loss is to smooth the target boundary through the Conditional Random Field (CRF). The boundary loss algorithm calculates the KL divergence loss function between the smoothed result and the segmentation graph. The KL divergence is used to clarify the difference between the distribution of the segmentation graph and the CRF smoothed result.

2. The weakly supervised semantic segmentation method based on cue learning as described in claim 1, characterized in that, In step (1), an image encoder is used to extract multi-layer feature maps. The size of the output feature map of each layer gradually decreases. The feature map of the last layer is processed by global average pooling (GAP), and an attention pooling layer is added after it. The formula is as follows: In the above formula, For image encoders, Let these be the image embeddings after passing through the encoder. Recorded as The representation after GAP operation. This is recorded as a GAP operation. This is denoted as the attention pooling operation. They are respectively denoted as those obtained through attention pooling operations. .

3. The weakly supervised semantic segmentation method based on cue learning as described in claim 2, characterized in that, The image encoder is a pre-trained contrastive image-text pre-trained CLIP network.

4. The weakly supervised semantic segmentation method based on cue learning as described in claim 1 or 2, characterized in that, In step (1), the CLIP text encoder requires class text as input, uses the text to generate text embeddings, and the cue adapter consists of a class text vocabulary and several learnable embeddings, used as an aid to the text input. Half of the learned embeddings is appended to the beginning of the text, and the other half is appended to the end of the text. The formula for the cue adapter is defined as follows: In the above formula, This represents learnable embeddings that are consistent with the dimensions of the text query. Let N represent the text words of the k-th category, and N is a hyperparameter that determines the number of learnable parameters.

5. The weakly supervised semantic segmentation method based on cue learning as described in claim 4, characterized in that, After obtaining image embeddings and text embeddings, the image embeddings can be queried by the text embeddings on the same number of channels. The query process involves calculating the cosine similarity between the two, and finally, the size of the queried image embeddings is scaled back to the original size.

6. The weakly supervised semantic segmentation method based on cue learning as described in claim 5, characterized in that, After performing upsampling and slicing operations, the segmentation results are obtained.

7. The weakly supervised semantic segmentation method based on cue learning as described in claim 6, characterized in that, To compensate for the loss of detailed information, a Feature Pyramid Network (FPN) is added to obtain better segmentation results. This result is then used as the final result and to calculate the loss of the CAM seed.

8. The weakly supervised semantic segmentation method based on cue learning as described in claim 1 or 2, characterized in that, In step (3), the seeding loss is calculated as follows: Seeding loss is a weighted cross-entropy loss between CAM seed supervision and the WeakCLIP network segmentation graph, where C is defined as the set of classes in the image. Represents background, definition For the set of pixels classified as category c, Representing the segmentation prediction of class c at position u, the seeding loss is defined as follows: .

9. The weakly supervised semantic segmentation method based on cue learning as described in claim 1 or 2, characterized in that, In step (3), the boundary loss is calculated as follows: The target boundary is smoothed using a Conditional Random Field (CRF). A boundary loss algorithm is then used to calculate the KL divergence loss function between the smoothed result and the segmentation map. The KL divergence is used to clarify the difference between the distribution of the segmentation map and the CRF smoothing result. In the above formula, I represents the original image, and n is the total number of positions. It represents the paired potential energy independent of the image pixels.

10. A weakly supervised semantic segmentation device based on cue learning, characterized in that: It includes at least one processor and a memory, which are connected via a data bus. The memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to perform the weakly supervised semantic segmentation method based on cue learning as described in any one of claims 1-9.

Images