Weak supervision semantic segmentation model based on fine gain type activation graph

Abstract

The invention provides a weak supervision semantic segmentation model based on a fine gain class activation graph, which relates to the technical field of semantic segmentation and is composed of two continuous components: a dynamic hierarchical CAM module and a self-adaptive affinity module. The dynamic hierarchical CAM module generates two complementary CAMs by using a hierarchical function and fuses a dynamic strategy of the CAMs; the affinity based on multi-head self-attention is adaptively re-weighted by the CAM in the adaptive affinity module, so that the quality of the CAM is effectively improved. The model can effectively overcome the problems of over-activation and under-activation of the CAM. A wide range of experiments carried out on Pascal VOC 2012 and MS COCO 2014 data sets show that the proposed LayerCLIP realizes new and most advanced (SOTA) performance, and shows a huge potential of improving the weak supervision semantic segmentation performance based on the CLIP.

Description

Technical Field

[0001] The present invention relates to the technical field of semantic segmentation, and specifically relates to a weakly supervised semantic segmentation model based on a fine-grained gain class activation map. Background Art

[0002] Currently, we are in and will long be in the big data era, and an extremely large amount of data information is generated every day. The storage method has transitioned from the initial paper document storage to electronic file storage. However, with the increase in the number of files, the local storage space no longer meets the current actual needs. With the development of cloud computing, the electronic file storage method has also risen from local storage to cloud storage.

[0003] Semantic segmentation, as one of the basic tasks in the field of computer vision [1，2，3] , focuses on accurately assigning appropriate labels to each pixel in an image. However, the acquisition of pixel-level labels is labor-intensive, time-consuming, and expensive. To reduce the dependence on such intensive annotation, weakly supervised semantic segmentation (WSSS) techniques have emerged, which only use partial or incomplete supervision to learn the segmentation model. WSSS methods usually depend on different forms of supervision, such as image-level labels [4，5] , bounding boxes [6，7] , point annotations [8，9] , and scribbles [10，11，12] . Among these supervision forms, WSSS with image-level annotations is the most common and challenging task, which is also the core concern of this context.

[0004] Previous WSSS methods [5，13，14，15] using image-level labels generally follow a framework consisting of three key steps

[16] . First, a classification model is trained on a specific dataset to generate an initial class activation map (CAM). Subsequently, these initial CAMs are further refined into pseudo-labels through a pixel affinity network [1，17] or a saliency map [18，19] . Finally, these pseudo-labels are used to train the final segmentation model. Since multi-stage methods focus on the refinement process to produce higher-quality masks, it is crucial to improve the quality of the CAM [20,21,22,23] .

[0005] Recently, in some studies, the contrastive language-image pre-training (CLIP) model

[24] has been explored for the WSSS field, such as CILPES. Due to extensive training on a large-scale dataset containing 4 million image-text pairs, CLIP has successfully established a deep association between images and texts and has powerful object localization capabilities [25，26，27，28，29]Existing CLIP-based WSSS methods utilize the potential of matching visual concepts with corresponding text labels to locate objects and improve the quality of CAMs. [30，31] However, the small spatial resolution and chaotic semantic information of the last layer reduce the details of CAMs and trigger problems of over-activation and under-activation (as Figure 1 shown). The insufficient spatial information provided by the downsampled resolution results in partial masking of foreground objects. In addition, long-range dependencies introduce some irrelevant object regions, leading to incorrect activation of background regions or non-target regions. Therefore, effectively alleviating the problems of over-activation and under-activation while using CLIP to generate CAMs has always been a key issue that needs to be solved. Summary of the Invention

[0006] Weakly-supervised semantic segmentation with image-level labels endeavors to generate pseudo-labels based on class activation maps (CAMs) for training a single segmentation model. Recently, methods based on the contrastive language-image pre-training model (CLIP) have made significant progress, which utilize the potential of CLIP to locate various categories without additional training. However, due to the limited spatial resolution and long-range dependencies of the final embedding layer, the CAMs generated by the CLIP model are still a rough region, suffering from problems of under-activation or over-activation. In addition, the rich multi-layer information of CLIP is crucial for dense prediction but has been overlooked.

[0007] Based on this, the present invention proposes a weakly-supervised semantic segmentation model based on a refined gain class activation map, which consists of two consecutive components: a dynamic hierarchical CAM module and an adaptive affinity module; the dynamic hierarchical CAM module utilizes a hierarchical function to generate two complementary CAMs, as well as a dynamic strategy for fusing these CAMs; the affinity based on multi-head self-attention is adaptively reweighted by the CAMs in the adaptive affinity module, effectively improving the quality of CAMs.

[0008] Class Activation Mapping (CAM) locates discriminative regions of target classes through a weighted combination of feature maps; for a ViT-based backbone, CLIP-ES introduces a softmax function in GradCAM and defines a class-related background to suppress non-target classes and background interference, achieving effective competition between classes and making the model output more class-exclusive; the CAM of class C at position (i, j) is calculated as follows:

[0009]

[0010] where The importance weight of the k-th feature map channel for class c, z is the number of pixels in the feature map, Y cis the logit score for class c, that is, the raw score output by the model for class c before passing through the softmax function. is the activation value of the k-th feature map at position (i, j);

[0011] Based on the above weights, the CAM map for class c at spatial position (i, j) is obtained, and the relu function is used to ignore features that have a negative impact on the target class.

[0012] Furthermore, the dynamic hierarchical CAM module uses the hierarchical function to generate two complementary CAMs, and the dynamic strategy for fusing these CAMs is as follows:

[0013] For the input image: Assume the input image I = R D×H×W , where D is the dimension of the image, and H and W represent the height and width of the image; the image I is input into the pre-trained CLIP encoder to generate hierarchical features Feature F is obtained by fusing features at different levels c , and the operation is as follows:

[0014]

[0015] Here, Mean(.) and Conat(.) represent the operations of averaging and concatenation respectively; the averaging operation performs arithmetic averaging on the features of each layer, aiming to extract the common information of the features of each layer; the concatenation operation concatenates the features of each layer in the channel dimension, thus retaining the uniqueness and complementarity of the features of each layer; in this way, feature F c not only contains the multi-level information of the image, but also enhances its expression ability and robustness through feature fusion;

[0016] From the generated hierarchical features Feature F is obtained, and according to the research in CLIP-ES, the obtained fused feature F c and the hierarchical feature F generate uCAM and MCAM through formulas (1) and (2); due to the diversity of objects in complex scenes, the intra-class variation is huge; it is far from enough for a single type of CAM to hard-cover all scenes of an object, and directly fusing other CAMs will also result in superposition errors; therefore, a dynamic strategy is proposed to select a suitable CAM for the current image, which is expressed as follows:

[0017]

[0018] Among them, when facing over-activated categories, screening is performed according to the discrimination criterion of formula (6), and when facing incompletely activated targets, the discrimination criterion of formula (6) will be flipped; based on this rule, high-quality initial CAMs are dynamically retained and low-quality CAMs are removed.

[0019] Furthermore, the affinity based on multi-head self-attention is adaptively reweighted by the CAM in the adaptive affinity module, specifically as follows:

[0020] First, a standard affinity matrix based on the CAM is obtained as follows:

[0021]

[0022] where S represents the sigmoid function, and A d means the standard affinity, which is used to evaluate the similarity with the MHSA;

[0023]

[0024] where A l is the MHSA mapping from the CLIP image encoder, and S l represents the similarity between A d and A l ; after obtaining the similarities of all MHSAs, the mean value of S l is calculated as the adaptive threshold to filter the MHSAs; those multi-head attention maps with similarity values greater than the threshold will be retained; finally, the constructed new LayerCLIP generates a high-quality CAM.

[0025] The present invention proposes a CAM that generates fine gains based on the CLIP model. LayerCLIP, that is, consists of two subsequent components: a dynamic hierarchical CAM (DHC) module and an adaptive affinity (ADA) module. Generally, CAMs originate from the last layer of the classification model, which has limited spatial resolution for dense prediction tasks. Although CLIP-ES

[30] utilizes the softmax function in GradCAM to improve the CAM, the function of generating CAMs is still at the last layer of the Vit-based CLIP. The semantic information of multiple layers is different. The high-level has rich semantic contexts, while the low-level has better details. However, the hierarchical features that are ignored in existing CLIP-based methods are usually used for pixel-level segmentation tasks. Therefore, a dynamic hierarchical CAMs module is proposed to utilize important hierarchical features, where the hierarchical features are used to generate complementary CAMs and a dynamic strategy for generating the initial CAM. Instead of directly fusing various layer features, a CAM aggregation module is proposed to merge the different results of multi-layer features at the CAM level because the goal of WSSS is to obtain high-quality pseudo-masks.

[0026] To further refine the CAM, an adaptive affinity module is proposed. The inheritance of multi-head self-attention (MHSA) in CLIP based on ViT reflects the semantic affinity between patches. Therefore, CLIP-ES proposes an affinity module based on class-aware attention to improve the initial CAM. FrozenCLIP utilizes these features to filter ordinary MHSA in one-stage WSSS. However, the features used for affinity are extracted from the segmentation backbone, and these features are highly coupled with the segmentation results. Errors in the segmentation results will lead to incorrect selection of the coupled features. Since the purpose of affinity is to modify the initial CAM, the present invention rethinks the affinity process and reweights the affinity through the CAM itself. The initial CAM is used as a criterion to judge the correlation between affinity and CAM, which will ensure consistency in the refinement process.

[0027] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0028] 1. The present invention proposes a new LayerCLIP framework, which reveals the defects of existing CLIP-based methods in CAM generation and improvement. The method of the present invention demonstrates the potential of generating fine CAMs.

[0029] 2. LayerCLIP consists of two key components, a dynamic hierarchical CAM module and an adaptive affinity module, which can effectively overcome the over-activation and under-activation problems of CAM.

[0030] 3. Extensive experiments conducted on the Pascal VOC 2012

[13] and MS COCO 2014

[32] datasets show that the proposed LayerCLIP achieves new state-of-the-art (SOTA) performance and demonstrates great potential for improving the performance of CLIP-based weakly supervised semantic segmentation. Brief Description of the Drawings

[0031] Figure 1 It is a comparison of the CAMs generated by CLIP-ES. On the left, the method has a problem of under-activation for some targets, and on the right, the method has a problem of over-activation for some targets.

[0032] Figure 2This is the overall framework of the weakly supervised semantic segmentation model proposed in the present invention based on fine-grained gain class activation maps. It mainly consists of two parts: the dynamic hierarchical CAMs module and the adaptive affinity module. The final features, intermediate-level features, or intermediate-level fused features are extracted from the CLIP image encoder through the dynamic hierarchical CAMs module. Then, these features are backpropagated to generate class activation maps (CAMs). Subsequently, these CAMs are evaluated and selected to obtain more comprehensive and accurate CAMs. The adaptive affinity module filters the multi-head attention matrix with the initially selected CAMs after evaluation as the selection criterion to obtain the affinity matrix. Finally, the filtered affinity matrix is used to refine the initially selected and evaluated CAMs to generate pseudo-labels.

[0033] Figure 3 is a comparison of class activation maps from different Transform blocks, randomly selected from the PASCAL VOC 2012 dataset

[42] In this case, TB represents different transform blocks of the image encoder in the CLIP model. Fusion is the class activation map generated by fusing the feature maps of different layer blocks from TB7 to TB11.

[0034] Figure 4 is a comparison of CAMs generated by different methods. Here, (a) represents the original image, (b) represents the CAM generated by CLIP-ES

[30] generated CAM, and (c) represents the CAM generated by the method of the present invention.

[0035] Figure 5 is a comparison of the method of the present invention with the CLIP-ES

[30] method in generating pseudo-labels on the voc and coco datasets. The left one is the PASCAL VOC 2012 dataset, and the right one is the MS COCO 2014 dataset. Detailed implementation manners

[0036] The present invention will be further described in detail below in conjunction with embodiments and the accompanying drawings.

[0037] Embodiment 1: Introduction to the model architecture

[0038] Weakly-supervised semantic segmentation uses degraded labels, commonly including image-level labels, bounding boxes, point annotations, and scribbles. Bounding boxes delimit the target area with a rectangle, defining the scope for segmentation; point annotations guide the model to understand the object shape and position by marking key positions; scribbles intuitively outline the general contour of the target. In recent years, weakly-supervised semantic segmentation has played an important role in multiple fields. In autonomous driving, it helps to accurately segment targets such as vehicles and pedestrians in road scenes, enhancing the environmental perception ability; in medical image analysis, it can assist in identifying lesion areas, providing support for disease diagnosis. Among these labels, image-level labels have attracted much attention in research and applications because they only need to label the category of the whole image without precisely outlining the target, and the annotation is simple and the cost is low.

[0039] Most existing WSSS with image-level labels use pseudo-labels generated based on the CAM of the trained classification model to train the segmentation network. However, the initial CAM is usually incomplete or redundant. Due to the task gap between image-level labels and dense pixel-level labels, the CAM usually focuses on the most discriminative regions of the target in the image. To improve the quality of pseudo-labels, several methods have been proposed to optimize the CAM. For example, adversarial erasing and prohibitive segmentation learning [33,34] Expand the segmentation area with unique algorithms. The former dynamically adjusts the model's focus through adversarial training, and the latter guides the model through constraint rules. PSA [2] Introduce an affinity network to directly predict the semantic affinity of adjacent pixels, laying the foundation for pixel relationships in segmentation. Some studies such as SIPE

[35] Mine prototype representations to optimize the class activation mapping (CAM), extract specific prototypes of the image to accurately locate objects. There are also studies that propose a pixel-to-prototype contrast strategy

[36] Generate high-quality pseudo-labels. As the research progresses, some works are carried out for Vision Transformer

[37] Unfold, such as MCTformer designs multi-class tokens in the transformer architecture

[38] to improve the segmentation fineness, and Token Contrast uses PTC and CTC modules to solve the over-smoothing problem of ViT. However, limited by the representation ability of the backbone network, these methods all encounter bottlenecks in performance improvement.

[0040] CLIP (Contrastive Language-Image Pretraining) is an innovative multi-modal machine learning model

[39] , it integrates two core components: an image encoder and a text encoder. By adopting a contrastive learning strategy, CLIP can be deeply trained on a large-scale dataset of text-image pairs. Since it can associate a wider range of visual concepts in images with their text labels in an open-world environment, CLIP has been applied to multiple semantic segmentation tasks, such as zero-shot semantic segmentation, decoupled zero-shot semantic segmentation, and extracting free dense labels from CLIP, which has attracted the attention of the weakly supervised semantic segmentation (WSSS) field.

[0041] Recently, some studies have explored the potential of the CLIP model in WSSS tasks. CLIMS

[40] was the first to apply CLIP to WSSS tasks and significantly improved the segmentation performance by activating more complete target regions and effectively suppressing background regions. However, in the CLIMS method, CLIP mainly plays an auxiliary role in evaluating the presence of objects, and the actual generation of class activation maps (CAMs) depends on another convolutional neural network (CNN) model. Different from CLIMS, CLIP-ES

[30] proposes a more direct and efficient strategy that makes full use of the softmax function in the CLIP model to calculate Grad-CAM

[41] , without additional training. At the same time, they redesigned the text input under the WSSS setting to give full play to the advantages of CLIP, thus obtaining more efficient and accurate segmentation results. Different from the multi-level CLIP-ES, WeCLIP

[32] further explores the potential of directly applying CLIP to segmentation and single-stage processes. This invention focuses on a multi-stage scheme because it can provide higher-quality CAMs by flexibly adopting more targeted refinement processes and advanced segmentation models. Therefore, this invention explores the defects in the CAM generation process and rethinks the refinement process in CLIP-ES to generate CAMs with fine improvements.

[0042] As Figure 2 shown, the proposed LayerCLIP framework in this invention consists of two consecutive components: a dynamic hierarchical CAM module and an adaptive affinity module. Specifically, the dynamic hierarchical CAM module first generates two complementary CAMs using hierarchical features and generates an initial CAM by designing a dynamic selection strategy. After that, the adaptive affinity module directly explores the CAM itself and adaptively re-weights the affinity for the redundant multi-head self-attention mechanism. By proposing the simple and effective LayerCLIP, the quality of the CAM is improved. To better express the core idea of the method of this invention, using CLIP-ES

[30] as a benchmark, each module of LayerCLIP is described in detail as follows:

[0043] 1 Dynamic Hierarchical CAM

[0044] Class Activation Mapping (CAM), as an effective method, is widely used to locate discriminative regions of target classes through weighted combinations of feature maps. However, its application is limited to specific Convolutional Neural Network (CNN) architectures, especially those models that are configured with a Global Average Pooling (GAP) layer immediately after the feature map. In contrast, Grad-CAM

[51] overcomes this architectural limitation by using gradient information to combine feature maps, showing broader applicability. For the ViT-based backbone, CLIP-ES introduces the softmax function in GradCAM and defines a class-related background to suppress non-target class and background interference, enabling effective competition between classes and making the model output more class-exclusive. The CAM of class c at position (i, j) can be calculated as follows:

[0045]

[0046] where is the importance weight of the k-th feature map channel for class c, z is the number of pixels in the feature map, and Y c is the logit score for class c, that is, the raw score output by the model for class c before passing through the softmax function, is the activation value of the k-th feature map at position (i, j). Based on the above weights, the CAM map of class c at spatial position (i, j) can be obtained, and the relu function is used to ignore features that have a negative impact on the target class.

[0047] 1.1 Dynamic Hierarchical Strategy

[0048] Although CAM can be modified by other classes in CLIP-ES through the above equation, the features are still at a single scale of the last layer, with limited spatial resolution. In addition, hierarchical features have not been emphasized, and the present invention finds that it can provide complementary information for fine CAM of objects (such as Figure 3As shown. Therefore, it is a straightforward insight to generate a more accurate CAM using hierarchical features. Since the goal is to obtain a high-quality CAM, the features at different levels are selected for fusion instead of the CAM results generated at different levels. In addition, the CLIP form generated by CAM does not require additional training, and the fusion of multi-level features may lead to a decrease in CAM quality due to semantic confusion. Therefore, in addition to the original CAM generated from the last layer, the present invention also generates an additional CAM obtained from the rejected multi-level features. These two CAMs are complementary to obtain a more accurate object region in the next step.

[0049] Specifically, for the input image: Assume the input picture I = R D×H×W , where D is the dimension of the picture, and H and W represent the height and width of the picture. The image I is input into the pre-trained CLIP encoder to generate hierarchical features The present invention obtains the feature F by fusing features at different levels c , and the operation is as follows:

[0050]

[0051] Here, Mean(.) and Conat(.) represent the operations of averaging and concatenation respectively. The averaging operation aims to extract the common information of each layer of features by performing arithmetic averaging on the features of each layer; the concatenation operation concatenates the features of each layer in the channel dimension, thus retaining the uniqueness and complementarity of each layer of features. In this way, the feature F c not only contains multi-level information of the image but also enhances its expression ability and robustness through feature fusion.

[0052] 1.2 Dynamic Selection Strategy

[0053] Obtain the feature F from the generated hierarchical features . According to the research in CLIP-ES, the obtained fused feature F c and the hierarchical feature F are used to generate uCAM and MCAM through formulas (1) and (2). Due to the diversity of objects in complex scenes, the intra-class variation is huge. It is far from enough for a single type of CAM to hard-cover all scenes of an object, and direct fusion of other CAMs will also result in superposition errors. Therefore, the present invention proposes a dynamic strategy to select a suitable CAM for the current image, which is expressed as follows:

[0054]

[0055]

[0056] Among them, when facing over-activated categories, the present invention screens according to the discrimination criterion of formula (6), and when facing incompletely activated targets, the discrimination criterion of formula (6) will be reversed. Based on this rule, high-quality initial CAMs can be dynamically retained and low-quality CAMs can be removed.

[0057] 2 Adaptive Affinity

[0058] After obtaining the initial CAM, it needs to be further refined to generate the final pseudo-mask for training the segmentation model. CLIP-ES

[30] The MHSA in ViT is used as semantic affinity to refine the initial CAM. However, the affinity based on MHSA is redundant, and all attention weights are calculated to obtain the corresponding affinity matrix, which will cause inaccurate localization of the target class, as Figure 4 shown. On the contrary, affinity involves a selection strategy to retain relevant attention weights. Different from WeCLIP

[32] standards, the affinity in WeCLIP is established from the features of the segmentation backbone, and the quality of the segmentation result often affects the selection of affinity. Since the method of the present invention provides high-quality CAMs, the initial CAM can be directly used as a standard to select affinity, and the error propagation of coupled features in WeCLIP is avoided.

[0059] Specifically, the present invention first obtains the standard affinity matrix based on CAM as follows:

[0060]

[0061] where S represents the sigmoid function, and A d means standard affinity. According to previous work, A d is used to evaluate the similarity with MHSA.

[0062]

[0063] where A l is the MHSA mapping from the CLIP image encoder, and S l represents the similarity between A d and A l . After obtaining the similarity of all MHSAs, the mean value of S l is calculated as the adaptive threshold to filter MHSAs. Those multi-head attention maps with similarity values greater than the threshold will be retained. The method of the present invention uses CAM itself as a standard to screen rich multi-head attention maps, which has high consistency compared with previous methods. Finally, the novel LayerCLIP of the present invention generates high-quality CAMs. As Figure 4As shown, the CAM generated by the method of the present invention is more accurate than the baseline and alleviates the problems of over-activation or under-activation.

[0064] Example 2: Experiment

[0065] 1 Dataset and Evaluation

[0066] To verify the effectiveness of the proposed framework, the present invention uses two public datasets for weakly supervised semantic segmentation: Pascal VOC 2012

[42] and MS COCO 2014

[43] . The Pascal VOC 2012 dataset covers 21 categories (including a background class), and the enhanced training set contains 10,582 images [5] . The MS COCO 2014 dataset is more complex, containing 80 object categories and a background class. The training set contains 82,081 images, and the validation set contains 40,137 images. It should be noted that during the process of generating class activation maps (CAMs), the present invention completely relies on image-level ground truth labels. All experiments use the mean intersection over union (mIoU) as the evaluation criterion.

[0067] 2 Specific Details

[0068] In the CAM generation stage, the present invention adopts the clip pre-trained vit-b-16 model

[56] , and selects the feature map before the last self-attention layer in the vit structure as the generation basis. The present invention sets the parameter n in the dynamic hierarchical CAM to 7 and the hierarchical feature to 9, achieving a significant performance improvement. During the inference process, the input image maintains the original size and does not adopt a multi-scale strategy. For the VOC and COCO datasets, the present invention keeps the parameters in CLIP-ES

[30] unchanged, and the generated CAMs are then processed by a dense CRF

[57] to generate the final pseudo-mask.

[0069] For the final execution of the segmentation task, the present invention follows the deep labv2 architecture based on resnet101 [24,27,46]When processing the VOC dataset, the image size is randomly scaled to [0.5, 0.75, 1.0, 1.25, 1.5] times and cropped to 321x321 pixels. The batch size is set to 10, the number of iterations is default set to 20k, and the initial learning rate is 2.5e-5. When processing the COCO dataset, the image size is randomly cropped to [0.5, 0.75, 1.0, 1.25, 1.5, 1.75, 2.0] times and adjusted to 481x481 pixels. At this time, the batch size is set to 5, the number of iterations is default set to 100k, and the initial learning rate is 2.5e-5. Both adopt the polynomial learning rate decay strategy with the γ value set to 0.9. In addition, according to CLIP-ES

[30] Set μ = 0.95 to ignore the pseudo-labels with low credibility. During the COCO training process, the present invention adopts the alternating cross-entropy loss. In the test stage, the multi-scale strategy and the default hyperparameters of the dense crf are used for post-processing.

[0070] Both the pseudo-code generation stage and the segmentation task stage are run on NVIDIA RTX 3090 GPUs.

[0071] 3 Comparison with the state-of-the-art methods

[0072] Due to the importance of CAM, the quality of CAM was first evaluated. Table 1 shows the mIoU of the CAMs generated by different methods on the PASCAL VOC 2012 dataset. The method of the present invention achieved the state-of-the-art performance on the initial seeds and significantly outperformed the baseline, improving the mIoU by 1.6%. Under the dCRF setting, the performance was improved to 77.4%, which was also better than all previous methods. Compared with the latest method CPAL

[54] CPAL introduced an additional prototype to improve the performance of CLIP-ES. The method of the present invention still performed better than it without additional training parameters.

[0073] Table 1 miou of the CAMs generated on the PACAL VOC 2012 training set. dCRF indicates using the dense crf

[21] to post-process the CAMs.

[0074]

[0075] To further compare the quality of the pseudo-labels, the segmentation results based on DeepLabV2 and ResNet-101 are shown in Table 2. The LayerCLIP of the present invention achieved the state-of-the-art performance on the validation and test sets of PASCAL VOC 2012, reaching 75.0% and 75.1% mIoU respectively. Compared with CLIP-ES

[30] In comparison, the LayerCLIP of the present invention improves the mIoU by 1.2% on both the val and test sets. Even when compared with the latest CLIP-ES improvement method CPAL

[54] the method of the present invention also achieves state-of-the-art results.

[0076] Table 2 Evaluation results on the Pascal voc 2012 validation and test sets. The best results are shown in bold. Segment. represents the segmentation network. and represent using the voc and ms coco pre-trained models respectively.

[0077]

[0078] Traditionally, the performance on MS COCO 2014 was also evaluated. As shown in Table 3, the method of the present invention also outperforms other methods, with an mIoU of 46.9%, which also improves the baseline performance by 1.5%. The evaluation metrics prove the effectiveness and advantages of the LayerCLIP of the present invention.

[0079] Table 3 Evaluation results on the MS coco 2014 validation set. The best results are shown in bold. Seg. represents the segmentation network, and Sup represents the weakly supervised type.

[0080]

[0081] The qualitative results of the method of the present invention and the baseline on the PASCAL VOC 2012 and MS COCO 2014 datasets are also shown in Figure 5 . The visual results show that for both datasets, the LayerCLIP of the present invention can generate more accurate and complete pseudo-labels than the baseline, and both over-activation and under-activation can be alleviated.

[0082] 4 Ablation experiments

[0083] Table 4: Reports the impact of feature fusion with different numbers of transformer layers. That is, n in formula (3). As the number of layers decreases, the performance improves. This is because fusing these shallow features leads to semantic confusion and affects the discrimination of target classes. As the number of layers decreases, better feature representations will be fused. However, if n is greater than 7, the performance will decline. A possible reason is that the lack of fused features leads to the loss of detailed features. Therefore, it is reasonable for the parameter n to decline after increasing to 7.

[0084] Table 4 Ablation study of n on the number of transformer layers in formula (3). The miou value is evaluated on the pascal voc2012train set.

[0085]

[0086] The LayerCLIP of the present invention has two key components: dynamic hierarchical CAM and adaptive affinity. An ablation study was conducted on the PASCAL VOC 2012 dataset to explore the contribution of each module. The experimental results are shown in Table 5. Either the single dynamic hierarchical CAM module or the adaptive affinity module can promote the performance of the baseline. In the DHC module, complementary CAMs are generated using hierarchical features, and fine CAMs are obtained after dynamic selection. Therefore, DHC can improve the CAM quality of the baseline. For the ADA module, the affinity relationships based on redundant self-attention are adaptively reweighted by the CAM itself, and the highly relevant affinity relationships are retained to refine the CAM. Therefore, the performance of the baseline can be improved. In addition, due to the high-quality CAM of the DHC module, the mIoU of DHC is better than that of the ADA module. The combination of these models achieves an mIoU of 72.3%, verifying the effectiveness of the method of the present invention.

[0087] Table 5 Ablation experiment analysis of various methods, and the miou value is evaluated on the pascal voc 2012 training set.

[0088]

[0089] The method of the present invention is applied to compare the segmentation effects of the current state-of-the-art language-guided models on the voc dataset. As shown in Table 6, first, the method is applied to the CLIP-ES

[30] method, and the segmentation effect ranges from 73.8% to 75.0%, an increase of 1.2%. Secondly, in the WeCLIP

[32] method, this method borrows the features extracted by the CLIP model and realizes segmentation through a segmentation network stacked with visual transforms. The dynamic hierarchical CAM module is applied to this method, and finally the segmentation result ranges from 46.4% to 76.9%, an increase of 0.5%, reaching the current state-of-the-art level. The comparison between the two models once again shows the effectiveness of the method of the present invention.

[0090] Table 6 Effectiveness of the method of the present invention. Results after CRF on the Pascal voc 2012 val dataset

[0091]

[0092] In summary, the present invention proposes a new LayerCLIP framework and studies the potential of the hierarchical features of ViT-based CLIP in improving the quality of CAM. In addition, the present invention rethinks the role of self-attention-based affinity and proposes an adaptive affinity to correct the class activation map. The method of the present invention achieves SOTA performance on PASCAL VOC 2012 and COCO2014, and also verifies the effectiveness of the LayerCLIP of the present invention. A large number of experiments prove that the proposed method can further inspire the research of WSSS.

[0093] The above content is only an example and illustration of the concept of the present invention. Those skilled in the art of the present technology can make various modifications or supplements to the described specific embodiments or use similar methods to replace them, as long as they do not deviate from the concept of the invention or exceed the scope defined by this claim book, they should all fall within the protection scope of the present invention.

[0094] The references in the specification are listed as follows:

[0095] [1] J.Ahn, S.Cho, and S.Kwak, “Weakly supervised learning of in stancesegmentation with inter-pixel relations,” in Proceedings of the IEEE / CVFconference on computer vision and pattern recognition, 2019, pp.2209–2218.

[0096] [2] Y.Wei, X.Liang, Y.Chen, Z.Jie, Y.Xiao, Y.Zhao, and S.Yan, “Learning tosegment with image-level annotations,”Pattern Recogni tion, vol.59, pp.234–244, 2016.

[0097] [3]B. Zhang, J. Xiao, J. Jiao, Y. Wei, and Y. Zhao, “Affinity attention graph neural network for weakly supervised semantic segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 11, pp. 8082–8096, 2021.

[0098] [4]A. Kolesnikov and C. H. Lampert, “Seed, expand and constrain: Three principles for weakly-supervised image segmentation,” in Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14. Springer, 2016, pp. 695–711.

[0099] [5]J. Lee, E. Kim, and S. Yoon, “Anti-adversarially manipulated attributions for weakly and semi-supervised semantic segmentation,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2021, pp. 4071–4080.

[0100] [6]J. Dai, K. He, and J. Sun, “Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation,” in Proceedings of the IEEE international conference on computer vision, 2015, pp. 1635–1643.

[0101] [7]G. Papandreou, L.-C. Chen, K. P. Murphy, and A. L. Yuille, “Weakly and semi-supervised learning of a deep convolutional network for semantic image segmentation,” in Proceedings of the IEEE international conference on computer vision, 2015, pp. 1742–1750.

[0102] [8]A. Bearman, O. Russakovsky, V. Ferrari, and L. Fei-Fei, “What’s the point: Semantic segmentation with point supervision,” in European conference on computer vision. Springer, 2016, pp. 549–565.

[0103] [9]H. Chen, J. Wang, H. C. Chen, X. Zhen, F. Zheng, R. Ji, and L. Shao, “Seminar learning for click-level weakly supervised semantic segmentation,” in Proceedings of the IEEE / CVF International Conference on Computer Vision, 2021, pp. 6920–6929.

[0104]

[10] D. Lin, J. Dai, J. Jia, K. He, and J. Sun, “Scribblesup: Scribble supervised convolutional networks for semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 3159–3167.

[0105]

[11] Z. Pan, P. Jiang, Y. Wang, C. Tu, and A. G. Cohn, “Scribble-supervised semantic segmentation by uncertainty reduction on neural representation and self-supervision on neural eigenspace,” in Proceedings of the IEEE / CVF International Conference on Computer Vision, 2021, pp. 7416–7425.

[12] J. Xu, C. Zhou, Z. Cui, C. Xu, Y. Huang, P. Shen, S. Li, and J. Yang, “Scribble-supervised semantic segmentation inference,” in Proceedings of the IEEE / CVF International Conference on Computer Vision, 2021, pp. 15354–15363.

[0106]

[13] Y. Wang, J. Zhang, M. Kan, S. Shan, and X. Chen, “Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2020, pp. 12275–12284.

[0107]

[14] J. Xie, X. Hou, K. Ye, and L. Shen, “Clims: Cross language image matching for weakly supervised semantic segmentation,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2022, pp. 4483–4492.

[0108]

[15] L. Xu, W. Ouyang, M. Bennamoun, F. Boussaid, and D. Xu, “Multi class token transformer for weakly supervised semantic segmentation,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2022, pp. 4310–4319.

[0109]

[16] A. Kolesnikov and C. H. Lampert, “Seed, expand and constrain: Three principles for weakly-supervised image segmentation,” in Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14. Springer, 2016, pp. 695–711.

[0110]

[17] J. Ahn and S. Kwak, “Learning pixel-level semantic affinity with image level supervision for weakly supervised semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4981–4990.

[0111]

[18] P.-T. Jiang, Q. Hou, Y. Cao, M.-M. Cheng, Y. Wei, and H.-K. Xiong, “Integral object mining via online attention accumulation,” in Proceedings of the IEEE / CVF international conference on computer vision, 2019, pp. 2070–2079.

[0112]

[19] G. Sun, W. Wang, J. Dai, and L. Van Gool, “Mining cross-image semantics for weakly supervised semantic segmentation,” in Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II16. Springer, 2020, pp. 347–365.

[0113]

[20] Y. Du, Z. Fu, Q. Liu, and Y. Wang, “Weakly supervised semantic segmentation by pixel-to-prototype contrast,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 4320–4329.

[0114]

[21] T. Wu, G. Gao, J. Huang, X. Wei, X. Wei, and C. H. Liu, “Adaptive spatial-bce loss for weakly supervised semantic segmentation,” in European Conference on Computer Vision. Springer, 2022, pp. 199–216.

[0115]

[22] S.-H. Yoon, H. Kweon, J. Cho, S. Kim, and K.-J. Yoon, “Adversarial erasing framework via triplet with gated pyramid pooling layer for weakly supervised semantic segmentation,” in European Conference on Computer Vision. Springer, 2022, pp. 326–344.

[0116]

[23] F. Zhang, C. Gu, C. Zhang, and Y. Dai, “Complementary patch for weakly supervised semantic segmentation,” in Proceedings of the IEEE / CVF International conference on computer vision, 2021, pp. 7242–7251.

[0117]

[24] A. Radford, J.W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning. PMLR, 2021, pp. 8748–8763.

[0118]

[25] T. Huang, B. Dong, Y. Yang, X. Huang, R.W. Lau, W. Ouyang, and W. Zuo, “Clip2point: Transfer clip to point cloud classification with image-depth pre-training,” in Proceedings of the IEEE / CVF International Conference on Computer Vision, 2023, pp. 22157–22167.

[0119]

[26] S. Jiao, Y. Wei, Y. Wang, Y. Zhao, and H. Shi, “Learning mask-aware clip representations for zero-shot segmentation,” Advances in Neural Information Processing Systems, vol. 36, pp. 35631–35653, 2023.

[0120]

[27] J. Xie, X. Hou, K. Ye, and L. Shen, “Clims: Cross language image matching for weakly supervised semantic segmentation,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recogni tion, 2022, pp. 4483–4492.

[0121]

[28] G. Zhang, L. Wang, G. Kang, L. Chen, and Y. Wei, “Slca: Slow learner with classifier alignment for continual learning on a pre-trained model,” in Proceedings of the IEEE / CVF International Conference on Computer Vision, 2023, pp. 19148–19158.

[0122]

[29] H. Zhu, Y. Wei, X. Liang, C. Zhang, and Y. Zhao, “Ctp: Towards vision language continual pretraining via compatible momentum contrast and topology preservation,” in Proceedings of the IEEE / CVF International Conference on Computer Vision, 2023, pp. 22257–22267.

[0123]

[30] Y. Lin, M. Chen, W. Wang, B. Wu, K. Li, B. Lin, H. Liu, and X. He, “Clip is also an efficient segmenter: A text-driven approach for weakly supervised semantic segmentation,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 15305–15314.

[0124]

[31] J. Xie, X. Hou, K. Ye, and L. Shen, “Clims: Cross language image matching for weakly supervised semantic segmentation,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recogni tion, 2022, pp. 4483–4492.

[0125]

[32] B. Zhang, S. Yu, Y. Wei, Y. Zhao, and J. Xiao, “Frozen clip: A strong backbone for weakly supervised semantic segmentation,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recog nition, 2024, pp. 3796–3806.

[0126]

[33] E. Stammes, T. F. Runia, M. Hofmann, and M. Ghafoorian, “Find it if you can: end-to-end adversarial erasing for weakly-supervised semantic segmentation,” in Thirteenth International Conference on Digital Image Processing (ICDIP 2021), vol. 11878. SPIE, 2021, pp. 610–619.

[0127]

[34] Y. Wei, J. Feng, X. Liang, M.-M. Cheng, Y. Zhao, and S. Yan, “Object region mining with adversarial erasing: A simple classification to semantic segmentation approach,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1568–1576.

[0128]

[35] Q. Chen, L. Yang, J.-H. Lai, and X. Xie, “Self-supervised image-specific prototype exploration for weakly supervised semantic segmentation,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2022, pp. 4288–4298.

[0129]

[36] Y. Du, Z. Fu, Q. Liu, and Y. Wang, “Weakly supervised semantic segmentation by pixel-to-prototype contrast,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 4320–4329.

[0130]

[37] Y. Wang, R. Huang, S. Song, Z. Huang, and G. Huang, “Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition,” Advances in neural information processing systems, vol. 34, pp. 11960–11973, 2021.

[0131]

[38] L. Xu, W. Ouyang, M. Bennamoun, F. Boussaid, and D. Xu, “Multi-class token transformer for weakly supervised semantic segmentation,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2022, pp. 4310–4319.

[0132]

[39] Y. Li, F. Liang, L. Zhao, Y. Cui, W. Ouyang, J. Shao, F. Yu, and J. Yan, “Supervision exists everywhere: A data efficient contrastive language image pre-training paradigm,” arXiv preprint arXiv:2110.05208, 2021.

[0133]

[40] J. Xie, X. Hou, K. Ye, and L. Shen, “Clims: Cross language image matching for weakly supervised semantic segmentation,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recogni tion, 2022, pp. 4483–4492.

[0134]

[41] J. Dai, K. He, and J. Sun, “Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation,” in Pro ceedingsof the IEEE international conference on computer vision, 2015, pp. 1635–1643.

[0135]

[42] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisser man, “Thepascal visual object classes (voc) challenge,” International journal of computervision, vol. 88, pp. 303–338, 2010.

[0136]

[43] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll′ar, and C. L. Zitnick, “Microsoft coco: Common objects in con text,” in ComputerVision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer, 2014, pp. 740–755.

[0137]

[44] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning. PMLR, 2021, pp. 8748–8763.

[0138]

[45] P. Krähenbühl and V. Koltun, “Efficient inference in fully connected crfs with gaussian edge potentials,” Advances in neural information processing systems, vol. 24, 2011.

[0139]

[46] J. Lee, J. Choi, J. Mok, and S. Yoon, “Reducing information bottleneck for weakly supervised semantic segmentation,” Advances in neural information processing systems, vol. 34, pp. 27408–27421, 2021.

[0140]

[47] S.-H. Yoon, H. Kweon, J. Cho, S. Kim, and K.-J. Yoon, “Adversarial erasing framework via triplet with gated pyramid pooling layer for weakly supervised semantic segmentation,” in European conference on computer vision. Springer, 2022, pp. 326–344.

[0141]

[48] J.Lee,S.J.Oh,S.Yun,J.Choe,E.Kim,and S.Yoon,“Weakly supervised semantic segmentation using out-of-distribution data,”in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition,2022,pp.16897–16906.

[0142]

[49] L.Chen,C.Lei,R.Li,S.Li,Z.Zhang,and L.Zhang,“Fpr:False positive rectification for weakly supervised semantic segmentation,”in Proceedings of the IEEE / CVF International Conference on Computer Vision,2023,pp.1108–1118.

[50] H.Kweon,S.-H.Yoon,and K.-J.Yoon,“Weakly supervised semantic segmentation via adversarial learning of classifier and reconstructor,”in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition,2023,pp.11329–11339.

[0143]

[51] Z.Chen and Q.Sun,“Extracting class activation maps from nondiscriminative features as well,”in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition,2023,pp.3135–3144.

[0144]

[52] X. Zhao, F. Tang, X. Wang, and J. Xiao, “Sfc: Shared feature calibration in weakly supervised semantic segmentation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 7, 2024, pp. 7525–7533.

[0145]

[53] S.-H. Yoon, H. Kwon, H. Kim, and K.-J. Yoon, “Class tokens infusion for weakly supervised semantic segmentation,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 3595–3605.

[0146]

[54] F. Tang, Z. Xu, Z. Qu, W. Feng, X. Jiang, and Z. Ge, “Hunting attributes: Context prototype-aware learning for weakly supervised semantic segmentation,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 3324–3334.

[0147]

[55] X. Zhao, Z. Yang, T. Dai, B. Zhang, and J. Xiao, “Psdpm: Prototype based secondary discriminative pixels mining for weakly supervised semantic segmentation,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 3437–3446.

[0148]

[56] T. Wu, J. Huang, G. Gao, X. Wei, X. Wei, X. Luo, and C. H. Liu, “Embedded discriminative attention mechanism for weakly supervised semantic segmentation,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2021, pp. 16765–16774.

[0149]

[57] S. Lee, M. Lee, J. Lee, and H. Shim, “Railroad is not a train: Saliency as pseudo-pixel supervision for weakly supervised semantic segmentation,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2021, pp. 5495–5505.

[0150]

[58] P.-T. Jiang, Y. Yang, Q. Hou, and Y. Wei, “L2g: A simple local-to-global knowledge transfer framework for weakly supervised semantic segmentation,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2022, pp. 16886–16896.

[0151]

[59] T. Zhou, M. Zhang, F. Zhao, and J. Li, “Regional semantic contrast and aggregation for weakly supervised semantic segmentation,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 4299–4309.

[0152]

[60] C. Wang, R. Xu, S. Xu, W. Meng, and X. Zhang, “Treating pseudo labels generation as image matting for weakly supervised semantic segmentation,” in Proceedings of the IEEE / CVF International Conference on Computer Vision, 2023, pp. 755–765.

[0153]

[61] Z. Chen, T. Wang, X. Wu, X.-S. Hua, H. Zhang, and Q. Sun, “Class reactivation maps for weakly-supervised semantic segmentation,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2022, pp. 969–978.

[0154]

[62] M. Lee, D. Kim, and H. Shim, “Threshold matters in wsss: Manipulating the activation for the robust and accurate segmentation model against thresholds,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2022, pp. 4330–4339.

[0155]

[63] Z. Cheng, P. Qiao, K. Li, S. Li, P. Wei, X. Ji, L. Yuan, C. Liu, and J. Chen, “Out-of-candidate rectification for weakly supervised semantic segmentation,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 23673–23684.

[0156]

[64] L. Ru, H. Zheng, Y. Zhan, and B. Du, “Token contrast for weakly supervised semantic segmentation,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 3093–3102.

[0157]

[65] Y. Wu, X. Ye, K. Yang, J. Li, and X. Li, “Dupl: Dual student with trustworthy progressive learning for robust weakly supervised semantic segmentation,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 3534–3543.

[0158]

[66] Z. Yang, K. Fu, M. Duan, L. Qu, S. Wang, and Z. Song, “Separate and conquer: Decoupling co-occurrence via decomposition and representa tion for weaklysupervised semantic segmentation,” in Proceedings of the IEEE / CVF Conferenceon Computer Vision and Pattern Recognition, 2024, pp. 3606–3615.

[0159]

[67] T. Zhou, M. Zhang, F. Zhao, and J. Li, “Regional semantic contrast andaggregation for weakly supervised semantic segmentation,” in Pro ceedings ofthe IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 4299–4309.

[0160]

[68] Y. Li, Y. Duan, Z. Kuang, Y. Chen, W. Zhang, and X. Li, “Uncer taintyestimation via response scaling for pseudo-mask noise mitiga tion in weakly-supervised semantic segmentation,” in Proceedings of the AAAI conference onartificial intelligence, vol. 36, no. 2, 2022, pp. 14471455.

Claims

1. A weakly supervised semantic segmentation model based on fine-grained gain class activation maps, characterized in that It consists of two consecutive components: a dynamic hierarchical CAM module and an adaptive affinity module; the dynamic hierarchical CAM module utilizes the hierarchical function to generate two complementary CAMs, as well as a dynamic strategy for fusing these CAMs; the affinity based on multi-head self-attention is adaptively reweighted by the CAM in the adaptive affinity module, effectively improving the quality of the CAM.

2. The weakly supervised semantic segmentation model based on the fine-grained gain class activation map according to claim 1, wherein Class Activation Mapping (CAM) locates the discriminative regions of the target class through the weighted combination of feature maps; for the ViT-based backbone, CLIP-ES introduces the softmax function in GradCAM and defines a class-related background to suppress non-target class and background interference, enabling effective competition between classes and making the model output more class-exclusive; the CAM of class c at position (i, j) is calculated as follows: Among them, The importance weight of the k-th feature map channel for class c, z is the number of pixels in the feature map, Y c is the logit score for class c, that is, the original score output by the model for class c before passing through the softmax function, is the activation value of the k-th feature map at position (i, j); Based on the above weights, the CAM map of class c at spatial position (i, j) is obtained, and the relu function is used to ignore the features that have a negative impact on the target class.

3. The weakly supervised semantic segmentation model based on the fine-grained gain class activation map according to claim 2, wherein, The dynamic hierarchical CAM module utilizes the hierarchical function to generate two complementary CAMs, and the specific dynamic strategy for fusing these CAMs is as follows: For the input image: Assume the input picture I = R D×H×W , where D is the dimension of the picture, and H and W represent the height and width of the picture; input the image I into the pre-trained CLIP encoder to generate hierarchical features Obtain the feature F by fusing features at different levels c , and the operation is as follows: Here, Mean(.) and Conat(.) represent the operations of averaging and concatenation respectively; the averaging operation performs arithmetic averaging on the features of each layer, aiming to extract the common information of the features of each layer; the concatenation operation concatenates the features of each layer in the channel dimension, thus retaining the uniqueness and complementarity of the features of each layer; in this way, the feature F c not only contains the multi-level information of the image, but also enhances its expression ability and robustness through feature fusion; Obtain feature F from generating hierarchical features According to the research in CLIP-ES, the obtained fused feature F c and the hierarchical feature F are used to generate uCAM and MCAM through formulas (1) and (2); due to the diversity of objects in complex scenes, the intra-class variations are huge; it is far from enough for a single type of CAM to hard-cover all scenes of an object, and direct fusion of other CAMs will also result in superposition errors; therefore, a dynamic strategy is proposed to select a suitable CAM for the current image, which is expressed as follows: Among them, when facing over-activated classes, screening is performed according to the discrimination criterion of formula (6), and when facing incompletely activated targets, the discrimination criterion of formula (6) is flipped; based on this rule, high-quality initial CAMs are dynamically retained and low-quality CAMs are removed.

4. The weakly supervised semantic segmentation model based on the fine-grained gain class activation map according to claim 3, characterized in that, The affinity based on multi-head self-attention is adaptively reweighted by the CAM in the adaptive affinity module, specifically as follows: First, the standard affinity matrix based on the CAM is obtained as follows: where S represents the sigmoid function, and A d means the standard affinity, which is used to evaluate the similarity with MHSA; Among them, A l is the MHSA mapping from the CLIP image encoder, and S l represents the similarity between A d and A l ; after obtaining the similarities of all MHSAs, the mean of S l is calculated as an adaptive threshold to filter the MHSAs; those multi-head attention maps with similarity values greater than the threshold will be retained; finally, the constructed new LayerCLIP generates high-quality CAMs.

Images