A weakly supervised positioning method based on prompt learning
By using a cue-based learning approach and fusing class discriminative and representational features to optimize the denoising diffusion model, the problem of inaccurate localization in weakly supervised target detection is solved, achieving higher localization accuracy and stability.
Patent Information
- Application Number
- CN202310812108.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-04
- Publication Date
- 2025-12-30
- Estimated Expiration
- 2043-07-04
AI Technical Summary
Existing weakly supervised target detection algorithms have shortcomings in localization accuracy and stability, especially in the absence of precise location information, which leads to randomness and instability in detection.
A cue-based learning approach is adopted, which obtains class-discriminative features by encoding image categories, optimizes learnable features using a denoising diffusion model, and combines class-discriminative and representational features for feature fusion to generate a cross-attention map to determine the activation region of the target.
It significantly improves the accuracy and stability of target localization, outperforming traditional discriminative model methods, and can achieve accurate target localization while suppressing background noise.
Smart Images
Figure CN117011510B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a target localization and detection method, and more particularly to a weakly supervised localization method based on cue learning, belonging to the field of artificial intelligence technology. Background Technology
[0002] Object detection, a fundamental problem in the field of computer vision, is the basis for many visual applications. Traditional supervised object detection models often require precise annotation of the location of each object in a large number of images. While these methods can learn object recognition and localization information by relying on a large amount of annotation information, they place very high demands on the collection and production of the dataset. The process of accurately annotating each object in the image dataset is time-consuming and labor-intensive, especially as the dataset size increases. This significantly affects the scaling of algorithms on massive datasets and limits the utilization of large datasets by deep learning algorithms.
[0003] To reduce manual annotation and supervision in the target modeling process, weakly supervised target detection frameworks have been proposed in existing technologies. These frameworks only require labeling whether a certain type of target has appeared in the image; this labeling work is consistent with image classification, therefore, currently labeled image classification datasets can all serve as input data for weakly supervised target detection algorithms. This annotation method significantly reduces the workload and can directly utilize massive image classification datasets. However, current weakly supervised target detection algorithms suffer from significant localization randomness during training due to a lack of precise location information, affecting the stability and accuracy of detection.
[0004] To address the problem of weakly supervised target localization, existing technologies have proposed various methods, such as target localization methods that find discriminative regions, Transformer-based methods (TS-CAM, LCTR), BAS methods, PSOL methods, and C2AM methods, etc. However, most of these methods are based on discriminative models and use regularization, online localization optimization, classification-regression decoupling, and attention regularization to solve the problem. They do not fundamentally solve the difficulty of localization.
[0005] Therefore, it is necessary to explore weakly supervised target detection frameworks more deeply in order to improve positioning accuracy. Summary of the Invention
[0006] To overcome the above problems, the inventors conducted in-depth research and designed a weakly supervised localization method based on cue learning, comprising the following steps:
[0007] S1. Encode the image description containing image categories to obtain the category discriminative features corresponding to different categories of images;
[0008] S2. For any image category, learnable features are obtained by initializing with category-discriminative features. The noisy image of that category and the learnable features are then input into a denoising diffusion model. With the goal of removing noise from the image, the learnable features are optimized through backpropagation to obtain category-representative features.
[0009] The denoising diffusion model mentioned above is a pre-trained model, set to a non-learnable state;
[0010] S3. Set the denoising diffusion model to a learnable state, input the discriminative features and representational features into the denoising diffusion model, and optimize the denoising diffusion model through backpropagation with the goal of removing noise from the image.
[0011] S4. Set up a neural network classifier to classify the image to be located and obtain the image category;
[0012] S5. Obtain the corresponding class discriminative features and class representation features according to the image category. Fuse the class discriminative features and class representation features to obtain combined features. Input the combined features into the optimized denoising diffusion model to obtain multiple cross-attention maps. Fuse the multiple cross-attention maps into an activation map. Scale the activation map to the size of the original image to determine its activation region on the original image. The minimum bounding box of the activation region is the localization result of the target in the image.
[0013] In a preferred embodiment, in S1, the image description is a string combination of a fixed template and an image category.
[0014] In a preferred embodiment, the encoding is performed using a visual language model.
[0015] In a preferred embodiment, the denoising diffusion model employs a stable diffusion model.
[0016] In a preferred embodiment, in S2, learnable features are obtained by initializing with class discriminative features.
[0017] In a preferred embodiment, in S2, the optimization objective is set as follows:
[0018]
[0019] in, L(θ,f) represents the class representation feature that achieves the optimal denoising effect for a given set of noisy images. r ) represents the loss function of the denoised probability diffusion model, θ represents the network parameters of the denoised probability diffusion model, and f r Representing category-specific features, Let L(θ,f) be the minimum function. rf when taking the minimum value r Variable value.
[0020] In a preferred embodiment, in S3, the optimization objective is set as follows:
[0021]
[0022] Where, θ * This represents the network parameters of the probability diffusion model that achieves the optimal denoising effect for a given set of noisy images.
[0023] In a preferred embodiment, in S5, the category discriminative features and category representation features are fused in a linear weighted manner to obtain combined features.
[0024] In a preferred embodiment, the cross-attention graph m l,t Represented as:
[0025] m l,t =CrossAttn(F l,t ,f c ),l∈L,t∈T
[0026] Where l represents the cross-attention layer index in the probabilistic diffusion model, L represents the set of cross-attention layer indices in the probabilistic diffusion model, t represents the input time step of the probabilistic diffusion model, T represents the set of input time steps of the probabilistic diffusion model, usually T = 1, 2, 3, ..., 1000, and CrossAttn represents the attention mechanism that mixes two different sequences.
[0027] In a preferred embodiment, the fusion of the multiple cross-attention maps is achieved through normalization and averaging, as follows:
[0028]
[0029] Where M represents the fused activation graph.
[0030] The beneficial effects of this invention include:
[0031] (1) A systematic approach is provided to solve the mismatch between discriminative models and generative localization targets through an image denoising diffusion process;
[0032] (2) This invention proposes to query the discriminative features of a category from a pre-trained visual language model using the category name; and to learn the representational features of the category using a denoising diffusion process. By combining the discriminative features and representational features of the category, the accurate localization of the target can be obtained while suppressing background noise and achieving complete activation of the target.
[0033] (3) It significantly outperforms other discriminative model-based methods in terms of performance. Attached Figure Description
[0034] Figure 1 A schematic flowchart of a weakly supervised localization method based on cue learning according to a preferred embodiment of the present invention is shown.
[0035] Figure 2 A graph showing the impact of the weights of different categories of discriminative features on localization performance;
[0036] Figure 3 The results of localization of some images in Experiment Example 2 are shown. Detailed Implementation
[0037] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Through these descriptions, the features and advantages of the present invention will become clearer and more apparent.
[0038] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.
[0039] A weakly supervised localization method based on cue learning according to the present invention is characterized by comprising the following steps:
[0040] S1. Encode the image description containing image categories to obtain the category discriminative features corresponding to different categories of images;
[0041] S2. For any image category, learnable features are obtained by initializing with the discriminative features of the category. The noisy image of that category and the obtained learnable features are then input into the denoising diffusion model. With the goal of removing noise from the image, the learnable features are optimized through backpropagation to obtain the category representation features.
[0042] The denoising diffusion model mentioned above is a pre-trained model, set to a non-learnable state;
[0043] S3. Set the denoising diffusion model to a learnable state, input the discriminative features and representational features into the denoising diffusion model, and optimize the denoising diffusion model through backpropagation with the goal of removing noise from the image.
[0044] S4. Set up a neural network classifier to classify the image to be located and obtain the image category;
[0045] S5. Obtain the corresponding class discriminative features and class representation features according to the image category. Fuse the class discriminative features and class representation features to obtain combined features. Input the combined features into the optimized denoising diffusion model to obtain multiple cross-attention maps. Fuse the multiple cross-attention maps into an activation map. Scale the activation map to the size of the original image to determine its activation region on the original image. The minimum bounding box of the activation region is the localization result of the target in the image.
[0046] According to the present invention, in S1, the image description is a string combination of a fixed template and an image category.
[0047] For example, given the category "category", the fixed template "aphoto of a" is first combined with the category string to form a description of an image containing objects of that category, "a photo of a category". Then, a pre-trained visual language model is used to encode this description to obtain the discriminative features of the category.
[0048] More preferably, a representative word block of a category is used as the image category to obtain an image description, wherein the representative word block is the last word block of the first string of the category word.
[0049] Some categories of labels correspond to multiple strings, and each string corresponds to multiple word pieces. This can cause the cue generation model to produce multiple noisy activation maps for the same target. To address this issue, in this invention, the last word piece of the first string is heuristically selected (…). <category>The category can be represented by the first word of a string, or by the word itself. For example, selecting "goldfish" from the category "goldfish,Carassius auratus" or "ray" from "electric ray,crampfish,numbfish,torpedo" to represent the category, and then combining this with a fixed template and the category's representative word block to form a description of an image containing an object of that category, such as "a photo of a...". <category>For example, you could choose "fish" from the category "fish head, fish tail, shark fin" to represent this category.
[0050] In this invention, category-discriminative features are used to focus on the most salient features of a category. Using these features as language cues can help locate the salient regions of the target category.
[0051] Preferably, the encoding is performed using a visual language model. In this invention, no specific visual language model is limited; those skilled in the art can freely choose according to actual needs, such as using the CLIP model or an optimized model based on the CLIP model.
[0052] Preferably, the obtained class discriminative features f d It can be represented as:
[0053] f d = CLIP(v(p))
[0054] Here, CLIP represents the visual language model, p represents the description of an image containing objects of that category, i.e., "a photo of a category", and v(p) represents the description of the image after being encoded by a tokenizer. Specifically, v(p) is a list containing several numeric indices, each numeric index corresponding to a string, for example, 302 corresponds to "a", 1125 corresponds to "photo", and 539 corresponds to "of".
[0055] In S2, initializing learnable features with class discriminative features means that the learnable features (representative features of the class) are first initialized with class discriminative features before optimization, that is, the class discriminative features are assigned to the learnable features.
[0056] In S2, the denoising diffusion model is a pre-trained model, preferably a pre-trained stable diffusion model.
[0057] In S2, the optimization objective is set as follows:
[0058]
[0059] in, L(θ,f) represents the class representation feature that achieves the optimal denoising effect for a given set of noisy images. r ) represents the loss function of the denoised probability diffusion model, θ represents the network parameters of the denoised probability diffusion model, and f r Representing category-specific features, Let L(θ,f) be the minimum function. r f when taking the minimum value r Variable value.
[0060] Through iterative optimization, the learnable features reach a stable state, thereby obtaining categorical representation features.
[0061] In S3, the denoising diffusion model is fine-tuned. After obtaining the discriminative features of each category and learning the representational features of each category, the representational features of each category are set to a non-learnable state, while the denoising diffusion model is set to a learnable state. Then, the discriminative features and representational features of each category are input into the denoising diffusion model, which receives a noisy image and optimizes the model through backpropagation with the goal of removing noise from the image.
[0062] Preferably, the optimization objective is as follows:
[0063]
[0064] Where, θ * This represents the network parameters of the probability diffusion model that achieves the optimal denoising effect for a given set of noisy images.
[0065] Through iterative optimization of the network, the denoising diffusion model can reach a stable state.
[0066] According to the present invention, in the optimization processes of S2 and S3, the images involved are images with only image categories given and without target locations labeled.
[0067] In S4, the specific structure of the neural network classifier is not limited. Any classifier capable of image category recognition can be used, such as a CNN-based classifier. Preferably, the EfficientNet network is used, and the classifier is optimized and trained through the backpropagation algorithm.
[0068] In step S5, based on the results of steps S1 and S2, the obtained image categories are mapped to obtain the category discriminative features and category representation features corresponding to the image.
[0069] Preferably, the category discriminative features and category representation features are fused in a linear weighted manner to obtain combined features.
[0070] Experimental studies have revealed that using only discriminative features as input to a denoising diffusion model to obtain a cross-attention map for localization results in the inability to activate certain regions of the target. While using only representative features as input to a denoising diffusion model to obtain a cross-attention map for localization can activate the entire target region, it is affected by background noise. In this invention, by combining discriminative and representative features, the combined features activate the entire target region.
[0071] Preferably, the combined feature f c Represented as:
[0072]
[0073] Where w represents the weight of the categorical discriminative feature.
[0074] Furthermore, the value of the weight will have a limited impact on the positioning accuracy, such as... Figure 2 As shown, the preferred value of weight w is [0.6, 0.9], within which the positioning accuracy and the correct positioning rate are both relatively high.
[0075] Localization accuracy and correct localization rate are commonly used metrics for evaluating localization performance. Localization accuracy is defined as the image label classification prediction being correct and having an IoU greater than 50% with the true label. Correct localization (CorLoc) rate represents the localization performance for a given category label of each test image.
[0076] After using the combined features as input to the denoising diffusion model as language cues, the denoising diffusion model outputs image features F with different noise levels. l,t Combined features f c Image features F with different noise levels l,t A cross-attention graph m is formed between them l,t , is represented as:
[0077] m l,t =CrossAttn(F l,t ,f c ),l∈L,t∈T
[0078] Where l represents the cross-attention layer index in the probabilistic diffusion model, L represents the set of cross-attention layer indices in the probabilistic diffusion model, t represents the input time step of the probabilistic diffusion model, T represents the set of input time steps of the probabilistic diffusion model, usually T = 1, 2, 3, ..., 1000, and CrossAttn represents the attention mechanism that mixes two different sequences. CrossAttn is a commonly used algorithm in neural networks, which will not be described in detail in this invention.
[0079] In a preferred embodiment, the fusion of the multiple cross-attention maps is achieved through normalization and averaging, as follows:
[0080]
[0081] Where M represents the fused activation graph.
[0082] Furthermore, for weakly supervised point localization tasks, the point with the largest activation value in the activation results is selected as the point localization result; for weakly supervised bounding box localization results, an appropriate threshold is selected to obtain the minimum bounding box of the activated region, which is the bounding box localization result.
[0083] In this invention, a cross-attention map is obtained based on a diffusion model for localization. The diffusion model integrates target localization results from different time steps. The diffusion model at low time steps tends to locally activate and suppress background noise, while the diffusion model at high time steps tends to overactivate and activate background noise. Integrating the results of diffusion models at different time steps is beneficial for more accurate target localization.
[0084] Example
[0085] Example 1
[0086] Using the CUB-200-2011 dataset, we validated the localization accuracy and correct localization rate performance of the localization method.
[0087] The CUB-200-2011 dataset is a publicly available dataset commonly used for evaluating localization methods. It contains 11,788 images of 200 bird species, with 5,994 images used for training and 5,794 for testing. Based on biological taxonomy, the 200 bird species are divided into three levels, including 122 genera, 37 families, and 11 orders.
[0088] The localization method employs a weakly supervised localization approach based on cue learning, comprising the following steps:
[0089] S1. Encode the image description containing image categories to obtain the category discriminative features corresponding to different categories of images;
[0090] S2. For any image category, learnable features are obtained by initializing with category-discriminative features. The noisy image of that category and the learnable features are then input into a denoising diffusion model. With the goal of removing noise from the image, the learnable features are optimized through backpropagation to obtain category-representative features.
[0091] The denoising diffusion model mentioned above is a pre-trained model, set to a non-learnable state;
[0092] S3. Set the denoising diffusion model to a learnable state, input the discriminative features and representational features into the denoising diffusion model, and optimize the denoising diffusion model through backpropagation with the goal of removing noise from the image.
[0093] S4. Set up a neural network classifier to classify the image to be located and obtain the image category;
[0094] S5. Obtain the corresponding class discriminative features and class representation features according to the image category. Fuse the class discriminative features and class representation features to obtain combined features. Input the combined features into the optimized denoising diffusion model to obtain multiple cross-attention maps. Fuse the multiple cross-attention maps into an activation map. Scale the activation map to the size of the original image to determine its activation region on the original image. The minimum bounding box of the activation region is the localization result of the target in the image.
[0095] In S1, the image description is a string combination of a fixed template and an image category. The fixed template "aphoto of a" is combined with the category string, and the last word block of the first string is selected to represent the category.
[0096] The encoding uses the CLIP model, and the denoising diffusion model uses the stable diffusion model.
[0097] In S2, learnable features are obtained by initializing with class discriminative features.
[0098] In S2, the optimization objective is set as follows:
[0099]
[0100] In S3, the optimization objective is set as follows:
[0101]
[0102] In S4, the neural network classifier used is EfficientNet-B7.
[0103] In S5, the category discriminative features and category representation features are fused using a linear weighting method to obtain a combined feature, f. c Represented as:
[0104]
[0105] The cross-attention map m l,t Represented as:
[0106] m l,t =CrossAttn(F l,t ,f c ),l∈L,t∈T
[0107] The fusion of multiple cross-attention maps is achieved through normalization and averaging, as follows:
[0108]
[0109] Example 2
[0110] The same experiments as in Example 1 were conducted, except that the large-scale ILSVRC2016 dataset was used, in which 1.2 million images and 1,000 classes were used for training, and 5,000 images were used for testing in the validation set.
[0111] Comparative Example 1
[0112] The same experiments as in Example 1 were conducted, except that existing weakly supervised target localization methods such as CAM (Class Activation Map), TS-CAM, LCTR, CREAM, BAS, PSOL, and C2AM were used.
[0113] CAM was proposed in the paper "B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning deep features for discriminative localization. In IEEE CVPR, pages 2921–2929, 2016."
[0114] TS-CAM was proposed in the document "Wei Gao, Fang Wan, Xingjia Pan, Zhiliang Peng, Qi Tian, Zhenjun Han, Bolei Zhou, and Qixiang Ye. TS-CAM: token semantic coupled attentionmap for weakly supervised object localization. In IEEE ICCV, pages 2866–2875, 2021";
[0115] LCTR was proposed in the paper "Zhiwei Chen, Changan Wang, Yabiao Wang, Guannan Jiang, Yunhang Shen, Ying Tai, Chenjie Wang, Wei Zhang, and Liujuan Cao. LCTR: on awakening the local continuity of transformer for weakly supervised object localization. In AAAI, pages 410-418, 2022."
[0116] CREAM was proposed in the paper "Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng, Rui-Wei Zhao, Tao Zhang, Xuequan Lu, and Shang Gao. CREAM: weakly supervised object localization via class re-activation mapping. In IEEE CVPR, pages 9427–9436, 2022."
[0117] BAS was proposed in the paper "Pingyu Wu, Wei Zhai, and Yang Cao. Background activation suppression for weakly supervised object localization. In IEEE CVPR, pages 14228–14237, 2022."
[0118] PSOL was proposed in the paper "Chen-Lin Zhang, Yun-Hao Cao, and Jianxin Wu. Rethinking the route towards weakly supervised object localization. In IEEE CVPR, pages 13457–13466, 2020."
[0119] C2 AM was proposed in the document "Jinheng Xie, Jianfeng Xiang, Junliang Chen, Xianxu Hou, Xiaodong Zhao, and Linlin Shen. C2 AM: contrastive learn-ing of class-agnosticactivation map for weakly supervised object localization and semanticsegmentation. In IEEE CVPR, pages 979–988, 2022."
[0120] Comparative Example 2
[0121] The same experiments as Comparative Example 1 were conducted, except that the same ILSVRC2016 dataset as Example 2 was used.
[0122] Experimental Example 1
[0123] The results of comparing Example 1 and Comparative Example 1 are shown in Table 1.
[0124] Table 1
[0125]
[0126] As shown in Table 1, on the CUB-200-2011 dataset, the method in Example 1 improves the Top-1 / Top-5 localization accuracy by 5.2% / 5.0% compared to the previous state-of-the-art method C2 AM. If classification performance is disregarded, the method in Example 1 improves the correct localization rate by 2.9% compared to the previous state-of-the-art method BAS.
[0127] The results of comparing Example 2 and Comparative Example 2 are shown in Table 2.
[0128] Table 2
[0129]
[0130]
[0131] As shown in Table 2, on the ILSVRC2016 dataset, the method in Example 2 improves the Top-1 / Top-5 localization accuracy by 5.6% / 6.0% compared to the previous state-of-the-art method C2 AM. If classification performance is disregarded, the method in Example 2 improves the localization accuracy by 3.2% compared to the previous state-of-the-art method BAS.
[0132] As can be seen from Tables 1 and 2, regardless of the dataset, the methods in the examples all have better localization performance than traditional methods.
[0133] Experiment Example 2
[0134] Multiple experiments were conducted, and the differences compared to Example 2 are as follows:
[0135] In the first iteration, steps S2 and S3 are not performed, meaning that category representation features are not obtained and the denoising diffusion model is not optimized; and in step S5, the category discriminative features f are... d Alternative combined features are input into the unoptimized denoising diffusion model;
[0136] The second time, step S3 is skipped; and in step S2, the learnable features are initially set to empty, meaning that during the process of obtaining the category representation features, the class discriminative features are not used to initialize the learnable features; in step S5, the class representation features f obtained in step S2 are used. r Alternative combined features are input into the unoptimized denoising diffusion model;
[0137] In the third instance, step S3 is skipped; in step S5, the categorical representation feature f obtained in step S2 is used. r Alternative combined features are input into the unoptimized denoising diffusion model;
[0138] In the fourth instance, step S3 is skipped; and in step S2, the learnable features are initially set to empty, meaning that during the process of obtaining category representation features, category discriminative features are not used to initialize the learnable features; in step S5, the combined feature f is... c Input into an unoptimized denoising diffusion model;
[0139] In the fifth instance, step S3 is skipped; and in step S5, the combined feature f is... c Input into an unoptimized denoising diffusion model;
[0140] In the sixth instance, step S2 is skipped, and the class discriminative feature f is removed in step S5. d The alternative combined features are input into the optimized denoising diffusion model;
[0141] In the 7th step, S5, the categorical representation features f obtained in step S2 are used. r The alternative combined features are input into the optimized denoising diffusion model;
[0142] The results of multiple experiments were compared with the positioning results of Example 2, as shown in Table 3.
[0143] Table 3
[0144]
[0145]
[0146] The comparison between the first and fifth experiments shows that using combined features to obtain the localization results can achieve a 3.3% improvement in Top-1 and Top-5 localization accuracy and a 3.8% improvement in correct localization accuracy compared to directly using category discriminative features.
[0147] The comparison between the second and third experiments, as well as the fourth and fifth experiments, shows that if discriminative features are not used to initialize the representational features, both the localization accuracy and the correct localization rate are significantly reduced, which proves the importance of initialization.
[0148] The comparisons between experiments 1 and 6, experiments 3 and 7, and experiment 5 with Example 2 show that optimizing the denoising diffusion model in step S3 can significantly improve the localization effect.
[0149] The comparison of experiments 1, 3, and 5 shows that the localization effect of using combined features is better than that of using categorical representation features, and the localization effect of using categorical representation features is better than that of using categorical discriminative features.
[0150] Figure 3 The first row shows the results of experiments 1, 3, and 5 for images of the category "spaniel". When only class discriminative features are used as input to the denoising diffusion model to obtain the cross-attention map for localization, some regions of the target cannot be activated. When only class representational features are used as input to the denoising diffusion model to obtain the cross-attention map for localization, although the overall region of the target can be activated, it is affected by background noise. By combining class discriminative features and class representational features, the combined features activate the overall region of the target and reduce the impact of noise.
[0151] like Figure 3 The second row shows the effect of using different categories of language cues for localization. Categories similar to "spaniel," such as "dog" and "husky," can correctly locate foreground targets, indicating that the localization effect is robust to the classifier's classification results. Categories less associated with "spaniel," such as "man" and "tower," introduce too much background noise into the localization process, leading to localization failure.
[0152] like Figure 3 As shown in the third and fourth rows, for test images containing multiple categories, the target can be accurately located if the correct category cue is given. This indicates that the cue generation model can not only generatively locate the target, but also discriminate the target.
[0153] In the description of this invention, it should be noted that the terms "upper," "lower," "inner," "outer," "front," and "rear," etc., indicate the orientation or positional relationship based on the orientation or positional relationship in the working state of this invention. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention. In addition, the terms "first," "second," "third," and "fourth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.
[0154] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.
[0155] The present invention has been described above with reference to preferred embodiments; however, these embodiments are merely exemplary and illustrative. Various substitutions and modifications can be made to the present invention based on these embodiments, all of which fall within the scope of protection of the present invention.< / category> < / category>
Claims
1. A weakly supervised localization method based on prompt learning, characterized in that, The method comprises the following steps: S1, encoding an image description comprising an image category to obtain category discriminative features corresponding to different categories of images; S2, for any image category, using the category discriminative features to initialize to obtain learnable features, inputting the noisy images of the category and the learnable features into a denoising diffusion model, and optimizing the learnable features by back propagation to remove noise in the images to obtain category representative features, Wherein the denoising diffusion model is a pre-trained model, which is set to a non-learnable state; S3, set the denoising diffusion model to a learnable state, input the discriminative features and the representative features into the denoising diffusion model, and optimize the denoising diffusion model by back propagation to remove noise in the images; S4, set a neural network classifier to classify the image to be positioned to obtain the category of the image; S5, according to the category of the image, obtain the corresponding category discriminative features and category representative features, fuse the category discriminative features and category representative features to obtain combined features, input the combined features into the optimized denoising diffusion model, obtain multiple cross attention maps, fuse the multiple cross attention maps into an activation map, and scale the activation map to the original image size to determine the activation area on the original image. The minimum bounding box of the activation area is the positioning result of the target in the image.
2. The weakly supervised positioning method based on prompt learning according to claim 1, wherein In S1, the image description is a combination of a fixed template and a string of image categories.
3. The weakly supervised positioning method based on prompt learning according to claim 1, wherein The encoding is performed by a visual language model.
4. The weakly supervised positioning method based on prompt learning according to claim 1, wherein The denoising diffusion model uses a stable diffusion model.
5. The weakly supervised positioning method based on prompt learning according to claim 1, wherein In S2, the optimization target is set as: wherein, represents a class representative feature reaching an optimal denoising effect for a given noise image set, L(θ, f r ) represents a loss function of a denoising diffusion model, θ represents a network parameter of the denoising diffusion model, f r represents a class representative feature, is a minimum function, and represents a value of f r when L(θ, f r ) takes a minimum value.
6. The weakly supervised positioning method based on prompt learning according to claim 1, wherein In S3, the optimization target is set as: where θ * denotes the network parameters of the denoising diffusion model that achieves the optimal denoising effect for a given set of noisy images.
7. The weakly supervised positioning method based on prompt learning according to claim 1, wherein In S5, the category discriminative features and the category representative features are fused by linear weighting to obtain the combined features.
8. The weakly supervised positioning method based on prompt learning according to claim 1, wherein The cross-attention map m l,t is represented as: m l,t = CrossAttn(F l,t ,f c ), l e L, t e T Wherein, l represents the cross attention layer index in the denoising diffusion model, L represents the cross attention layer index set in the denoising diffusion model, t represents the input time step of the denoising diffusion model, T represents the input time step set of the denoising diffusion model, CrossAttn represents the attention mechanism of mixing two different sequences, 9. The weakly supervised positioning method based on prompt learning according to claim 1, wherein The fusion of the multiple cross attention maps is realized by normalization and averaging, which is represented as: Wherein, M represents the fused activation map.