Zero sample image classification method based on double contrast prompt learning
By adopting the dual contrast prompt learning method in zero-sample image classification, through visual and text comparison learning, the problem of insufficient overfitting and generalization capabilities in the existing technology is solved, and the model's distinction and generalization capabilities are significantly improved.
Patent Information
- Application Number
- CN202411967774.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-04-25
AI Technical Summary
The prior art has problems with insufficient overfitting and generalization capabilities in zero-sample image classification, especially when facing complex or visually similar images, it is difficult to distinguish nuances of different categories.
A zero-sample image classification method based on dual contrast prompt learning is proposed. Through visual contrast prompt learning and text comparison prompt learning, the model's distinction ability is enhanced. The specific steps include stitching the learnable vectors and class names to build a learnable propt, performing data augmentation, comparative learning in text and visual space, training to obtain a propt, and using this propt to perform zero-sample image classification.
Through the dual comparison tip learning strategy, the performance of zero-sample image classification is significantly improved, and the model's distinction ability and generalization ability are enhanced. Especially when facing complex or visually similar images, it can better distinguish different categories.
Smart Images
Figure CN119888346A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of zero-shot image classification, and in particular relates to a zero-shot image classification method based on double contrast cue learning. Background Art
[0002] The development of visual-language models such as CLIP has attracted widespread attention in the field of artificial intelligence because they have demonstrated unprecedented capabilities in understanding and processing image and text data. These models learn to embed visual content and corresponding text descriptions into a common semantic space by pre-training on a large number of image-text pairs. CLIP (Contrastive Language-Image Pre-training) is particularly noteworthy because it is able to achieve impressive results in a variety of downstream tasks through zero-shot learning without any task-specific training.
[0003] During the pre-training phase of CLIP, the model learns to encode images and text by maximizing the similarity of matched image-text pairs and minimizing the similarity of mismatched ones. This approach enables CLIP to capture the complex relationship between images and text and preserve semantic similarity in a shared embedding space. This capability not only enables CLIP to generalize to new visual concepts, but also gives it strong zero-shot reasoning capabilities, that is, it can recognize and classify a category without seeing samples of a specific category. In actual downstream tasks, CLIP usually uses manually designed prompts, such as "a photo of a{CLASS}", to generate predictions. These prompts help the model align the output text embedding with the image embedding, thereby achieving accurate classification of the image content. However, this approach that relies on fixed manual prompts has limitations because it cannot fully take into account the specific knowledge of each task and may cause the model to overfit on a specific task, limiting its generalization ability on new or unseen categories.
[0004] To overcome this challenge, existing studies have proposed Context Optimization (CoOp) as a novel prompt adjustment method. CoOp replaces traditional hand-crafted prompts by introducing trainable vectors, allowing the model to dynamically adjust prompts based on specific task contexts. This approach has achieved significant performance improvements over the basic categories because it allows the model to learn more refined prompts that better adapt to the needs of specific tasks. Knowledge-guided Context Optimization (KgCoOp) alleviates the problem of forgetting basic knowledge by minimizing the difference between the text embeddings generated by learned prompts and those generated by hand-crafted prompts.
[0005] Nevertheless, similar methods such as CoOp and KgCoOp mainly focus on strengthening the positive associations between cues and images, that is, those text cues that semantically match the image content. These methods often ignore the importance of negative associations, that is, those text cues that semantically do not match the image content. This neglect may cause the model to have difficulty distinguishing the subtle differences between different categories when faced with complex or visually similar images, thus affecting the model's discriminative ability. Summary of the invention
[0006] In order to solve the deficiencies in the prior art, this application proposes a zero-shot image classification method based on dual contrast cue learning. The performance of zero-shot image classification can be enhanced through a dual contrast cue learning strategy, namely visual contrast cue learning and text contrast cue learning.
[0007] The present invention achieves the above technical objectives through the following technical means.
[0008] A zero-shot image classification method based on dual contrast cue learning, comprising:
[0009] Step 1: Concatenate the learnable vector and the category name to construct a learnable prompt, and send it to the text encoder of the Contrastive Language-Image Pre-training (CLIP) model to obtain text features;
[0010] Step 2: Perform data enhancement on the image data and send it to the image encoder of CLIP to obtain image features;
[0011] Step 3: Compare and learn the obtained text features and image features in the text space and visual space respectively, and train to obtain prompts;
[0012] Step 4: Use the trained prompt to perform zero-shot image classification to improve classification performance.
[0013] In the first aspect, the specific steps of the above step 1 are as follows:
[0014] Step 1.1, concatenate the vector and the category to construct a learnable prompt. Each category has a corresponding prompt, denoted as t i =[v] 1 [v] 2 ...[v] M {CLASS};
[0015] Step 1.2, send prompt to the text encoder of CLIP, represented as g(t i ), and get the text embedding T = {g(t i )|i=1,2,…,C}, where C is the number of categories contained in the dataset.
[0016] In the second aspect, the specific steps of the above step 2 are as follows:
[0017] Step 2.1, divide the data set into base class and new class, use only base class data in training, and perform data augmentation operations such as random rotation, flipping and cropping on the base class image data to expand the data set size;
[0018] Step 2.2: Send the training data to the image encoder f(·) in batches to obtain the image feature set X = {x i |i=1,2,…,N}; where N is the training batch size.
[0019] In the third aspect, the specific steps of the above step 3 are as follows:
[0020] Step 3.1, in the text contrast cue learning, put x i As an anchor point, g(t i ) as the text positive sample T p , take the text features other than this as text negative samples T n , respectively calculate the cosine similarity between the positive and negative samples and the anchor point, and use the triplet loss form to pull the negative samples away while pulling the positive samples closer;
[0021] Step 3.2, in visual contrast cue learning, g(t i ) as the anchor point, and x i As the positive sample X p , take the image features other than these as visual negative samples X n , respectively calculate the cosine similarity between the positive and negative samples and the anchor point, and use the triplet loss form to pull the negative samples away while pulling the positive samples closer;
[0022] Step 3.3 combines the two contrast prompts to learn and obtain the final loss function, thereby training the prompt.
[0023] In the fourth aspect, the specific steps of the above step 4 are as follows:
[0024] The final trained prompt is used to replace the original manual prompt of CLIP, and then zero-shot image classification is performed directly on the new class.
[0025] Compared with the prior art, the present invention has the following beneficial effects:
[0026] 1. We propose a Dual Contrastive Prompt Learning (DCPL) framework, which enhances the discrimination ability of various models by optimizing the association between positive and negative samples.
[0027] 2. Our proposed dual-contrast cue learning strategy can be adapted to existing methods as a plug-in module, showing strong flexibility. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 Flowchart of the zero-shot image classification method based on dual contrast cue learning.
[0029] Figure 2 A detailed description of the double contrast cue learning strategy.
[0030] Figure 3 A general overview of the double contrast cue learning framework.
[0031] Figure 4 This paper compares the performance of Dual Contrastive Cue Learning (DCPL) with existing methods context-optimized CoOp and class-specific knowledge context-optimized KgCoOp on 11 datasets.
[0032] Figure 5 We compare the performance of Dual Contrastive Cue Learning (DCPL) with existing methods Context-Optimized CoOp and Class-Specific Knowledge Context-Optimized KgCoOp in a cross-dataset setting. DETAILED DESCRIPTION
[0033] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0034] The present invention aims to propose a zero-shot image classification method based on double contrast cue learning for zero-shot image classification, so as to effectively improve the performance of zero-shot image classification. The invention provides a complete double contrast cue learning framework and conducts sufficient experiments to prove the feasibility and effectiveness of the method.
[0035] like Figure 1 As shown, a zero-sample image classification method based on double contrast cue learning of the present invention comprises:
[0036] Step 201 concatenates the learnable vector and the category name to construct a learnable prompt, and sends it to the text encoder of CLIP to obtain text features.
[0037] In the implementation case of the present invention, the learnable vector and the category name are concatenated to construct a learnable prompt. The purpose is that CILP originally used a hand-crafted prompt, which cannot fully consider the specific knowledge of each task and has certain limitations. It cannot fully consider the specific knowledge of each task and may cause the model to overfit on a specific task, limiting its generalization ability on new or unseen categories. Therefore, we construct a learnable prompt and learn the prompt through a double comparison method.
[0038] Step 2011 uses M learnable vectors to concatenate the corresponding category names to construct a learnable prompt represented by t i =[v] 1 [v] 2 ...[v] M {CLASS}, where t i is the i-th learnable prompt, v is the learnable vector, M is the number of learnable vectors, different numbers of vectors have a certain impact on the final classification result, {CLASS} is the category name; each category corresponds to a prompt.
[0039] Step 2012 sends prompt to the text encoder of CLIP, which is represented by g(t i ), and get the text embedding T = {g(t i )|i=1,2,…,C}, where C is the number of categories contained in the dataset.
[0040] Step 202 performs data enhancement on the image data and sends it to the image encoder of CLIP to obtain image features.
[0041] Step 2021 uses random flipping, rotation, etc. to expand the data set and divide it into base classes and new classes. Only base class data is used during training.
[0042] Step 2022 inputs the training data into the image encoder f(·) in batches to obtain the image feature set X={x i |i=1,2,…,N}, where N is the training batch size.
[0043] In step 203, the obtained text features and image features are compared and learned in text and vision respectively, so as to train and obtain prompt.
[0044] Step 2031 In the text contrast prompt learning, put x i As an anchor point, g(t i ) as the text positive sample T p , take the text features other than this as text negative samples T n , respectively calculate the cosine similarity between the positive and negative samples and the anchor point, and use the triplet loss form to pull the negative samples away while pulling the positive samples closer.
[0045] The process of learning text contrast prompts includes:
[0046] (1) For a given image feature x of the i-th image i , the text contrast cue is learned using this image as an anchor, expressed as:
[0047] Anchor t =x i
[0048] According to x i Category, Anchor t The text comparison positive sample can be defined as T p =g(t i ), the text contrast negative sample can be defined as T n ={g(t j )|t j ≠t i}.
[0049] (2) Then calculate the similarity between the anchor point and the positive and negative samples of the text comparison, expressed as Pos t and Neg t :
[0050] Pos t =cos(Anchor t ,T p )
[0051] Neg t ={cos(Anchor t ,g(t j ))∣g(t j )∈T n}
[0052] Where cos is the cosine similarity.
[0053] (3) Finally, we get the loss function for text prompt learning
[0054]
[0055] Margin is an adjustable hyperparameter.
[0056] Step 2032 In visual contrast cue learning, g(t i ) as the anchor point, and x i As the positive sample X p , take the image features other than these as visual negative samples X n , respectively calculate the cosine similarity between the positive and negative samples and the anchor point, and use the triplet loss form to pull the negative samples away while pulling the positive samples closer.
[0057] The process of visual contrast cue learning includes:
[0058] (1) For a given image feature x of the i-th image i , visual contrast cues are learned using their textual cues embedding g(t i ) as the anchor point, expressed as:
[0059] Anchor v =g(t i )
[0060] According to g(t i ) category, Anchor v The visual contrast positive sample can be defined as X p =x i , the visual comparison negative samples are composed of all the samples in the batch that are the same as x i The image features with different textual hints are composed, defined as X n ={x j ∣t j ≠t i}.
[0061] (2) Then calculate the similarity between the anchor point and the positive and negative samples of the visual comparison, expressed as Pos v and Neg v :
[0062] Pos v =cos(Anchor v ,X p )
[0063] Neg v ={cos(Anchorv ,x j )|x j ∈X n}
[0064] Where cos is the cosine similarity.
[0065] (3) Finally, we get the loss function for text prompt learning
[0066]
[0067] Margin is an adjustable hyperparameter.
[0068] Step 2033: Combine the original loss We can get the final loss function
[0069]
[0070] Among them, α and β are weights, which control the proportion of learning of the two contrasting cues.
[0071] Step 204 uses the trained prompt to replace the original manual prompt, and the predicted probability is given by the following formula:
[0072]
[0073] Where τ is the temperature parameter.
[0074] Figure 2 The specific details of the double contrast cue learning strategy are presented, using visual anchors (Anchor v ) and text anchors (Anchor t ) to promote cue learning. Positive associations are represented by dark double-headed arrows and are strengthened, while negative associations are represented by light double-headed arrows and are suppressed. This approach ensures the robustness of the learning process by aligning the representations of semantically similar instances and distinguishing those that are semantically dissimilar. Figure 3 An overview of the double contrast cue learning framework, showing the complete training process.
[0075] In order to verify the effectiveness of the invented zero-shot image classification method based on dual contrast cue learning, the invention is compared with the baseline methods CoOp and KgCoOp, and rigorous evaluation is carried out on 11 different datasets, including ImageNet, SUN397 (scene classification dataset), FGVC Aircraft (fine-grained aircraft dataset), EuroSAT (European satellite image dataset), StanfordCars (Stanford car dataset), Food101 (food 101 dataset), OxfordPets (Oxford pet dataset), Flowers (flower dataset), Caltech101 (California Institute of Technology 101 dataset), DTD (texture description dataset) and UCF101 (action recognition dataset). Figure 4 As shown in the figure, on average, after integrating the method of the present invention, the performance of CoOp on both the base and new categories has improved. For KgCoOp, the performance of the base category has slightly decreased, but the performance of the new category has improved. Overall, through the harmonic mean (H), the method of the present invention can enhance the overall performance of the baseline method.
[0076] Figure 5 The performance under the cross-dataset migration setting is demonstrated. The present invention explores the effect of transferring the knowledge learned from ImageNet to ten other datasets. Specifically, we use ImageNet as the source dataset (Source) and then evaluate it on the remaining ten target datasets (Target). On average, after adopting the method of the present invention, the cross-dataset migration ability of the two baseline models is enhanced.
Claims
1. A zero-shot image classification method based on dual contrast cue learning, characterized in that: The steps include: Step 1: Concatenate the learnable vector and the category name to construct a learnable prompt, and feed it into the text encoder of the comparative language-image pre-training model CLIP to obtain text features; Step 2, enhance the image data and send it to the CLIP image encoder to obtain image features; Step 3: Compare and learn the obtained text features and image features in the text space and visual space respectively, and train to obtain prompts; Step 4: Use the trained prompt to perform zero-shot image classification to improve classification performance.
2. The method according to claim 1, characterized in that: The specific implementation of step 1 includes the following steps: Step 1.1: Use limited labeled data to train the context vector, concatenate the vector and the category to construct a learnable prompt. Each category has a corresponding prompt, denoted as t i =[v]1[v]2...[v] M {CLASS}, where t i is the i-th learnable prompt, v is the learnable vector, M is the number of learnable vectors, and {CLASS} is the class name; Step 1.2, send prompt to the text encoder of CLIP, represented as g(t i ), and get the text embedding T = {g(t i )|i=1,2,…,C}, where C is the number of categories contained in the dataset.
3. The method according to claim 1, characterized in that: The specific implementation of step 2 includes the following steps: Step 2.1, divide the data set into base class and new class, use only base class data in training, and perform data augmentation operations such as random rotation, flipping and cropping on the base class image data to expand the data set size; Step 2.2: Send the training data to the image encoder f(t i ), we get the image feature X = {x i |i=1,2,…,N}, where N is the training batch size.
4. The method according to claim 1, characterized in that: The specific implementation of step 3 includes the following steps: Step 3.1, in the text contrast cue learning, put x i As an anchor point, g(t i ) as the text positive sample T p , take the text features other than this as text negative samples T n , respectively calculate the cosine similarity between the positive and negative samples and the anchor point, and use the triplet loss form to pull the negative samples away while pulling the positive samples closer; Step 3.2, in visual contrast cue learning, g(t i ) as the anchor point, and x i As the positive image sample X p , take the image features other than these as visual negative samples X n , respectively calculate the cosine similarity between the positive and negative samples and the anchor point, and use the triplet loss form to pull the negative samples away while pulling the positive samples closer; Step 3.3 combines the two contrast prompts to learn and obtain the final loss function, thereby training the prompt.
5. The method according to claim 1, characterized in that: In step 4: Using the final trained prompt, we can directly perform zero-shot image classification on the new class.