Zero sample CLIP image classification method based on attribute guidance
Through the attribute-guided method, the CLIP model is used to predict pseudo-labels and select superclass attribute sets, which improves the classification accuracy of similar objects, solves the problem of low accuracy of CLIP model in distinguishing similar species, and realizes the interpretability and migration of zero-sample classification.
Patent Information
- Application Number
- CN202510711321.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-29
- Publication Date
- 2025-08-15
AI Technical Summary
The existing CLIP image classification model has low accuracy when distinguishing similar objects, and requires additional data or training fine-tuning to migrate to other visual-language models, making it difficult to meet the needs of zero-sample classification.
Through an attribute-guided method, the visual-language model CLIP is used to predict pseudo-labels and select superclass attribute sets, and combine cosine similarity calculations of images and texts to realize image classification.
It improves the accuracy of the model's classification of similar objects, simplifies the migration process, and does not require additional data training, and model predictions are more interpretable.
Smart Images
Figure CN120495784A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image classification methods, and in particular relates to an attribute-guided zero-sample CLIP image classification method. Background Art
[0002] Image classification, as a core task in computer vision, has undergone a technological evolution from traditional manual feature extraction to deep learning-driven approaches. Early supervised learning methods relied on large-scale labeled data for model training, making it difficult to cope with the dynamic expansion of categories and the scarcity of labeled data. With the development of multimodal learning, visual-language models such as CLIP have achieved semantic alignment between images and text through contrastive learning, opening up a new paradigm for zero-shot classification. However, visual-language models such as CLIP suffer from low classification accuracy for objects of similar types when performing zero-shot image classification. For example, they are not very effective at distinguishing between different subcategories of airplanes, cats, and remote sensing images. Existing improvement technologies either require a small amount of sample data for model training and fine-tuning, or cannot be easily transferred to other visual-language models. Summary of the Invention
[0003] The purpose of the present invention is to provide an attribute-guided zero-shot CLIP image classification method to solve the problems raised in the above background technology.
[0004] To achieve the above object, the present invention provides the following technical solution: an attribute-guided zero-sample CLIP image classification method, comprising the following steps: S1: For the input image and labels to be classified , first fill in the category labels into the pre-made prompt template , get a set of text prompts ; S2: Image and each prompt filled with category labels The features are extracted and encoded by the image encoder and text encoder respectively to obtain the image representation. and a set of text representations ; S3: In the same feature space, vectors with similar directions will have higher cosine similarity. and The cosine similarity of the image is predicted A pseudo-label for ; S4: Select the probability Highest label as pseudo-labels; S5: The pseudo label predicted in the previous step After encoding by the text encoder, the text representation of the pseudo label is obtained and a textual representation of a set of attribute set names , respectively calculated and The cosine similarity of S6: Select the probability Highest superclass : S7: Select the corresponding attribute set based on the mapping relationship between the superclass and the attribute set ; S8: Set the attribute set selected in the previous stage Attribute filling prompt template in Among them Replace the attributes, is a pseudo label, and we get a set of prompts ; S9: Send each prompt to the text encoder to get the text representation after encoding and represented with the image Calculate cosine similarity; S10: Select so that the probability The highest [ATTR] is the attribute a of the image i : S11: Perform classification prediction.
[0005] Preferably, the image encoder includes ViT-B / 32, and the text encoder includes a 12-layer Transformer.
[0006] Preferably, the pseudo-label is The probability calculation formula is: ; in, Indicates that it contains classification labels The text representation of .
[0007] Preferably, the superclass The calculation formula is: .
[0008] Preferably, in S5, the classification probability is calculated as: .
[0009] Preferably, in S9, the predicted probability calculation formula is: .
[0010] Preferably, the calculation formula of the attribute ai is: .
[0011] Preferably, the S10 specifically includes the steps of: S111: Attributes to be obtained Embed prompt template , get the classification hint containing attributes Where [CLS] will be replaced by the classification label; S112: Send each prompt text to the text encoder for encoding to obtain text representation , and the image representation Calculate cosine similarity; S113: Select The largest c is used as the final classification prediction .
[0012] Preferably, in S112, the prediction probability calculation formula is: .
[0013] Preferably, in said S113, said classification prediction The calculation formula is: .
[0014] Compared with the prior art, the present invention has the following beneficial effects: The present invention significantly improves the accuracy of the model in distinguishing similar objects, and also improves the accuracy of dissimilar objects to a certain extent. The model migration effect is good, the accuracy is improved, and no additional data or external training is required. It can be "plug and play" and can extract attributes in the image, making the prediction more explainable. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 Schematic diagram of the structure of the visual encoder of the present invention; Figure 2 Schematic diagram of the CLIP model structure of the present invention; Figure 3 A schematic diagram of the attribute set construction of the present invention; Figure 4 Schematic diagram of pseudo label prediction of the present invention; Figure 5 It is a schematic diagram of an algorithm for a specific field of the present invention. DETAILED DESCRIPTION
[0016] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0017] See also Figure 1-Figure 5 As shown, the present invention provides the following technical solutions: This method is based on the visual-language model CLIP and can be migrated to other visual-language models. The visual-language model aligns the representations of images and text in the same embedding space by jointly training a large number of images and text descriptions. During the training process, the image is encoded into a feature vector (image representation), and the text (usually a category label or description) is also encoded into a corresponding feature vector (text representation). By learning the alignment relationship between the image and the text, the model can obtain the semantic information of the category from the text. During classification prediction, after a new image is input into the model, the model converts the image into a feature vector (image representation) through an image encoder (such as CNN, Transformer, etc.). The model then fills all categories into pre-set prompts, and converts the generated text description into a text feature vector (text representation), which is then calculated for similarity with the image representation (usually using cosine similarity), and the best matching category is selected as the prediction result. For example, when a picture of "dog" is input, the model fills the classification label into the pre-set prompt Among them Will be replaced by each category label to get a text description Etc. Next, the image and text description are encoded through the image encoder and text encoder respectively to obtain the feature vectors of the image and text (representation of the image and text), calculate the cosine similarity between them, and select the text label with the largest similarity as the final classification prediction result.
[0018] In this invention, regarding the principle of visual encoder, the Vision Transformer used in this method is taken as an example: Step 1: Divide the image into patches First, the input image is divided into multiple patches. The size of each patch is usually P×P pixels (the specific size can be selected according to the design). Assuming the size of the image is H×W, the image will be divided into A small piece.
[0019] Step 2: Flatten each image block Each small block (e.g., a P×P pixel image block) is flattened into a one-dimensional vector. For example, if each small block is an RGB image, its size is P×P×C (where C is the number of channels), then each small block becomes a one-dimensional vector after flattening. In this way, each image block will become a vector of length vector.
[0020] Step 3: Linear projection to high-dimensional space Each flattened block (size The vector of each small block is projected into a fixed high-dimensional space D through a linear layer. Usually D is a large value. The dimensional vector is mapped to a new space so that the representation dimension of each small block becomes D.
[0021] Step 4: Add position code Because the Transformer model inherently lacks sequential information, positional encoding is necessary. Positional encoding provides positional information for each image patch, ensuring the model captures the positional relationships between small patches within the image. The positional encoding is element-wise added to the projection vector of each image patch, ensuring that the representation of each image patch includes its position within the image.
[0022] Step 5: Add [CLS] Token In addition to image patches, ViT also introduces a special [CLS] token. This token does not correspond to any actual image patch, but is an additional marker used by the model to aggregate information about the entire image. This [CLS] token is input to the Transformer encoder at the beginning of the input sequence along with the image patch.
[0023] Step 6: Input to Transformer encoder to obtain global image representation [CLS] All image patches (including the [CLS] token) that have undergone linear projection and positional encoding are passed together as input to the Transformer encoder. Each layer of the Transformer consists of a self-attention mechanism and a feedforward neural network. The self-attention mechanism fuses the features of each image patch with those of other patches, ensuring that each patch not only contains local information but also integrates global information from other patches. After processing through the multi-layer Transformer encoder, the final [CLS] token, through the self-attention mechanism, integrates information from other patches (i.e., the complete image) and is used to represent the entire image. This high-dimensional vector can be used as an image feature (i.e., image representation) for subsequent tasks (such as classification, retrieval, and generation).
[0024] Text encoder principle: Step 1: Text input processing The text encoder first receives input text data, usually a sentence or a group of words. In order for the model to be able to process these texts, some preprocessing is first required: Tokenization: Text input is broken down into individual words or subwords (tokens). For example, the sentence "a cat on the mat" is broken down into the following words or subwords: ["a", "cat", "on", "the", "mat"].
[0025] Word embeddings: Each word or subword is mapped to a high-dimensional word embedding vector, which is usually obtained from pre-trained word vectors (such as GloVe or Word2Vec) or randomly initialized embedding matrices.
[0026] Step 2: Add position code The Transformer model itself lacks positional information, so positional encoding is required to help the model understand the order of words in a sentence. Positional encoding encodes the index of each word and adds it to the word embedding, allowing the model to distinguish the position of different words in a sentence.
[0027] Step 3: Final output After multiple self-attention and feedforward neural network layers, the text encoder outputs a sequence of embedding vectors of the same length as the input. For each input word, it has a corresponding representation vector that captures the word's context. Finally, the embedding vector of the [CLS] token or the last word of the sentence is typically chosen as the representation of the entire text.
[0028] Furthermore, in the present invention, the specific steps for calculating cosine similarity are as follows: For vector and , the cosine similarity is calculated by the following method Step 1: Normalization To ensure consistency when comparing two vectors, CLIP performs L2-norm normalization on both vectors. This means that each vector is normalized to the unit vector, meaning its L2-norm is 1. Normalizing both vectors yields: and ; The purpose of this step is to ensure that when calculating similarity, the comparison of two vectors is based only on their orientation and not on their absolute magnitude.
[0029] Step 2: Calculate cosine similarity Cosine similarity is achieved by calculating the dot product of two vectors and normalizing them. The specific formula is: ; When calculating image representation and text representation, it can be expressed as: ; In the attribute set selection process, the cosine similarity between the pseudo label and the superclass can be expressed as: ; because and The vectors have been L2 normalized, and the cosine similarity is equivalent to their dot product.
[0030] The CLIP model mainly consists of two encoders: an image encoder and a text encoder. During the training process, CLIP adjusts the parameters of these two encoders through a large number of image-text pairs, so that the representations of similar image-text pairs in the feature space are as close as possible. In zero-shot image classification prediction, CLIP feeds the image into the image encoder to obtain the image representation, and feeds the hint filled with the classification label into the text encoder to obtain the text representation. Then, the image representation vector is calculated. and text representation vector 、 and The cosine similarity between them is used to select the text with the highest similarity as the classification result.
[0031] Furthermore, in the present invention, an attribute set is a group of attributes used to describe or characterize objects of the same type. For example, dogs and cats are both animals, and animals may have attributes such as running, jumping, and flying. Tables and sofas are both furniture, and furniture may have attributes such as wooden, smooth, and flat. Attribute sets may also overlap; for example, different attribute sets may share the same color attribute. Specifically, a plant attribute set would not include attributes such as running, flying, or sad. Attribute set selection before prediction is performed to eliminate impossible attributes and narrow the range of attribute selection, thereby reducing algorithm complexity and improving attribute prediction accuracy.
[0032] Dogs and cats are both animals, tables and sofas are both furniture, and objects of similar categories have similar properties. Here we introduce a superclass For example, here dogs and cats are category labels, animals are the superclass of dogs and cats; tables and sofas are category labels, furniture is the superclass of tables and sofas. In this way, a superclass can be made Mapping a property set , here That is, the possible attributes in the attribute set. When selecting the attribute set in the following method, you can determine which superclass the pseudo label belongs to, and then select the corresponding attribute set based on the superclass.
[0033] Automatically construct the attribute set by asking GPT-3, the template is , [cls] is replaced with the guide word of the attribute set you want to build. For example, To construct the attribute set of animals, take at most the first 50 attributes and assign a corresponding attribute set to each superclass. One superclass corresponds to one attribute set, and one attribute set can be assigned to multiple superclasses. For example, using The constructed attribute set can be assigned to the superclasses of "Pet" and "Animal," but "Pet" or "Animal" can only correspond to one attribute set each. Without the concept of superclasses and assigning an attribute set to each superclass, selecting an attribute set for each classification label would be a significant workload. For example, if a dataset has 1,000 categories, without superclasses, a specific attribute set would be required for each category to predict attributes. With the introduction of the superclass concept, assuming there are only 10 superclasses, only the mapping of attribute sets needs to be specified for these 10 superclasses, significantly reducing the workload.
[0034] In addition, in the present invention, a zero-sample CLIP image classification method based on attribute guidance is proposed, which includes the following steps: S1: Pseudo-label prediction When predicting attributes using visual language models such as CLIP, each attribute needs to be filled in the prompt template , among which The label is unknown and cannot be obtained, so a pseudo label must be predicted first to select the appropriate attribute set and construct a prompt for attribute prediction.
[0035] For the input image and labels to be classified , first fill in the category labels into the pre-made prompt template , get a set of text prompts The image and each prompt filled with category labels The image is fed into an image encoder (such as ViT-B / 32) and a text encoder (such as a 12-layer Transformer) to extract features and encode them to obtain an image representation. and a set of text representations In the same feature space, vectors with similar directions will have higher cosine similarity. and The cosine similarity of the image is predicted A pseudo-label for The pseudo label is The probability calculation formula is: ; in Indicates that it contains classification labels The text representation of .
[0036] Choose so that the probability Highest label As pseudo labels ; S2: Attribute set selection (2-1) Similar texts should be more similar in vector space. For example, the vector of "lily" encoded by the text encoder should be closer to the vector of "plant" than to the vector of "animal". Using this feature, the pseudo label predicted in the previous step is , such as "daffodil" and its superclasses, such as animals, plants, materials, etc. After encoding by the text encoder, the pseudo-label text representation is obtained and a textual representation of a set of attribute set names , respectively calculated and The cosine similarity of , the classification probability is calculated as: (2-2) Select the probability Highest superclass : (2-3) Select the corresponding attribute set based on the mapping relationship between the superclass and the attribute set When constructing the property set, for example After the attribute set is constructed, each superclass that applies to the attribute set will be manually assigned a mapping with the attribute set, and a mapping relationship between the superclass and the attribute set will be established. For example, you can specify that the two superclasses "mammals" and "pets" correspond to The constructed attribute set is manually specified, and a json file can be used to store the mapping relationship between superclass and attribute set.
[0037] S3: Attribute Prediction (3-1) The attribute set selected in the previous stage Attribute filling prompt template in Among them Replace the attributes, is a pseudo label, and we get a set of prompts .
[0038] (3-2) Send each prompt to the text encoder to get the text representation and represented with the image Calculate the cosine similarity and the prediction probability calculation formula is: , (3-3) Choose so that the probability The highest [ATTR] is the attribute a of the image i : S4: Classification Prediction (4-1) Attributes to be obtained Embed prompt template , get the classification hint with attributes Where [CLS] will be replaced by the classification label.
[0039] (4-2) Send each prompt text to the text encoder to encode and obtain text representation , and the image representation Calculate the cosine similarity and the prediction probability calculation formula is: (4-3) Select The largest c is used as the final classification prediction : .
[0040] Application of this method in remote sensing image recognition: For image recognition in a specific domain, this method can be simplified, eliminating the step of selecting an attribute set, as only the attribute set for that domain can be provided for attribute selection. For example, in remote sensing image recognition applications, the attribute set can be constructed by using the question "What attributes may terrain have?" and adding some color attributes. The algorithm flow is simplified to: pseudo-label prediction, attribute prediction, and classification prediction. The pseudo-label prediction and classification prediction stages remain unchanged, while attribute prediction is now performed by selecting attributes from a specified attribute set. Experiments on the classic EuroSAT remote sensing image recognition dataset significantly improved the accuracy of the CLIP model from 37.56% to 43.04%.
[0041] In addition, to achieve the above objectives, an embodiment of the present invention further proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the attribute-guided zero-shot CLIP image classification method of an embodiment of the present invention.
[0042] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, embodiments of the present invention may take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable devices (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.
[0043] The embodiments of the present invention are described with reference to flowcharts and / or block diagrams of methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or boxes.
[0044] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a product including an instruction system, which is implemented in the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.
[0045] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0046] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.
Claims
1. Attribute-guided zero-shot CLIP image classification method, characterized by: The steps include: S1: For the input image and labels to be classified , first fill in the category labels into the pre-made prompt template , get a set of text prompts ; S2: Image and each prompt filled with category labels The features are extracted and encoded by the image encoder and text encoder respectively to obtain the image representation. and a set of text representations ; S3: In the same feature space, vectors with similar directions will have higher cosine similarity. and The cosine similarity of the image is predicted A pseudo-label for ; S4: Select the probability Highest label as pseudo-labels; S5: The pseudo label predicted in the previous step After encoding by the text encoder, the text representation of the pseudo label is obtained and a textual representation of a set of attribute set names , respectively calculated and The cosine similarity of S6: Select the probability Highest superclass : S7: Select the corresponding attribute set based on the mapping relationship between the superclass and the attribute set ; S8: Set the attribute set selected in the previous stage Attribute filling prompt template in Among them Replace the attributes, is a pseudo label, and we get a set of prompts ; S9: Send each prompt to the text encoder to get the text representation after encoding and represented with the image Calculate cosine similarity; S10: Select so that the probability The highest [ATTR] is the attribute a of the image i : S11: Perform classification prediction.
2. The attribute-guided zero-shot CLIP image classification method according to claim 1, characterized in that: The image encoder includes ViT-B / 32, and the text encoder includes a 12-layer Transformer.
3. The attribute-guided zero-shot CLIP image classification method according to claim 1, characterized in that: The pseudo label is The probability calculation formula is: ； in, Indicates that it contains classification labels The text representation of .
4. The attribute-guided zero-shot CLIP image classification method according to claim 1, characterized in that: The superclass The calculation formula is: 。 5. The attribute-guided zero-shot CLIP image classification method according to claim 1, characterized in that: In S5, the classification probability is calculated as: 。 6. The attribute-guided zero-shot CLIP image classification method according to claim 1, characterized in that: In S9, the prediction probability calculation formula is: 。 7. The attribute-guided zero-shot CLIP image classification method according to claim 1, characterized in that: The attribute a i The calculation formula is: 。 8. The attribute-guided zero-shot CLIP image classification method according to claim 1, characterized in that: The S10 specifically includes the following steps: S111: Attributes to be obtained Embed prompt template , get the classification hint with attributes Where [CLS] will be replaced by the classification label; S112: Send each prompt text to the text encoder for encoding to obtain text representation , and the image representation Calculate cosine similarity; S113: Select The largest c is used as the final classification prediction .
9. The attribute-guided zero-shot CLIP image classification method according to claim 8, characterized in that: In S112, the prediction probability calculation formula is: 。 10. The attribute-guided zero-shot CLIP image classification method according to claim 8, characterized in that: In S113, the classification prediction The calculation formula is: 。