A CLIP-based zero-shot image recognition method

Abstract

The application discloses a zero sample image recognition method based on CLIP, which is excellent in zero sample knowledge migration capacity and obtains more accurate zero sample image classification results by positioning semantic guided fine-grained local visual features and fine-tuning attribute feature prompts by using feature alignment of a double network. On three classical image classification data sets CUB, SUN and AWA2, the traditional zero sample benchmark test reaches 84%, 89.8% and 97.4% of the correct rate respectively, and the generalized zero sample benchmark test reaches 75.9%, 68.4% and 92.6% of the H value respectively, which is the best model in the current result. It is proved that the method can learn the prior knowledge of seen class images, and accurately identify unseen class images by using feature alignment of a double network and fine-tuning attribute feature prompts.

Description

TECHNICAL FIELD

[0001] The present application relates to an image recognition method, in particular to a zero-shot image recognition method. BACKGROUND

[0002] Image classification is an important task in the field of computer vision, which aims to assign input images to predefined categories. This task has wide applications in many fields, including medical image analysis, autonomous driving, security monitoring, and face recognition.

[0003] Zero-shot learning is a special machine learning task that aims to classify unseen classes by learning the mapping relationship between known classes and unknown classes. Unlike traditional supervised learning, zero-shot learning allows the model to classify without any training samples about unknown classes. To achieve zero-shot learning, traditional methods usually rely on manually annotated class attributes from seen classes and unseen classes. These class attributes are shared knowledge that connects the disjoint sets of seen classes and unseen classes. Class attributes can describe the features of each class, such as color, shape, or function. By associating these class attributes with image features, the model can reason and classify when encountering unseen classes. The use of class attributes is also closer to human cognition. Humans can distinguish instances and unseen classes by using their prior knowledge of seen images and descriptive information about seen and unseen classes. This method mimics the way humans use prior knowledge and descriptive information when perceiving the world.

[0004] Zero-shot classification has the characteristic of not needing to see all class images, so it has wide applications in various aspects. For example, zero-shot image classification can be used in intelligent security systems to identify new objects or events, such as untrained faces, unusual behaviors, or suspicious items, to provide more accurate security monitoring and alarms; it can also be used to identify new plant and animal species, even without training images of these species, which is of great significance in fields such as ecology research, biodiversity protection, and wildlife monitoring; through zero-shot image classification, new food categories can be identified and classified, helping users to accurately assess nutrition and manage diet, etc. These application examples highlight the practical applications of zero-shot image classification in various fields, demonstrating its potential in solving practical problems and improving life.

[0005] Early leading zero-shot learning methods usually learn a joint embedding space by associating class attributes (semantic vectors) with global image representations of seen classes. However, due to the less-than-ideal mining of fine-grained class attributes, these global embedding methods almost always fail to adequately locate discriminative regional features. At the same time, recent zero-shot learning methods have verified that class attribute description vectors are effective in assisting models to obtain discriminative regional features, thereby improving zero-shot learning results and providing new clues. These methods directly align local regional features and class attribute description vectors, achieving coarse positioning of different attribute regions. However, due to the unavailability of unseen class images, the transferability of coarse regional positioning is insufficient, often leading to misclassification of unseen class images and seen class images. In contrast, the emergence of powerful contrastive language-image pre-training (CLIP) models uses class names to achieve more general zero-shot-like predictions. Due to the use of a large amount of image-text paired training data, CLIP generally exhibits significant image-text alignment capabilities and generalization performance on coarse-grained downstream datasets, but cannot locate information attribute regions. SUMMARY

[0006] The purpose of the application is to propose a CLIP-based zero-shot image recognition method to improve the knowledge transferability of the pre-trained CLIP model to the downstream zero-shot learning framework and to pursue ideal feature representation.

[0007] Technical solution: A CLIP-based zero-shot image recognition method, comprising:

[0008] S1: Given a dataset, divide it into seen classes and unseen classes;

[0009] S2: Use ResNet and CLIP visual encoders to represent the features of the images, and use the CLIP text encoder to represent the features of the class names, and introduce attribute description vectors;

[0010] S3: Perform visual-semantic interaction attention operations on the ResNet local visual features and the CLIP local visual features with the attribute description vectors to obtain attribute prompt attention-based feature vectors, thereby obtaining local class prediction vectors for the two networks;

[0011] S4: Construct a cross-network alignment loss L pal to limit cross-network feature alignment; and use cross-entropy loss L cl and self-calibration loss L cal to supervise the training of the local class prediction vectors of the two network models, respectively;

[0012] S5: Calculate the cosine similarity between the CLIP visual global feature and the class name feature to obtain the global prediction, fuse the final local prediction of the two network models, and introduce a bias vector to allow a part of the seen class prediction to migrate to the unseen class to obtain the final prediction score.

[0013] Beneficial effects: The present application can obtain excellent zero-shot knowledge transfer capability and more accurate zero-shot image classification results by positioning the fine-grained local visual features guided by semantics and fine-tuning the attribute feature prompt by using the feature alignment of the double network. On the traditional zero-shot benchmark test of three classic image classification data sets Caltech-USCD Birds-200-2011 (CUB), SUN and Animals with Attributes2 (AWA2), the accuracy rates are 84%, 89.8% and 97.4% respectively, and on the generalized zero-shot benchmark test, the H value (harmonic mean of the correct rates of seen classes and unseen classes) is 75.9%, 68.4% and 92.6% respectively, which is the best model result at present. It is proved that the present method can accurately identify the unseen class images by learning the prior knowledge of the seen class images and fine-tuning the attribute feature prompt by using the feature alignment of the double network. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 It is a flowchart of a zero-shot image recognition method based on CLIP;

[0015] Figure 2 It is a schematic diagram of the network framework proposed;

[0016] Figure 3 It is a schematic diagram of global prediction probability calculation. DETAILED DESCRIPTION

[0017] The present application will be further explained below in combination with the drawings.

[0018] As shown in the drawings, a zero-shot image recognition method based on CLIP comprises: Figure 1

[0019] S1: Given a data set It is divided into seen classes and unseen classes Where y i corresponds to the true class label of the image x i , respectively represent the image set of seen classes and unseen classes, respectively represent the label set of seen classes and unseen classes, let y i represent the class attribute probability score vector of the class, ​respectively represent the class attribute probability score vector set of seen classes and unseen classes.

[0020] The training set only has seen classes, and the test set only has unseen classes for the zero-shot setting, and includes both seen classes and unseen classes for the generalized zero-shot.

[0021] S2: Use ResNet and CLIP visual encoders to represent the features of the image respectively, and use the CLIP text encoder to represent the features of the class name, and introduce the attribute description vector.

[0022] Specifically, as shown in Figure 2 , the ResNet101 convolutional neural network and the CLIP visual encoder ViT 16 / B are used to extract image features x res , x clip , wherein x res has a dimension of [2048, 7, 7], and x clip has a dimension of [197, 512].

[0023] Use the CLIP text encoder Transformer to extract the feature of each class name, take the prompt "a photo of a {classname}" to get the class name feature c with a dimension of [|Y|, 512], wherein |Y| represents the number of classes. Introduce the attribute description vector set V = {v1, v2, …, v K}, which is obtained by inputting the attribute description text "bill color red, throat color white" into the text network in the form of a prompt "{att}" for feature extraction, wherein K represents the number of attribute description vectors.

[0024] S3: x res The corresponding feature vector takes the last dimension as the ResNet local visual feature, and x clip takes other heads except the classification head as the CLIP local visual feature. The ResNet local visual feature and the CLIP local visual feature are respectively subjected to visual-semantic interaction attention operation with the attribute description vector v, to obtain the feature vector based on attribute prompt attention, and thus the local class prediction vectors of the two networks are obtained.

[0025] Specifically, the obtained x res feature dimension becomes [2048, 49], which is regarded as 49 local visual features with a dimension of 2048 The obtained x clip is truncated to obtain a dimension of [196, 512], which is regarded as 196 local visual features with a dimension of 512 The local visual features are respectively subjected to visual-semantic interaction attention operation with the attribute description vector set V = {v1, v2, …, vK}: K

[0026]

[0027]

[0028]

[0029]

[0030] wherein, represents visual-semantic interaction attention operation, v k represents the kth attribute description vector, k = 1, 2, …, K; that is, respectively represents the visual feature of the mth block region of the ith image, M represents the number of local regions; g R (·) / g C (·) represents inputting the text into a simple linear neural network. respectively represent the attribute prompt attention-based feature vectors obtained by multiplying the attention matrices of the two branches with the local visual features; respectively represent the attribute-related vectors obtained by dot multiplication of the attribute prompt attention-based feature vectors and the mapped attribute description vectors;h R (·) / h C (·) represents two simple linear neural networks; respectively represent the final class prediction scores obtained by multiplying the attribute-related vectors with the class attribute probability scores and the attribute attention weights; represents the class y i with k attributes; wherein, the attribute attention weight The function w C / R (·) is a simple linear neural network. Wherein, the simple linear neural network is a simple linear layer that maps the input to the output space, such as: is a learnable matrix.

[0031] Two local class prediction vectors p i are finally obtained through different visual-semantic interaction attention networks.

[0032] S4: A cross-network alignment loss L pal is constructed for the local class prediction vectors of the two network models to limit cross-network feature alignment; cross-entropy losses L cl ​and self-calibration loss L cal Conduct supervised training.

[0033] Specifically, a cross-network alignment loss is constructed from the obtained local class prediction vectors of the two networks:

[0034]

[0035] in,

[0036] Calculate the cross-entropy loss L for the local class prediction vectors of the two networks. cl :

[0037]

[0038] Where |N| is the number of samples in the training set. This loss ensures that seen classes are included in training but does not consider unseen classes. This may cause the model to predict unseen classes with low probability and exhibit bias towards seen classes. To address this issue, a self-calibration loss L is used. cal To increase the prediction probability of unseen classes during training:

[0039]

[0040] in, It is an indicator function, representing the deviation vector. If but Otherwise, it is -1. This represents the two final category prediction scores for category c.

[0041] S5: The cosine similarity between the CLIP visual global features and the category name features is calculated to obtain a global prediction, which is then fused with the final local predictions of the two network models. Since the model only encounters image data of seen categories during training, it may overfit to those categories. To avoid this, the concept of a bias vector is introduced, allowing some predicted features from seen categories to be transferred to unseen categories. Finally, the prediction score of the network model is obtained.

[0042] Specifically, such as Figure 3 As shown, x obtained from S2 clip Feature extraction classification head As a global feature, cosine similarity is calculated with category name feature c to obtain the global prediction probability. Finally, the three probability vectors are weighted and summed to obtain the final prediction score:

[0043]

[0044] Wherein, β1, β2 are weight parameters, the prediction proportion of the double branches is adjusted according to different data sets, such as (β1, β2) = (0.1, 0.04) in the CUB data set.

[0045] The above merely describes the preferred embodiments of the present application, and it should be noted that, for those skilled in the art, several improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements should also be considered as falling within the scope of the present application.

Claims

1. A CLIP-based zero-shot image recognition method, characterized in that, Comprise: S1: Given a data set, divide it into seen classes and unseen classes; S2: respectively using ResNet and CLIP visual encoders to perform feature representation on the images, and respectively extracting image features x res 、 x clip , while using a CLIP text encoder to perform feature representation on the category name and introducing an attribute description vector; S3: ResNet local visual features and CLIP local visual features are respectively subjected to visual-semantic interaction attention operation with attribute description vectors to obtain feature vectors based on attribute prompt attention, thereby obtaining local class prediction vectors of two networks; S4: Construct cross-network alignment loss L on the local class prediction vectors of the two network models pal to limit cross-network feature alignment; The local class prediction vectors of the two network models are respectively subjected to cross-entropy loss L cl and self-calibration loss L cal Supervised training; S5: Calculate the cosine similarity between the CLIP visual global feature and the class name feature to obtain the global prediction, fuse the final local prediction of the two network models, and introduce a bias vector to allow a part of the seen class prediction to migrate to the unseen class to obtain the final prediction score; The S3 includes: x res The corresponding feature vector regards the last dimension as a ResNet local visual feature, x clip Taking other heads except the classification head as CLIP local visual features, the ResNet local visual features and the CLIP local visual features are respectively subjected to visual-semantic interaction attention operation with the attribute description vector v, the attention matrices obtained by the two branches are respectively multiplied with the corresponding local visual features to obtain attribute prompt attention-based feature vectors; then, attribute-related vectors are obtained by dot multiplication of the attribute prompt attention-based feature vectors and the mapped attribute description vectors; finally, the final class prediction scores are obtained by multiplying the attribute-related vectors with the class attribute probability scores and the attribute attention weights Thus, the local class prediction vectors p of the two networks are obtained i , 2. The CLIP-based zero-shot image recognition method of claim 1, wherein, In the S1, a given dataset is divided into seen classes and unseen classes where denote the image collections of seen classes and unseen classes, respectively, y i denotes the true class label of the corresponding image x i denotes the class attribute probability score vector of class y i denote the sets of class attribute probability score vectors of seen classes and unseen classes, respectively.​​ 3. The CLIP-based zero-shot image recognition method of claim 2, wherein, In S2, image features x are extracted using ResNet101 convolutional neural network and CLIP visual encoder ViT 16 / B respectively res clip The category name feature c is extracted using CLIP text encoder Transformer, and a set of attribute description vectors V = {v1, v2, …, v K K represents the number of attribute description vectors.​ 4. The CLIP-based zero-shot image recognition method of claim 3, wherein, In S4, a cross-network alignment loss is constructed for the local class prediction vectors of the two networks: wherein, |Y| represents the number of categories; The local class prediction vectors of the two networks are computed cross-entropy loss L cl : where |N| is the number of samples in the training set, Reusing self-calibration loss L cal to increase the prediction probability of unseen classes during training: wherein, is an indicator function, representing the bias vector, if then else -1; denotes the two final class prediction scores for class c.

5. The CLIP-based zero-shot image recognition method of claim 4, wherein, In S5, the x clip Feature intercept classification head As global feature, compute cosine similarity with class name feature c to get global prediction probability Weighted sum of the three probability vectors to get the final prediction score: Where β1, β2 are weight parameters.

Images