A few-shot classification method based on an image-text pre-training model
By using a four-branch architecture for few-shot classification networks, combined with multimodal feature extraction and optimal transport strategies, the accuracy problem of few-shot classification algorithms when there are large differences in visual features is solved, and high precision and generalization ability are improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NORTHEASTERN UNIV CHINA
- Filing Date
- 2024-07-04
- Publication Date
- 2026-05-29
AI Technical Summary
Existing few-shot classification algorithms struggle to accurately identify objects of the same category when visual features differ significantly, and they fail to effectively utilize the generalization ability of pre-trained models, resulting in suboptimal classification performance.
A few-shot classification network with a four-branch architecture is used, which combines multimodal feature extraction, primitive-level and cue-level optimal transport, and similarity calculation. Through optimization of cross-entropy loss, unbalanced consistency loss and prototype distance loss function, image-text feature matching and alignment are achieved.
It improves the accuracy and generalization ability of few-shot classification. The experimental results are excellent on the Caltech101, DTD and EuroSAT datasets, with classification accuracy of 96.3%, 73.7% and 92.7% respectively.
Smart Images
Figure CN118747313B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and in particular to a few-shot classification method based on an image-text pre-trained model. Background Technology
[0002] Few-shot classification is an important technique in machine learning, aiming to design classification models with efficient learning and accurate prediction capabilities using only a small number of samples. Early few-shot classification algorithms primarily focused on the extraction and utilization of visual features. However, when visual features vary significantly, few-shot classification algorithms based on a single visual modality struggle to accurately identify objects of the same category. This poses a significant challenge to few-shot object classification tasks. Furthermore, the development of vision-language multimodal technologies has further expanded the application scenarios of few-shot object classification. Currently, few-shot classification has been widely applied in many fields, such as robot manipulation, surface defect detection, and remote sensing image understanding. Therefore, researching vision-language multimodal few-shot classification algorithms is of great significance. With the development of deep learning and model fine-tuning techniques, few-shot classification algorithms have also made significant progress. Many works extract multimodal features through pre-trained vision-language models and use contrastive learning to distinguish and determine whether the semantic information represented by different features is the same. In this process, a multimodal high-dimensional embedding space is learned through the network, so that positive image-text pairs have higher similarity and negative image-text pairs have lower similarity, thereby completing the matching and alignment of multimodal features.
[0003] The paper *International Journal of Computer Vision, 2337-2348, 2022* models contextual cues using a set of learnable text vectors, which can be optimized by minimizing the cross-entropy classification loss. However, this method only uses a single text vector to adapt to the downstream task, failing to consider multiple learnable visual and textual cues simultaneously, resulting in low classification performance. The paper *Neural Information Processing Systems, 52792-52810, 2023* uses primitive-level and cue-level optimal transport methods to learn multiple visual and textual cues simultaneously, and achieves multimodal feature alignment through the cross-entropy loss function. However, this method only focuses on the few-sample information contained in the trained cues, causing the model to only consider its ability to fit the downstream task, neglecting the generalization ability of the pre-trained zero-shot CLIP model, thus leading to unsatisfactory model performance. Summary of the Invention
[0004] The purpose of this invention is to propose a few-shot classification method based on an image-text pre-trained model, which improves the accuracy and generalization of few-shot classification tasks.
[0005] The technical solution of the present invention is as follows: a few-shot classification method based on an image-text pre-trained model, wherein a few-shot classification network is established for classification;
[0006] The few-shot classification network has a four-branch architecture: the first branch is the zero-shot text branch, the second branch is the few-shot image branch, the third branch is the few-shot text branch, and the fourth branch is the zero-shot image branch. The zero-shot text branch takes unlearnable text as input and outputs unlearnable primitive-level text features and unlearnable cue-level text features. The zero-shot image branch takes unlearnable images as input and outputs unlearnable primitive-level image features and their prototypes, as well as unlearnable cue-level image features and their prototypes. The few-shot image branch takes learnable images as input and outputs learnable primitive-level image features and learnable cue features. The few-shot text branch takes learnable text as input and outputs learnable primitive-level text features and learnable cue-level text features. The zero-shot text branch and zero-shot image branch are composed of a multimodal feature extraction module, a cue-level optimal transport module, and a similarity calculation module, respectively. The optimization process of the few-shot classification network is accomplished through three loss functions: cross-entropy loss function, unbalanced consistency loss function, and cue-level prototype distance loss function.
[0007] The few-sample image branch extracts image features using the following formula:
[0008] (1)
[0009] Where z represents the learnable cue-level image features, f The CLIP image encoder represents the few-shot image branch. and These represent image patch vectors and visual primitive vectors in learnable primitive-level image features, respectively. The input represents learnable primitive-level image features; <cls>Represents a learnable category primitive;
[0010] Learnable text vector t and the first k Fixed text vectors corresponding to each category The input text features are used to construct the output text features, which are obtained by the following formula:
[0011] (2)
[0012] in, This represents learnable prompt-level text features. and Represents learnable primitive-level text features; g The CLIP text encoder represents the few-sample text branch. <eos>This represents the end primitive of the input text.
[0013] Given an object category, compute unlearnable primitive-level image features using a CLIP image encoder with a zero-sample image branch. :
[0014] (3)
[0015] These are non-learnable cue-level image features;
[0016] Unlearnable primitive-level image feature prototypes The calculation method is as follows:
[0017] (4)
[0018] in, Represents the selected Training samples for each category; It is the number of image samples in a few-shot classification dataset; using Euclidean distance. To measure unlearnable primitive-level image feature prototypes With learnable primitive-level image patch vectors The differences between them are specifically expressed as follows:
[0019] (5)
[0020] Distance to unlearnable primitive-level image feature prototype center Closer learnable primitive-level image patches exhibit smaller intra-class variability; Euclidean distance With learnable primitive-level image patch weights Inversely proportional; this mapping process is achieved using the following Gaussian function:
[0021] (6)
[0022] in, It is the standard deviation of the Gaussian function; based on the learnable primitive-level image patch weights. The reweighted learnable primitive-level image patch representation is as follows:
[0023] (7).
[0024] Learnable primitive-level image features and learnable primitive-level text features The model is based on two empirical distributions, as shown below:
[0025] (8)
[0026] in, express The corresponding Dirac function value, express The corresponding Dirac function value;
[0027] Define the primitive-level cost matrix as follows ,in Learnable primitive-level image features and learnable primitive-level text features The optimal transport distance between them is expressed as follows:
[0028] (9)
[0029] in, >0 indicates a hyperparameter; express R and S The optimal transport matrix between them.
[0030] use and Two empirical distributions are used to model learnable cue-level image features. and learnable prompt-level text features This yields the optimal transportation strategy at the cue level with cross-alignment:
[0031] (10)
[0032] in, Indicates the first Learnable, cue-level image features; Indicates the first n One learnable prompt-level text feature; feature express The corresponding Dirac function value, express The corresponding Dirac function value;
[0033] Compute unlearnable primitive-level text features from the CLIP text encoder using the zero-sample text branch. and unlearnable prompt-level text features :
[0034] (11)
[0035] Unlearnable cue-level image features and non-learnable cue-level text features Corresponding distribution and It is expressed as follows:
[0036] (12)
[0037] in, Indicates the first m An unlearnable cue-level image feature, Indicates the first n An unlearnable prompt-level text feature; express The corresponding Dirac function value, express The corresponding Dirac function value;
[0038] The optimal transport distance at the prompt level is represented as follows:
[0039] (13)
[0040] in, express P and Q The optimal transportation matrix between them; yes P and Q The cost matrix between them is defined as follows:
[0041] (14)
[0042] The optimal transport distance for crossover at the prompt level is defined as follows:
[0043] (15)
[0044] (16)
[0045] The cost matrix in formulas (15) and (16) is defined as follows:
[0046] (17)
[0047] (18)
[0048] The category prediction probability with the optimal transportation matrix is represented as follows:
[0049] (19).
[0050] The unbalanced consistency loss with prototype distance uses cross-entropy loss as the basic loss function:
[0051] (20)
[0052] in, It is a category k Image Tags;
[0053] By applying unbalanced consistency loss and To maintain consistency between learnable and non-learnable cue features, the formula is defined as follows:
[0054] (twenty one)
[0055] (twenty two)
[0056] in, This represents a consistency loss that includes unlearnable cue-level image features. This represents the consistency loss that includes non-learnable cue-level text features; Represents KL divergence; using training samples from a few-shot dataset. Computing unlearnable cue-level image feature prototypes :
[0057] (twenty three)
[0058] in These are unlearnable cue-level image features; prototype loss is used to supervise learnable cue-level image features. With unlearnable cue-level image feature prototypes Distance between:
[0059] (twenty four)
[0060] in, k Representing an image Category labels; combined with unbalanced consistency loss and and prototype loss Unbalanced consistency loss based on prototype distance It is expressed as follows:
[0061] (25)
[0062] The overall loss function is shown below:
[0063] (26).
[0064] The beneficial effects of this invention: Addressing the low accuracy problem in few-shot classification algorithms, this invention proposes a few-shot classification method based on an image-text pre-trained model to solve few-shot classification tasks in different scenarios. First, for primitive-level optimal transport, we propose an image patch reweighting mechanism based on prototype distance. We convert the Euclidean distance between the prototype and the image patch into a weight for each image patch, thus assigning different weights to each patch. For the cue-level stage, we propose a cascaded regularized optimal transport module. This module considers not only the optimal transport between few-shot image-text features but also the cross-matching between zero-shot and few-shot CLIP multimodal features. Finally, we propose a prototype-based unbalanced logarithmic loss function to supervise the network. This loss function consists of a prototype distance metric and a logarithmic consistency metric. The prototype part calculates the loss based on the distance between the cue-level prototype and the image feature embedding. The logarithmic constraint makes the few-shot logarithmic distribution approximate the zero-shot logarithmic distribution. Finally, tests are conducted on the Caltech01, DTD, and EuroSAT datasets. The experimental results demonstrate that the proposed network model performs well, validating the effectiveness of the algorithm. Attached Figure Description
[0065] Figure 1 This is a flowchart of a few-shot classification method based on an image-text pre-trained model. Detailed Implementation
[0066] Figure 1 This is the main flowchart of the technical solution of this invention. For example... Figure 1 As shown, the few-shot classification method based on an image-text pre-trained model proposed in this invention includes the following steps:
[0067] Step 1, Training Data: This invention constructs a relevant training set using RGB images from the datasets Caltech101, EuroSAT, and DTD, and provides input images for the network;
[0068] Step 2, Model Training: The network framework proposed in this invention consists of a four-branch architecture: the first branch is the zero-sample text branch, the second branch is the few-sample image branch, the third branch is the few-sample text branch, and the fourth branch is the zero-sample image branch. Each branch consists of three forward stages: multimodal feature extraction, optimal transportation planning, and similarity calculation. In the multimodal feature extraction stage, it is implemented through a text encoder or an image encoder. The few-sample images and their linguistic descriptions are input into each branch of the few-sample classification network. After image-text feature extraction, the transportation matrix is calculated through optimal transportation planning. In the few-sample text and few-sample image branches, the optimal transportation module consists of primitive-level optimal transportation and cue-level optimal transportation. The optimization process of the few-sample classification network is accomplished through three loss functions: cross-entropy loss function, unbalanced logarithmic consistency loss function, and cue-level prototype distance loss function. These will be described in detail below.
[0069] a. Primitive-level optimal transportation based on image patch prototype distance
[0070] Given multiple images of an object category, different image embeddings can be obtained from a pre-trained CLIP image encoder. In this case, intra-class differences arise between image patch embeddings. To better adjust text and image embeddings during cue-level optimal transport, more attention should be paid to image patch embeddings with smaller intra-class differences. Therefore, we propose a primitive-level reweighting mechanism for image patches.
[0071] Given an object category, different image embeddings can be obtained from a pre-trained CLIP image encoder. These are non-learnable primitive-level image feature prototypes. As shown below:
[0072] (4)
[0073] in, Represents the selected Training samples for each category. It is the number of image samples in a few-shot classification dataset. The CLIP image encoder, representing the zero-sample image branch, computes unlearnable primitive-level image features:
[0074] (5)
[0075] Then, use Euclidean distance To measure unlearnable primitive-level image feature prototypes With learnable primitive-level image patch vectors The differences between them are specifically expressed as follows:
[0076] (6)
[0077] Distance to unlearnable primitive-level image feature prototype center Blocks that are closer together have less intra-class diversity, and these blocks need to be assigned more weight during transportation optimization. Therefore, Euclidean distance... It is inversely proportional to the block weight value. This mapping process is achieved using the following Gaussian function:
[0078] (7)
[0079] in, It is the standard deviation of the Gaussian function. Based on block weights. The reweighted image patch can be represented as follows:
[0080] (8)
[0081] To link multimodal cues with optimal transportation, learnable primitive-level image features are needed. and learnable primitive-level text features As shown below:
[0082] (9)
[0083] in, It is a length of A combination of image patches and learnable image primitives. It is a length of The combination of learnable and original text primitives.
[0084] Furthermore, the primitive-level cost matrix is defined as follows: ,in Learnable primitive-level image features and learnable primitive-level text features The optimal transport distance between them is expressed as follows:
[0085] (10)
[0086] in, >0 indicates a hyperparameter. Primitive-level transport matrix. Indicates the first j The first visual primitive and the first l The correlation between text primitives.
[0087] b. Optimal transport with cross-matching prompts
[0088] To ensure that the network maintains the generalization performance of the pre-trained CLIP model while fitting the downstream classification task, we consider both zero-shot image-text features and few-shot image-text features and perform cross-matching in the cue-level optimal transport stage.
[0089] Use of cross-matching hint-level optimal transportation strategies and Two experience distributions to simulate learnable cues for visual language:
[0090] (11)
[0091] in, Indicates the first Learnable, cue-level image features; Indicates the first n Learnable prompt-level text features; Meanwhile... and All are dimension Feature embeddings in the shared space. To cross-align the learnable embeddings and zero-point features, we compute zero-sample features from the pre-trained CLIP model using Equation (5) and the following formula:
[0092] (12)
[0093] Similar to formula (11), the distribution of zero-sample features and It is expressed as follows:
[0094] (13)
[0095] in, Indicates the first m An unlearnable cue-level image feature, Indicates the first n An unlearnable cue-level text feature. The optimal cue-level transport distance representation is as follows:
[0096] (14)
[0097] in It is the cue-level cost matrix between the set of visual cues and the set of text cues. This is the prompt-level transportation matrix. The prompt-level cost matrix is defined as follows:
[0098] (15)
[0099] The optimal transport distance for crossover at the prompt level is defined as follows:
[0100] (16)
[0101] (17)
[0102] The cost matrix in formulas (16) and (17) is defined as follows:
[0103] (18)
[0104] (19)
[0105] The category prediction probability with optimal transportation planning is represented as follows:
[0106] (20)
[0107] c. Unbalanced consistency loss with prototype distance
[0108] To train the proposed few-shot classification network model based on the image-text pre-trained model, we employ a set of mixed loss functions, including the cross-entropy loss function. and non-equilibrium consistency loss with prototype distance The overall loss function is shown below:
[0109] (twenty one)
[0110] First, the cross-entropy loss is used as the basic loss function:
[0111] (twenty two)
[0112] in, It is an image of category k. The label. By imposing an unbalanced logarithmic level consistency loss. and To maintain consistency between learnable and non-learnable cue features, the formula is defined as follows:
[0113] (twenty three)
[0114] (twenty four)
[0115] in, This represents a consistency loss that includes unlearnable cue-level image features. This represents the consistency loss that includes non-learnable cue-level text features; This represents the KL divergence. For cue-level alignment, different image feature embeddings should approximate their corresponding prototypes. Therefore, training samples from a few-shot dataset are used. Computing unlearnable cue-level image feature prototypes :
[0116] (25)
[0117] in These are non-learnable cue-level image features; to cluster image embeddings into the same class, each image feature embedding should be close to its cue-level prototype. Therefore, prototype loss is used to supervise learnable cue-level image features. With unlearnable cue-level image feature prototypes Distance between:
[0118] (26)
[0119] Where k represents the image Category labels. Combined with unbalanced logarithmic level consistency loss. and and prototype loss Unbalanced consistency loss based on prototype distance It is expressed as follows:
[0120] (27)
[0121] Step 3, Few-shot Classification: In the testing phase, we first construct corresponding descriptive text from the category labels of the classification task according to the learnable cue template. Then, the descriptive text is fed into the text encoder of the CLIP network to obtain the corresponding text features. Next, the image to be predicted is fed into the image encoder of the CLIP network to obtain the corresponding image features. Then, the cosine similarity between the image features and the text features is calculated and multiplied with the cue-level optimal transport matrix. The category corresponding to the text with the largest product result is selected as the classification result.
[0122] To verify the effectiveness of the algorithm in few-shot classification accuracy, this invention was tested on the Caltech101, DTD, and EuroSAT datasets. According to the experimental results, the proposed algorithm achieves few-shot classification accuracy of 96.3% in indoor and outdoor natural scenes, 73.7% in surface texture scenes, and 92.7% in satellite image scenes.< / eos> < / cls>
Claims
1. A few-shot classification method based on an image-text pre-trained model, characterized in that, Establish a few-shot classification network for classification; The few-shot classification network has a four-branch architecture: the first branch is the zero-shot text branch, the second branch is the few-shot image branch, the third branch is the few-shot text branch, and the fourth branch is the zero-shot image branch. The zero-shot text branch takes unlearnable text as input and outputs unlearnable primitive-level text features and unlearnable cue-level text features. The zero-shot image branch takes an unlearnable image as input and outputs unlearnable primitive-level image features and their prototypes, as well as unlearnable cue-level image features and their prototypes. The few-shot image branch takes a learnable image as input and outputs learnable primitive-level image features and learnable cue-level image features. The few-shot text branch takes learnable text as input and outputs learnable primitive-level text features and learnable cue-level text features. The zero-shot text branch and zero-shot image branch are composed of a multimodal feature extraction module, a cue-level optimal transport module, and a similarity calculation module, respectively; the few-shot text branch and few-shot image branch are composed of a multimodal feature extraction module, a primitive-level optimal transport module, a cue-level optimal transport module, and a similarity calculation module, respectively; the optimization process of the few-shot classification network is accomplished through three loss functions, namely the cross-entropy loss function, the unbalanced consistency loss function, and the cue-level prototype distance loss function; The primitive-level optimal transportation module will learn primitive-level image features. and learnable primitive-level text features The model is based on two empirical distributions, as shown below: （8） in, express The corresponding Dirac function value, express The corresponding Dirac function value; This represents a reweighted learnable primitive-level image patch; Represents visual primitive vectors in learnable primitive-level image features; and Represents learnable primitive-level text features; Define the primitive-level cost matrix as follows ,in Learnable primitive-level image features and learnable primitive-level text features The optimal transport distance between them is expressed as follows: （9） in, >0 indicates a hyperparameter; express R and S The optimal transportation matrix between them; The prompt-level optimal transportation module uses and Two empirical distributions are used to model learnable cue-level image features. and learnable prompt-level text features We obtain the cue-level optimal transportation strategy with cross-alignment; and compute unlearnable primitive-level text features from the CLIP text encoder using the zero-sample text branch. and unlearnable prompt-level text features and unlearnable cue-level image features and non-learnable cue-level text features Corresponding distribution and The optimal transport distance at the prompt level is obtained.
2. The few-shot classification method based on an image-text pre-trained model according to claim 1, characterized in that, The few-sample image branch extracts image features using the following formula: （1） Where z represents the learnable cue-level image features, f The CLIP image encoder represents the few-shot image branch. Represents an image patch vector in learnable primitive-level image features; The input represents learnable primitive-level image features; <cls> Represents a learnable category primitive;< / cls> Learnable text vector t and the first k Fixed text vectors corresponding to each category The input text features are used to construct the output text features, which are obtained by the following formula: （2） in, Represents learnable prompt-level text features; g The CLIP text encoder represents the few-sample text branch. <eos> This represents the end primitive of the input text.< / eos> 3. The few-shot classification method based on an image-text pre-trained model according to claim 2, characterized in that, Given an object category, compute unlearnable primitive-level image features using a CLIP image encoder with a zero-sample image branch. : （3） These are non-learnable cue-level image features; Unlearnable primitive-level image feature prototypes The calculation method is as follows: （4） in, Represents the selected Training samples for each category; It is the number of image samples in a few-shot classification dataset; using Euclidean distance. To measure unlearnable primitive-level image feature prototypes With learnable primitive-level image patch vectors The differences between them are specifically expressed as follows: （5） Distance to unlearnable primitive-level image feature prototype center Closer learnable primitive-level image patches exhibit smaller intra-class variability; Euclidean distance With learnable primitive-level image patch weights Inversely proportional; this mapping process is achieved using the following Gaussian function: （6） in, It is the standard deviation of the Gaussian function; based on the learnable primitive-level image patch weights. The reweighted learnable primitive-level image patch representation is as follows: （7）。 4. The few-shot classification method based on an image-text pre-trained model according to claim 1, characterized in that, use and Two empirical distributions are used to model learnable cue-level image features. and learnable prompt-level text features This yields the optimal transportation strategy at the cue level with cross-alignment: （10） in, Indicates the first Learnable, cue-level image features; Indicates the first n One learnable prompt-level text feature; feature express The corresponding Dirac function value, express The corresponding Dirac function value; Compute unlearnable primitive-level text features from the CLIP text encoder using the zero-sample text branch. and unlearnable prompt-level text features : （11） Unlearnable cue-level image features and non-learnable cue-level text features Corresponding distribution and It is expressed as follows: （12） in, Indicates the first m An unlearnable cue-level image feature, Indicates the first n An unlearnable prompt-level text feature; express The corresponding Dirac function value, express The corresponding Dirac function value; The optimal transport distance at the prompt level is represented as follows: （13） in, express P and Q The optimal transportation matrix between them; yes P and Q The cost matrix between them is defined as follows: （14） The optimal transport distance for crossover at the prompt level is defined as follows: （15） （16） The cost matrix in formulas (15) and (16) is defined as follows: （17） （18） The category prediction probability with the optimal transportation matrix is represented as follows: （19）。 5. The few-shot classification method based on an image-text pre-trained model according to claim 1, characterized in that, The unbalanced consistency loss with prototype distance uses cross-entropy loss as the basic loss function: （20） in, It is a category k Image Tags; By applying unbalanced consistency loss and To maintain consistency between learnable and non-learnable cue features, the formula is defined as follows: （21） （22） in, This represents a consistency loss that includes unlearnable cue-level image features. This represents the consistency loss that includes non-learnable cue-level text features; Represents KL divergence; using training samples from a few-shot dataset. Computing unlearnable cue-level image feature prototypes : （23） in These are unlearnable cue-level image features; prototype loss is used to supervise learnable cue-level image features. With unlearnable cue-level image feature prototypes Distance between: （24） in, k Representing an image Category labels; combined with unbalanced consistency loss and and prototype loss Unbalanced consistency loss based on prototype distance It is expressed as follows: （25） The overall loss function is shown below: （26）。