Long tail distribution-oriented vision-language model prompt learning framework

Through the visual-language model prompt learning framework, multiple prompts are generated to describe semantics, solving the problem of poor model effectiveness in the long-tail distribution dataset, achieving full learning of tail categories and prompt diversity, and avoiding the additional text corpus. Dependence improves the generalization ability of the model.

CN118917276APending Publication Date: 2024-11-08SOUTHEAST UNIV
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202410270687.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-03-11
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

In the long-tail distribution dataset, traditional machine learning algorithms have poor model effects due to the skewness of the training set. The existing methods require additional text corpus and complex prompt engineering, making it difficult to effectively learn tail categories with scarce training samples.

Method used

A visual-language model prompt learning framework for long-tail distribution is proposed. By obtaining pre-trained vision-language model and target dataset, multiple prompts are generated using a learnable prompt distribution generator, and the image is trained in combination with a comparative learning method. and text encoder to ensure the diversity of hint distributions and the generalization capability of the model.

Benefits of technology

It realizes the generation of multiple prompts on the long-tail distribution dataset to describe semantics, establish a powerful integrated learning algorithm, fully learn tail categories with scarce training samples, avoid dependence on additional text corpus, and improve the generalization ability of the model. and diversity of tips.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917276A_ABST
    Figure CN118917276A_ABST
Patent Text Reader

Abstract

The invention discloses a long tail distribution-oriented vision-language model prompt learning framework, which comprises the following steps of: firstly, generating a prompt distribution by using a class-specific or class-universal prompt distribution generator, then sampling a certain number of prompts from the distribution, and inputting the prompts and a class text into a text encoder to obtain a text encoding vector; and then inputting the picture into an image encoder to obtain an image encoding vector, and guiding training of the text encoding vector and the image encoding vector on semantic alignment by utilizing comparison loss. The learning of the prompts is formalized into a variational problem, and the framework can simultaneously generate a plurality of prompts to describe categories, so that a powerful integrated learning algorithm is established, and the model can fully learn tail categories with scarce training samples. Actual research shows that the provided prompt learning framework is beneficial to successfully applying the pre-trained vision-language model to a downstream vision identification task of data long-tail distribution.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of long-tail distribution, and in particular relates to a vision-language model prompt learning framework for long-tail distribution. Background Art

[0002] Typically, datasets collected from the real world exhibit a long-tail distribution. A long-tail distribution refers to a distribution in which a few categories or values ​​have very high frequencies (heads) in a dataset, while most categories or values ​​have lower frequencies (tails). In a long-tail distribution, the categories or values ​​at the head are often called "hot", while the categories or values ​​at the tail are called "long tails". This form of distribution is common in many real-world datasets, such as sales, website visits, product reviews, and so on. For traditional machine learning algorithms, if there is no additional processing to reduce the skewness of the training set, the model's performance tends to be poor. Currently, there are datasets for long-tail distributions including CIFAR10-LT, Places-LT, and ImageNet-LT.

[0003] To address the imbalanced data problem, the researchers proposed a series of algorithms, including adjustments to the sampling strategy (upsampling less frequent categories), data augmentation to create synthetic samples (synthesizing less frequent samples), a sample weight-based reweighting scheme (assigning greater weights to less frequent samples), and knowledge distillation techniques.

[0004] Recently, large model-based methods have achieved impressive results on zero-shot tasks. Therefore, the focus of current research is to select appropriate pre-trained large models and use the text encoder or visual encoder in the large model according to the data of downstream tasks. Among these large models, CLIP is a highly-regarded text-visual large model that can efficiently decode high-level semantic information of text and visual data in zero-shot learning settings, demonstrating its strong generalization ability.

[0005] Hint engineering has been shown to be effective in transferring knowledge from pre-trained base models to downstream tasks. The basic idea is to learn a suitable textual context as a hint to build the input of the model's text encoder around the main text. Unlike hand-crafted hints, the differentiable hints proposed in CoOp are automatically learned in the word embedding space, greatly accelerating the hint tuning process for downstream tasks. In addition to the idea of ​​automatically learning hints like CoOp, the recent ProDA work trains a fixed set of hints for the CLIP model by estimating the Gaussian mean and covariance on the output embeddings of the text encoder. VL-LTR aims to solve the learning of imbalanced downstream datasets using the CLIP model, considering distilling knowledge from the CLIP model while training a student visual-language network from scratch. However, VL-LTR requires users to collect additional text corpora from the web, rather than just using the data provided in the downstream long-tail learning task as hints. Summary of the invention

[0006] To solve the above problems, the present invention discloses a visual-language model prompt learning framework for long-tail distribution, which can simultaneously generate multiple prompts to describe semantics, thereby establishing a powerful ensemble learning algorithm, enabling the model to fully learn the tail categories with scarce training samples.

[0007] To achieve the above object, the technical solution of the present invention is as follows:

[0008] A visual-language model hint learning framework for long-tail distribution, including the following steps:

[0009] Step 1: Obtain a large visual-language model and its pre-trained model parameter file, prepare a target long-tail distribution data set and pre-process it; the large visual-language model includes an image encoder, a text encoder, and a prompt distribution generator;

[0010] Step 2: Sample text descriptions of images and categories in the dataset, generate a prompt distribution through the prompt distribution generator, and sample a prompt set from the distribution to obtain a set of prompt embedding vectors;

[0011] Step 3: Input the sampled image into the image encoder to obtain the image embedding vector;

[0012] Step 4: Input the prompt embedding vector and the category text description into the text encoder to obtain the text encoding embedding vector;

[0013] Step 5: Use contrastive learning to select appropriate loss functions and regularization terms to train the image encoder and prompt distribution generator;

[0014] Step 6: During testing, input the test image into the image encoder to obtain the image embedding vector; input the prompt set and category description generated by the prompt distribution generator corresponding to different categories into the text encoder to obtain the text embedding vector, and obtain the category of the test image by comparison.

[0015] Furthermore, the visual-language model used in step 1 is Contrastive Language Image Pre-training (CLIP), which is pre-trained using the WIT dataset. The datasets required for downstream tasks are CIFAR10-LT, Places-LT, and ImageNet-LT.

[0016] Furthermore, the prompt distribution generator used in step 2 is a learnable module and can be divided into three categories: a category-general prompt distribution generator, a category-specific prompt distribution generator, and a conditional category-specific prompt distribution generator, and the formula is expressed as:

[0017] Category General Prompt Generator:

[0018]

[0019] Category specific prompt distribution generator:

[0020]

[0021] Conditional category specific prompt distribution generator:

[0022]

[0023] Among them, G agn ,G spec ,G condi They represent three prompt distribution generators, θ represents the learnable parameters in the generator, y represents the category, C represents the number of categories, and w y Represents the learnable conditional parameter vector for class y. represents normal distribution, μ and Σ represent the mean and variance of normal distribution. TEXT y Represents the text description of category y. During training or testing, first select a prompt distribution generator for a specified category, then generate a prompt distribution from the prompt distribution generator, and finally sample K prompts from the prompt distribution. Represents the k-th hint vector for category y.

[0024] Furthermore, the image encoder used in step 3 is ResNet50, which is composed of four residual structures. It can also be Vision Transformer, which is composed of 12 ViT units in cascade.

[0025] Furthermore, the text encoder used in step 4 is Text Transformer, which is composed of 12 cascaded Bert units.

[0026] Furthermore, the loss function and regularization term mathematical formula of step 5 are shown as follows:

[0027] Loss function:

[0028]

[0029] Among them, l represents the loss function, and y represents the category of x. y In practice, it is usually set to the inverse of the class prior probability, and τ is an additional free parameter that can be used to adjust π y The offset of f(x) is a vector, and the y'th component can be expressed as:

[0030]

[0031] also

[0032] v(x)=f img (x),#(14)

[0033]

[0034] Among them, f img and f txt Represent image encoder and text encoder respectively, x represents input image, v(x) represents image x embedding vector, represents the k-th text embedding vector of category y (here y also needs to be converted to y'), and h(·) represents the mapping function that embeds the text into a vector.

[0035] Regularization term:

[0036]

[0037] in, Represents the KL divergence. The purpose of this regularization term is to make the generated prompt distribution a standard normal distribution, so as to ensure the diversity of the sampled prompts.

[0038] The beneficial effects of the present invention are:

[0039] (1) The present invention proposes a prompt learning algorithm to address the long-tail problem. The algorithm determines a set of processes for generating any number of prompts, explicitly setting multiple prompts for each category to describe the semantics, thereby establishing a powerful ensemble learning algorithm that enables the model to fully learn the tail categories with a scarce number of training samples.

[0040] (2) The present invention imposes a normal distribution restriction on the prompt distribution generated by the prompt distribution generator, which ensures the diversity of the sampled prompts.

[0041] (3) The present invention does not require users to collect additional text corpora from the Internet, and only uses the data provided in the downstream long-tail learning tasks to fine-tune the network and prompt distribution generator. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is the visual-language large model prompt learning framework diagram proposed by the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0044] As shown in the figure, the visual-language model prompt learning framework for long-tail distribution described in the present invention includes the following steps:

[0045] Step 1: Dataset preparation: This paper uses three datasets, CIFAR10-LT, Places-LT, and ImageNet-LT, as experimental datasets. CIFAR10-LT includes low-resolution images of ten categories, using an imbalance ratio of {10, 50, 100, 200}, where a higher ratio means that the training data is more skewed; Places-LT is a long-tail version of Places365, containing 365 scene / environment categories; finally, the highly unbalanced ImageNet-LT contains 1000 categories of natural images, with the largest category cardinality being approximately 1300 samples, while the smallest category cardinality is only five samples.

[0046] Step 2: Obtain a large visual-language model and its pre-trained model parameter file, wherein the large visual-language model includes an image encoder, a text encoder, and a prompt distribution generator. The large visual-language model used is Contrastive Language Image Pre-training (CLIP), which is pre-trained using the WIT dataset.

[0047] Step 3: Sample images and text descriptions of categories in the dataset, generate a distribution of prompts through the prompt distribution generator, and sample a prompt set from the distribution to obtain a set of prompt embedding vectors; the prompt distribution generator used is a learnable module and can be divided into three categories: category-general prompt distribution generator, category-specific prompt distribution generator, and conditional category-specific prompt distribution generator, the formula is expressed as:

[0048] Category-general prompt distribution generator:

[0049]

[0050] Category specific prompt distribution generator:

[0051]

[0052] Conditional category specific prompt distribution generator:

[0053]

[0054] Among them, G agn ,G spec ,G condi They represent three prompt distribution generators, θ represents the learnable parameters in the generator, y represents the category, C represents the number of categories, and w y Represents the learnable conditional parameter vector for class y. represents normal distribution, μ and Σ represent the mean and variance of normal distribution. TEXT y Represents the text description of category y. During training or testing, first select the specified prompt distribution generator, then generate a prompt distribution from the prompt distribution generator, and finally sample K prompts from the prompt distribution. Represents the k-th hint vector for category y.

[0055] Step 4: Input the sampled image into the image encoder to obtain the image embedding vector; the image encoder used is ResNet50, which consists of four residual structures. It can also be Vision Transformer, which consists of 12 ViT units cascaded. The formula is expressed as:

[0056] v(x)=f img (x),#(20)

[0057] Among them, f img represents the image encoder and text encoder, x represents the input image, and v(x) represents the embedding vector of image x.

[0058] Step 5: Input the prompt embedding vector and the category text description into the text encoder to obtain the text encoding embedding vector. The text encoder used is TextTransformer, which is composed of 12 cascaded Bert units. The formula is:

[0059]

[0060] Among them, f img and f txtdenote image encoder and text encoder respectively, x denotes input image, and h(·) denotes the mapping function that embeds text into a vector. Represents the k-th text embedding vector of category y.

[0061] Step 6: Use the contrastive learning method to select appropriate loss functions and regularization terms to train the image encoder and prompt distribution generator.

[0062] Loss function:

[0063]

[0064] Where y represents the label of x. y In practice, it is usually set to the inverse of the class prior probability, and τ is an additional free parameter that can be used to adjust π y The offset of f(x) is a vector, and the y'th component can be expressed as:

[0065]

[0066] Regularization term:

[0067]

[0068] in, Represents the KL divergence. The purpose of this regularization term is to make the generated prompt distribution a standard normal distribution, so as to ensure the diversity of the sampled prompts.

[0069] Step 7: For the test data x, first input it into the image encoder:

[0070] v(x)=f img (x),#(25)

[0071] Second, for each category y', K hints are sampled from the corresponding hint distribution generated by the hint distribution generator Input to the text encoder to get K text embedding vectors, where the kth text embedding is expressed as:

[0072]

[0073] Then, the K text embedding vectors are averaged and compared with v(x) for similarity:

[0074]

[0075] Finally, the category y of x x Judged as:

[0076] y x = argmaxy' f y' (x)#(28)。

Claims

1. A visual-language model prompt learning framework for long-tail distribution, characterized in that: The following steps are involved: Step 1: Obtain a large visual-language model and its pre-trained model parameter file, prepare a target long-tail distribution dataset and pre-process it; the large visual-language model includes an image encoder, a text encoder and a prompt distribution generator; Step 2: Sample text descriptions of images and categories in the dataset, generate a prompt distribution through the prompt distribution generator, and sample a prompt set from the distribution to obtain a set of prompt embedding vectors; Step 3: Input the sampled image into the image encoder to obtain the image embedding vector; Step 4: Input the prompt embedding vector and the category text description into the text encoder to obtain the text encoding embedding vector; Step 5: Use contrastive learning to select appropriate loss functions and regularization terms to train the image encoder and prompt distribution generator; Step 6: During testing, input the test image into the image encoder to obtain the image embedding vector; input the prompt set and category description generated by the prompt distribution generator corresponding to different categories into the text encoder to obtain the text embedding vector, and obtain the category of the test image by comparison.

2. The long-tail distribution-oriented visual-language model prompt learning framework according to claim 1, characterized in that: The visual-language model used in step 1 is Contrastive Language Image Pre-training, which is pre-trained using the WIT dataset; the datasets used for downstream tasks are CIFAR10-LT, Places-LT, and ImageNet-LT.

3. The long-tail distribution-oriented visual-language model prompt learning framework according to claim 1, characterized in that: The prompt distribution generator described in step 2 is a learnable module and is divided into three categories: category-general prompt distribution generator, category-specific prompt distribution generator, and conditional category-specific prompt distribution generator, expressed as: Category General Prompt Generator: Category specific prompt distribution generator: Conditional category specific prompt distribution generator: Among them, G agn ,G spec ,G condi They represent three types of prompt distribution generators, θ represents the learnable parameters in the prompt distribution generator, y represents the category, C represents the number of categories, w y Represents the learnable conditional parameter vector of category y; TEXT y Represents the text description of category y; represents the normal distribution, μ and Σ represent the mean and variance of the normal distribution. During training or testing, first select the prompt distribution generator of a specified week category, then generate a prompt distribution from the prompt distribution generator, and finally sample K prompts from the prompt distribution. Represents the k-th hint vector for category y.

4. The long-tail distribution-oriented visual-language model prompt learning framework according to claim 1, characterized in that: The default image encoder used in step 3 is ResNet50, which consists of four residual structures; the image encoder is VisionTransformer, which consists of 12 ViT units cascaded.

5. The long-tail distribution-oriented visual-language model prompt learning framework according to claim 1, characterized in that: The text encoder used in step 4 is Text Transformer, which is composed of 12 cascaded Bert units.

6. The long-tail distribution-oriented visual-language model prompt learning framework according to claim 1, characterized in that: The mathematical formulas for the loss function and regularization term in step 5 are: Loss function: Among them, y represents the label of x; π y In practice, it is set to the inverse of the class prior probability, and τ will be an additional free parameter used to tune π y offset; f(x) is a vector, and the y'th component can be expressed as: v(x)=f img (x),#(6) Among them, f img and f txt denote image encoder and text encoder respectively, x denotes input image, and h(·) denotes the mapping function for embedding text into vector; Regularization term: in, represents the KL divergence; this regularization term unifies the generated prompt distribution into a standard normal distribution, thereby ensuring the diversity of the sampled prompts.

Citation Information

Cited By

  • Method for finely adjusting image classification model and computing equipment

    CN120259772A

  • A method and a computing device for fine-tuning an image classification model

    CN120259772B

  • Long-tail image recognition method based on multi-modal semantic generation and image-text fusion

    CN120495814A