Zero-shot image recognition method based on adaptive text prompt

Abstract

Although a large-scale pre-training model is excellent in performance in the fields of natural language processing and computer vision, the large-scale pre-training model is limited in performance in specific downstream tasks such as zero sample image recognition and the like and needs to be optimized urgently. The invention discloses a prefix and suffix self-adaptive text prompting method based on a data set label. The performance of a model in zero sample recognition is improved. Firstly, learnable prefix and suffix vectors are introduced before and after data labels, text prompts are dynamically adjusted, the semantic understanding ability of a large model is effectively utilized, and then the adaptability and generalization ability of the model to new categories are improved. Secondly, a self-attention mechanism is used for extracting and fusing data set label features, so that a more effective self-adaptive text prompt is generated; experimental results show that the method remarkably improves the zero sample recognition capability of the model, and particularly provides powerful support for further development and application of a large-scale pre-training model when being applied to complex visual tasks.

Description

Technical Field

[0001] The present invention relates to the field of efficient fine-tuning of parameters of large-scale pre-trained models, and mainly revolves around adaptive text prompts. By utilizing the learnability of prefix and suffix vectors, the model can automatically optimize text prompts according to specific downstream tasks, and dynamically adjust text prompts by analyzing the label features of a data set, effectively integrating the label information in the data set and improving the model's responsiveness to specific tasks. Background Art

[0002] Background technology involved in the present invention:

[0003] (1) Large-scale pre-trained models: The emergence of large-scale pre-trained models is one of the important research advances in the field of deep learning. Early pre-trained models mainly focused on word embedding methods, which learned vector representations of vocabulary by pre-training on a large amount of text data. These vectors can capture the semantic relationship between words. Subsequently, the concept of contextual word embedding was proposed, which enables the same word to have different representations in different contexts. The BERT model uses a bidirectional Transformer architecture and is pre-trained on a large amount of unlabeled text. It can better understand the bidirectional context of language and greatly improve the performance of multiple NLP tasks. With the improvement of computing power and the increase in data volume, subsequent research has pointed to larger-scale pre-trained models, such as GPT-2, GPT-3 and T5, and explored multimodal pre-trained models, such as CLIP and DALL-E. These models have learned rich language or visual features by pre-training on a large amount of data sets, and have demonstrated excellent capabilities in various understanding and generation tasks. Compared with previous methods, large-scale pre-trained models can not only understand more complex patterns, but also have better generalization and adaptability.

[0004] (2) Fine-tuning methods based on text prompts: Text prompts play a vital role in large models. They are the bridge between visual information and language descriptions, enabling models to learn and understand image content without direct supervision. Text prompts are usually natural language descriptions that are used as input together with images. Models understand visual concepts by learning the relationship between images and their corresponding text descriptions. At present, text prompt methods can be divided into three categories, including prompt engineering, automatic prompt generation, and multi-prompt fusion. Prompt Engineering: This is a method of manually designing prompts to improve model performance. Researchers try different text prompt forms and structures to find the best prompt to guide the model to better understand and perform specific tasks. Automatic prompt generation: In order to overcome the limitations of manually designed prompts, some research works focus on developing algorithms to automatically generate text prompts. These methods use the model's own knowledge to create customized prompts that can maximize model performance. Multi-prompt fusion: Another method is to use multiple text prompts simultaneously and then fuse their outputs to improve the accuracy of the task. This method believes that different prompts may capture different aspects of the image, and their combined use can improve the generalization ability of the model.

[0005] (3) Image prompt-based tuning methods: Compared with text prompts, image prompts are a technology used to assist models in understanding and processing visual information. They are usually used as part of the model input to improve the model's ability to recognize and understand image content. In some scenarios, image prompts may include labels, borders, color codes, etc., which are used to guide the model to pay attention to specific parts or attributes in the image. Common image prompt methods are mainly divided into three categories: data augmentation, attention mechanism, and generative adversarial network. Data augmentation methods can generate diverse image prompts by rotating, cropping, adjusting colors, etc., and enhance the model's ability to adapt to changes; attention mechanism methods introduce attention mechanisms to enable the model to automatically focus on important areas or objects in the image. This method does not require explicit image prompts, but can be regarded as an implicit prompt optimization technology; generative adversarial networks (GANs) use GANs to generate images containing specific features or objects as prompts to train models. This method is particularly suitable for situations where data is scarce. Summary of the invention

[0006] Main issues:

[0007] (1) Limitations of static text prompts: Traditional text prompt methods usually rely on fixed templates or predefined texts. Such static methods show obvious limitations in dealing with dynamically changing zero-shot image recognition tasks. Static text prompts lack the ability to flexibly adapt to different image contents and cannot effectively cope with the diversity and complexity of images, which limits the model's recognition ability when dealing with novel or rare image categories.

[0008] (2) Limitations of learnable textual cues: Although learnable textual cues provide a certain degree of adaptability, allowing the model to adjust the textual cues according to task requirements, these methods often rely on simple text vectors and fail to deeply explore and utilize the complex semantic relationship between images and text. In addition, this method may not be fully trained in practical applications due to the lack of sufficient training data, resulting in the model still facing generalization and adaptability issues in diverse and complex practical scenarios.

[0009] Technical solution:

[0010] The present invention proposes a tag-based adaptive context prompt method (TACP) based on tags of specific datasets. In the process of customizing prompt templates for different image tags, TACP adopts a learnable mechanism to construct a flexible text prompt structure by inserting adaptively adjustable prefix and suffix vectors before and after the data tags. These prefix and suffix vectors are learned end-to-end from specific data as the model is trained to accurately capture contextual information related to specific image tags, while reducing the risk of overfitting and the burden of manually adjusting text prompts by freezing the core parameters of the pre-trained model. TACP aims to integrate the specific label information of the dataset into the text prompt generation process and automatically adjust the prefix and suffix vectors to more accurately reflect the characteristics of each label.

[0011] The design of the TACP method draws on the encoder structure of the Transformer model. Using the self-attention mechanism, TACP accurately extracts dataset label information and generates customized text prompts that adapt to different datasets. These prompts contain learnable prefix and suffix vectors. The implementation process of TACP is divided into four main stages: Label Embedding, Self-Attention Layer, Feed-Forward Neural Network (FFN), and Vector Mapping. In the label embedding stage, complex category names are converted into vector representations to capture the semantic meaning of the labels.

[0012] First, the labels need to be tokenized to split the text sequence into words or finer-grained tokens. For example, in Caltech101, "Siamese cat" is tokenized into "Siamese" and "cat", and in Flowers102, "rose shrub" is decomposed into "rose" and "shrub". Next, a vocabulary is built, with each unique word or token corresponding to a unique index. Here, the settings in the Transformer model are used. Then, the embedding vector of each word is found in the embedding matrix using the index of the word. The embedding matrix is ​​a trainable parameter matrix, and each row corresponds to the embedding vector of a word in the vocabulary. Subsequently, the embedding layer will return the embedding vector of each word. In order to be consistent with the text prompt input of the CLIP model, the dimension of the embedding vector is uniformly set to 512, that is, each word is converted into a vector with 512 values. Next, the self-attention layer captures the relationship between labels by calculating the similarity between label embeddings. The self-attention layer is mainly used to calculate the representation of the input dataset label, so that each element can utilize the global information in the sequence. This mechanism allows the model to directly transfer information between different positions, thereby capturing complex internal dependencies. The self-attention layer is implemented by calculating attention scores for all pairs of elements in the input, which reflect the importance of each element in the sequence to the current element. Specifically, the self-attention layer uses the query (Q), key (K), and value (V) vectors generated by three sets of linear transformations to calculate the attention scores. Subsequently, the feedforward neural network further processes and optimizes these relationships. The role of the feedforward neural network is to provide independent nonlinear processing for each input position, and to map and remap the data to high dimensions through a series of fully connected layers. This structure allows the model to improve its ability to process and understand sequence data by increasing depth and complexity while maintaining the parallelism of information processing. Finally, the vector mapping stage generates the final prefix and suffix vectors for text prompt generation. Two fully connected layers are used to convert the features extracted by the feedforward network into fixed-length vectors, thereby unifying the length of the prefix and suffix vectors. The purpose of this is to ensure that the information extracted from the entire dataset labels is retained, and that text inputs of different lengths can generate prefix and suffix vectors of consistent dimensions.

[0013] Experimental results:

[0014] (1) A flexible text hint structure is constructed by inserting adaptive prefix and suffix vectors before and after the data labels. These prefix and suffix vectors are learned end-to-end from the specific data as the model is trained to accurately capture the contextual information related to a specific image label.

[0015] (2) The TACP method integrates the specific label information in the dataset into the text prompt generation process and dynamically adjusts the prefix and suffix vectors, ensuring a high degree of consistency between the text prompt and the label, thereby improving the recognition accuracy of the model when dealing with label-related tasks.

[0016] In order to comprehensively evaluate the ability of the proposed text prompt method based on learnable context in zero-shot image recognition, four standard datasets with different characteristics and challenges were selected: CIFAR-10, CIFAR-100, DTD and UCF101, and the results in Tables 1 and 3 were finally obtained. These datasets cover various scenes from static images to dynamic videos, and they are different not only in the complexity of the images, but also in the difficulty of zero-shot learning. They can comprehensively test the adaptability and performance of the method in different scenes.

[0017]

[0018] Table 1 Experimental results of different methods of learnable text prompts on four datasets

[0019]

[0020] Table 2 Experimental results of dataset label mapping methods on four datasets

[0021]

[0022] Table 3 Experimental results of different types of text prompts on four datasets BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a prompt engineering schematic diagram;

[0024] Figure 2 It is a schematic diagram of a method for learning text prompts with prefixes and suffixes;

[0025] Figure 3 It is a schematic diagram of the adaptive text prompt method based on dataset labels. DETAILED DESCRIPTION

[0026] The flexibility and adaptability of text prompts are enhanced by sharing a set of learnable prefix and suffix prompt vectors across all categories. The design of text prompts adopts a structured form, [Label] is used to represent a specific category, usually a specific word or phrase, to indicate the target category that the model is to recognize. For example, if the model is to recognize "dog" in a picture, then the label is the embedding vector of the word "dog". [V]1[V]2...[V] M and [V]1[V]2...[V]N Represents the continuous vector sequence in the prefix and suffix respectively. These three parts are combined to form the final prompt tp, which is used to guide the model to match images and texts. Each vector is in the same dimensional space as the word embedding (in the CLIP model, this dimension is set to 512), while M and N represent the specific number of vectors in the prefix and suffix respectively.

[0027] tp=[V]1[V]2...[V] M [Label][V]1[V]2...[V] N

[0028] Through this structure, the prompt tp is input into the text encoder Et() to generate the classification weight vector corresponding to the visual concept, which still comes from the position of the [EOS] tag in the CLIP model. Therefore, the predicted probability of each image can be calculated according to the following formula:

[0029]

[0030] Where y represents the predicted category of image x, i is the specific category index, and K is the total number of categories. i The category labels in are replaced by the corresponding word embedding vectors of the i-th category name in the target dataset. cos(Et(tpi)),Ei(f)) calculates the cosine similarity between the text feature vector Et(tpi) and the image feature vector Ei(f). τ is the temperature parameter learned by the model, which is used to adjust the "smoothness" of the similarity distribution. During the training process, optimization is performed by minimizing the standard classification loss function based on cross entropy, while allowing the gradient to backpropagate through the text encoder Et(). This not only utilizes the deep knowledge encoded in the model parameters to optimize the context, but the design of the continuous vector also makes it possible to fully explore the word embedding space, which is conducive to better learning task-related context information.

[0031] The self-attention layer is mainly used to calculate the representation of the input dataset label so that each element can utilize the global information in the sequence. This mechanism allows the model to directly transfer information between different positions, thereby capturing complex internal dependencies. The self-attention layer is implemented by calculating the attention scores of all pairs of elements in the input. These scores reflect the importance of each element in the sequence to the current element. Specifically, the self-attention layer uses the query (Q), key (K), and value (V) vectors generated by three sets of linear transformations to calculate the attention score, which is expressed as follows:

[0032]

[0033] Where QKT represents the dot product of query and key, which is used to measure the similarity between elements. Represents a scaling factor, which is used to reduce the variance of the calculation. The Softmax function ensures that the obtained weights are positive and sum to 1, so that the output is the weighted sum of the values ​​(V), thereby fully integrating the information of the entire input sequence. At the same time, in order to enable the self-attention mechanism to simultaneously focus on information from different representation subspaces, a multi-head attention mechanism is introduced. The above process will be replicated multiple times (once for each "head"), and each head uses a different linear transformation matrix to generate Q, K, and V vectors. Subsequently, the outputs generated by each head are concatenated and processed through an additional linear transformation layer to produce the final output vector. The self-attention layer enables the model to capture complex input dependencies and greatly improves the ability to handle long-distance dependencies.

[0034] A feedforward neural network usually consists of two linear transformations with an activation function inserted in between. Its basic form can be expressed as:

[0035] FFN(x)=(max(0,xW1+b1))W2+b2

[0036] Among them, x is the input vector, W1 and W2 are weight matrices, b1 and b2 are bias terms, and the activation function uses ReLU, which is defined as max(0,z). The process is mainly divided into three steps. First, a linear transformation (i.e., xW1+b1) is applied to the input x, which maps the input vector to a usually higher-dimensional space for nonlinear transformation. Subsequently, the ReLU activation function is applied. This step is the key to introducing nonlinearity. Without this step, no matter how many layers the network has, it is ultimately equivalent to a linear model. Finally, the data processed by the activation function is again transformed through a linear transformation (i.e., (max(0,xW1+b1))W2+b2) to map the data from the high-dimensional space back to a lower dimension, close to the original input dimension.

[0037] Residual connections and normalization are required when processing self-attention layers and feedforward neural networks. Residual connections add the input itself to each output of these two layers, which helps alleviate the gradient vanishing problem in deep networks and keeps information flowing in the deep layers of the network. Normalization normalizes the output of each layer, stabilizes the training process and accelerates convergence. These mechanisms together optimize the learning efficiency and expression ability of the network.

[0038] Two fully connected layers are used to convert the features extracted by the feedforward network into fixed-length vectors, thereby unifying the lengths of the prefix and suffix vectors. This is done to ensure that the information extracted from all dataset labels is retained, while also ensuring that text inputs of different lengths can generate prefix and suffix vectors of consistent dimensions. The specific mapping formula is as follows:

[0039] V pre =V suf =FC2(ReLU(FC1(x)))

[0040] Where x represents the input dataset label feature vector, Vpre and Vsu f represent the final generated prefix and suffix vectors respectively.

[0041] The TACP method effectively processes the label features of the dataset by combining the self-attention and feedforward neural network of the Transformer architecture, and further converts them into learnable prefix and suffix vectors to generate adaptive text prompts.

Claims

1. A learnable text prompting method integrating prefix and suffix prompts, characterized in that: In the process of customizing prompt templates for different image labels, this method adopts a learnable mechanism to construct a flexible text prompt structure by inserting adaptively adjustable prefix and suffix vectors before and after the data label; then, the dataset label information is introduced on the basis of the prefix and suffix prompts to further enhance the adaptability of the model, enabling it to generate customized prompt vectors for each category label. These vectors are dynamically adjusted during the training process to accurately adapt to different datasets and task requirements.

2. The learnable text prompting method for prefix and suffix prompting according to claim 1, characterized in that: It optimizes by minimizing the standard classification loss function based on cross entropy, while allowing the gradient to backpropagate through the text encoder, using the encoding in the model parameters to optimize the context. The design of the continuous vector also makes it possible to fully explore the word embedding space, which can better learn task-related contextual information.

3. The learnable text prompting method for prefix and suffix prompting according to claim 1, characterized in that: The specific label information of the dataset is incorporated into the text prompt generation process, and the prefix and suffix vectors are automatically adjusted to more accurately reflect the characteristics of each label.

4. The learnable text prompting method for prefix and suffix prompting according to claim 1, characterized in that: Reduce the risk of overfitting and alleviate the burden of manually tuning text prompts by freezing the core parameters of the pre-trained model.

5. The learnable text prompting method for prefix and suffix prompting according to claim 1, characterized in that: It includes the following four steps: 1) In the label embedding stage, the dataset labels are first converted into embedding vectors; 2) The self-attention layer captures the relationship between labels by calculating the similarity between label embeddings; 3) Feedforward neural networks further process and optimize these relationships; 4) The vector mapping stage generates the final prefix and suffix vectors for text prompt generation.

Images