Image classification method based on adaptive multi-mode prompt learning

Abstract

The invention discloses a self-adaptive multi-mode prompt learning method, which utilizes the potential of a large-scale visual language model, and improves the generalization ability inside and outside a sample through the cooperation of image prompt and text prompt. By masking meaningless blocks in the image and adopting a self-adaptive interactive learning mechanism, mutual cooperation and sharing of multi-modal information are effectively enhanced, so that deep semantic relation between the image and the text can be learned steadily when a small number of samples and unseen categories are processed. Under the condition that large-scale fine adjustment of model parameters is not needed, the advantages of an existing prompt learning method are combined, and an efficient solution is provided for a multi-modal task. A large number of experiments show that the performance of the method on a plurality of reference data sets is superior to that of the existing most advanced method.

Description

Technical Field

[0001] The present invention belongs to the technical field of image classification, and more specifically, relates to an image classification method based on adaptive multimodal cue learning. Background Art

[0002] While large pre-trained vision-language models such as CLIP have shown great potential for text-image alignment, prompt learning (PL) is often designed to learn different alignments for various downstream tasks. Specifically, prompt learning techniques aim to fine-tune the input data to better align images and text in the shared space defined by the large pre-trained model. In particular, such techniques allow the reuse of pre-trained models without having to tune their large number of parameters and fit different downstream tasks.

[0003] Previous PL methods can be divided into three categories, namely, unimodal PL methods, non-interactive multimodal PL methods, and interactive multimodal PL methods. Specifically, unimodal PL methods design a single prompt to use a large pre-trained model. For example, VPT prompts images effectively use a pre-trained image encoder. Since unimodal PL methods only prompt one modality and ignore prompts from the other modality, non-interactive multimodal PL methods are designed to prompt both the image modality and the text modality. For example, IVLP learns two prompts of image and text to show generalization to known classes on unknown data, which is referred to as in-sample generalization in this paper. However, previous non-interactive multimodal PL methods cannot effectively design prompts for widely unseen classes on unseen data, and thus easily lead to overfitting problems. Recently, interactive multimodal PL methods are designed to learn two prompts and obtain generalization to unseen classes for unseen data, which is referred to as out-of-sample generalization in this paper. For example, MaPLe achieves out-of-sample generalization by promoting the interaction between two prompts. Although multimodal PL methods have been widely used in large-scale project pre-trained models, they still have limitations that need to be addressed.

[0004] First of all, not all patches in an image are useful. For each image, patches that are irrelevant to the image category (referred to as meaningless patches) may have an adverse effect on determining the image category, which is ignored by previous PL methods. In practical applications, meaningless patches in one downstream task (or image category) may be useful for other downstream tasks (or image categories), so dealing with meaningless patches is necessary and challenging.

[0005] Secondly, the second problem is that previous PL methods did not consider achieving in-sample generalization and out-of-sample generalization. Specifically, downstream tasks include classes or categories seen during training, and classes not seen during training. Generally, previous PL methods that are robust to both types of generalization are considered to be robust class transfer. In the literature, unimodal and non-interactive multimodal PL methods attempt to improve in-sample generalization, but are prone to overfitting problems. In contrast, interactive multimodal PL methods focus on achieving out-of-sample generalization while ignoring in-sample generalization. Therefore, previous fast learning methods are not robust class transfer. Summary of the invention

[0006] The present invention aims to overcome the shortcomings of the prior art and propose a new interactive multimodal learning method, namely adaptive multimodal prompt learning (AMMPL). The AMMPL proposed in the present invention includes three modules, namely text prompt learning, image prompt learning and adaptive interactive learning. Specifically, we follow coop to generate text representation and perform text prompt learning. The image prompt learning method proposed in the present invention first learns a probability matrix and then uses Bernoulli sampling to detect and mask meaningless patches of each category. The mask image filled with learnable parameters and text information is fed into the image encoder. Therefore, it solves the first problem by reasonably processing the meaningless patches in the image. In addition, the probability matrix sets meaningful patches with a large probability and sets meaningless patches with a small probability to improve the sample generalization ability. In addition, the probability matrix sets meaningful patches with a large probability and sets meaningless patches with a small probability to improve the sample generalization ability. Bernoulli sampling makes it possible for larger values ​​in the probability matrix to have a smaller chance of being selected, and this randomness improves the out-of-sample generalization ability. Therefore, our image prompt learning solves the second problem. In addition, our adaptive interactive learning performs information interaction between the two modes. Specifically, the lightweight network learns textual cues by propagating image information, which improves the effectiveness of textual cues learning.

[0007] Compared with previous methods, the main contributions of our method are two-fold. First, we propose a novel image cue learning method to address two issues in previous PL methods. To the best of our knowledge, this is the first work to explore the impact of meaningless patches on image cue learning. Second, we investigate two lightweight networks (i.e., two fully connected layers) for semantic and dimensional transformation of modal information to promote effective cross-modal interaction. Therefore, they enhance each other. In addition, the text encoder-based network enhances the image cue learning function and addresses the problems of previous PL methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1It is a flowchart of a specific implementation of the image classification method based on adaptive multimodal prompt learning of the present invention;

[0009] Figure 2 It is a framework diagram of the adaptive multimodal prompt learning module in the present invention;

[0010] Figure 3 4 is a flowchart of image prompt learning in this embodiment. DETAILED DESCRIPTION

[0011] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0012] Example

[0013] Figure 1 FIG. 1 is a flowchart of a specific implementation of the image classification method based on adaptive multi-modality of the present invention. Figure 1 As shown, the specific steps of the image classification method based on adaptive multimodal prompt learning of the present invention include:

[0014] S101: Building a text prompt learning module:

[0015] In this paper, text hint learning is first used to generate text representation. Specifically, referring to the existing CoCoOp method, text hint learning is performed by representing each context word as a learnable vector. Specifically, we first use a learnable vector with the same length as the word embedding Each context token is represented by l = 512 in CLIP, and then the fixed context “aphoto of a” is replaced by M learnable vectors to learn the context. Therefore, the text prompt of the i-th class is represented as where c i Represents the name of the i-th class, and the text hint tensor for all classes is:

[0016]

[0017] Where k is the class number. We further input the text prompt T into the text encoder TextEn(·) to obtain the text representation, and then input the text representation into the text projection function TextProj(·) to obtain the final text representation Z in the following way:

[0018]

[0019] Where d represents the dimension of text representation. During training, the parameters of TextEn(·) and TextProj(·) are frozen, while the parameters in T are adaptively adjusted and can be flexibly fixed for various tasks.

[0020] After learning text hints through the above formula, the text hint T learns the specific context of a single class, enabling the text encoder to extract fine-grained text representations for each class or category.

[0021] S102: Constructing image prompt learning module:

[0022] In order to process meaningless image blocks in images and achieve in-sample and out-of-sample generalization, the present invention proposes a new image cue learning method. Specifically, the method includes the following steps: Figure 3 The two steps are shown.

[0023] S201: Image block masking:

[0024] First, the input image is divided into b×b small blocks, and the size of each image block is determined by the actual application. The purpose of this is to subdivide the image into smaller units so that these blocks can be processed independently in subsequent steps. In order to identify whether each image block belongs to a meaningless block, a probability matrix is ​​generated for each category Each element P i Represents the probability that the image block belongs to a meaningless block. The probability matrix of all categories can be represented as a three-dimensional tensor Where k is the total number of categories. Through this probability matrix, each category can be flexibly masked.

[0025] Next, Bernoulli sampling is performed on the probability matrix of each category to generate the masking matrix (i.e., Bernoulli masking matrix), which is calculated as follows:

[0026]

[0027] Among them, r represents the probability of being sampled as 1, the Clamp(·,0,1) function limits each element in the probability matrix to between 0 and 1, and j and c represent the row and column coordinates in the matrix respectively. If the element in the masking matrix is ​​0, it means that the corresponding image block is a meaningless block and needs to be masked in subsequent steps.

[0028] Using the generated Bernoulli masking matrix The masking operation is performed on the input image I. First, the masking matrix Perform element-wise Hadamard product with the original image I to obtain a new image representation:

[0029]

[0030] Where u represents the number of channels of the image (for example, u = 3 for RGB images). If an element in the masking matrix is ​​0, the image block at that position is masked and becomes 0. After this step, the meaningless blocks in the image are successfully masked, thereby reducing the interference of these blocks on model learning.

[0031] S202: Image block filling:

[0032] After masking the meaningless image blocks, it is necessary to fill these masked blocks to reduce the pixel value difference between them and the meaningful image blocks, thereby improving the learning effect of the model. The present invention adopts a filling method based on learnable parameters to enhance the information richness of the masked image blocks. Specifically, for each category, a fillable parameter matrix is ​​learned Where q×q represents the size of the image patch and u represents the number of channels of the image. In this way, each category can obtain a set of specific filling parameters. The filling process applies these parameters to the masked image patch through a broadcast mechanism to form a padded image representation, as follows:

[0033]

[0034] in, represents the masked image block, N represents the padding parameter for all categories, represents a broadcast addition operation. Through the above operation, each masked image block is replaced by the corresponding padding parameter, which contains task-related information. In addition, the padded image It will be input into the image encoder and the final image representation X will be generated through the image projection function (ImageProj):

[0035]

[0036] In this process, the parameters of the image encoder (ImageEn) and the image projection function (ImageProj) remain unchanged, while the padding parameter N is dynamically adjusted according to the model's loss. This padding method not only alleviates the pixel value difference between the masked block and other image blocks, but also optimizes the padding parameter N so that the padded image blocks can better adapt to different tasks and input data, thereby improving the overall performance of the model.

[0037] S103: Adaptive interactive learning:

[0038] In order to establish an effective connection between image and text prompts and thus enhance the learning effect of both, the present invention introduces an adaptive interactive learning module. The goal of this module is to use a lightweight network to achieve information interaction between images and texts, and further improve the generalization ability. Specifically, the text representation Z and the image representation X are first processed by the lightweight network to generate interactive information. For the i-th category, its text representation z i and image representation x i Input into two lightweight networks respectively to generate information for interaction: and in and Lightweight networks for representing text and images respectively. E i The dimension is Used to update image prompts; h i is a scalar used to update the text prompt. Next, the interaction information is fed into the text and image prompts. First, each context vector in the text prompt is updated:

[0039]

[0040] The formula represents that by i is added to each context vector of the text prompt, so that the text prompt can incorporate information from the image representation. Then, the generated interaction information E i To update the padding parameters in the image hint:

[0041] N i =N i +E i ,

[0042] This operation is done by i The filling parameters of the image prompt are added to enhance the filling effect of the image block using the information of the text prompt. Through this two-way interactive learning, both the text prompt and the image prompt can use the auxiliary information of the other to improve their own representation, thereby achieving better in-sample and out-of-sample generalization capabilities.

[0043] S104: Set the number of iterations t=1.

[0044] S105: Generate image representation:

[0045] The input image is divided into multiple image blocks, and the current probability matrix P is used for Bernoulli sampling to generate the masking matrix M. Then, the new image representation is obtained through the masking operation Next, use the image filling parameter N to Fill the masked image block in the image to get the filled image Will Input into the image encoder (ImageEn) and obtain the image representation X through the image projection function (ImageProj).

[0046] S106: Adaptive interactive learning:

[0047] In each training batch, a lightweight network is used to interactively learn the text representation Z and the image representation X to generate mutual information. This generated mutual information is used to update the context vector in the text prompt T and the padding parameter N in the image prompt, thereby further improving the effect of text prompts and image prompts.

[0048] S107: Training model:

[0049] The predicted probability is calculated by the cosine similarity between the text representation Z and the image representation X:

[0050]

[0051] Among them, sim(·,·) represents the cosine similarity function, τ is the temperature parameter, y is the predicted category, and K is the total number of categories.

[0052] The cross entropy loss function is used to calculate the prediction loss of the model. Due to the discontinuity of Bernoulli sampling in the image prompt, standard back propagation cannot be directly used to update the gradient of the probability matrix P. To solve this problem, the present invention proposes an optimization method to reconstruct the calculation graph of the Bernoulli masking matrix M:

[0053] M=Detach(MP)+P,

[0054] In this way, the gradient of the discrete masking matrix can be transferred to the continuous probability matrix P, thereby achieving effective optimization of P.

[0055] S108 determines whether t<t max , t max Indicates the predicted maximum number of iterations. If yes, go to step S109, otherwise go to step S110.

[0056] S109: Let t=t+1, and use the gradient obtained by back propagation to update the context vector T in the text prompt, the probability matrix P in the image prompt, and the filling parameter N. Return to step S105.

[0057] S110: Use the trained model to perform image classification.

[0058] In order to better illustrate the technical effect of the present invention, the present invention is experimentally verified by using specific examples. In this experimental verification, its performance on in-sample generalization (Few-Shot Learning) is evaluated in a variety of downstream tasks. Specifically, a set of experiments was designed with different numbers of samples as training data and compared with existing prompt learning methods (CoCoOp and MaPLe). Five benchmark datasets were selected in the experiment, including two fine-grained datasets (Food101, FGVCAircraft), one general object dataset (Caltech101), one action recognition dataset (UCF101), and one scene recognition dataset (Sun397).

[0059] In order to evaluate the in-sample generalization performance, two different numbers of training samples, 1-shot and 8-shot, were used. Here, "shot" refers to the number of samples used in the training set for each category. For example, 1-shot means that only 1 sample is used for training for each category. To ensure the reliability of the results, all experiments were conducted on NVIDIA TeslaV100S GPUs (32GB each), and each setting was run 3 times independently, and the average was taken as the final result. The model initialization and training process follows the standard prompt learning framework, and all methods are fine-tuned on the pre-trained CLIP model (ViT-B / 16) to ensure fairness in the comparison.

[0060] Table 1 is a statistical table of classification accuracy of the present invention and the comparative method under different shot numbers in all data sets in this embodiment.

[0061]

[0062] Table 1

[0063] As shown in Table 1, it can be seen from the results in Table 1 that the method proposed in the present invention achieves the best performance. Our method outperforms the interactive multimodal methods (i.e., CoCoOp and MaPLe) by a large margin.

[0064] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. An image classification method based on adaptive multimodal cue learning, characterized in that: The following steps are involved: S1: Constructing a text prompt learning module: Specifically, referring to the existing CoCoOp method, text hint learning is performed by representing each context word as a learnable vector. The text hint T learns the specific context of a single class, enabling the text encoder to extract fine-grained text representation for each class or category. S2: Building an image prompt learning module: In order to process meaningless image blocks in images and achieve in-sample and out-of-sample generalization, the present invention proposes a new image cue learning method. The method first divides the image into multiple patches, then processes the meaningless patches, and then feeds the image representation containing the meaningless patches and the meaningful patches back to the image encoder. At the same time, it is also expected to obtain in-sample generalization and out-of-sample generalization. S3: Building an adaptive multimodal cue learning module: In order to provide auxiliary information between text and image prompts, the present invention introduces an adaptive interaction learning module. Given a text representation Z and an image representation X, two lightweight networks are used to process the text representation and image representation of the i-th category respectively, obtain the interaction information, and then input the interaction information into the text and image prompts, and update the context and filling parameters of each category to improve the adaptability of the model to different tasks. S4: Set the number of iterations t=1. S5: Divide the input image into multiple image blocks, and use the current probability matrix P to perform Bernoulli sampling to generate a masking matrix M. Then, the new image representation is obtained through the masking operation Next, use the image filling parameter N to Fill the masked image block in the image to get the filled image Will Input into the image encoder (ImageEn) and obtain the image representation X through the image projection function (ImageProj). S6: In each training batch, a lightweight network is used to interactively learn the text representation Z and the image representation X to generate interactive information. The generated interactive information is used to update the context vector in the text prompt T and the padding parameter N in the image prompt, thereby further improving the effect of the text prompt and the image prompt. S7: Training target model: The predicted probability is calculated by the cosine similarity between the text representation Z and the image representation X: Among them, sim(·,·) represents the cosine similarity function, τ is the temperature parameter, y is the predicted category, and K is the total number of categories. The cross entropy loss function is used to calculate the prediction loss of the model. Due to the discontinuity of Bernoulli sampling in the image prompt, standard back propagation cannot be directly used to update the gradient of the probability matrix P. To solve this problem, the present invention proposes an optimization method to reconstruct the calculation graph of the Bernoulli masking matrix M: M=Detach(MP)+P, In this way, the gradient of the discrete masking matrix can be transferred to the continuous probability matrix P, thereby achieving effective optimization of P. S8 determines whether t<t max , t max Indicates the predicted maximum number of iterations. If yes, go to step S109, otherwise go to step S110. S9: Let t=t+1, and use the gradient obtained by backpropagation to update the context vector T in the text prompt, the probability matrix P in the image prompt, and the filling parameter N. Return to step S105. S10: Use the trained model to perform image classification.

2. The image classification method according to claim 1, characterized in that: The image prompt learning in step S2 is performed using the following method: First, the input image is divided into b×b small blocks, and the size of each image block is determined by the actual application. The purpose of this is to subdivide the image into smaller units so that these blocks can be processed independently in subsequent steps. In order to identify whether each image block belongs to a meaningless block, a probability matrix is ​​generated for each category Each element P i Represents the probability that the image block belongs to a meaningless block. The probability matrix of all categories can be represented as a three-dimensional tensor Where k is the total number of categories. Through this probability matrix, each category can be flexibly masked. Next, Bernoulli sampling is performed on the probability matrix of each category to generate the masking matrix (i.e., Bernoulli masking matrix), which is calculated as follows: Among them, r represents the probability of being sampled as 1, the Clamp(·,0,1) function limits each element in the probability matrix to between 0 and 1, and j and c represent the row and column coordinates in the matrix respectively. If the element in the masking matrix is ​​0, it means that the corresponding image block is a meaningless block and needs to be masked in subsequent steps. Using the generated Bernoulli masking matrix The masking operation is performed on the input image I. First, the masking matrix Perform element-wise Hadamard product with the original image I to obtain a new image representation: Where u represents the number of channels of the image (for example, u = 3 for RGB images). If an element in the masking matrix is ​​0, the image block at that position is masked and becomes 0. After this step, the meaningless blocks in the image are successfully masked, thereby reducing the interference of these blocks on model learning. After masking the meaningless image blocks, it is necessary to fill these masked blocks to reduce the pixel value difference between them and the meaningful image blocks, thereby improving the learning effect of the model. The present invention adopts a filling method based on learnable parameters to enhance the information richness of the masked image blocks. Specifically, for each category, a fillable parameter matrix is ​​learned Where q×q represents the size of the image patch and u represents the number of channels of the image. In this way, each category can obtain a set of specific filling parameters. The filling process applies these parameters to the masked image patch through a broadcast mechanism to form a padded image representation, as follows: in, represents the masked image block, N represents the padding parameter for all categories, represents a broadcast addition operation. Through the above operation, each masked image block is replaced by the corresponding padding parameter, which contains task-related information. In addition, the padded image It will be input into the image encoder and the final image representation X will be generated through the image projection function (ImageProj): In this process, the parameters of the image encoder (ImageEn) and the image projection function (ImageProj) remain unchanged, while the padding parameter N is dynamically adjusted according to the model's loss. This padding method not only alleviates the pixel value difference between the masked block and other image blocks, but also optimizes the padding parameter N so that the padded image blocks can better adapt to different tasks and input data, thereby improving the overall performance of the model.

3. The image classification method according to claim 1, characterized in that: The step S3 constructs the adaptive multimodal prompt learning using the following strategy: First, a lightweight network is used to realize information interaction between images and texts to further improve the generalization ability. Specifically, the text representation Z and image representation X are processed by the lightweight network to generate interactive information. For the i-th category, its text representation z i and image representation x i Input into two lightweight networks respectively to generate information for interaction: and in and Lightweight networks for representing text and images respectively. E i The dimension is Used to update image prompts; h i is a scalar used to update the text prompt. Next, the interaction information is fed into the text and image prompts. First, each context vector in the text prompt is updated: The formula represents that by transforming the mutual information h i is added to each context vector of the text prompt, so that the text prompt can incorporate information from the image representation. Then, the generated interaction information E i To update the padding parameters in the image hint: N i =N i +E i , This operation is done by i The filling parameters of the image prompt are added to enhance the filling effect of the image block using the information of the text prompt. Through this two-way interactive learning, both the text prompt and the image prompt can use the auxiliary information of the other to improve their own representation, thereby achieving better in-sample and out-of-sample generalization capabilities.

Images