Multi-modal prompt learning method based on retrieval enhancement

Abstract

The invention discloses a multi-modal prompt learning method based on retrieval enhancement, which can align a text encoder and a visual encoder of VLMs at the same time, adopts a retrieval enhancement strategy, captures cross-modal information through a cross-attention mechanism, realizes efficient multi-modal interaction, generates an adaptive embedded prompt based on the cross-modal information, and improves the learning efficiency. A multi-modal representation space is flexibly aligned, a learnable vector library (LTL) is introduced, the prompt generation effect is further optimized, excellent performance in various scenes is ensured, the accuracy of self-adaptive prompt can be enhanced through a retrieval enhancement strategy and a cross-modal cooperative sensing technology, efficient interaction of multi-modal information is achieved in combination with the learnable vector library, and the self-adaptive prompt effect is improved. Computing resources needed by fine tuning are reduced, the generalization ability of the model and the processing performance of complex tasks are remarkably improved, and a new solution is provided for efficient adaptation and optimization of large-scale pre-training VLMs.

Description

[Technical field]

[0001] The present invention relates to the technical field of prompt learning, in particular to the technical field of multimodal prompt learning methods based on retrieval enhancement. [Background technology]

[0002] Vision-Language Models (VLMs) have demonstrated excellent performance in multiple downstream tasks. These models project visual and language information into a shared feature space by training image encoders and text encoders simultaneously, thereby capturing the deep semantic associations between the two. For example, CLIP significantly improves the performance of the model in multi-task processing by contrastive learning of 400 million text-image pairs. However, the sensitivity of these models to input cues and the huge computational resource requirements limit their widespread application.

[0003] To solve the above problems, researchers proposed a lightweight method - prompt learning. Prompt learning originated from the field of natural language processing (NLP). By introducing learnable prompt vectors, the model can adapt to different downstream tasks while keeping the parameters of the pre-trained model unchanged. Methods such as CoOp and CoCoOp significantly improved the text embedding effect by using learnable continuous vectors; Visual Prompt (VP) adds a learnable prompt vector at the input of the image encoder to enhance the model's understanding of visual information. However, existing prompt learning methods mainly focus on single modality and fail to give full play to the multimodal synergy advantages of VLMs. [Summary of the invention]

[0004] The purpose of the present invention is to solve the problems in the prior art and propose a multimodal prompt learning method based on retrieval enhancement, which can enhance the accuracy of adaptive prompts through retrieval enhancement strategies and cross-modal collaborative perception technology, and combine with a learnable vector library to achieve efficient interaction of multimodal information, reduce the computing resources required for fine-tuning, significantly improve the generalization ability of the model and the processing performance of complex tasks, and provide a new solution for the efficient adaptation and optimization of large-scale pre-trained VLMs.

[0005] To achieve the above object, the present invention proposes a multimodal prompt learning method based on retrieval enhancement, comprising the following steps:

[0006] S1: Given an input image and a pre-trained visual model, the key features are captured by a lightweight feature extractor (LFE) and then imported into a visual cue generator (VPG). The visual cue generator (VPG) searches for historical data and task-related information through a learnable vector library (LTL), captures and fuses cross-modal information through a cross-attention mechanism to generate visual cue information, adds the visual cue information to the input image to form a visual cue image, and freezes the visual cue image into the visual encoder.

[0007] S2: Given the input text and the pre-trained text model, the text features are encoded and constructed and then imported into the text prompt generator (TPG). The text prompt generator (TPG) searches for historical data and task-related information through the learnable vector library (LTL), captures and fuses cross-modal information through the cross-attention mechanism to generate text prompt information, and freezes the text prompt information into the text encoder;

[0008] S3: Cross-fusing the visual cue image frozen in the visual encoder and the text cue information frozen in the text encoder to generate adaptive cue information.

[0009] Preferably, in step S1, given an input image I i ∈R 3×H×W and the pre-trained visual model Φ V , divide the image into s×s small blocks and get {p1,p2,···,p s×s}, where each small block The small patches are encoded using a linear layer L(·) to construct the visual input; the visual prompt generator (VPG) is implemented by a lightweight neural network g(.) and a learnable vector library (LTL), which converts the image into a feature vector F through a depth-wise separable convolution. i ∈R L×C , L is the feature sequence length, C is the feature dimension, and the learnable vector library (LTL): L V ∈R N×C , which contains N learnable feature vectors, each with dimension C, and these image features are combined with K visual cue vectors generated by the learnable vector library (LTL) through a cross-attention mechanism In this process, the image feature F i As the query vector Q, the visual token library L V As key K and value V, the specific calculation of prompt generation is as follows:

[0010]

[0011] Where g(·) represents the 3×H×W to RK×N Finally, the cue information is added to the input image to form a visual cue image The vector is input into the frozen visual encoder to generate latent visual features. Passed as input to downstream tasks.

[0012] Preferably, in step S2, given the input text T i ∈R L and the pre-trained language model Φ T , where L is the length of the text, the text is divided into K subwords, and {t1, t2, ···, t K}, where each subword t i ∈R d , d is the word embedding dimension, and the embedding layer E(·) is used to encode the subwords to construct the text input; the text prompt generator (TPG) is composed of the category text C i It is implemented in collaboration with the Learnable Vector Library (LTL) to convert the category text C i ∈R L×C Convert to embedding vector T i =E(C i ), where L is the length of the text sequence, C is the embedding dimension, and the embedding vector T i and the learnable vector library L T P i =(T i ; L T )∈R (L+N)×C , through the cross-attention mechanism (Cross-Attention) i and P i Perform text processing to generate the final K text prompt vectors In this process, the text token library L T As the query vector Q, P i As key K and value V, the specific calculation of prompt generation is as follows:

[0013]

[0014] The final generated text hint vector is fused with the original category text embedding vector to obtain the enhanced text feature representation:

[0015]

[0016] Beneficial effects of the present invention: The present invention can simultaneously align the text encoder and visual encoder of VLMs, adopt a retrieval enhancement strategy, capture cross-modal information through a cross-attention mechanism (Cross-Attention), and achieve efficient multimodal interaction. Based on the cross-modal information, adaptive embedding prompts are generated to flexibly align the multimodal representation space. A learnable vector library (LTL) is introduced to further optimize the prompt generation effect and ensure excellent performance in various scenarios. The accuracy of adaptive prompts can be enhanced through retrieval enhancement strategies and cross-modal collaborative perception technology, and combined with a learnable vector library, efficient interaction of multimodal information can be achieved, the computing resources required for fine-tuning can be reduced, and the generalization ability of the model and the processing performance of complex tasks can be significantly improved, providing a new solution for the efficient adaptation and optimization of large-scale pre-trained VLMs.

[0017] The features and advantages of the present invention will be described in detail through embodiments in conjunction with the accompanying drawings.

Brief Description of the Drawings

[0018] Figure 1 It is a general architecture diagram of the retrieval-enhanced multimodal prompt learning method of the present invention;

[0019] Figure 2 This is a comparison chart of the experimental results of a few-shot experiment on 6 different data sets based on the comparative experiment of the multimodal prompt learning method for retrieval enhancement of the present invention;

[0020] Figure 3 This is a comparison chart of learning performance on 6 different data sets based on the comparative experiment of the retrieval-enhanced multimodal prompt learning method of the present invention;

[0021] Figure 4 This is a comparison chart of experimental results of a comparative experiment of the retrieval-enhanced multimodal prompt learning method of the present invention using multiple visual backbone networks on 6 different data sets. [Specific implementation method]

[0022] See also Figure 1 The present invention provides a retrieval-enhanced multimodal prompt learning method, comprising the following steps:

[0023] S1: Given an input image and a pre-trained visual model, the key features are captured by a lightweight feature extractor (LFE) and then imported into a visual cue generator (VPG). The visual cue generator (VPG) searches for historical data and task-related information through a learnable vector library (LTL), captures and fuses cross-modal information through a cross-attention mechanism to generate visual cue information, adds the visual cue information to the input image to form a visual cue image, and freezes the visual cue image into the visual encoder.

[0024] S2: Given the input text and the pre-trained text model, the text features are encoded and constructed and then imported into the text prompt generator (TPG). The text prompt generator (TPG) searches for historical data and task-related information through the learnable vector library (LTL), captures and fuses cross-modal information through the cross-attention mechanism to generate text prompt information, and freezes the text prompt information into the text encoder;

[0025] S3: Cross-fusing the visual cue image frozen in the visual encoder and the text cue information frozen in the text encoder to generate adaptive cue information;

[0026] In step S1, given an input image I i ∈R 3×H×W and the pre-trained visual model Φ V , divide the image into s×s small blocks and get {p1,p2,···,p s×s}, where each small block The small patches are encoded using a linear layer L(·) to construct the visual input; the visual prompt generator (VPG) is implemented by a lightweight neural network g(.) and a learnable vector library (LTL), which converts the image into a feature vector F through a depth-wise separable convolution. i ∈R L×C , L is the feature sequence length, C is the feature dimension, and the learnable vector library (LTL): L V ∈R N×C , which contains N learnable feature vectors, each with a dimension of C. These image features are combined with the learnable vector library (LTL) through a cross-attention mechanism to generate the final K visual cue vectors In this process, the image feature F i As the query vector Q, the visual token library L V As the key and value K, V, the specific calculation of the prompt generation is as follows:

[0027]

[0028] Where g(·) represents the 3×H×W to R K×N Finally, the cue information is added to the input image to form a visual cue image The vector is input into the frozen visual encoder to generate latent visual features. As input to the downstream task, in step S2, given the input text T i ∈R L and the pre-trained language model Φ T , where L is the length of the text, the text is divided into K subwords, and {t1, t2, ···, tK}, where each subword t i ∈R d , d is the word embedding dimension, and the embedding layer E(·) is used to encode the subwords to construct the text input; the text prompt generator (TPG) is composed of the category text C i It is implemented in collaboration with the Learnable Vector Library (LTL) to convert the category text C i ∈R L×C Convert to embedding vector T i =E(C i ), where L is the length of the text sequence, C is the embedding dimension, and the embedding vector T i and the learnable vector library L T P i =(T i ; L T )∈R (L+N)×C , through the cross-attention mechanism (Cross-Attention) i and P i Perform text processing to generate the final K text prompt vectors In this process, the text token library L T As the query vector Q, P i As key K and value V, the specific calculation of prompt generation is as follows:

[0029]

[0030] The final generated text hint vector is fused with the original category text embedding vector to obtain the enhanced text feature representation:

[0031]

[0032] Experiments were conducted on six public image recognition datasets: DTD, OxfordPets, Flowers102, Food101, EuroSAT, and Caltech101. In addition, Food101, OxfordPets, and Caltech101 were selected for ablation experiments.

[0033] In the experiment, all images were resized to 224×224 pixels. The context vector of the retrieval-enhanced multimodal prompt learning method (CRMPrompt) was randomly initialized from a Gaussian distribution with a mean of zero and a standard deviation of 0.02. Stochastic gradient descent (SGD) was used for training, with an initial learning rate of 1 and decayed by a cosine annealing rule. To alleviate the gradient explosion problem in early training, a warm-up technique was used, and the learning rate was fixed to 1e-5 in the first training round. Random flipping and cropping were used for image enhancement. In the Line Probe experiment, the experimental settings were aligned, and all experiments were run on the NVIDIA RTX 3080 GPU.

[0034] Full-Training and few-shot experimental settings are used to evaluate the proposed method. Few-shot experiments are trained using 1, 2, 4, 8, and 16 samples. After training, the model is deployed and evaluated on the full test set. To ensure the stability of the results, each experiment is repeated three times and the average is taken as the final result. ResNet50, ResNet101, ViT-B-16, and ViT-B-32 are used as the backbone networks of the image encoder, and the number of context tokens M is set to 16.

[0035] As shown in Table 1, although the accuracy of few-shot training is slightly lower than that of full-shot training, the performance gap is small and still performs well. Few-shot training has obvious advantages, including saving data annotation costs, reducing training time and computing resources, and improving the adaptability of the model to new tasks and changing data. Therefore, few-shot training has important application value in resource-constrained environments.

[0036] Table 1. Performance comparison between few samples and full samples

[0037]

[0038] First, the proposed CRMPrompt is compared with three classic benchmark methods. The first benchmark method is zero-shot CLIP, which directly classifies from the features of pre-trained data through manual prompt design; the second benchmark method is linear probe CLIP, which trains simple linear classification based on pre-trained features; the third benchmark method is CoOp, which aims to enhance model performance by optimizing contextual prompts. Figure 2 The results of few-shot experiments on 6 different datasets are shown; Figure 3 (a) is the overall average performance of few-shot under 6 different datasets. Figure 3 (b) is the performance comparison of the zero-shot of CRMPrompt of the present invention relative to CLIP, Figure 3(c) is the performance comparison under different backbones. In order to further analyze the performance of the model, the present invention is compared with CLIP, CoOp, CoCOOP, ProGrad, KgCoOp, DAPT, TaskRes, MaPLe, PromptSRC and TCP for a small sample comparison analysis with a sampling number of 4 when the backbone is Vit-B-16, as shown in Table 1.

[0039] See also Figure 2 , where M represents the context length, end and mid indicate placing the class token at the end or in the middle, and CSC represents the class-specific contextual cue.

[0040] like Figure 2 and Figure 3 As shown in (a), the proposed method outperforms other methods on all six datasets. As the number of training samples increases, the performance continues to improve, increasing by about 26.7% compared to Linear Probe CLIP. Figure 3 As shown in (a), on EuroSAT, it is improved by 52.80% compared with manual hints; on Flowers102, it is improved by 31.83%; and on DTD, it is improved by 26.13%. These significant performance improvements fully demonstrate the effectiveness and advantages of CRMPrompt of the present invention.

[0041] Through comparative analysis of Table 2 (backbone is ViT-B-16), the present invention performs excellently on EuroSAT, reaching 80.35%, significantly better than other methods. On DTD, the present invention also maintains leading performance, reaching 60.69%. On Food101, the present invention shows the best performance (86.29%), exceeding the second-performing MaPLe by 5.28 percentage points. At the same time, on the Flowers dataset, the present invention also performs well, reaching 92.18%. This may be due to the complexity and feature diversity of these datasets, which enables the present invention to better capture these features. In addition, the average performance of the present invention on all datasets (85.81%) is better than all other methods, further demonstrating its powerful capabilities and wide applicability in few-sample learning tasks.

[0042] Table 2 Comparison of few-shot learning using 4 samples

[0043]

[0044] like Figure 4As shown in the figure, the present invention significantly outperforms CLIP and CoOp on various visual backbone networks, demonstrating its powerful ability in few-shot learning. Especially on complex datasets, CRMPrompt of the present invention shows strong adaptability and wide applicability. Advanced backbone networks (such as ViT) significantly improve performance, further demonstrating the important influence of backbone network selection on model performance.

[0045] Table 3 shows the ablation experiments of few-shot learning on different datasets (the backbone network is ViT-B-16). The results show that VPG is less effective than TPG. This is mainly because the visual-language model has a deeper understanding of the text and richer semantic information, while the complexity of visual cues in cross-modal learning, data quality, and feature extraction makes it more difficult to optimize. The CRMPrompt of the present invention combines VPG and TPG, and its average performance reaches 95.64%, which is significantly better than single-modal cue learning. This may be because visual cues and text cues are complementary in feature extraction and representation, thus providing richer information in few-shot learning tasks.

[0046] Table 3 Performance comparison of different prompt generators

[0047]

[0048] Table 4 shows the impact of different numbers of prompts on model performance. Experiments show that when the number of prompts increases from 4 to 8, the performance improves significantly; but when the number exceeds 16, the performance improvement is not obvious or even decreases due to redundant information.

[0049] Table 5 shows the effect of different Token Library sizes on the performance of the CRMPrompt of the present invention. A larger TokenLibrary provides more feature vectors, making the prompts richer and more diverse, and adapting to different input data. Experiments show that the performance gradually improves as the TokenLibrary size increases, but after exceeding 256 feature vectors, the performance improvement is no longer significant, which may be affected by repeated or redundant information.

[0050] The above experimental results show that when the number of prompts and feature vectors exceeds a certain range, it will affect the performance improvement or even reduce the performance, so the right amount of prompts is optimal.

[0051]

[0052] Table 6 shows the impact of different Cross-Attention layers on the performance of the CRMPrompt model of the present invention. The data shows that the number of layers has little impact on the performance, but using more attention layers may introduce redundant information, resulting in performance degradation. According to the task requirements, the appropriate number of attention layers can be freely selected to strike a balance between performance and complexity.

[0053] Table 6 Effect of different cross attention layers on model performance

[0054]

[0055] This experiment aims to analyze the impact of different numbers of cross-attention heads on the performance of multimodal prompts (including image prompts VPG and text prompts TPG). Tables 7 and 8 show the experimental results.

[0056] Table 7 Number of VPG heads

[0057]

[0058] Table 8 Number of TPG headers

[0059]

[0060] Experiments show that increasing the number of attention heads can improve model performance to a certain extent, but too many attention heads can lead to performance degradation. This may be due to the increased model complexity and the introduction of redundant information. When choosing the number of attention heads, a balance should be found between performance improvement and computing resources. For VPG heads, 8 heads are the best choice, while for TPG heads, 16 heads perform best, but 8 heads are also close to optimal with low resource consumption.

[0061] The above experiments prove that the design based on conditional cue learning performs well in various scenarios. In addition, by combining the retrieval enhancement method and using the learnable vector library to retrieve historical data and task-specific information, the performance of the model in the few-shot learning environment can be significantly improved. The experimental results show that conditional cue learning not only performs well in diverse tasks, but also reduces the complexity and energy consumption of manual design.

[0062] The above embodiments are intended to illustrate the present invention, not to limit the present invention. Any solution that is a simple transformation of the present invention belongs to the protection scope of the present invention.

Claims

1. A multimodal prompt learning method based on retrieval enhancement, characterized by: The following steps are involved: S1: Given an input image and a pre-trained visual model, the key features are captured by the lightweight feature extractor (LFE) and then imported into the visual prompt generator (VPG). The visual prompt generator (VPG) searches for historical data and task-related information through the learnable vector library (LTL), captures and fuses cross-modal information through the cross-attention mechanism (Cross-Attention) to generate visual prompt information, adds the visual prompt information to the input image to form a visual prompt image, and freezes the visual prompt image into the visual encoder; S2: Given the input text and the pre-trained text model, the text features are encoded and constructed and then imported into the text prompt generator (TPG). The text prompt generator (TPG) searches for historical data and task-related information through the learnable vector library (LTL), captures and fuses cross-modal information through the cross-attention mechanism to generate text prompt information, and freezes the text prompt information into the text encoder; S3: Cross-fusing the visual cue image frozen in the visual encoder and the text cue information frozen in the text encoder to generate adaptive cue information.

2. The multimodal prompt learning method based on retrieval enhancement as claimed in claim 1, characterized in that: In step S1, given an input image I i ∈R 3×H×W and the pre-trained visual model Φ V , divide the image into s×s small blocks and get {p1,p2,···,p s×s }, where each small block The small patches are encoded using a linear layer L(·) to construct the visual input; the visual prompt generator (VPG) is implemented by a lightweight neural network g(.) and a learnable vector library (LTL), which converts the image into a feature vector F through a depth-wise separable convolution. i ∈R L×C , L is the feature sequence length, C is the feature dimension, and the learnable vector library (LTL): L V ∈R N×C , which contains N learnable feature vectors, each with dimension C, and these image features are combined with K visual cue vectors generated by the learnable vector library (LTL) through a cross-attention mechanism In this process, the image feature F i As the query vector Q, the visual token library L V As key K and value V, the specific calculation of prompt generation is as follows: Where g(·) represents the 3×H×W to R K×N Finally, the cue information is added to the input image to form a visual cue image The vector is input into the frozen visual encoder to generate latent visual features. Passed as input to downstream tasks.

3. The multimodal prompt learning method based on retrieval enhancement as claimed in claim 1, characterized in that: In step S2, given the input text T i ∈R L and the pre-trained language model Φ T , where L is the length of the text, the text is divided into K subwords, and {t1, t2, ···, t K }, where each subword t i ∈R d , d is the word embedding dimension, and the embedding layer E(·) is used to encode the subwords to construct the text input; the text prompt generator (TPG) is composed of the category text C i It is implemented in collaboration with the Learnable Vector Library (LTL) to convert the category text C i ∈R L×C Convert to embedding vector T i =E(C i ), where L is the length of the text sequence, C is the embedding dimension, and the embedding vector T i and the learnable vector library L T P i =(T i ; L T )∈R (L+N)×C , through the cross-attention mechanism (Cross-Attention) i and P i Perform text processing to generate the final K text prompt vectors In this process, the text token library L T As the query vector Q, P i As key K and value V, the specific calculation of prompt generation is as follows: The final generated text hint vector is fused with the original category text embedding vector to obtain the enhanced text feature representation:

Images