Multi-modal large model prompt learning method based on class knowledge enhancement

Abstract

The invention discloses a multi-modal large model prompt learning method based on class knowledge enhancement, and belongs to the technical field of prompt learning, and the method comprises the following steps: initializing a group of prompt words for a target task; obtaining word vectors of each category of the target task, and splicing the word vectors to cue words to obtain category specific cue words; designing an interval self-adjusting picture-text matching loss function; sequentially inputting the category specific cue words and the pictures into a pre-trained multi-modal large model CLIP to obtain text features and picture features; inputting the text features and the picture features into an interval self-adjusting picture-text matching loss function to calculate classification loss, performing back propagation on the classification loss, and updating prompt words; and carrying out picture classification through the updated prompt word. According to the method, the problem that the performance of the model on the target task is better and the performance on the unknown new task is poorer in the existing prompt learning method is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of prompt learning, and in particular relates to a multimodal large model prompt learning method based on class knowledge enhancement. Background Art

[0002] With the rise of multimodal large models represented by CLIP, exploring and utilizing the "free" general knowledge in pre-trained models to enhance the learning of downstream target tasks has received widespread attention and has achieved great success in many fields. The multimodal large model prompt learning method uses the training samples in the downstream target task to learn task-specific prompt words - a lightweight network module composed of a small number of learnable parameters, so as to generalize the pre-trained multimodal large model to the target task. Although the model learned by the existing method can achieve good generalization performance on the target task, the model is prone to overfitting to the target task and cannot be successfully generalized to unknown new tasks. To solve this problem, some researchers proposed to use image knowledge to constrain the prompt word update process, and to enhance the learning of class-independent prompt words during the model training process to improve the generalization level of the learned model on unknown new tasks. Although this strategy can improve the performance of the learned model on unknown new tasks to a certain extent, it is easy to cause the performance of the model on the target task to decline.

[0003] This means that the models learned by existing prompt learning methods face a trade-off between target task and unknown new task performance: the better the model performs on the target task, the worse it performs on unknown new tasks, and vice versa. This problem greatly limits the development of current multimodal large model prompt learning methods. Summary of the invention

[0004] The present invention provides a multimodal large model prompt learning method based on class knowledge enhancement, which solves the problem that the better the performance of the model on the target task, the worse the performance on unknown new tasks in the existing prompt learning method.

[0005] In order to solve the above technical problems, the technical solution of the present invention is: a multimodal large model prompt learning method based on class knowledge enhancement, comprising the following steps:

[0006] S1, initialize a set of prompt words for the target task;

[0007] S2, obtain the word vectors of each category of the target task, and concatenate the word vectors to the prompt word to obtain the category-specific prompt word;

[0008] S3, design interval self-adjusting image-text matching loss function;

[0009] S4, inputting the category-specific prompt words and pictures into the pre-trained multimodal large model CLIP in sequence to obtain text features and picture features;

[0010] S5, inputting the text features and the image features into the interval self-adjusting image-text matching loss function to calculate the classification loss, and back-propagating the classification loss to update the prompt word;

[0011] S6. Classify the images using the updated prompt words.

[0012] Furthermore, the prompt word in step S1 includes a plurality of learnable vectors of dimension d.

[0013] Furthermore, the specific steps of step S2 are:

[0014] S21, obtain all category names of the target task, and input all category names into the pre-trained tokenizer network Tokenizer in sequence for tokenization to obtain category word vectors;

[0015] S22, inputting the category word vector into the multi-layer perceptron network MLP for feature embedding to obtain the category embedding representation;

[0016] S23, adding the category embedding representation to the prompt word channel by channel to obtain the category-specific prompt word.

[0017] Furthermore, the multimodal large model prompt learning method also includes:

[0018] The learnable vectors in the prompt words are set to be updateable; the network parameters of the multi-layer perceptron network MLP are set to be updateable; the network parameters of the tokenizer network Tokenizer are set to be non-updatable; and the pre-trained parameters of the multimodal large model are set to be non-updatable.

[0019] Furthermore, the expression of all the category names is:

[0020]

[0021] in, Indicates the category name of the i-th category in the target task, M base Indicates the number of categories in the target task;

[0022] The calculation formula of the category-specific prompt word is:

[0023]

[0024] in, represents the mth accumulated learnable vector, l represents the number of learnable vectors in the prompt word, and v m represents the mth learnable vector, represents the category embedding representation, represents the i-th category-specific cue word, CLS i Represents the name of the i-th category.

[0025] Furthermore, the expression of the interval self-adjusting picture-text matching loss function in step S3 is:

[0026]

[0027]

[0028] Among them, L margin represents the interval self-adjusting image-text matching loss function, y i represents the true label of the image, P'(·) represents the posterior probability, and x j represents the jth image, <·> represents the cosine similarity, g(·) represents the CLIP text encoder network, and f j represents the image feature of the jth image, τ represents the temperature parameter, s ti express and The affinity between Represents the category word vector of the t-th category, represents the category word vector of the i-th category, and ò represents the scaling hyperparameter.

[0029] Furthermore, the specific steps of step S4 are:

[0030] S41, extracting features of the category-specific prompt words through the text encoder in the pre-trained multimodal large model CLIP to obtain text features of each category;

[0031] S42. Extract features from the image through the image encoder in the pre-trained multimodal large model CLIP to obtain image features.

[0032] Furthermore, the specific steps of step S5 are:

[0033] S51, inputting the text features and the image features into the interval self-adjusting image-text matching loss function to calculate the classification loss;

[0034] S52, back-propagating the classification loss, and updating the network parameters of the prompt words and the multi-layer perceptron network MLP in turn;

[0035] S53, determine whether the preset number of iterations has been reached, if so, end the iterative training, save the updated network parameters and prompt words, and proceed to step S6, otherwise, return to step S2 and proceed to the next round of iterations.

[0036] Furthermore, the specific steps of performing image classification in step S6 are: obtaining the image features and text features corresponding to the image through the pre-trained multimodal large model CLIP, and determining the category to which the image belongs by calculating the distance between the image features and the text features.

[0037] The beneficial effects of the present invention are: (1) a pre-trained tokenizer network Tokenizer and a multi-layer perceptron network MLP are used in sequence to extract target task category knowledge, and the knowledge is integrated into the prompt word to guide the prompt word parameter update process;

[0038] (2) We designed a self-regulating image-text matching loss function that helps maintain the generalization level of the pre-trained multimodal large model on unknown new tasks by avoiding pulling similar category features too far apart during the prompt learning process and preventing the model from overfitting to the target task category features.

[0039] (3) The present invention can be embedded as a plug-and-play component into existing multimodal large model prompt learning methods to improve the generalization of the models learned by such methods on target tasks and unknown new tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The present invention is a flowchart of the multimodal large model prompt learning method based on class knowledge enhancement.

[0041] Figure 2 This is a framework diagram of the multimodal large model prompt learning method based on class knowledge enhancement of the present invention. DETAILED DESCRIPTION

[0042] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

[0043] Example 1

[0044] like Figure 1 As shown, the present invention provides a multimodal large model prompt learning method based on class knowledge enhancement, comprising the following steps:

[0045] S1, initialize a set of prompt words for the target task;

[0046] S2, obtain the word vectors of each category of the target task, and concatenate the word vectors to the prompt word to obtain the category-specific prompt word;

[0047] S3, design interval self-adjusting image-text matching loss function;

[0048] S4, inputting the category-specific prompt words and pictures into the pre-trained multimodal large model CLIP in sequence to obtain text features and picture features;

[0049] S5, inputting the text features and the image features into the interval self-adjusting image-text matching loss function to calculate the classification loss, and back-propagating the classification loss to update the prompt word;

[0050] S6. Classify the images using the updated prompt words.

[0051] The prompt word in step S1 includes a plurality of learnable vectors of dimension d.

[0052] The specific steps of step S2 are:

[0053] S21, obtain all category names of the target task, and input all category names into the pre-trained tokenizer network Tokenizer in sequence for tokenization to obtain category word vectors;

[0054] S22, inputting the category word vector into the multi-layer perceptron network MLP for feature embedding to obtain the category embedding representation;

[0055] S23, adding the category embedding representation to the prompt word channel by channel to obtain the category-specific prompt word.

[0056] The multimodal large model prompt learning method also includes:

[0057] The learnable vectors in the prompt words are set to be updateable; the network parameters of the multi-layer perceptron network MLP are set to be updateable; the network parameters of the tokenizer network Tokenizer are set to be non-updatable; and the pre-trained parameters of the multimodal large model are set to be non-updatable.

[0058] In this embodiment 1, the prompt word p = {v1, ..., v l}, the multimodal large model CLIP is represented by ω θ , θ is the pre-training parameter of CLIP, and the pre-trained tokenizer network Tokenizer is expressed as The network is responsible for mapping any given category name into a vector of dimension d.

[0059] The multi-layer perceptron network MLP is h α (·), whose network parameter is α and can be updated, and the dimensions of the input layer and output layer of the MLP are both d.

[0060] The expression for all the class names is:

[0061]

[0062] in, Indicates the category name of the i-th category in the target task, M base Indicates the number of categories in the target task;

[0063] The calculation formula of the category-specific prompt word is:

[0064]

[0065] in, represents the mth accumulated learnable vector, l represents the number of learnable vectors in the prompt word, and v m represents the mth learnable vector, represents the category embedding representation, represents the i-th category-specific cue word, CLS i Represents the name of the i-th category.

[0066] The expression of the interval self-adjusting picture-text matching loss function in step S3 is:

[0067]

[0068] Among them, L margin represents the interval self-adjusting image-text matching loss function, y i represents the true label of the image, P'(·) represents the posterior probability, and x j represents the jth image, <·> represents the cosine similarity, g(·) represents the CLIP text encoder network, and f j represents the image feature of the jth image, τ represents the temperature parameter, s ti express and The affinity between Represents the category word vector of the t-th category, represents the category word vector of the i-th category, and ò represents the scaling hyperparameter.

[0069] The specific steps of step S4 are:

[0070] S41, extracting features of the category-specific prompt words through the text encoder in the pre-trained multimodal large model CLIP to obtain text features of each category;

[0071] S42. Extract features from the image through the image encoder in the pre-trained multimodal large model CLIP to obtain image features.

[0072] like Figure 2As shown in the figure, E T and E I They are the text encoder and image encoder of the multimodal large model CLIP respectively.

[0073] The specific steps of step S5 are:

[0074] S51, inputting the text features and the image features into the interval self-adjusting image-text matching loss function to calculate the classification loss;

[0075] S52, back-propagating the classification loss, and updating the network parameters of the prompt words and the multi-layer perceptron network MLP in turn;

[0076] S53, determine whether the preset number of iterations has been reached, if so, end the iterative training, save the updated network parameters and prompt words, and proceed to step S6, otherwise, return to step S2 and proceed to the next round of iterations.

[0077] The specific steps of classifying the image in step S6 are: obtaining the image features and text features corresponding to the image through the pre-trained multimodal large model CLIP, and determining the category to which the image belongs by calculating the distance between the image features and the text features.

[0078] Example 2

[0079] The method of the present invention can be widely used in downstream application scenarios such as remote sensing image recognition and military target detection based on multimodal large models. In this embodiment 2, the remote sensing image recognition task is taken as an example, and the data set used in the task is EuroSAT (including 10 categories).

[0080] In the training phase, the 10 category names in the task are sequentially passed through the pre-trained word segmentation network and the MLP network to extract category-specific features. Afterwards, the extracted category features are fused with the prompt words of the task to obtain category-adaptive prompt words. The training images are then fed into the image encoder E of the multimodal large model. I Extract image features, the real category name corresponding to the image is concatenated to the constructed prompt word and input into the text encoder E of the multimodal large model T Extract text features.

[0081] Finally, the classification loss is calculated by the designed interval self-adjusting image-text matching loss function, and the prompt word parameters and MLP network parameters are updated. I Extract the image features, and then calculate the image features and the text encoder E T The nearest distance of the 10 extracted category name features is used to determine the category to which the image belongs.

[0082] The classification performance comparison between the method proposed in this invention and the current advanced prompt learning method DePT on the EuroSAT dataset is shown in Table 1, where Base ACC represents the classification accuracy achieved by the method on the target task, NewACC represents the classification accuracy achieved on an unknown new task, and H AC is the harmonic mean of the former two.

[0083] Table 1

[0084]

[0085] As can be seen from Table 1, the proposed method achieves better performance than the most advanced method DePT in both the target task and the unknown new task.

[0086] In summary, the method provided by the present invention enhances the prompt learning process of the multimodal pre-trained model by using the category information in the target task, which can effectively avoid the generalization performance of the pre-trained multimodal large model on other unknown new tasks while improving the learning performance of the target task.

Claims

1. A multimodal large model prompt learning method based on class knowledge enhancement, characterized in that: The following steps are involved: S1, initialize a set of prompt words for the target task; S2, obtain the word vectors of each category of the target task, and concatenate the word vectors to the prompt word to obtain the category-specific prompt word; S3, design interval self-adjusting image-text matching loss function; S4, inputting the category-specific prompt words and pictures into the pre-trained multimodal large model CLIP in sequence to obtain text features and picture features; S5, inputting the text features and the image features into the interval self-adjusting image-text matching loss function to calculate the classification loss, and back-propagating the classification loss to update the prompt word; S6. Obtain updated category-specific prompt words through the updated prompt words, and input the updated category-specific prompt words and the pictures into the pre-trained multimodal large model CLIP in sequence for picture classification.

2. The multimodal large model prompt learning method based on class knowledge enhancement according to claim 1 is characterized in that: The prompt word in step S1 includes a plurality of learnable vectors of dimension d.

3. The multimodal large model prompt learning method based on class knowledge enhancement according to claim 2 is characterized in that: The specific steps of step S2 are: S21, obtain all category names of the target task, and input all category names into the pre-trained tokenizer network Tokenizer in sequence for tokenization to obtain category word vectors; S22, inputting the category word vector into the multi-layer perceptron network MLP for feature embedding to obtain the category embedding representation; S23, adding the category embedding representation to the prompt word channel by channel to obtain the category-specific prompt word.

4. The multimodal large model prompt learning method based on class knowledge enhancement according to claim 3 is characterized in that: The multimodal large model prompt learning method also includes: The learnable vectors in the prompt words are set to be updateable; the network parameters of the multi-layer perceptron network MLP are set to be updateable; the network parameters of the tokenizer network Tokenizer are set to be non-updatable; and the pre-trained parameters of the multimodal large model are set to be non-updatable.

5. The multimodal large model prompt learning method based on class knowledge enhancement according to claim 3 is characterized in that: The expression for all the class names is: in, Indicates the category name of the i-th category in the target task, M base Indicates the number of categories in the target task; The calculation formula of the category-specific prompt word is: in, represents the mth accumulated learnable vector, l represents the number of learnable vectors in the prompt word, v m represents the mth learnable vector, represents the category embedding representation, represents the i-th category-specific cue word, CLS i Represents the name of the i-th category.

6. The multimodal large model prompt learning method based on class knowledge enhancement according to claim 5 is characterized in that: The expression of the interval self-adjusting picture-text matching loss function in step S3 is: Among them, L margin represents the interval self-adjusting image-text matching loss function, y i represents the true label of the image, P'(·) represents the posterior probability, and x j represents the jth image, <·> represents the cosine similarity, g(·) represents the CLIP text encoder network, and f j represents the image feature of the jth image, τ represents the temperature parameter, s ti express and The affinity between Represents the category word vector of the t-th category, represents the category word vector of the i-th category, and ò represents the scaling hyperparameter.

7. The multimodal large model prompt learning method based on class knowledge enhancement according to claim 1 is characterized in that: The specific steps of step S4 are: S41, extracting features of the category-specific prompt words through the text encoder in the pre-trained multimodal large model CLIP to obtain text features of each category; S42. Extract features from the image through the image encoder in the pre-trained multimodal large model CLIP to obtain image features.

8. The multimodal large model prompt learning method based on class knowledge enhancement according to claim 3 is characterized in that: The specific steps of step S5 are: S51, inputting the text features and the image features into the interval self-adjusting image-text matching loss function to calculate the classification loss; S52, back-propagating the classification loss, and updating the network parameters of the prompt words and the multi-layer perceptron network MLP in turn; S53, determine whether the preset number of iterations has been reached, if so, end the iterative training, save the updated network parameters and prompt words, and proceed to step S6, otherwise, return to step S2 and proceed to the next round of iterations.

9. The multimodal large model prompt learning method based on class knowledge enhancement according to claim 1 is characterized in that: The specific steps of classifying the image in step S6 are: obtaining the image features and text features corresponding to the image through the pre-trained multimodal large model CLIP, and determining the category to which the image belongs by calculating the distance between the image features and the text features.

Images