Pre-training multi-modal model feature uniform alignment method based on meta-learning

Abstract

The invention discloses a meta-learning-based pre-training multi-modal model feature uniform alignment method, which comprises the following steps of: taking a pre-training multi-modal model as a reference model, replacing a word embedding coding module by adopting a text end prompt module, replacing a block embedding coding module by adopting an image end prompt module, and adding a weight self-adaptive adjustment model; a meta-enhanced contrast learning model is constructed, a prompt parameter, a weight adaptive adjustment model parameter and a loss function weight are initialized, and then the prompt parameter and the weight adaptive adjustment model parameter are learned by alternately adopting a training data set and a metadata set; and adding the text end prompt module and the image end prompt module which determine the prompt parameters into the multi-modal model to form a double-prompt multi-modal model, and finishing fine tuning of the multi-modal model. According to the method, the feature extraction capability of images and texts is improved by introducing a double-prompt framework and alignment and uniformity constraints, so that the performance of a multi-modal model in a downstream task is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and more specifically, relates to a method for uniformly aligning features of a pre-trained multimodal model based on meta-learning. Background Art

[0002] In recent years, in the field of artificial intelligence, researchers have been constantly exploring how to better utilize the connections between multimodal data to achieve more intelligent information processing models. Among them, visual language pre-training models (such as CLIP and ALIGN) have received widespread attention and rapid development. With the help of modern computer storage technology and GPU computing power, these models have demonstrated strong zero-sample reasoning capabilities by training on massive image-text pair data, and have achieved remarkable results in tasks such as image classification, object detection, and segmentation. With the success of these pre-trained models, how to efficiently fine-tune them to adapt to specific downstream tasks has become a research focus.

[0003] Existing fine-tuning methods can be mainly divided into three categories: adapter, prompt learning, and low-rank adaptation (LoRA). Among them, LoRA technology improves the efficiency of model fine-tuning by decomposing the weight matrix into the product of low-rank matrices and fine-tuning these low-rank parameter matrices. However, the low-rank decomposition assumption of LoRA is not always applicable to all tasks, especially when faced with very complex or strongly nonlinear data sets. The adapter method adds learnable additional layers to specific layers of the pre-trained model to adapt to different task requirements. However, this method sometimes introduces additional model complexity, thereby increasing computational overhead. Prompt learning draws on the prompting technology in the field of natural language processing, and modifies the input of the model so that it takes into account the specific requirements of the task at the input stage. At present, prompt learning has become the mainstream method for fine-tuning multimodal large models. However, there are some key problems with existing prompt learning-based methods:

[0004] (1) Insufficient alignment and uniformity of feature distribution within a single modality: Current cue learning methods mainly focus on utilizing scarce labeled domain data to design various efficient cue forms to achieve alignment of feature spaces between the visual and language modalities. However, these methods often ignore the feature distribution within a single modality. When the embeddings of each modality are more evenly arranged in the latent space, the alignment between modalities will be more effective.

[0005] (2) Insufficient task generalization ability: Since soft prompts tend to prioritize task-specific knowledge, the model is prone to overfitting on the target task, resulting in a decrease in task generalization ability. Although some methods introduce new regularization constraints to alleviate the overfitting problem in multimodal fine-tuning, regularization constraints are not effective for all tasks. How to adaptively adjust the strength of the regularization term according to the characteristics of different samples remains a difficult problem that needs to be solved. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a meta-learning-based pre-trained multimodal model feature uniform alignment method, which improves the feature extraction capability for images and texts by introducing a dual prompt framework as well as alignment and uniformity constraints, thereby significantly improving the performance of the multimodal model in downstream tasks.

[0007] In order to achieve the above-mentioned object of the invention, the method for uniformly aligning features of a pre-trained multimodal model based on meta-learning of the present invention comprises the following steps:

[0008] S1: Build and pre-train a multimodal model according to actual needs, including a word embedding encoding module, a text encoder block, an embedding encoding module, and an image encoder, where:

[0009] The word embedding encoding module is used to perform word embedding encoding on the input text m to obtain word embedding and sent to the text encoder;

[0010] Text encoder is used to embed words Perform encoding processing to obtain text embedding t;

[0011] The block embedding coding module is used to perform block embedding coding on the input image x to obtain the block embedding And send to the image encoder;

[0012] Image encoder is used to embed the block Perform encoding processing to obtain image embedding z;

[0013] S2: According to the actual application scenarios of the multimodal model, M image samples and corresponding texts are collected to form a training dataset S, and then a meta-dataset is constructed based on the training dataset S.

[0014] S3: Using the pre-trained multimodal model as the baseline model, the text-side prompt module is used to replace the word embedding encoding module, the image-side prompt module is used to replace the block embedding encoding module, and a weight adaptive adjustment module is added to construct a meta-enhanced contrastive learning model, where:

[0015] The text-side prompt module is used to perform word embedding encoding on the input text m to obtain word embedding The parameters of the word embedding encoding are the parameters of the pre-trained word embedding encoding module, and then the learnable text prompt vector v = [v1, v2, ..., v L ] and the true category label y to obtain the text prompt p = [v1, v2, ..., v L ,y], and then embed the text prompt p with the word The concatenated text prompt embedding m′ is output to the text encoder;

[0016] The image end prompt module is used to perform image embedding encoding on the input image x to obtain block embedding The parameters of the image embedding coding are the parameters of the pre-trained image embedding coding module, and then the learnable image prompt vector u = [u1, u2, ..., u L ], and get the image prompt q = [CLS,u1,u2…,u L ,E], and then embed the image prompt q and the block The concatenated image cue embedding x′ is output to the image encoder;

[0017] The weight adaptive adjustment module is used to receive the text embedding t output by the text encoder and the image embedding z output by the image encoder, and estimate the loss function weight λ;

[0018] S4: Randomly initialize the prompt parameter w (0) = {v (0) ,u (0)}, weight adaptive adjustment model parameters θ (0) ;

[0019] S5: Let training round t = 1;

[0020] S6: Input the training samples in the training data set S into the meta-enhanced contrastive learning model and calculate the loss function using the following formula

[0021]

[0022] Among them, λ i represents the loss function weight obtained by the weight adaptive adjustment module for the i-th training sample in the training data set S; represents the multimodal model loss function of the i-th training sample in the training dataset S, i = 1, 2, ..., B, B represents the number of samples in the training dataset S, and the calculation formula is as follows;

[0023]

[0024] in, Represents the image sample x in the i-th training sample i Predict the probability of belonging to category c, Represents image x i The predicted class label of

[0025] represents the alignment loss function of the i-th training sample in the training data set S, and the calculation formula is as follows:

[0026]

[0027] Among them, y i Represents the image sample x in the i-th training sample i The true label, c represents the category number; 1 [yi＝c] Represents a binary variable, that is, when y i = c, 1 [yi＝c] =1, otherwise 1 [yi＝c] =0; z i and z j Respectively represent the image samples x in the training dataset S output by the image encoder i and x j The corresponding image embedding, j = 1, 2, ..., B; || || 2 means to obtain the L2 norm;

[0028] represents the uniformity loss function of the i-th training sample in the training data set S, and the calculation formula is as follows:

[0029]

[0030] Among them, exp() represents the exponential function, σ represents the width parameter of the Gaussian kernel function, t i and t j Respectively represent the text m in the training dataset S output by the text encoder i and m j The corresponding text embedding;

[0031] Then calculate the loss function Gradient And use the following formula to update the prompt parameter w (t) :

[0032]

[0033] Among them, α represents the preset learning rate;

[0034] S7: Metadata set The training sample input element in the enhanced contrast learning model uses the following formula to calculate the loss function

[0035]

[0036] Where N represents the metadata set The number of training samples, Metadata sets Training samples True and predicted probabilities for category c, n=1,2,…,N;

[0037] Then calculate the loss function Gradient And use the following formula to update the weight adaptive adjustment model parameter θ (t) :

[0038]

[0039] Among them, β represents the preset learning rate;

[0040] S8: Determine whether the iteration end condition is met, if yes, go to step S9, otherwise go to step S10;

[0041] S9: Set training round t=t+1, and return to step S6;

[0042] S10: Determine the text prompt vector v = [v1, v2, ..., v L ] and image hint vector u=[u1,u2,…,u L ], the weight adaptive adjustment model is removed from the meta-enhanced contrastive learning model to obtain the dual-cue multimodal model and complete the multimodal model fine-tuning.

[0043] The present invention is based on a meta-learning-based pre-trained multimodal model feature uniform alignment method, which takes the pre-trained multimodal model as a baseline model, adopts a text-side prompt module to replace a word embedding coding module, adopts an image-side prompt module to replace a block embedding coding module, and adds a weight adaptive adjustment model, thereby constructing a meta-enhanced contrastive learning model, initializing prompt parameters, weight adaptive adjustment model parameters and loss function weights, and then alternately using training data sets and meta data sets to learn the prompt parameters and the weight adaptive adjustment model parameters, adding the text-side prompt module and the image-side prompt module that determine the prompt parameters to the multimodal model to form a dual-prompt multimodal model, and completing the multimodal model fine-tuning.

[0044] The present invention has the following beneficial effects:

[0045] 1) Improving modal alignment: This paper introduces alignment constraints to reduce the intra-class distance of samples of the same category in the embedding space, thereby enhancing the feature alignment effect between visual and textual modalities, thereby improving the performance of multimodal models in cross-modal tasks;

[0046] 2) Improve feature uniformity: The present invention ensures that the model can be evenly distributed in the latent space when learning text features through uniformity constraints, reduces the risk of excessive feature aggregation, and improves the generalization ability of the model, especially on complex and changeable data sets;

[0047] 3) Adaptive regularization tuning: This paper introduces an adaptive regularization tuning mechanism based on meta-learning, which can dynamically adjust the strength of the regularization constraint according to the training loss gradient of each sample, effectively avoiding the overfitting problem and significantly improving the adaptability and stability of the model in different tasks;

[0048] 4) Improving the generalization ability of pre-trained models: Through the combination of two-layer optimization problems and online approximation technology, the present invention can efficiently train and adaptively adjust model parameters, so that the model can quickly adapt to different tasks, thereby achieving higher accuracy and stability on multiple downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram of the technical principle of the present invention;

[0050] Figure 2 It is a flowchart of a specific implementation method of the method for uniformly aligning features of a pre-trained multimodal model based on meta-learning of the present invention;

[0051] Figure 3 It is a structural diagram of the meta-enhanced contrastive learning model in the present invention. DETAILED DESCRIPTION

[0052] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0053] In order to better illustrate the technical solution of the present invention, the fine-tuning idea of ​​the pre-trained multimodal model of the present invention is first briefly described.

[0054] Figure 1 It is a schematic diagram of the technical principle of the present invention. Figure 1As shown in FIG. 1 , this embodiment takes the prompt learning CLIP model as an example to illustrate the multimodal model. The traditional CLIP model includes an image feature extraction module, an image encoder, a text feature extraction module and a text encoder. Given an image text set M and a sample set X, for each text m i and image x i , respectively, through the text feature extraction module and the image feature extraction module to obtain word embedding and block embedding Then, the text embedding t can be obtained by using the text encoder and image encoder respectively. i and image embedding z i They are as follows:

[0055]

[0056] Among them, i represents the training sample number, g() represents the text encoding operation, and f() represents the image encoding operation.

[0057] Given C image categories, the predicted probability of an image belonging to each category can be calculated using the softmax function, so the following formula is used to calculate the image x i The probability of belonging to each category is:

[0058]

[0059] in, Represents image x i The predicted category label is exp(), which represents the exponential function, the superscript T represents the transpose, τ represents the temperature parameter, and t c Represents the text embedding of the c-th class label.

[0060] Define the training loss of the multimodal model as Cross entropy loss is usually used, and its specific calculation formula is as follows:

[0061]

[0062] Among them, B represents the number of samples in the current training data.

[0063] For multimodal models such as the CLIP model mentioned above, a prompt framework can be built on both the text side and the image side, replacing the image feature extraction module with the image side prompt module and replacing the text feature extraction module with the text side prompt module, that is, adding image prompts and text prompts to block embedding and word embedding to facilitate alignment between modalities. Specifically, the text prompt p is represented as a learnable vector v = [v1, v2, ..., v L] and the true category label y, where L represents the length of the learnable vector. In order to maintain generality, this embodiment sets the length of L to 16. Then, the input of the text encoder, i.e., the text prompt p, is:

[0064] p=[v1,v2,…,v L ,y]

[0065] In the training phase, the true category label in the text prompt of each training sample is the label corresponding to the image in the training sample. In the inference phase, for example, if the multimodal model is used for classification, the learnable vector can be concatenated with the true category label of each category, and then the text embeddings under different categories can be obtained, and the similarity between the image embedding and the different text embeddings can be calculated to determine the category corresponding to the image. For other multimodal tasks, the true category label in the text prompt can be set according to the actual needs of the task.

[0066] Similarly, because image X i It can also be converted into a token form through a transformer by inserting a learnable variable u = [u1, u2, ..., u L ], the input of the image encoder, i.e. the image prompt q, is obtained as:

[0067] q=[CLS,u1,u2…,u L ,E]

[0068] Then the text prompt p is embedded with the word Splice, embed the image prompt q with the block Splicing, embed the obtained text prompt into Image prompt embed Input the text encoder and image encoder respectively for the next step of processing.

[0069] In order to unify the description, the present invention constructs the text-side learnable vector v and the image-side learning variable u as the prompt model parameter w={v,u}.

[0070] Current cue learning methods mainly focus on utilizing scarce labeled domain data and designing various efficient cue forms to achieve alignment of feature spaces between the two modalities of vision and language. However, these methods often ignore the feature distribution within a single modality. Alignment between modalities will be more effective when the feature representations of each modality are more evenly arranged in the unit hypersphere. Inspired by the contrastive learning theory of unimodality, it should be promoted that the modal feature representations learned by multimodal large models should satisfy alignment and uniformity, where alignment means that similar samples should be mapped to nearby features. And uniformity means that the feature representations should be roughly evenly distributed on the unit hypersphere, retaining as much data information as possible. Starting from the ideas of alignment and uniformity, the present invention forces the learned feature representations to meet the above requirements while fine-tuning the multimodal model.

[0071] A) Alignment constraints:

[0072] Specifically, since there are multiple images per class in the dataset, various image embeddings are obtained from the image encoder. To better align the text and image embeddings for a given class, the image embeddings of samples from the same class should be close to each other. i and z j The intra-class distance of this paper is based on the L2 norm to constrain the image embedding of samples of the same class to be close, that is, to construct an alignment loss function

[0073]

[0074] Among them, y i Represents image sample x i The true label, c represents the category number; Represents a binary variable, that is, when y i = c, otherwise z i and z j Respectively represent the image samples x output by the image encoder i and x j The corresponding image embedding, || ||2 means to obtain the L2 norm.

[0075] B) Uniformity Constraint:

[0076] Small distances between text (label) embeddings may lead to misclassification and make it difficult to align visual features. To solve this problem, the present invention introduces a Gaussian kernel function to optimize text representation and constructs a uniformity loss function.

[0077]

[0078] Among them, exp() represents the exponential function, σ represents the width parameter of the Gaussian kernel function, t i and t j Respectively represent the text m output by the text encoder i and m j The corresponding text embedding.

[0079] Based on the above analysis, the training loss of the multimodal model, the alignment loss, and the uniformity loss are combined to obtain the total training loss of the multimodal model of the present invention: for:

[0080]

[0081] Among them, λ represents the weight of the loss function.

[0082] Since soft prompts tend to prioritize task-specific knowledge, the model is prone to overfitting on the target task, resulting in a decrease in task generalization ability. Although the present invention alleviates the overfitting problem in multimodal fine-tuning by introducing new alignment and uniformity regularization constraints, the regularization constraints are not effective for all tasks. In order to better adaptively adjust the regularization strength, the present invention uses the training loss gradient of each sample to adaptively adjust the strength of the regularization constraint based on the meta-learning idea. Specifically, a weight adaptive adjustment model λ=h(z,t;θ) is designed to predict the appropriate hyperparameter λ for each sample, and θ represents the parameter of the weight adaptive adjustment model. That is, the present invention uses image embedding z and text embedding t as the input of the weight adaptive adjustment model, so that the weight adaptive adjustment model can be set according to the automatic prediction of the weight hyperparameter of the training sample.

[0083] Therefore, based on the above analysis, the present invention can perform model training by constructing the following two-layer optimization problem:

[0084]

[0085] in, and They represent the total loss calculated based on the training dataset and the metadata dataset, respectively. X and M represent the image set and text set in the training dataset S, respectively. Z and T represent the image embedding set and text embedding set generated based on the image set X and the text set M, respectively. Metadata sets The training dataset is used to learn the prompt parameters, while the meta dataset is used to learn the adaptive adjustment model parameters.

[0086] Based on the above principles, the present invention proposes a pre-trained multimodal model fine-tuning method based on meta-learning. Figure 2 : is a flowchart of a specific implementation of the method for fine-tuning a pre-trained multimodal model based on meta-learning of the present invention. Figure 2 As shown, the pre-trained multimodal model fine-tuning method based on meta-learning of the present invention includes the following steps:

[0087] S201: Build and pre-train a multimodal model:

[0088] Construct and pre-train a multimodal model according to actual needs, including a block embedding encoding module, an image encoder, a word embedding encoding module, and a text encoder, where:

[0089] The word embedding encoding module is used to perform word embedding encoding on the input text m to obtain word embedding and sent to the text encoder.

[0090] Text encoder is used to embed words Perform encoding processing to obtain text embedding t.

[0091] The block embedding coding module is used to perform block embedding coding on the input image x to obtain the block embedding And sent to the image encoder.

[0092] Image encoder is used to embed the block Perform encoding processing to obtain image embedding z.

[0093] S202: Constructing a data set:

[0094] According to the actual application scenarios of the multimodal model, M image samples and corresponding texts are collected to form a training dataset S, and then a meta-dataset is constructed based on the training dataset S.

[0095] In order to increase the generalization and diversity of the model, the metadata set in this embodiment The mixup data enhancement method is used to generate based on the training data set S, that is:

[0096]

[0097] in, represents the image samples and corresponding texts generated by the mixup data enhancement method, x i 、x j Represent two different image samples of the training data set S, m i 、m j Represents image samples x i 、x jThe corresponding text is i,j=1,2,…,B, where B represents the number of samples in the training data set S, and μ represents the ratio of data enhancement mixing, which can be obtained by random sampling through Beta distribution.

[0098] S203: Constructing a meta-enhanced contrastive learning model:

[0099] In order to achieve fine-tuning of the pre-trained multimodal model, the present invention takes the pre-trained multimodal model as the baseline model, adopts a text-side prompt module to replace the word embedding coding module, adopts an image-side prompt module to replace the block embedding coding module, and adds a weight adaptive adjustment module, thereby constructing a meta-enhanced contrastive learning model. Figure 3 It is a structural diagram of the meta-enhanced contrastive learning model in the present invention. Figure 3 As shown, in the meta-enhanced contrastive learning model of the present invention:

[0100] The text-side prompt module is used to perform word embedding encoding on the input text m to obtain word embedding The parameters of the word embedding encoding are the parameters of the pre-trained word embedding encoding module, and the learnable text prompt vector v = [v1, v2, ..., v L ] and the true category label y to obtain the text prompt p = [v1, v2, ..., v L ,y], and then embed the text prompt p with the word The concatenated text hint embedding m′ is output to the text encoder.

[0101] The image end prompt module is used to perform image embedding encoding on the input image x to obtain block embedding The parameters of the image embedding coding are the parameters of the pre-trained image embedding coding module, and then the learnable image prompt vector u = [u1, u2, ..., u L ], and get the image prompt q = [CLS,u1,u2…,u L ,E], and then embed the image prompt q and the block The concatenated image cue embedding x′ is output to the image encoder.

[0102] The weight adaptive adjustment module is used to receive the text embedding t output by the text encoder and the image embedding z output by the image encoder, and estimate the loss function weight λ. In this embodiment, the weight adaptive adjustment model includes a multi-layer perceptron and a Sigmoid function layer, where:

[0103] The multi-layer perceptron is used to process the image embedding z and text embedding t, and send the obtained features to the Sigmoid function layer.

[0104] The Sigmoid function layer is used to process the received features using the Sigmoid function to obtain the loss function weight λ.

[0105] According to the above structure, the forward process of the weight adaptive model in this embodiment can be expressed as:

[0106] h(z,t;θ)=σ(MLP(z,t))

[0107] Among them, θ represents the parameters of the adaptive adjustment model, σ() represents the Sigmod function, which outputs the input information between 0 and 1; MLP() represents the operation of the multi-layer perceptron (MLP, Multi-Layer Perceptron).

[0108] S204: Initialization parameters:

[0109] Randomly initialize the prompt parameter w (0) = {v (0) ,u (0)} and weights to adaptively adjust the model parameters θ (0) .

[0110] S205: Let training round t=1.

[0111] S206: Training data set learning:

[0112] The training samples in the training data set S are input into the meta-enhanced contrastive learning model, and the loss function is calculated using the following formula

[0113]

[0114] Among them, λ i represents the loss function weight obtained by the weight adaptive adjustment module for the i-th training sample in the training data set S; represents the multimodal model loss function of the i-th training sample in the training dataset S, i = 1, 2, ..., B, B represents the number of samples in the training dataset S, and the calculation formula is as follows;

[0115]

[0116] in, Represents the image sample x in the i-th training sample i Predict the probability of belonging to category c, Represents image x i The predicted class label of

[0117] represents the alignment loss function of the i-th training sample in the training data set S, and the calculation formula is as follows:

[0118]

[0119] Among them, y i Represents the image sample x in the i-th training sample i The true label, c represents the category number; Represents a binary variable, that is, when y i = c, otherwise z i and z j Respectively represent the image samples x in the training dataset S output by the image encoder i and x j The corresponding image embedding, j = 1, 2, …, B; || || 2 means to obtain the L2 norm.

[0120] represents the uniformity loss function of the i-th training sample in the training data set S, and the calculation formula is as follows:

[0121]

[0122] Among them, exp() represents the exponential function, σ represents the width parameter of the Gaussian kernel function, t i and t j Respectively represent the text m in the training dataset S output by the text encoder i and m j The corresponding text embedding.

[0123] Then calculate the loss function Gradient And use the following formula to update the prompt parameter w (t) :

[0124]

[0125] Among them, α represents the preset learning rate.

[0126] S207: Metadata set learning:

[0127] Metadata Set The training sample input element in the enhanced contrast learning model uses the following formula to calculate the loss function

[0128]

[0129] Where N represents the metadata set The number of training samples, Metadata sets Training samples True and predicted probabilities for class c, n=1,2,…,N.

[0130] Then calculate the loss function Gradient And use the following formula to update the weight adaptive adjustment model parameter θ (t) :

[0131]

[0132] Among them, β represents the preset learning rate.

[0133] S208: Determine whether the iteration end condition is met. If not, proceed to step S209, otherwise proceed to step S210. The iteration end condition can be set according to actual needs. In this embodiment, the iteration end condition is to reach the preset maximum number of iterations t max .

[0134] S209: Let t=t+1, and return to step S206.

[0135] S210: Determine the dual prompt multimodal model:

[0136] Determine the text prompt vector v = [v1, v2, ..., v L ] and image hint vector u=[u1,u2,…,u L ], the weight adaptive adjustment model is removed from the meta-enhanced contrastive learning model to obtain the dual-cue multimodal model and complete the multimodal model fine-tuning.

[0137] In order to better illustrate the technical effect of the present invention, the present invention is experimentally verified using specific examples. In this experimental verification, two commonly used data sets Caltech101 and FlowerS202 are used for fine-tuning common pre-trained multimodal models. The model generalization ability experiments of base class-base class and base class-new class are carried out on these two data sets, where training prompts are set with 16 shots (16 images per class) on the base class, and the performance of the prompt method on the base class and the new class is evaluated. Under this setting, the model cannot see new classes during the training phase.

[0138] Three comparison methods were set up in this experimental verification, namely Zero-CLIP (zero-sample CLIP), CoOp (context optimized CLIP), and CoCoOp (conditional context optimized CLIP).

[0139] In this experimental verification, the method of the present invention was implemented using PyTorch and trained on an NVIDIA RTX3090 GPU. The backbone networks used in the present invention all adopted the CLIP network architecture, and the meta-feature enhancer adopted a multi-layer MLP network.

[0140] Table 1 is a statistical table of classification accuracy of the present invention and the comparative method under different tasks in this embodiment.

[0141]

[0142] Table 1

[0143] From the results in Table 1, it can be seen that the present invention achieves the best results in the two generalization tasks of the two datasets Caltech101 and FlowerS202, thereby verifying the effectiveness of the present invention.

[0144] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A meta-learning-based pre-trained multimodal model feature uniform alignment method, characterized in that: The following steps are involved: S1: Build and pre-train a multimodal model according to actual needs, including a word embedding encoding module, a text encoder block, an embedding encoding module, and an image encoder, where: The word embedding encoding module is used to perform word embedding encoding on the input text m to obtain word embedding and sent to the text encoder; Text encoder is used to embed words Perform encoding processing to obtain text embedding t; The block embedding coding module is used to perform block embedding coding on the input image x to obtain the block embedding And send to the image encoder; Image encoder is used to embed the block Perform encoding processing to obtain image embedding z; S2: According to the actual application scenarios of the multimodal model, M image samples and corresponding texts are collected to form a training dataset S, and then a meta-dataset is constructed based on the training dataset S. S3: Using the pre-trained multimodal model as the baseline model, the text-side prompt module is used to replace the word embedding encoding module, the image-side prompt module is used to replace the block embedding encoding module, and a weight adaptive adjustment module is added to construct a meta-enhanced contrastive learning model, where: The text-side prompt module is used to perform word embedding encoding on the input text m to obtain word embedding The parameters of the word embedding encoding are the parameters of the pre-trained word embedding encoding module, and then the learnable text prompt vector v = [v1, v2, ..., v L ] and the true category label y to obtain the text prompt p = [v1, v2, ..., v L ,y], and then embed the text prompt p with the word The concatenated text prompt embedding m′ is output to the text encoder; The image-side prompt module is used to perform image embedding encoding on the input image x to obtain block embedding The parameters of the image embedding coding are the parameters of the pre-trained image embedding coding module, and then the learnable image prompt vector u = [u1, u2, ..., u L ], and get the image prompt q = [CLS,u1,u2…,u L ,E], and then embed the image prompt q and the block The concatenated image cue embedding x′ is output to the image encoder; The weight adaptive adjustment module is used to receive the text embedding t output by the text encoder and the image embedding z output by the image encoder, and estimate the loss function weight λ; S4: Randomly initialize the prompt parameter w (0) = {v (0) ,u (0) }, weight adaptive adjustment model parameters θ (0) ; S5: Let training round t = 1; S6: Input the training samples in the training data set S into the meta-enhanced contrastive learning model and calculate the loss function using the following formula Among them, λ i represents the loss function weight obtained by the weight adaptive adjustment module for the i-th training sample in the training data set S; represents the multimodal model loss function of the i-th training sample in the training dataset S, i = 1, 2, ..., B, B represents the number of samples in the training dataset S, and the calculation formula is as follows; in, Represents the image sample x in the i-th training sample i Predict the probability of belonging to category c, Represents image x i The predicted class label of represents the alignment loss function of the i-th training sample in the training data set S, and the calculation formula is as follows: Among them, y i Represents the image sample x in the i-th training sample i The true label, c represents the category number; Represents a binary variable, that is, when y i = c, otherwise z i and z j Respectively represent the image samples x in the training dataset S output by the image encoder i and x j The corresponding image embedding, j = 1, 2, ..., B; || || 2 means to obtain the L2 norm; represents the uniformity loss function of the i-th training sample in the training data set S, and the calculation formula is as follows: Where exp( ) represents the exponential function, σ represents the width parameter of the Gaussian kernel function, t i and t j Respectively represent the text m in the training dataset S output by the text encoder i and m j The corresponding text embedding; Then calculate the loss function Gradient And use the following formula to update the prompt parameter w (t) : Among them, α represents the preset learning rate; S7: Metadata set The training sample input element in the enhanced contrast learning model uses the following formula to calculate the loss function Where N represents the metadata set The number of training samples in Metadata sets Training samples True and predicted probabilities for category c, n=1,2,…,N; Then calculate the loss function Gradient And use the following formula to update the weight adaptive adjustment model parameter θ (t) : Among them, β represents the preset learning rate; S8: Determine whether the iteration end condition is met, if yes, go to step S9, otherwise go to step S10; S9: Set training round t=t+1, and return to step S6; S10: Determine the text prompt vector v = [v1, v2, ..., v L ] and image hint vector u=[u1,u2,…,u L ], the weight adaptive adjustment model is removed from the meta-enhanced contrastive learning model to obtain the dual-cue multimodal model and complete the multimodal model fine-tuning.

2. The pre-trained multimodal model fine-tuning method according to claim 1, characterized in that: The metadata set in step S2 The mixup data enhancement method is used to generate based on the training data set S.

3. The pre-trained multimodal model fine-tuning method according to claim 1, characterized in that: The weight adaptive adjustment model in step S3 includes a multi-layer perceptron and a Sigmoid function layer, wherein: The multi-layer perceptron is used to process the image embedding z and the text embedding t, and the obtained features are sent to the Sigmoid function layer; The Sigmoid function layer is used to process the received features using the Sigmoid function to obtain the loss function weight λ.

Images