A multi-modal transfer learning method for large language models

Abstract

This invention provides a multimodal transfer learning method for large language models. It utilizes a frozen image encoder to extract multi-scale image features, which are then concatenated with a learnable modal indication representation and all text features and input into a large language model for modeling. Lightweight adapter modules are placed after the self-attention layers of both the image encoder and the large language model; during transfer learning, only the parameters of these modules need to be updated. For image-text problems, this invention also utilizes a cross-attention mechanism to generate fusion prompts that connect the image and text in the input features, better guiding the large language model to generate more accurate answers. Simultaneously, based on the ∈-greedy algorithm, the optimal placement layer K of the fusion prompts in the large language model is pre-searched. This invention enables the rapid and cost-effective transfer of general knowledge from large language models to specific multimodal tasks, achieving automatic switching between unimodal and multimodal text instructions.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a multimodal transfer learning method for large language models. Background Technology

[0002] The current fields of natural language processing and multimodal computing face the challenge of effectively transferring the general knowledge of large language models (LLMs) to vision-language (VL) tasks. Although large language models have made significant progress in handling language tasks, some key issues remain when applying them to multimodal scenarios.

[0003] The main challenge of large language models in multimodal learning lies in the transfer of general knowledge. First, current multimodal learning methods typically require large-scale pre-training to adapt to new multimodal tasks. This process is not only time-consuming but also consumes significant computational resources, limiting the flexibility and scalability of large language models in practical applications. Second, while directly using linear mappings of image features to obtain textual information describing images is simple, it introduces computational complexity issues. Due to the high dimensionality of visual features, mapping them to the input sequence of large language models leads to a sharp increase in computational complexity, especially when processing large-scale image data. This poses a severe challenge to the efficiency and practicality of current large language model multimodal transfer learning in real-world applications. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a multimodal transfer learning method for large language models, so as to realize the rapid transfer of general knowledge of large language models to specific downstream multimodal tasks in a low-cost and high-efficiency manner.

[0005] This invention provides a multimodal transfer learning method for large language models, comprising the following steps:

[0006] Step 1: Extract multi-scale visual features F from image Img in the multimodal task using the frozen image encoder CLIP-ViT. [CLS] , will F [CLS] Perform linear projection W t The input text Txt is encoded using a large language model to obtain the corresponding features F. Txt Initialize two learnable modal embedding maps W m Used to represent plain text modalities and image-text modalities, the modal representation F is determined based on the modalities contained in the training samples. m ;

[0007] Step 2: Concatenate the features contained in different modalities to obtain the actual input features F of the large language model;

[0008] Step 3: Freeze the parameters of the large language model. After each self-attention sublayer, set up a multilayer perceptron and two randomly initialized adapter modules. The adapter modules are used to model the processing paths of unimodal and multimodal tasks. The multilayer perceptron uses modal representation F... m Obtain the assigned weights for single-modal and multi-modal tasks;

[0009] Step 4: Equip each layer with an independent cross-attention layer. Use cross-attention layers to generate fusion suggestions and search for the optimal placement layer K of the fusion suggestions in the large language model;

[0010] Step 5: Extract the image features F from the visual encoder. Img The plain text feature F corresponding to the output of the (K-1)th layer of the large language model txt Through cross-attention layer Obtain the corresponding fusion hint F P Then F P The multi-scale visual features and text features corresponding to the output of the (K-1)th layer are added to the optimal placement layer K for further interactive learning, and finally the large language model generates text results for the input.

[0011] Furthermore, in step 2, the feature representation input to the large language model is F, and c is the longest input sequence of the large language model:

[0012] F = [F m ;F [CLS] W t ;F Txt Image-Text

[0013] F = [F m ;F Txt Plain text

[0014] If the length of F is less than c, it is padded with a padding vector; if it exceeds c, it is truncated.

[0015] Furthermore, in step 3, the Adapter module consists of a downsampling layer. An upsampling layer It consists of a SiLU activation function δ, and the input of the Adapter module is defined as... The formula for calculating the Adapter's output O is:

[0016] O=X+U(δ(D(X)))

[0017] d hLet d be the intermediate dimension of the Adapter module, and scale it to the text dimension d. Txt 1 / 64;

[0018] The multilayer perceptron consists of two linear projection matrices W1 and W2 and an activation function σ. For each decoder layer, there are two adapters. The formula for calculating the weights [α1, α2] is:

[0019] [α1,α2]=Softmax(W2(σb1(W1(F m )+b1))+b2)

[0020] Where b1 and b2 are bias terms;

[0021] Therefore, F is expressed as follows through the two Adapter modules at each layer:

[0022]

[0023] Where F represents the sequence features of the large language model, and F' represents the weighted output features of the input sequence features of the large language model after passing through two Adapter modules.

[0024] Furthermore, step 4 specifically includes:

[0025] Step 41: Using a search method based on the ∈-greedy algorithm, each decoder layer of the large language model is regarded as the search space, and a preference H is assigned to represent the probability of each decoder layer being selected. At the same time, each layer is equipped with an independent cross-attention layer.

[0026] Step 42: During each training step t, randomly sample an insertion layer K from the M decoder layers of the large language model, and utilize the cross-attention layer. Generate visual features F that can be integrated with the final output of the image encoder. Img and the output text features F of the (K-1)th layer Txt Fusion hint F P Using F Txt The mean pooling features are used as the query vector, F Img As both the key vector and the value vector, F P It is added at the Kth layer of the large language model, concatenated with the output of the previous layer, and continuously passed to subsequent layers. Then, the Adapter module and modal representation F are updated. m Visual projection of text W t Corresponding to the current Parameters;

[0027] Step 43: After the training step t is completed, the evaluation phase with n sampling iterations begins. During each evaluation sampling, there is a probability of 1 - ∈ based on the weights of preference H, and a probability of ∈ being randomly sampled. During the evaluation process, a validation dataset D is set up. v According to D v The reward score R for inserting cross-cues at different layers in the above n samplings is obtained using the loss function. t Then based on R t Update preference H;

[0028] Repeat steps 42 and 43. After the search phase ends, the optimal placement layer K of the fusion hint in the large language model is obtained based on the weight of preference H.

[0029] Furthermore, if the features are image-text, a cross-attention layer can be used. Fenigma for generating large language models P The specific calculation formula is as follows:

[0030] F P =Concat(head1,..,head) h W O

[0031]

[0032] Where h is the number of attention heads, This represents the intermediate dimension of the cross-attention layer. W represents the mapping of the query vector. K i W represents the mapping of key vectors. V i W represents a mapping of value vectors. O This represents the final splicing mapping.

[0033] Furthermore, in step 42, at each training step t, the objective function of the large language model is:

[0034]

[0035] in, This represents the parameters of the Adapter module; W t F represents the projection matrix from the visual image to the text. m It is a learnable modal representation; represents the parameters of the cross-attention layer; L(G(F)) represents the loss function of the large language model.

[0036] Furthermore, in step 42, at each training step t, the weights π of the Kth layer in the preference H corresponding to the large language model are... tThe formula for calculating (K) is:

[0037]

[0038] After sampling n times, based on R t The formula for updating preference H is:

[0039] H t+1 (K)=H t (K)+α(R t -R b )π t (K)(1-π t (K))

[0040] Where M represents the number of layers in the large model, H t (K) represents the weight of the Kth layer, H t (m) represents the weights of the m-th layer, α is the learning rate for preference updates, set to 1; R t It is 100 times the negative loss function power of the natural logarithm e, i.e., 100*e -loss R b Let be the average reward from n samples.

[0041] Furthermore, in step 5, based on the image-text features corresponding to the multimodal task, the large language model obtains the optimal placement layer K using an ∈-greedy algorithm search strategy, and the image encoder extracts the image features F. Img The features F output by the (K-1)th layer of the large language model K-1 Through cross-attention layer Obtain the corresponding fusion hint F P , will F P Join F K-1 The input features F of the Kth layer are then fed into the Kth layer. K for:

[0042]

[0043] in, This represents the modal representation of the output of the (K-1)th layer. This represents the text representation corresponding to the output of the (K-1)th layer.

[0044] The present invention has the following technical effects or advantages:

[0045] (1) Visual information is provided by the image [CLS] features of each multi-scale image, and fusion cues are added at specific layers to better connect the representations of images and text. For cross-cues, the pooling features of the text output corresponding to the specific layer are used as the cross-attention layer. The query vector interacts with the image features output by the image encoder, ensuring that only one feature of length is introduced while acquiring global image information, thus ensuring that the sequence position of other input information is not occupied.

[0046] (2) Two lightweight Adapter modules are used in each layer to form a switching mechanism between single-modal and multi-modal modes. At the same time, the ∈-greedy algorithm can help the large language model find the best position for fusion prompts, thereby better guiding the large language model to generate more accurate answers.

[0047] (3) The intermediate dimension in the cross-attention layer The mapping from visual to text W t Since the feature dimension of a large language model is very high, the idea of ​​the Adapter module is used here to... Set to the intermediate dimension d of the Adapter module. h Similar to low-dimensionality, W t It is decomposed into two low-rank matrices. Attached Figure Description

[0048] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0049] Figure 1 This is a flowchart illustrating the execution of a multimodal transfer learning method for large language models according to the present invention.

[0050] Figure 2 This is a schematic block diagram illustrating the principle of a multimodal transfer learning method for large language models according to the present invention.

[0051] Figure 3 This is a schematic diagram illustrating the process of searching for the optimal placement layer of fusion suggestions based on the ∈-greedy algorithm in the large language model of this invention.

[0052] Figure 4 This is a schematic diagram of the human-computer interaction interface in a specific application of the multimodal transfer learning method for large language models proposed in this invention. Detailed Implementation

[0053] This invention provides a multimodal transfer learning method suitable for large language models. The overall technical approach is as follows: In multimodal tasks, we utilize a frozen visual encoder to extract multi-scale [CLS] features from images to provide visual information, and use a visual-to-text mapping W... t The representation is converted to match the dimensions of the text features, and a learnable modal representation F is added to the input. m Two Adapter modules are added after the self-attention layer in each decoder layer of the frozen large language model to implement a switching mechanism between unimodal and multimodal modes, based on F. m Make weight assignments. Assign F...Txt The pooling features output from the corresponding layer are used as the cross-attention layer. The query vector, the final visual feature F output by the image encoder. Img As a cross-modal attention layer The key and value vectors are used to generate fusion hints. Then, a search method based on the ∈-greedy algorithm is used to obtain the optimal placement layer K of the fusion hints in the large language model. Based on the optimal placement layer K obtained by the search, transfer learning is performed on the large language model. On multimodal tasks, the fusion hint F is applied. P The features output from layer K-1 are added and fed into layer K. This invention, based on a large language model, utilizes a small number of multi-scale image features, an adapter module and its switching mechanism, and fusion prompts to complete multimodal transfer learning. This not only significantly reduces the number of training parameters and computational overhead but also simultaneously completes single-modal and multimodal generation tasks, stimulating the transfer learning capability of the large language model in the multimodal domain, and thus enabling its application in human-computer interaction.

[0054] To better understand the technical solution of the present invention, the above technical solution will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] Please see Figure 1 and Figure 2 This invention provides a multimodal transfer learning method for large language models, comprising the following steps:

[0056] Step 1: Extract multi-scale visual features of image Img from the multimodal task using the frozen image encoder CLIP-ViT. Here, the global representation of the [CLS] image of each encoder layer in CLIP-ViT is extracted, where N is the layer number and d is the global representation of the encoder layer. v This refers to the dimension of the image features; the corresponding features are obtained by encoding the input text Txt using a large language model. L is the length of this text, and two learnable modal embedding maps are initialized. d Txt It is the dimension of the text, and the modal representation is determined based on the modalities contained in the training samples. To align dimensions, Perform linear projection For example, 0 / 1 labels can be used to distinguish between text modalities and image-text modalities, and based on the 0 / 1 labels, a modality embedding map W can be created. m Convert to modal representation F m .

[0057] Step 2: Concatenate the features contained in different modalities (i.e., the three features mentioned above) to obtain the actual input features F of the large language model;

[0058] Step 3: Freeze the parameters of the large language model. After each self-attention sublayer, set up a multilayer perceptron and two randomly initialized adapter modules. The latter is used to model the processing paths of unimodal and multimodal tasks, while the former uses modal representation F... m Obtain the assigned weights for single-modal and multi-modal tasks;

[0059] Step 4: Equip each layer with an independent cross-attention layer. A cross-attention layer is used to generate fusion cues, and the optimal placement layer K for the fusion cues in the large language model is searched.

[0060] Step 5: Based on the optimal placement layer K found, the image features F extracted by the visual encoder are... Img (The image encoder CLIP-ViT, frozen in step 1, extracts both the multi-scale [CLS] visual features of the image and the extracted image features F in the final layer.) Img The plain text features F corresponding to the output of the (K-1)th layer of the large language model. Txt Through cross-attention layer Obtain the corresponding fusion hint F P Then F P The multi-scale visual features and text features corresponding to the output of the (K-1)th layer are added together (generally, the output of the (K-1)th layer is F). m Image features F extracted by the visual encoder Img The text is concatenated with plain text features, and the output of the previous layer is used as the input of the next layer. Then it is fed into the Kth layer for further interactive learning, and finally the text result generated by the large language model for the input is obtained.

[0061] In one specific embodiment, step 4 specifically includes:

[0062] Step 41: Using a search method based on the ∈-greedy algorithm, treat each decoder layer of the large language model as a search space and assign preferences. This represents the probability of each decoder layer being selected, and each layer is equipped with an independent cross-attention layer.

[0063] Step 42: During each training step t, randomly sample an insertion layer K from the M decoder layers of the large language model, and use the corresponding cross-attention layer to generate fusion prompts. To better align the features of the image and text, utilize the cross-attention layer. Generate visual features that can be integrated with the final output of the image encoder. and the output text features F of the (K-1)th layer Txt Fusion hint F Ps is the length of the visual feature output by the image encoder, which is F here. Txt The mean pooling features are used as the query vector, F Img As key vectors and value vectors. F P It is added at the Kth layer of the large language model, concatenated with the output of the previous layer, and continuously passed to subsequent layers. Then, the Adapter module and modal representation F are updated. m Visual projection of text W t Corresponding to the current Parameters;

[0064] Step 43: After training step t is completed, the evaluation phase begins with n sampling iterations. During each evaluation sampling, there is a probability of 1 - ∈ based on the weights of preference H, and a probability of ∈ being randomly sampled. During the evaluation process, a validation dataset D is set up. v According to D v The reward score R for inserting cross-cues at different layers in the above n samplings is obtained using the loss function. t Then based on R t Update preference H;

[0065] After repeating steps 42 and 43 and completing a short search phase, the optimal placement layer K for the fusion cue in the large language model is obtained based on the weights of the preference H.

[0066] In one specific embodiment, in step 2, the feature representation input into the large language model is as follows: c is the longest input sequence for a large language model:

[0067] F = [F m ;F [CLS] W t ;F Txt Image-Text

[0068] F = [F m ;F Txt Plain text

[0069] If the length of F is less than c, it is padded with a padding vector; if it exceeds c, it is truncated. W t The projected space is decomposed into

[0070] F = [F m ;F [CLS] W t1 W t2 ;F Txt ]

[0071] In one specific embodiment, the Adapter module consists of a downsampling layer. An upsampling layer It consists of a SiLU activation function δ, and the input of the Adapter module is defined as... The formula for calculating the Adapter's output O is:

[0072] O=X+U(δ(D(X)))

[0073] d h Let d be the intermediate dimension of the Adapter module, and scale it to the text dimension d. Txt 1 / 64. The multilayer perceptron consists of two linear projection matrices. It consists of an activation function σ, d n This represents the intermediate dimension of the multilayer perceptron, for each of the two adapters in each decoder layer. The formula for calculating the weights [α1, α2] is:

[0074] [α1,α2]=Softmax(W2(σb1(W1(F m )+b1))+b2);

[0075] Where b1 and b2 are bias terms;

[0076] Therefore, F is expressed as follows through the two Adapter modules at each layer:

[0077]

[0078] Where F represents the sequence features of the large language model, F' represents the weighted output features of the input sequence features of the large language model after passing through two adapter modules, and the weights represent the allocation to the two adapter modules as routing selection;

[0079] In one specific embodiment, in step 4, if the feature is an image-text feature, a cross-attention layer is used. Fenigma for generating large language models P The specific calculation formula is as follows:

[0080] F P =Concat(head1,..,head) h W O

[0081]

[0082] Where h is the number of attention heads, This represents the intermediate dimension of the cross-attention layer, where These four matrices represent the projection matrices in the multi-head attention layer, respectively representing the mapping of the query vector, the key vector, the value vector, and the final concatenation mapping. This is the calculation formula for the multi-head attention mechanism. Considering that the search based on the ∈-greedy algorithm requires M cross-attention layers, to avoid too many training parameters, we also use dimensionality reduction to reduce the intermediate dimension d. h Set it to 64.

[0083] If it is plain text, there is no need to insert a fusion feature.

[0084] In one specific embodiment, in step 42, at each training step t, the objective function of the large language model is:

[0085]

[0086] in, This represents the parameters of the Adapter module; W t F represents the projection matrix from the visual image to the text. m It is a learnable modal representation; represents the parameters of the cross-attention layer; L(G(F)) represents the loss function of the large language model, which uses the ordinary cross-entropy function.

[0087] In one specific embodiment, in step 42, at each training step t, the weights π of the Kth layer in the preference H corresponding to the large language model are... t The formula for calculating (K) is:

[0088]

[0089] After sampling n times, based on R t The formula for updating preference H is:

[0090] H t+1 (K)=H t (K)+α(R t -R b )π t (K)(1-π t (K))

[0091] Where M represents the number of layers in the large model, H t (K) represents the weight of the Kth layer, H t (m) represents the weights of the m-th layer, α is the learning rate for preference updates, set to 1; R t It is 100 times the negative loss function power of the natural logarithm e, i.e., 100*e -loss Rb Let be the average reward from n samples.

[0092] In one specific embodiment, in step 5, based on the image-text features corresponding to the multimodal task, the large language model obtains the optimal placement layer K based on the ∈-greedy algorithm search strategy, and the image encoder extracts the image features F. Img The features F output by the (K-1)th layer of the large language model K-1 Through cross-attention layer Obtain the corresponding fusion hint F P , will F P Join F K-1 The input features F of the Kth layer are then fed into the Kth layer. K for:

[0093]

[0094] Similarly, the sequence length is limited to c. Wherein, This represents the modal representation of the output of the (K-1)th layer. This represents the text representation corresponding to the output of the (K-1)th layer.

[0095] The technical solutions provided in this application have at least the following technical effects or advantages: The multimodal transfer learning method for large language models in this application can quickly transfer the general knowledge of large language models to multimodal generation tasks, such as ScienceQA, in a low-cost and high-efficiency manner. It can keep the original parameters of the large language model unchanged, reduce the high computational cost brought by visual features, and use a search strategy based on the ∈-greedy algorithm to obtain compact features of connecting images and text, guiding the large model to generate more accurate answers.

[0096] The following is a specific implementation process of the model in a particular embodiment:

[0097] Step 1: Model Implementation Process

[0098] 1.1 Model Input:

[0099] The model's input is an image (Img) for a multimodal task, with its size trimmed to 224×224×3 to fit the CLIP visual encoder. Multi-scale [CLS] features of the image are extracted and mapped. Simultaneously, the features of the corresponding image description text (Txt) and the learnable modal features (F) are also mapped. m The three data points are concatenated and input into the large language model, with a fixed length of 512. Figure 1 As shown.

[0100] 1.2 Large Language Model:

[0101] This invention is based on the large language model LLaMA-7B, and the relevant parameters are shown in Table 1. In multimodal tasks, we treat it as a classification model, and add two Adapter modules after the self-attention layer in each decoding layer of LLaMA-7B.

[0102] Table 1. Relevant Parameters of LLaMA-7B

[0103]

[0104] 1.3 Generation of fusion prompts:

[0105] For image-text, the corresponding F in F Txt Pooling features as cross-attention layers The query vector, the final output F of the image encoder Img As a cross-attention layer The key vector and value vector, in order to reduce The training parameter size will be determined by the intermediate dimension of the cross-attention layer. Setting it to 64 generates text-related, compact blending tips.

[0106] F P =Concat(head1,..,head) h W O

[0107]

[0108] Where h is the number of attention heads, which is set to 8 here.

[0109] Step 2: Fusion Hints - Search Strategy Based on the ∈-Greedy Algorithm

[0110] As shown in the pseudocode flow of Algorithm 1 below, we treat the 32 autoencoder layers of the large language model LLaMA-7B as the search space, and the selection of each layer as a policy action. First, we assign preferences to the 32 layers. This represents the probability of each layer being selected. Each layer is also equipped with an independent cross-modal attention layer.

[0111] At each iteration time step t of the search process, a placement layer K is randomly sampled from H. The corresponding cross-attention layer is then used. To generate fusion prompts, the feature output of layer (K-1) is added before layer K of the large language model, and then passed to subsequent layers to update the visual-to-text mapping W. t1 W t2 Adapter module The parameters.

[0112] After the update, n samplings are performed, with a probability of 1-∈ based on the weights of preference H, and a probability of ∈ random sampling. During this process, a validation dataset D is set up. v Calculate the reward score R after each sample. t Then based on R t The preference H is updated. After the search phase, the optimal layer K for LLaMA-7B on the current multimodal task is obtained based on the weights of preference H. The entire algorithm flow is as follows: Figure 3 As shown.

[0113]

[0114]

[0115] In this search process, the number of samples n in the evaluation phase is set to 10, and the probability of ∈ is 20% to avoid getting trapped in local optima. At the same time, the search process is kept within a suitable time range; we control the epochs to 1-2. In this process, we use the Adam optimizer to validate the dataset D. v Set the batch size to the training size. As shown in Table 2.

[0116] Table 2 shows the parameter settings related to the optimal placement of K in the LLaMA-7B search fusion suggestion on SQA.

[0117] Model Dataset Epoch batchsize Training learning rate LLaMA-7B ScienceQA 2 32 0.01

[0118] Step 3: Model Training Process

[0119] Step two searches to find the optimal placement layer K for the multimodal task fusion suggestion in the large language model LLaMA-7B. The suggestion is then placed before the Kth layer of the large language model. The generated fusion hint F P The output of the (K-1)th layer is added to the subsequent layers of LLaMA-7B, and the Adapter is updated via gradient descent. And visual-to-text mapping W t1 W t2 The trainable parameters are as follows. Only about 10M parameters need to be trained throughout the entire training process. We use the Adam optimizer and set the epoch and batch size according to different vision-language tasks, as shown in Table 3. The batch size is set to 256.

[0120] Table 3. Training parameter settings for multimodal transfer learning using LLaMA-7B on SQA.

[0121] Model Dataset Epoch batchsize Learning rate LLaMA-7B ScienceQA 20 32 0.01

[0122] Step 4: Model Deployment Process

[0123] After performing multimodal transfer learning in step three using LLaMA-7B, we can obtain the generated results of the large language model by inputting the corresponding image and text instructions. By switching between the input unimodal and multimodal representations in the two Adapters at each layer, flexible switching between language unimodal and vision-language multimodal representations can be achieved. This is encapsulated using the open-source Python Grdio library, and the human-computer interaction interface is as follows: Figure 4 As shown, it outputs the corresponding text result based on the input text and / or image.

Claims

1. A multi-modal transfer learning method for large language models, characterized in that: Includes the following steps: Step 1: Extract multi-scale visual features of images in a multimodal task using the frozen image encoder CLIP-ViT. ,Will Perform linear projection The input text is encoded using a large language model to obtain corresponding features. Initialize two learnable modality embedding maps. Used to represent plain text modalities and image-text modalities, the modal representation is determined based on the modalities contained in the training samples. ; Step 2: Concatenate the features contained in different modalities to obtain the actual input features of the large language model. ; Step 3: Freeze the parameters of the large language model. After each self-attention sublayer, set up a multilayer perceptron and two randomly initialized adapter modules. The adapter modules are used to model the processing paths of unimodal and multimodal tasks. The multilayer perceptron uses modal representation... Weight allocation for unimodal and multimodal tasks; Step 4: Equip each layer with an independent cross-attention layer. Cross-attention layers are used to generate fusion cues, and the optimal placement layer for the fusion cues in the large language model is searched. ; Step 5: Extract image features from the visual encoder And the first of the large language models The plain text features corresponding to the layer's output Through cross-attention layer Get the corresponding fusion prompt Next Joined the The output of the layer corresponds to multi-scale visual features and text features, which are then fed into the optimal placement layer. Continue interactive learning to ultimately obtain the text generation results of the large language model for the input; In step 3, the Adapter module consists of a downsampling layer. One upsampling layer and a SiLU activation function The composition defines the input of the Adapter module as... Then the formula for calculating the Adapter's output O is: The intermediate dimension of the Adapter module, scaled down to the text dimension. 1 / 64; The multilayer perceptron consists of two linear projection matrices. , and an activation function Composition, for each decoder layer, two Adapters ( The weight of ) The calculation formula is: Where b1 and b2 are bias terms; So The expressions for the two Adapter modules at each layer are as follows: Where F represents the input sequence features of the large language model, This represents the weighted output features of the input series features of a large language model after passing through two Adapter modules; Step 4 specifically includes: Step 41, using based on The greedy algorithm's search method treats each decoder layer of the large language model as a search space and assigns preferences accordingly. This represents the probability of each decoder layer being selected, and each layer is equipped with an independent cross-attention layer. ; Step 42, in each training step During the process, an insertion layer is randomly sampled from the M decoder layers of the large language model. Utilizing cross-attention layers Generate visual features that can be integrated with the final output of the image encoder. and the output text features of the (K-1)th layer Fusion prompts ,use The mean pooling features are used as the query vector. As key vectors and value vectors The first in the large language model Each layer is added, concatenated with the output of the previous layer, and continuously passed to subsequent layers. Then, the Adapter module and modal representation are updated. Visual projection of text Corresponding to the current K Parameters; Step 43: After the training step t update is completed, proceed to sampling. Each evaluation phase has 1- The probability is based on preferences Sampling of weight magnitude, The probability is randomly sampled, and a validation dataset is set up during the evaluation process. ,according to The reward score for inserting cross-cues at different layers in the above n samplings is obtained using the loss function. Then based on Preference Update; Repeat steps 42 and 43. After the search phase ends, based on preferences... The weight values ​​are used to obtain the optimal placement of layers in the large language model. ; If the feature is image-text, use a cross-attention layer. Fusion hints for generating large language models The specific calculation formula is as follows: Where h is the number of attention heads, This represents the intermediate dimension of the cross-attention layer. This represents a mapping of the query vector. Represents a mapping of key vectors. Represents a mapping of value vectors. .

2. The multimodal transfer learning method for large language models as described in claim 1, characterized in that: In step 2, the feature representation input into the large language model is as follows: , It is the longest input sequence for a large language model: If Insufficient length The vector is then used to fill the gap; if it exceeds the limit, it is truncated.

3. The multimodal transfer learning method for large language models as described in claim 1, characterized in that: In step 42, at each training step t, the objective function of the large language model is: in, Indicates the parameters of the Adapter module; The projection matrix representing the visual image onto the text; It is a learnable modal representation; Parameters representing the cross-attention layer (F) represents the loss function of a large language model.

4. The multimodal transfer learning method for large language models as described in claim 1, characterized in that: In step 42, at each training step t, the weights of the Kth layer in the preference H corresponding to the large language model are... The calculation formula is: sampling Subsequently, based on The formula for updating preference H is: Where M represents the number of layers in the large model, H t (K) represents the weight of the Kth layer, H t (m) represents the weight of the m-th layer. The learning rate for preference updates is set to 1; It is 100 times the power of the negative loss function of the natural logarithm e. Let be the average reward from n samples.

5. The multimodal transfer learning method for large language models as described in claim 1, characterized in that: In step 5, the large language model is based on the image-text features corresponding to the multimodal task. The greedy algorithm search strategy obtains the optimal placement layer K, and the image encoder extracts the image features. Features output from the (K-1)th layer of the large language model Through cross-attention layer Get the corresponding fusion prompt ,Will join in The input features are then fed into the Kth layer. for: in, This represents the modal representation of the output of the (K-1)th layer. This represents the text representation corresponding to the output of the (K-1)th layer.

Images