A heterogeneous multimodal hybrid expert adapter

Abstract

The present invention proposes a heterogeneous multimodal hybrid expert adapter, which transforms the traditional single-type multimodal MoE architecture into a combination of multiple expert types. The adapter expands the input of each expert from a single modality to multiple modalities, and realizes cross-modal interaction within each expert. Specifically, the present invention introduces a heterogeneous expert structure including cross-attention experts and channel attention experts, maps the inputs of different modalities to a low-rank space, performs modal interaction fusion in this space, and then maps back to the original dimension, so that the trainable parameters can effectively adjust the gradient of the frozen pre-trained model and make full use of the collaborative interaction between different modalities. The global router assigns weights to each group of experts, and the local group router controls the gating weight of each expert. The adapter can be integrated into the encoder layer of the multimodal pre-trained model, and only fine-tunes the trainable parameters on the downstream tasks, keeping the number of parameters in the reasoning stage basically unchanged.

Description

Technical Field

[0001] The present invention relates to the fields of natural language processing and multimodal learning, and in particular to a technology for efficiently fine-tuning parameters of a large multimodal language model. Background Art

[0002] In recent years, multimodal large language models have shown successful applications in various cross-modal tasks. However, these models contain billions of parameters, and fine-tuning them on specific tasks requires a lot of computing resources. The parameter-efficient fine-tuning (PEFT) technology achieves efficient fine-tuning of large models by inserting small trainable components while freezing pre-trained parameters. The MoE (Mixture-of-Experts) adapter can expand the encoding capabilities of the model while keeping the number of parameters unchanged during the inference phase. However, existing methods are mainly based on the design of unimodal large language models, and lack MoE adapters tailored for multimodal large models. Summary of the invention

[0003] In order to solve the problem in the prior art of lack of hybrid expert adapters tailored for multimodal large models, the present invention proposes a heterogeneous multi-modal mixture of experts (HMMoE) adapter, which transforms the traditional single-type multimodal hybrid expert architecture into a combination of different expert types.

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] A heterogeneous multimodal hybrid expert adapter for fine-tuning a multimodal language model, wherein the multimodal language model includes an encoder and a decoder, the encoder includes a plurality of encoder layers, and the heterogeneous multimodal hybrid expert adapter is inserted into each encoder layer;

[0006] The heterogeneous multimodal hybrid expert adapter is composed of a global router, a local router and three expert groups, wherein the three expert groups are a single-modal expert group, a cross-attention expert group and a channel attention expert group, and each expert group contains M experts of the corresponding type;

[0007] The single-modality experts in the single-modality expert group are used to perform single-modality representation learning on the input first-modality embedding features or second-modality embedding features, the cross-attention experts in the cross-attention expert group are used to perform cross-attention fusion on the first-modality embedding features and the second-modality embedding features in a low-rank space, and the channel attention experts in the channel attention expert group are used to perform attention weighting on the channel dimensions of the first-modality embedding features and the second-modality embedding features;

[0008] The global router is used to allocate global routing mixed weight coefficients to different expert groups, and the local router is used to select K experts in each expert group to activate and allocate local routing mixed weight coefficients to the K experts;

[0009] The outputs of the experts in the expert group are weighted and summed by the local routing mixing weight coefficient as the output of the expert group, and the outputs of the expert groups are weighted and summed by the global routing mixing weight coefficient as the output of the adapter to participate in the calculation processing of the encoder layer.

[0010] On the other hand, the present invention also proposes a method for fine-tuning a multimodal language model based on the above-mentioned heterogeneous multimodal hybrid expert adapter, the method comprising the following steps:

[0011] 1) obtaining a training data set, wherein the training data set includes first modality data and second modality data corresponding to the first modality data;

[0012] 2) constructing a pre-trained multimodal language model, the multimodal language model comprising an encoder and a decoder, the encoder comprising a plurality of encoder layers, and inserting the heterogeneous multimodal hybrid expert adapter into each encoder layer;

[0013] 3) Inputting the training data set into the multimodal language model and training with a total loss function, wherein the total loss function is obtained by adding a loss term based on knowledge transfer and a generation loss term of the second modality data. During the training process, only the heterogeneous multimodal hybrid expert adapter in the training encoder and the classifier in the decoder are updated, and the other parts are frozen.

[0014] Compared with the prior art, the present invention has the following beneficial effects:

[0015] (1) The traditional single-modal hybrid expert architecture is expanded to a heterogeneous multi-modal combination, improving the ability to handle complex multi-modal scenarios;

[0016] (2) Realize interactive fusion between modalities in low-rank space, so that the trainable parameters can effectively adjust the gradient of the frozen pre-trained model and make full use of the collaborative interaction between different modalities;

[0017] (3) Introducing a variety of interactive experts, such as cross-attention experts and channel-attention experts, to enhance the expressiveness of modal fusion;

[0018] (4) While keeping the number of parameters in the inference phase basically unchanged, the performance of large multimodal models on downstream tasks is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The encoder architecture diagram of the heterogeneous multi-modal hybrid expert adapter of the present invention;

[0020] Figure 2 FIG. 4 is a diagram showing the architecture of different types of expert groups in the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] like Figure 1 As shown, the heterogeneous multimodal hybrid expert adapter module of the present invention is systematically integrated into the encoder layer of the multimodal pre-trained model. In this embodiment, after the heterogeneous multimodal hybrid expert adapter module is inserted into the feedforward neural network layer, the insertion position of the adapter module is not limited in actual implementation.

[0023] like Figure 2 As shown in the upper part, the heterogeneous multimodal hybrid expert adapter module sets three different types of expert groups. Taking text and vision as an example, for the lth encoder layer, the text input feature is set to T l ∈R B×St×D , the visual input feature is V l ∈R B×Sv×D , based on the text input feature T l and visual input features V l , using the global router R g Assign a global routing hybrid weight coefficient R to each expert group g (x):

[0024] R g (x) = SoftMax(W rg (x))

[0025] Where W rg (· is the global router R g, x represents the visual input features and / or text input features of the expert group, and SoftMax(·) is the activation function.

[0026] The three expert groups are unimodal expert group, cross-attention expert group and channel-attention expert group. Each expert group contains M experts of the same type, such as unimodal experts, cross-attention experts or channel-attention experts. For each expert group of M experts, a local router R l The first K experts are selected from each expert group for activation and assigned local routing hybrid weight coefficients, calculated as:

[0027] R l (x) s =SoftMax(TopK(W r (x) s ))

[0028] R l (x) c =SoftMax(TopK(W r (x) c ))

[0029] R l (x) ca =Softmax(TopK(W r (x) ca ))

[0030] Among them, R l (x) s The local routing hybrid weight coefficient of the single-modal expert, R l (x) ca is the local routing mixing weight coefficient of the cross-attention expert, R l (x) c is the local routing hybrid weight coefficient of the channel attention expert, x represents the modal embedding feature input to the expert group, and W r (·) ca is the trainable parameter matrix of the local routing network of the criss-cross attention expert, W r (·) s is the trainable parameter matrix of the local routing network of the unimodal attention expert, W r (·) c is the trainable parameter matrix of the local routing network of the channel attention expert, TopK(W r ) means selecting K experts with the largest weights to activate, K <M。

[0031] like Figure 2The general structure of different types of experts is shown in Figure 1. Single-modal experts are used to learn single-modal representations, cross-attention experts are used to perform cross-attention fusion of multi-modalities in low-rank space, and channel-attention experts are used to perform attention weighting on the channel dimension of multi-modalities. The inputs of different modalities are mapped to low-rank space, where modal interaction is fused and then mapped back to the original dimension. This approach enables trainable parameters to effectively adjust the gradients of frozen pre-trained models and make full use of the collaborative interaction between different modalities.

[0032] refer to Figure 2 The two unimodal expert structures shown in , one is a text unimodal expert, and the other is a visual unimodal expert, and the number of the two unimodal experts is the same. The output calculation formula of the two unimodal experts is:

[0033] T out1 =T in F relu (T in *W down )*W up

[0034] V out1 =V in F relu (V in *W down )*W up

[0035] Among them, T in 、V in are the text modality embedding features and visual modality embedding features input to the adapter module, W down ∈R D×r is the down-projection matrix, W up ∈R r×D is the upper projection matrix, r is the bottleneck dimension, F relu is a nonlinear activation function;

[0036] Figure 2 The lower part shows the structure of the cross-attention expert and channel-attention expert. For the cross-attention expert, the two modal features of the input are first mapped to a low-rank space through down-projection, in which cross-attention calculations are performed to achieve modal interaction fusion, and finally mapped back to the original dimensional space through up-projection. The specific calculation process is as follows:

[0037] 1) Embed the text modality into feature T in and the visual modality embedding feature V in Through the projection matrix W down Mapped to the low-rank space, the mapped text modality feature T′ and visual modality feature V′ are obtained, and the calculation formula is:

[0038] T′=F relu (T in *W down )

[0039] V′=F relu (V in *W down )

[0040] 2) For the mapped text modal feature T′, query the mapping matrix W q Generate the query matrix Q, and map the mapped visual modality features V′ to the key mapping matrix W k Generate key matrix K, through value mapping matrix W v Generate the value matrix V, the calculation formula is:

[0041] Q=T′*W q

[0042] K=V′*W k

[0043] V=V′*W v

[0044] 3) Calculate the attention weight matrix:

[0045]

[0046] Among them, d k is the last dimension of the value matrix K. The calculation of the attention weight matrix is ​​the conventional attention calculation formula.

[0047] 4) Multiply the attention weight matrix by the value matrix V to get the attention representation A t ;

[0048] 5) Attention represents A t Added to the mapped text modality feature sequence T' and passed through the projection matrix W up Mapping back to the original dimension, we get the cross-attention fused text modality feature T out2 , the calculation formula is:

[0049] T out2 =(A t +T′)*W up

[0050] Similarly, the visual modality feature V of cross-attention fusion can be calculated out2 ,

[0051] V out2 =(A t V′)*W up

[0052] For the channel attention expert, the specific implementation steps are as follows:

[0053] 1) Embedding feature V of visual modality cn Along S v Dimensions are averaged and pooled to obtain the channel vector v of the visual modality;

[0054] 2) The channel vector v of the visual modality c With the text modality embedding feature T in Perform element-by-element multiplication to obtain the product matrix M;

[0055] 3) Perform a sigmoid operation on the product matrix M to obtain the channel attention weight A of the text mode c ;

[0056] 4) Embed the feature vector sequence T of the text modality in Mapped to the low-rank space, the mapped text modality feature sequence T′ is obtained, and the calculation formula is:

[0057] T′=F relu (T in *W down )

[0058] 5) The mapped text modal feature sequence T ′ and channel attention weight A c Perform element-wise multiplication and pass the projection matrix W up Mapping back to the original dimension, we get the channel attention weighted text modality feature T out3 , the calculation formula is:

[0059] T out3 =(T′*A c )*W up

[0060] Similarly, the visual modality feature V can be obtained out3 :

[0061] V out3 =Frelu(V in *W down )W up (1+Sigmoid(AvgPool(T in )⊙V in ))

[0062] Finally, the outputs of the activated experts in the expert group are weighted and summed by the local routing mixed weight coefficient as the output of the expert group, and the outputs of each expert group are weighted and summed by the global routing mixed weight coefficient to obtain T OUT and V OUT , as the output of the adapter.

[0063] T OUT and V OUT The calculation formula is:

[0064]

[0065]

[0066] Where E v 、E t are the single-modal expert groups that learn their respective modalities, E ca 、E c They are the cross attention expert group and the channel attention expert group, R g1 , R g2 and R g3 are the global routing hybrid weight coefficients for the unimodal expert group, cross-attention expert group, and channel-attention expert group, respectively. l1i , R l2i and R l3i are the local routing mixing weight coefficients of the i-th expert in the unimodal expert group, cross-attention expert group, and channel-attention expert group, respectively.

[0067] Repeat the above process and integrate the heterogeneous multimodal hybrid expert adapter module into all encoder layers of the pre-trained model. The output feature sequence of the last encoder layer is input into the decoder, and the decoder autoregressively generates the target sequence.

[0068] On downstream tasks, only the trainable parameters of the heterogeneous hybrid expert adapter are fine-tuned, keeping the pre-trained model parameters frozen, thus achieving efficient parameter fine-tuning.

[0069] Based on the above heterogeneous multimodal hybrid expert adapter module, the present invention proposes a fine tuning method for a large multimodal language model, comprising the following steps:

[0070] 1) obtaining a training data set, wherein the training data set includes text modality data and corresponding visual modality data;

[0071] 2) Based on the above heterogeneous multimodal MoE adapter, an encoder-decoder model is constructed;

[0072] The encoder is composed of multiple encoder layers stacked together, each of which includes: a normalization layer (LN), a MoE adapter, a residual connection, and a feedforward neural network (FFN), and the calculation formula is:

[0073] X′=LN(X)

[0074] X"=X′+MoE Adapter (X′)

[0075] X out =X"+FFN(X")

[0076] Where X is the encoder layer input, X out is the encoder layer output.

[0077] Each decoder layer includes: occluded multi-head self-attention mechanism (MHA), layer normalization (LN), MoE adapter, residual connection and feed-forward neural network (FFN), and the calculation formula is:

[0078] X′=LN(X)

[0079] X"=X′+MHA(X′)

[0080] X"′=LN(X")

[0081] X""=X"′+MoE Adapter (X"′)

[0082] X out =X""+FFN(X"")

[0083] Where X is the decoder layer input, X out is the decoder layer output.

[0084] 3) Inputting the training data set into the encoder-decoder model for training, the training loss function includes: the loss term L based on knowledge transfer kt , which is used to transfer knowledge from visual modality data and guide the semantic representation of text modality data encoding; the generation loss term L of visual modality data T , used to guide the decoder to generate visual modality data; the total loss function is L = L kt +L T ;

[0085] The loss term L based on knowledge transfer kt The calculation formula is:

[0086] L kt =∑ i,j (cosv i ,v j )-cos(s i ,s j )) 2

[0087] where v i 、v j are the embedding vectors of the i-th and j-th text modality data, s i 、s jare the pre-trained embedding vectors of the i-th and j-th sentences in the visual modality data, respectively, and cos(u,v) represents the cosine similarity of the two vectors u and v.

[0088] The generation loss term L for visual modality data T The calculation formula is:

[0089] L T =∑ u ∑ d logP(y d |y ＜d ,X)

[0090] Where u is the decoding step number, d is the index of the target word in the vocabulary, and y d is the dth target word, y ＜d is the word sequence generated by the decoder before step u, and X is the input text modality data.

[0091] The calculation formulas for the loss term based on knowledge transfer and the generation loss term of visual modality data are well known to those skilled in the art.

[0092] In order to reflect the technical effect of the present invention, experiments were conducted on multiple video-text tasks and audio-video tasks. The experiments are mainly divided into three groups: visual-text classification tasks based on the vlmo model, video question-answering tasks based on the VALOR model, and audio-video positioning analysis tasks based on the Swin-Transfomer model. In the above three groups of experiments, experiments on full model parameter fine-tuning (Full Paramters Finetune) and efficient parameter fine-tuning (Parameter EfficientFinetue) were carried out respectively: when using the full model parameter fine-tuning method, the original model parameters will be used for training; when using the efficient parameter fine-tuning method, the original pre-trained model parameters will be frozen uniformly, and only the heterogeneous multimodal hybrid expert adapter module in the training encoder and a small number of classifier head parameters in the decoder will be updated. In the evaluation, the output accuracy of the model is used as an indicator for comparison.

[0093] Comparison 1. Lora method: This method fine-tunes the pre-trained model by inserting a low-rank matrix, thereby reducing the number of training parameters while maintaining performance.

[0094] Comparison 2. Lora-FA method: This is a variant of the Lora method, which further optimizes the performance by fine-tuning the single-layer parameters of the Lora structure.

[0095] Contrast 3. Series-Adapter: The traditional serial adapter is a method of serially inserting adapter modules in a pre-trained model to fine-tune the model through these modules.

[0096] Comparison 4. Parallel-Adapter: Traditional parallel adapter is a method of inserting adapter modules in parallel in a pre-trained model, similar to serial adapter, but the modules work in parallel.

[0097] Table 1: Experimental results of the visual language classification task of the present invention on the VQA and NLVR2 datasets

[0098] Fine-tuning methods Training parameter percentage VQA NLVR2 Lora 1.5％ 73.27 80.01 Lora-FA 1.3％ 73.35 79.93 Series-Adapter 1.5％ 74.38 80.35 Parallel-Adapter 1.5％ 74.41 80.21 The present invention 4.9％ 75.22 82.19

[0099] Table 2: Experimental results of the present invention on the AVE dataset in the audio and video event localization (AVEL) task

[0100] Fine-tuning methods Training parameter percentage Model Accuracy Lora 4.4％ 78.84 Lora-FA 4.2％ 78.39 Series-Adapter 4.8％ 78.92 Parallel-Adapter 4.8％ 79.06 The present invention 6.6％ 81.05

[0101] Table 3: Experimental results of the present invention on the audio-visual video parsing (AVVP) task

[0102]

[0103] From the experimental results in Table 1, it can be seen that the method proposed in the present invention significantly outperforms other fine-tuning methods: Lora scores 73.27 on VQA and 80.01 on NLVR2; Lora-FA scores 73.35 on VQA and 79.93 on NLVR2; Series-Adapter scores 74.38 on VQA and 80.35 on NLVR2; Parallel-Adapter scores 74.41 on VQA and 80.21 on NLVR2. It can be seen that the method of the present invention scores 76.89 on the VQA task, which is about 2.5 percentage points higher than the closest Parallel-Adapter; and scores 82.19 on the NLVR2 task, which is about 1.8 percentage points higher than the closest Series-Adapter. This shows that the method improves performance while maintaining the advantage of parameter efficiency.

[0104] The experimental results in Table 2 show that the moe-adapter method proposed in this invention shows obvious superiority in the audio and video event localization (AVEL) task. Although the parameter volume is 6.6%, slightly higher than other parameter efficiency tuning methods (4.2% to 4.8%), the accuracy rate reaches 81.05%, which is significantly better than the highest value of 79.06% in other parameter efficiency tuning methods, demonstrating the excellent balance and optimization ability of this method between performance and parameter efficiency.

[0105] The experimental results in Table 3 show that the moe adpater method proposed in the present invention shows significant superiority in the audio-visual video parsing (AVVP) task. Although the parameter amount is 2.7%, which is slightly higher than other parameter efficiency tuning methods (2.0% to 2.1%), it has achieved better performance in all evaluation indicators. Specifically, in each subtask of Segment-level and Event-level, the method of the present invention has surpassed other methods in the scores of A, V, AV, Type and Event, showing comprehensive advantages at different levels.

[0106] Through the above specific implementation methods, the heterogeneous multimodal hybrid expert adapter of the present invention can effectively fuse multiple modal information, significantly improve the performance of large multimodal language models on downstream tasks, while maintaining low parameters and computational complexity, and has important application value.

[0107] The above-mentioned embodiments only express several implementation modes of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. For those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A heterogeneous multimodal hybrid expert adapter for fine-tuning a multimodal language model, characterized in that: The multimodal language model includes an encoder and a decoder, the encoder includes a plurality of encoder layers, and the heterogeneous multimodal hybrid expert adapter is inserted into each encoder layer; The heterogeneous multimodal hybrid expert adapter is composed of a global router, a local router and a plurality of expert groups, wherein the expert groups include a single-modal expert group, a cross-attention expert group and a channel-attention expert group, and each expert group contains M experts of the corresponding type; The single-modal experts in the single-modal expert group are used to perform single-modal representation learning on the input first modal embedding features or second modal embedding features, the first modal features are text modal features, and the second modal features are visual modal features. The cross-attention experts in the cross-attention expert group are used to perform cross-attention fusion on the first modal embedding features and the second modal embedding features in a low-rank space. The channel attention experts in the channel attention expert group are used to perform attention weighting on the channel dimensions of the first modal embedding features and the second modal embedding features. The global router is used to allocate global routing mixed weight coefficients to different expert groups, and the local router is used to activate K experts in each expert group and allocate local routing mixed weight coefficients to the K experts; The outputs of the activated experts in the expert group are weighted and summed by the local routing mixing weight coefficient as the output of the expert group, and the outputs of each expert group are weighted and summed by the global routing mixing weight coefficient as the output of the adapter. The output result of the adapter participates in the calculation processing of the encoder layer.

2. The heterogeneous multimodal hybrid expert adapter according to claim 1, characterized in that: The unimodal expert group includes two types of unimodal experts of the same number, namely text unimodal experts and visual unimodal experts. The text unimodal experts are used to perform unimodal representation learning on the first modality embedding features of the input, and the visual unimodal experts are used to perform unimodal representation learning on the second modality embedding features of the input.

3. The heterogeneous multimodal hybrid expert adapter according to claim 1, characterized in that: The global router allocates global routing mixed weight coefficients to different expert groups according to the following formula: R g (x)=SoftMax(W rg (x)) Among them, R g (x) is the global routing hybrid weight coefficient of the expert group, W rg (·) is the global router R g The trainable parameter matrix of , x represents the modal embedding features input to the expert group, and SoftMax( ) is the activation function.

4. The heterogeneous multimodal hybrid expert adapter according to claim 1, characterized in that: The local router activates K experts in each expert group and allocates local routing mixed weight coefficients to the K experts according to the following formula: R l (x) s =softMax(TopK(W r (x) s )) R l (x) c =SoftMax(TopK(W r (x) c )) R l (x) ca =SoftMax(TopK(W r (x) ca )) Among them, R l (x) s The local routing hybrid weight coefficient of the single-modal expert, R l (x) ca is the local routing mixing weight coefficient of the cross-attention expert, R l (x) c is the local routing hybrid weight coefficient of the channel attention expert, x represents the modal embedding feature input to the expert group, and W r (·) ca is the trainable parameter matrix of the local routing network of the criss-cross attention expert, W r (·) s is the trainable parameter matrix of the local routing network of the unimodal attention expert, W r (·) c is the trainable parameter matrix of the local routing network of the channel attention expert, TopK(W r ) means selecting K experts with the largest weight values ​​to activate, K<M.

5. The heterogeneous multimodal hybrid expert adapter according to claim 1, characterized in that: The specific implementation method of the unimodal expert to perform unimodal representation learning on the input first modal embedding features or second modal embedding features is: T out1 =T in +F relu (T in *W down )*W up V out1 =V in +F relu (V in *W down )*W up Among them, T in 、V in is the first modality embedding feature and the second modality embedding feature of the input, T out1 、V out1 is the output of the unimodal expert after unimodal representation learning, W down is the down-projection matrix, W up is the upper projection matrix, F relu is a non-linear activation function.

6. The heterogeneous multimodal hybrid expert adapter according to claim 1, characterized in that: The specific implementation method of the cross-attention expert to cross-attentionally fuse the first modality embedding features and the second modality embedding features in the low-rank space is: 1) Embed the first modality into feature T in and the second modality embedding feature V in Through the projection matrix W down Map to the low-rank space to obtain the mapped first modal feature T′ and second modal feature V′; 2) For the mapped first modal feature T′, a query matrix is ​​generated by querying the mapping matrix; for the mapped second modal feature V′, a key matrix is ​​generated by the key mapping matrix, and a value matrix is ​​generated by the value mapping matrix; 3) Calculate the attention weight matrix based on the query matrix, key matrix and value matrix; 4) Multiply the attention weight matrix by the value matrix to get the attention representation A t ; 5) Attention represents A t Added to the mapped first modal feature sequence T′, and then through the upper projection matrix W up Mapping back to the original dimension, we get the first modal feature T of cross-attention fusion out2 ; Attention represents A t Added to the mapped second modal feature sequence V′, and through the upper projection matrix W up Mapping back to the original dimension, we get the second modal feature V of cross-attention fusion out2 .

7. The heterogeneous multimodal hybrid expert adapter according to claim 1, characterized in that: The specific implementation method of the channel attention expert to weight the channel dimension of the first modality embedding feature and the second modality embedding feature is: T out3 =F relu (T in *W down )W up (1+Sigmoid(AvgPool(V in )⊙T in )) V out3 =F relu (V in *W down )W up (1+Sigmoid(AvgPool(T in )⊙V in )) Among them, T in 、V in is the first modality embedding feature and the second modality embedding feature of the input, T out3 、V out3 is the output of the channel attention expert after attention weighting, W down and W up are the down-projection and up-projection matrices, respectively, and F relu is a nonlinear activation function, AvgPool is the average pooling operation, Sigmoid is the activation function, and ⊙ represents the dot product.

8. The heterogeneous multimodal hybrid expert adapter according to claim 1, characterized in that: The input first modality embedding features and second modality embedding features are outputs of the network before the heterogeneous multimodal hybrid expert adapter is inserted into the position.

9. A method for fine-tuning a multimodal language model based on the heterogeneous multimodal hybrid expert adapter according to any one of claims 1 to 8, characterized in that The method comprises the following steps: 1) obtaining a training data set, wherein the training data set includes first modality data and second modality data corresponding to the first modality data; 2) constructing a pre-trained multimodal language model, the multimodal language model comprising an encoder and a decoder, the encoder comprising a plurality of encoder layers, and inserting the heterogeneous multimodal hybrid expert adapter into each encoder layer; 3) Inputting the training data set into the multimodal language model and training with a total loss function, wherein the total loss function is obtained by adding a loss term based on knowledge transfer and a generation loss term of the second modality data. During the training process, only the heterogeneous multimodal hybrid expert adapter in the training encoder and the classifier in the decoder are updated, and the other parts are frozen.

Images