A general multi-modal learning method based on a deep interaction adaptation network model

Abstract

The application discloses a kind of general multimodal learning methods based on depth interaction adaptive network model.The application steps:1, obtain the pre-training model of image single mode and text single mode, add multimodal adapter module between two pre-training models,2, respectively load the pre-training weight parameters of image and text single mode model and keep it unchanged, fine-tune the weight parameters of the designed adapter on the multimodal downstream task,3, inference deployment is carried out on the multimodal downstream task using the fine-tuned multimodal adapter model.The application extracts hierarchical features of two single mode models by constructing external adapter network for the pre-training model of image and text two single modes, and uses the constructed adapter to carry out multimodal fine-grained alignment fusion, so that the single mode pre-training model can be quickly applied to various multimodal tasks, and the performance can be compared with large-scale multimodal pre-training model.

Description

TECHNICAL FIELD

[0001] The application relates to a text and image feature fusion method, in particular to a general multi-modal learning method based on a deep interaction adaptive network model. BACKGROUND

[0002] A pre-trained model refers to a deep neural network model pre-trained on a large amount of data, which is used to first learn a general feature representation, and then fine-tuned on a specific downstream task to improve performance. In recent years, with the increase of large-scale data sets in various fields, the emergence of various self-supervised training methods and the development of computing hardware, methods based on pre-trained models have been widely used in natural language processing, computer vision, speech recognition and other fields. Among them, BERT, GPT, RoBERTa, T5 and other natural language pre-training models, and ResNet, ViT, Swin Transformer and other computer vision pre-training models have achieved outstanding results.

[0003] Multi-modal learning refers to jointly modeling data from different modalities to improve model performance, among which visual and language multi-modal learning is an important branch. In visual and language multi-modal learning, image and text information are generally jointly modeled, and a deep neural network is used to fuse and interact the features obtained by processing image and text data using image and text pre-training models, respectively, and then predict different downstream tasks. After various multi-modal pre-training methods are proposed, multi-modal models begin to abandon single-modal pre-training models and directly pre-train multi-modal models on a large amount of image and text data, such as ALBEF, UNITER, OSCAR, etc. After obtaining the pre-training model, it is fine-tuned on various specific multi-modal downstream tasks to obtain a special model. Common multi-modal downstream tasks include visual question answering, visual positioning, image description, etc.

[0004] Adapter technique is a new technology in the field of deep learning in recent years. The traditional "pre-training-fine-tuning" paradigm needs to retrain based on a large amount of labeled data of specific downstream tasks on the basis of pre-trained models, which not only consumes a lot of computing resources, but also easily leads to overfitting and performance degradation. At the same time, fine-tuning a completely different model for different downstream tasks brings a lot of training and deployment costs. Adapter technique can insert some trainable small adapter modules into the middle layer of the model while keeping the structure and weights of the pre-trained model unchanged, which can quickly learn the relevant knowledge for specific tasks and pass it to the pre-trained model to achieve the effect of rapid fine-tuning, while reducing the deployment cost for different tasks. Currently, adapter technique has been widely used in natural language processing, computer vision and other fields. In the field of multi-modal, some adapter methods have also been proposed recently to quickly transfer the knowledge of pre-trained models to downstream tasks. For example, some researchers proposed to take the output of the pre-trained image model as the input of the large pre-trained language model, and keep the weights of the pre-trained model unchanged, and insert some trainable small adapter modules into the large pre-trained language model, combining two single-modal pre-trained large models into a multi-modal model. However, the performance of this adapter method on multi-modal tasks is weaker than that of traditional multi-modal pre-trained models, and the utilization of the feature analysis ability of single-modal pre-trained models is still not sufficient. Therefore, how to design an adapter method for multi-modal tasks to better utilize existing single-modal pre-trained large models and quickly adapt them to downstream multi-modal tasks is still a challenging task.

[0005] [1]Liu Y,Ott M,Goyal N,et al.Roberta:A robustly optimized bertpretraining approach[J].arXiv preprint arXiv:1907.11692,2019.

[0006] [2]Radford A,Kim J W,Hallacy C,et al.Learning transferable visualmodels from natural language supervision[C] / / International conference onmachine learning.PMLR,2021:8748-8763.

[0007] [3] Radford A, Wu J, Child R, et al. Language models are unsupervised multitask learners[J]. OpenAI blog, 2019, 1(8): 9.

[0008] [4] Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[J]. arXiv preprint arXiv:2010.11929, 2020. SUMMARY

[0009] Most of the existing multimodal pre-training models need to consume a lot of training time and computing power, and also need a large amount of image-text pair data, how to better use the existing single-modal pre-training model in the multimodal task is still a problem worth exploring. The present application constructs an external adapter network for image and text two single-modal pre-training models, extracts hierarchical features of the two single-modal models, and uses the constructed adapter for multimodal fine-grained alignment and fusion, so that the single-modal pre-training model can be quickly applied to various multimodal tasks.

[0010] The technical solution adopted by the present application to solve its technical problems comprises the following steps:

[0011] Step 1: Obtain an image single-modal pre-training model and a text single-modal pre-training model, and add a multimodal adapter module between the two pre-training models;

[0012] The image pre-training model is used to process the image to obtain an image feature map, and a deep neural network is used to infer the image features to obtain the final image features; the text pre-training model is used to segment the natural language sentence, and a deep neural network is used to infer the word vector to obtain the text features. The added multimodal adapter includes a self-attention module and a guided attention module, the self-attention module adjusts the intermediate layer output of the single-modal model to be more suitable for the multimodal task and more suitable for the intermediate layer output of the other modal model, and the guided attention module fuses the intermediate layer output from the other modal model and the intermediate layer output of the current modal model and then injects it back into the intermediate layer of the current modal.

[0013] Step 2: Load the pre-training weight parameters of the image and text single-modal models respectively and keep them unchanged, and fine-tune the weight parameters of the designed adapter on the multimodal downstream task;

[0014] Step 3: Deploy the finely tuned multimodal adapter model for inference on multimodal downstream tasks.

[0015] Furthermore, the module for obtaining the unimodal pre-trained model and the multimodal adapter between the pre-trained models described in step 1 will be further explained.

[0016] 1.1 The selected single-modal pre-trained model is stratified.

[0017] Currently, single-modal models based on deep learning often use stacked network layers to improve the model's feature understanding ability, thereby improving performance. Taking the typical visual single-modal model CLIP-ViT[3] as an example, the model consists of a slice coding network and a visual Transformer encoder. The visual Transformer encoder consists of multiple repeating encoder modules, which perform detailed feature extraction on the input image features. The number of encoder modules is determined by the size of the model, ranging from a few layers to dozens of layers. The layering operation of this invention is to layer the stacked repeating encoders, and each layer is assigned an equal number of encoders. Taking the CLIP-ViT-Base model as an example, it has 12 stacked encoder modules. Using the layering operation of this invention to perform three-layer layering, each layer is assigned 4 encoder modules. Note that this layering operation only layers the encoder modules and does not adjust the order of the encoders during the forward computation of the model. Popular text single-modal models such as RoBERTa, BERT, and GPT also have similar stacked Transformer encoders or decoders, so they can also be easily layered. For the selected image pre-trained model model V and text pre-trained model T Perform a layering operation so that the two models are divided into the same number of layers, represented as follows: and

[0018] 1.2 Add a multimodal adapter between the layers of two unimodal pre-trained models.

[0019] After layering the model, a multimodal adapter is added between every two layers. For a unimodal model, the added multimodal adapter is located at the Block level. i and Block i+1 Specifically, the multimodal adapter accepts a hierarchical structure of image and text models. and The output is used as input, and the adapter output with the same shape as the input is calculated and then added back to the layered outputs of the image model and the text model. The above process is expressed by the following formula:

[0020]

[0021]

[0022]

[0023]

[0024] wherein V i represents the image feature of the i-th hierarchical output, T i represents the text feature of the i-th hierarchical output. Ada represents a multi-modal adapter module, represents the image feature of the i-th hierarchical output of the multi-modal adapter, represents the text feature of the i-th hierarchical output of the multi-modal adapter.

[0025] The multi-modal adapter is described as follows:

[0026] After receiving the image feature V i and the text feature T i , the multi-modal adapter first uses a linear layer to reduce the dimension of each of them. The purpose of dimension reduction is to reduce the computational load inside the adapter, and to unify the features output by different single-modal models to the same dimension for subsequent calculations. Experiments on multiple multi-modal downstream tasks show that a certain degree of dimension reduction will not affect the performance of the adapter, and can reduce the parameter quantity of the adapter.

[0027] After dimension reduction, the image feature and the text feature are processed using a self-attention module respectively. The formula of the self-attention mechanism can be expressed as:

[0028]

[0029] wherein, in the self-attention module, q, k, and v are the input features. i is used to index different attention heads, N head represents the total number of attention heads, Softmax(·) represents a softmax normalization function, d head represents the dimension of the attention head, W i represents a learnable weight.

[0030] The dimension-reduced image feature and text feature are first calculated via the self-attention mechanism, and the obtained features are then processed using a feedforward neural network. The specific process can be expressed as the following formula:

[0031] feat=Attention(q,k,v=feat)

[0032] feat=FFN(feat)

[0033] Wherein, feat represents the input feature, which can be image feature or text feature. FFN represents feed-forward neural network.

[0034] After the calculation, the image feature and the text feature processed by the self-attention module are obtained. After using a guided attention module to calculate the image feature and the text feature, the fusion feature is obtained. The final fusion feature is calculated by using a feed-forward neural network, and then added back to the original single-modal feature output. The formula is as follows:

[0035] V i ′=Attention(q=V i ′,k,v=T I ′)

[0036] V i =V i +FFN(V i ′)

[0037] T i =T i +T i ′

[0038] Wherein, V i ′ and T I ′ represent the features of the i-th single-modal model block output processed by the dimension reduction and self-attention module. It should be noted that before adding the original features V i and T i , the calculated features are dimensioned by using a linear layer to make them return to the same shape as the original features, and then added. Finally, the obtained V i and T i will be used as the input of the next single-modal model block.

[0039] The application provides a deep interaction adaptive network for general multi-modal learning, which comprises the following contents: using an existing image single-modal pre-training model to process images; using an existing text single-modal pre-training model to process natural language sentences; while keeping the weights of the two single-modal pre-training models from being updated, the two single-modal models are layered. A multi-modal adapter is added to the middle layer of the two single-modal models, and an attention mechanism is used to guide the two single-modal pre-training models to use the middle layer feature output of the other modal to optimize the middle layer feature of the modal.

[0040] The prominent substantial features of the present application are that: by adding an additional adapter network between the intermediate layers of the image single-modal pre-training model and the text single-modal pre-training model, the excellent modal understanding ability of the single-modal pre-training model can be effectively utilized, and it can be promoted to multi-modal alignment ability, which can quickly adapt to multi-modal tasks that require joint understanding of images and texts. Compared with the current optimal multi-modal pre-training model, the present application has achieved competitive precision with greatly reduced model training parameters. In addition, the multi-modal adapter provided by the present application has less requirements for the structure of the single-modal pre-training model, and can be adapted to a variety of different single-modal pre-training models to be suitable for different multi-modal tasks, and make the model perform better in multi-modal tasks. At the same time, since the adapter designed by the present method has small parameter quantity, and the weight parameters of the single-modal pre-training model are kept unchanged, only different adapters need to be loaded for different multi-modal downstream tasks, which can make the model easier to deploy. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 A flowchart of a multi-modal task adapter method based on a CLIP-ViT model and a Roberta model constructed according to an example embodiment of the present application;

[0042] Figure 2 An old paradigm diagram of a multi-modal task based on multi-modal pre-training;

[0043] Figure 3 A new paradigm diagram of a multi-modal task based on an adapter designed according to the present application;

[0044] Figure 4 A schematic diagram of the overall structure of a multi-modal task adapter based on a CLIP-ViT model and a Roberta model constructed according to an example embodiment of the present application;

[0045] Figure 5 A schematic diagram of the internal structure of a multi-modal adapter constructed according to an example embodiment of the present application; DETAILED DESCRIPTION

[0046] The present application will be further described in detail below with reference to the accompanying drawings. Unless otherwise specifically stated, the relative arrangement, numerical expressions and values of the components and steps of the examples set forth in the drawings do not limit the scope of the present application.

[0047] As shown in Figure 1 , the deep interactive adapter network construction for multi-modal multi-task learning provided by the present application includes three steps.

[0048] Among them, step (1) is specifically implemented as follows:

[0049] For the image single-modal model and the text single-modal model, a multi-modal adapter is externally added, and the image single-modal model and the text single-modal model are taken as examples of CLIP-ViT and RoBERTa, it should be noted that the present application has less restrictions on single-modal models, and most image or text single-modal models can be used.

[0050] The present application adopts a new paradigm of multi-modal tasks, as shown in Figure 2 and Figure 3 . Figure 2 represents a multi-modal pre-training paradigm, Figure 3 represents an example of the new paradigm adopted by the present application. In the old paradigm of multi-modal pre-training, first, an image model and a text model are designed to preliminarily process the image-text pairs of a large-scale image-text pair dataset to obtain image features and text features, and then a multi-modal model is used to jointly model and align the obtained image features and text features. For different multi-modal pre-training models, the design of the image model and the text model will be different, and the multi-modal model for jointly modeling and aligning the features of the two is mostly a multi-layer stacked deep attention network. After using the above model structure to pre-train on a large-scale image-text pair dataset using various multi-modal pre-training methods, a special-purpose model is obtained by fine-tuning the obtained multi-modal pre-training model for different multi-modal downstream tasks. In the new paradigm shown in Figure 3 , first, the single-modal models are pre-trained on their respective single-modal datasets, then a multi-modal adapter is added between the two single-modal models, and then fine-tuning is performed on the dataset of the multi-modal downstream task. Since there are currently a large number of publicly available image single-modal pre-training models and text single-modal pre-training models, the pre-training step can be omitted, and the existing models can be directly selected for loading.

[0051] The specific multi-modal adapter adding method can be referred to Figure 4 . Taking CLIP-ViT as the image single-modal pre-training model and RoBERTa as the text single-modal pre-training model as an example, Figure 4 represents the overall structure of the model after adding the multi-modal adapter, Figure 5 represents the specific structure of the designed multi-modal adapter. Specifically, first, the image single-modal pre-training model and the text single-modal pre-training model are blocked according to their number of layers, the blocking number of the two models is kept the same, and a multi-modal adapter module is inserted between every two blocks. As shown in Figure 5As shown, the multi-modal adapter accepts the intermediate layer output from the image single-modal model and the intermediate layer output from the text single-modal model, processes the features from the single-modal using a self-attention module respectively, and then inputs the two single-modal features processed into a guided attention module to obtain the image feature after the fusion of the text feature. Finally, the image feature and the text feature output by the multi-modal adapter module are added back to the original single-modal features, and are input back into the single-modal model.

[0052] The following takes Figure 4 One of the hierarchical outputs of the embodiment is taken as an example to specifically illustrate the operation of the image feature and the text feature in the multi-modal adapter.

[0053] In a multi-modal adapter, the image feature and the text feature of the intermediate hierarchical output of the single-modal model are first reduced in dimension, and two linear layers are used to reduce the two features to 256 dimensions. The original input image feature is V, and the original input text feature is T. The above operation is expressed by the following formula:

[0054] V 256 =Linear(V), T 256 =Linear(T)

[0055] V 256 , T 256 are the reduced dimension features. Linear(·) represents the linear layer used. After dimension reduction, the obtained image feature and text feature are processed using a self-attention module respectively.

[0056] V sa =Attention(q, k, v = V 256 )

[0057] T sa =Attention(q, k, v = T 256 )

[0058] V′ sa =FFN(V sa ), T′ sa =FFN(T sa )

[0059] wherein V sa , T sa represent the image feature and the text feature calculated via the self-attention mechanism, V′ sa , T′ sa represent the image feature and the text feature calculated via the feedforward neural network. It should be noted that the features calculated above are processed via a Dropout layer and a layer normalization layer.

[0060] After the calculation is completed, a guided attention module is used to calculate the obtained features to obtain fused features. The final fused features are calculated using a feedforward neural network and added back to the original single-modal feature output. The formula is as follows:

[0061] V' ga = Attention(q = V' sa , k, v = T' sa )

[0062] V = V + Linear(FFN(V' ga ))

[0063] T = T + Linear(T' sa )

[0064] wherein V' ga represents the image feature calculated by the guided attention, which is dimensioned by a linear layer after being calculated by the feedforward neural network, and then added to the original image feature after returning to the same shape as the original image feature. The text feature calculated before is also dimensioned by a linear layer and then added to the original text feature. The final output V i and T i will be used as the input of the next single-modal model block.

[0065] It should be pointed out that this embodiment is mainly applicable to the image question answering task in the multi-modal downstream task, which mainly focuses on the content understanding in the image, so when the guided attention mechanism is used, the calculation result is guided to the image feature side. When other multi-modal downstream tasks such as image description are used, the model can simply modify the guided attention to the text feature side to better adapt to the downstream task.

[0066] The specific implementation of step (2) is as follows:

[0067] For different selected single-modal models, load their pre-training parameters and keep them unchanged, randomly initialize the parameters of the added multi-modal adapter, and fine-tune the whole model. The fine-tuning process can select the corresponding data set according to different downstream tasks, and set the corresponding training rounds, batch size, optimizer and learning rate according to the model accuracy and other requirements to fine-tune and update the parameters of the added multi-modal adapter.

[0068] The specific implementation of step (3) is as follows:

[0069] The fine-tuned multi-modal adapter model can be used in various multi-modal downstream task scenarios such as image question answering, visual positioning, image description, etc. The image question answering task is used to infer and answer for an image and a natural language question, the visual positioning task is used to detect and position the target for an image and a natural language sentence, and the image description task is used to infer and generate a natural language description for an image.

[0070] The model training process of the present application can be performed offline on a server or in the cloud. The trained model can be deployed to an electronic device to implement the multi-modal task reasoning. The electronic device can be a terminal device or a server. The terminal device includes a mobile phone, a tablet computer, or any terminal device. The server includes but is not limited to an application server or a Web server, and can be an independent server, a cluster server, or a cloud server, etc. Since the pre-training parameters of the single-modal model are kept unchanged in the present application, and the amount of multi-modal adapter parameters added is small, a single-modal model parameter can be reserved for various different multi-modal downstream tasks during deployment, and different multi-modal adapter parameters can be loaded for different multi-modal tasks, thereby reducing the parameter storage cost.

[0071] The present model has been verified in various multi-modal downstream tasks, and the experimental details are as follows.

[0072] Embodiment 1

[0073] (1) Image question answering experiment

[0074] In the experiment, the VQAv2 dataset is used to verify the effect of the present application on the image question answering task. The visual pre-training model uses CLIP-ViT, and the text pre-training model uses RoBERTa, and the pre-training weights thereof are initialized and fixed. The two models are divided into three layers, and three designed multi-modal adapters are inserted, and the adapter parameters are randomly initialized. The model is trained for 13 rounds, the batch size is 128, the optimizer is Adam, and the learning rate is 0.0005.

[0075] The experimental results are shown in Table 1. The multi-modal adapter method designed in the application can well adapt two single-modal pre-training models to a multi-modal task and achieve a performance comparable to that of a multi-modal pre-training model. The accuracy of the Base model of CLIP-ViT and the Base model of RoBERTa after training on the test-dev set of VQAv2 is 73.70, which is higher than that of the traditional multi-modal pre-training model UNITER and OSCAR, and also higher than that of the model reported by METER, which also uses the Base model of CLIP-ViT and the Base model of RoBERTa, and uses a multi-modal encoder to understand the single-modal output of the model. It should be particularly pointed out that the parameter amount required by the method proposed by us is much smaller than that of the above-mentioned other models. The parameter amount of the newly added multi-modal adapter is only 6.9M, while the trainable parameter amount of the other models in the table is about 80M.

[0076] Table 1

[0077] Model VQA accuracy UNITER-Base 72.70 OSCAR-Base 73.16 METER(CLIP-B&RoBERTa) 71.75 Ours(CLIP-B&RoBERTa) 73.70

[0078] Embodiment 2

[0079] (2) Visual positioning experiment

[0080] The experiment uses the RefCOCO+ dataset to verify the effect of the application in the visual positioning task. The model used in this experiment is the same as that in the image question answering experiment, only the linear layer in the output part is modified from outputting the answer index to outputting four coordinates. The experiment is trained for 120 iterations, the training batch size is 128, and the learning rate is 0.0004. Finally, the accuracy on the validation set reaches 78.00, which is higher than that of the traditional multi-modal pre-training model UNITER, ROSITA, and also higher than that of the TransVG++, Dynamic MDETR and other methods trained using models such as CLIP-ViT.

[0081] Table 2

[0082]

[0083]

[0084] Embodiment 3

[0085] (3) Image description experiment

[0086] The image description experiment is verified using the COCO dataset. The CLIP-ViT-Base model is used for the experiment with the GPT2-Base model, and the two models are also divided into three layers, and three designed multi-modal adapters are inserted. The model is trained for 18 rounds, the training batch size is 80, the optimizer is Adam, the learning rate is 0.00007, and the cross-entropy loss is used for optimization. The CIDEr score of the experimental results is 119.85, which is better than the I-Tuning method using the GPT2-Base model and the CLIP-ViT-Base model.

[0087] Table 3

[0088] Model CIDEr score OSCAR-Base 123.7 VL-T5 116.5 VL-Adapter 116.0 I-Tuning(GPT2-Base) 116.7 Ours(CLIP-B&GPT2-Base) 119.8

Claims

1. A general multi-modal learning method based on a deep interaction adaptation network model, characterized in that The method comprises the following steps: Step 1: obtaining an image single-modal pre-training model and a text single-modal pre-training model, and adding a multi-modal adapter module between the two pre-training models; Step 2: loading the pre-training weight parameters of the single-modal model and keeping them unchanged, and fine-tuning the weight parameters of the designed adapter on a multi-modal downstream task; Step 3: using the fine-tuned multi-modal adapter model to perform inference on a plurality of different multi-modal downstream tasks; The multi-modal adapter module added in step 1 comprises the following sub-steps: Step 1.1: layering the selected single-modal pre-training model; Layering the repeated encoder of the single-modal model, and assigning an equal number of encoders to each layer block; The layering operation is only performed on the encoder module, and the order of the encoders before and after the forward calculation of the model is not adjusted; The selected image pre-training model and the text pre-training model The hierarchical operation is performed, and the two models are divided into the same number of layer blocks, respectively represented as and ; Step 1.2: adding a multi-modal adapter between the layering of the two single-modal pre-training models; After layering the model, a multi-modal adapter is added between each two layer blocks; For the single modality model, the added multi-modality adapter position is between and ; The multi-modal adapter accepts as input the outputs of the image model and the text model and of the hierarchical model. The calculated output has the same shape as the input, and then the calculation output is added back to the layered output of the image model and the text model; The multi-modal adapter in step 1.2 accepts as input the hierarchical outputs of the image model and the text model, computes an adapter output of the same shape as the input, and adds it back to the hierarchical outputs of the image model and the text model; the formula is as follows: and the hierarchical outputs of the image model and the text model, computes an adapter output of the same shape as the input, and adds it back to the hierarchical outputs of the image model and the text model; the formula is as follows: ； ； ； ； wherein, represents image features of the i-th hierarchical output, represents text features of the i-th hierarchical output; represents a multi-modal adapter module, represents image features of the i-th hierarchical output of the multi-modal adapter output, represents text features of the i-th hierarchical output of the multi-modal adapter output.

2. The universal multi-modal learning method based on the deep interaction adaptation network model according to claim 1, characterized in that, Step 2 comprises the following sub-steps: using the model constructed in step 1, importing the weight of the two selected single-modal pre-training models and fixing the weight; the parameters of the multi-modal adapter part are randomly initialized; the model is fine-tuned on the downstream image question answering dataset, visual positioning dataset and image description dataset.

3. The universal multi-modal learning method based on the deep interaction adaptation network model according to claim 1, characterized in that Step 3 comprises the following sub-steps: only the parameters of the single-modal pre-training model and the multi-modal adapter parameters for different tasks need to be saved when deploying.

4. A multi-modal adapter, adapted for a general multi-modal learning method based on a deep interaction adaptation network model according to claim 1, characterized in that, The self-attention module for single modality and the guided attention module for multi-modality, the hierarchical output of the input image model and the hierarchical output of the text model and After the outputs of the self-attention module, the fusion feature is calculated by the guided attention module, and the obtained fusion feature is calculated by a feedforward neural network as the output of the multi-modal adapter.

5. A multi-modality adapter according to claim 4, wherein, The multi-modal adapter internally receives image features and text features afterwards; Linear layers are used to reduce the dimensionality of both, and the purpose of dimensionality reduction is to reduce the computational complexity of the adapter and unify the features output by different single-modal models to the same dimension for subsequent calculation; ； to obtain the reduced dimension features; denotes the linear layer used; After dimensionality reduction, the image features and text features are processed by a self-attention module respectively; The formula of the self-attention mechanism is as follows: ； wherein in the self-attention module, , , are the input features; is used to index different attention heads, denotes the total number of attention heads, denotes the softmax normalization function, denotes the dimension of the attention heads, denotes the learnable weights; After dimensionality reduction, the image features and text features are calculated by the self-attention mechanism, and the obtained features are processed by a feedforward neural network, and the formula is as follows: ； ； ； wherein, denote the image feature and the text feature calculated via the self-attention mechanism, denote the image feature and the text feature calculated via the feedforward neural network; The features obtained by the above calculation are processed by a Dropout layer and a layer normalization layer; After calculation, the image features and text features processed by the self-attention module are obtained; after calculation by a guided attention module, the fusion features are obtained; The fusion features are calculated by a feedforward neural network and added back to the original single-modal feature output; the formula is as follows: ； ； ； wherein, represents the image feature after attention is guided, will be added to the original image feature after using a linear layer to upgrade the dimension calculated by the feedforward neural network, so that it returns to the same shape as the original image feature. The output is obtained and will be input as the next single-modal model block.

Images