A lightweight adaptive network learning method for multi-modal multi-task learning

Abstract

The application discloses a lightweight adaptive network learning method for multi-modal multi-task learning, comprising the following steps: 1, constructing a downstream task dataset, 2, constructing a deep self-attention network model, 3, pre-training weight pruning, 4, constructing a task adapter, 5, adapting a pre-training model, 6, designing a progressive guided distillation training and training a model. The method prunes part of the weights of the pre-training model and adapts it with an efficient task adapter. The application proposes a progressive guided distillation training algorithm to better fill the gap between the pre-training task and the downstream task, ensuring the performance of the model on the downstream task. The application can be combined with any existing pre-training model based on a deep self-attention network, and through training, an adapter model with superior performance in the downstream task, total storage overhead during model deployment, calculation overhead during model inference and model configuration flexibility is obtained.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of lightweight multi-modal learning, and particularly relates to a lightweight adapter learning method for multi-modal multi-task learning. BACKGROUND

[0002] In recent years, various research fields of artificial intelligence have benefited from the rise of deep self-attention network architecture and self-supervised pre-training paradigm, and have achieved great success. For example, in the multi-modal field, researchers use large-parameter deep self-attention network architecture model and this training paradigm to first pre-train on a large corpus of image-text pairs to learn multi-modal task general knowledge, and then fine-tune the parameters for different multi-modal tasks, such as visual question answering, visual target positioning, image description, image-text retrieval, natural language visual reasoning, and visual entailment reasoning. This large model based on the "pre-training-fine-tuning" paradigm changes the full parameters of the model when fine-tuning for different downstream tasks, meaning that multiple large-scale models with different weights need to be deployed for different downstream tasks when the model is landed, which consumes a lot of storage space.

[0003] Therefore, an adapter fine-tuning method that does not change the structure and parameter weights of the pre-trained model but inserts lightweight learnable parameters into the model has emerged. Specifically, by inserting a small number of learnable parameters called adapters into the pre-trained model, only these parameters are trained during downstream task fine-tuning while the original parameters of the pre-trained model remain unchanged, making the pre-trained model more efficient when fine-tuning for multiple downstream tasks.

[0004] Although the existing adapter fine-tuning methods are parameter-efficient and reduce storage costs when deploying the model for multiple tasks, they all maintain the original parameters of the pre-trained model and adapt to multiple downstream tasks by adding a small number of parameters, which results in a larger model size than the original model, increasing memory usage during model training and inference overhead during application. If the representation capabilities of different parts of the pre-trained model can be analyzed in depth, the general part of the downstream task can be retained and the part that is useless for the downstream task can be pruned, and then a lightweight adapter can be introduced, which not only makes the parameter quantity efficient, but also makes the model inference computationally efficient. Therefore, designing a multi-aspect efficient adapter fine-tuning method for pre-trained models not only has certain application value in the deployment field of pre-trained models, but also has certain academic value for research in other fields.

[0005] In summary, how to design an efficient adapter fine-tuning method and combine it with existing pre-trained models is a topic worthy of further research. This patent intends to explore from several key points in this task, solve the difficulties and key points of the current method, and form a complete and efficient lightweight adapter fine-tuning method. SUMMARY

[0006] In view of the deficiencies in the prior art, the application provides a lightweight adaptive network learning method for multi-modal multi-task learning, which can be combined with any existing pre-training model based on deep self-attention network, and through training, an adapter model with superiorities in downstream task performance, total storage overhead during model deployment, calculation overhead during model inference and model configuration flexibility is obtained.

[0007] The application mainly includes two points:

[0008] 1. By analyzing the representation ability of the weights of different parts of the pre-training model, the application prunes part of the pre-training weights, designs an efficient adapter to adapt to them, and proposes a pre-training model adapter architecture of pruning first and filling later, so that the adapted model is lighter than the original model.

[0009] 2. In order to better fill the difference between the pre-training task and the downstream task, the application proposes a progressive guided distillation training algorithm for adapter fine-tuning, so that the adapter model is trained step by step to obtain better downstream task performance.

[0010] The application provides a lightweight adaptive network learning method for multi-modal multi-task learning, which is suitable for the scene of multi-task deployment of pre-training models. In order to reduce the calculation overhead during model inference, the application prunes part of the weights of the pre-training model, thereby reducing the calculation overhead by reducing the size of the model. At the same time, in order to reduce the storage overhead during deployment of multi-modal models, the application uses an efficient adapter structure, adopts adapter fine-tuning to reduce trainable parameters, and thereby reduces the storage overhead during deployment. Finally, in order to better fill the difference between the pre-training task and the downstream task, a progressive guided distillation training algorithm is proposed to ensure the performance of the model on the downstream task. The application can be combined with any existing pre-training model based on deep self-attention network, and through training, an adapter model with superiorities in downstream task performance, total storage overhead during model deployment, calculation overhead during model inference and model configuration flexibility is obtained.

[0011] A lightweight adaptive network learning method for multi-modal multi-task learning, comprising the following steps:

[0012] Step (1): constructing a downstream task data set;

[0013] The application takes the multi-modal research field as a breakthrough point, and selects four multi-modal downstream tasks of visual question answering, natural language visual reasoning, visual entailment reasoning and visual target positioning to construct training, verification and test datasets. For image data in the dataset, the existing trained Faster R-CNN target detection network is used to extract regional image features; for text data in the dataset, the semantic features are extracted by using word embedding vectors. Then the extracted image and text features are spliced to obtain the final input features.

[0014] Step (2): constructing a deep self-attention network model;

[0015] The deep self-attention network is stacked by multiple structures, and each layer is composed of a multi-head attention module and a feedforward layer. The deep self-attention network model is constructed to deeply understand and process the input features, so as to obtain multi-modal features with richer meanings.

[0016] Step (3): pre-training weight pruning;

[0017] In order to reduce the size of the pre-training model and improve the inference speed of the model, part of the pre-training weights is pruned. According to the structure of the deep self-attention network model, the weight pruning is divided into two types of cutting multi-head attention module and cutting feedforward layer.

[0018] Step (4): constructing a task adapter;

[0019] The task adapter refers to a small amount of learnable parameters inserted into each layer of the deep self-attention network model. The model only trains these parameters when fine-tuning the downstream task, and keeps the original pre-training weights unchanged, so that the model still has the learning ability of the downstream task while retaining the pre-training general knowledge. A lightweight task adapter contains two learnable parameter matrices, and a nonlinear activation function is contained in the middle.

[0020] Step (5): adapting the pre-training model;

[0021] The pre-training model after step (3) is combined with the task adapter of step (4) to obtain the adapted pre-training model. For different downstream tasks, they share the pruned pre-training weights and have different task adapters. Similarly, according to the structure of the deep self-attention network model, the adapted pre-training model is divided into two types of adapted multi-head attention module and adapted feedforward layer.

[0022] Step (6): designing progressive guided distillation training and training the model;

[0023] In order to stabilize the training of the adapter model obtained in step (5), the application proposes a progressive guided distillation training algorithm. Specifically, a full fine-tuning model under the traditional "pre-training-fine-tuning" paradigm is used as a teacher model, and the student model is the adapter model obtained in step (5). In each training iteration, the teacher model and the student model are trained together, and the knowledge learned by the teacher model is gradually distilled to the student adapter model in a step-by-step guided manner. In order to better align the parameter distribution of each layer of the student model, in addition to the output features of the distillation model, the training algorithm simultaneously distills the output features of each layer of the teacher model.

[0024] Further, the construction of the downstream task dataset in step (1) is as follows:

[0025] Among the multi-modal downstream tasks, the visual question answering task uses the VQA-v2 dataset, the natural language visual reasoning task uses the NLVR2 dataset, the visual entailment reasoning task uses the SNLI-VE dataset, and the visual target positioning task uses the Ref-COCO, Ref-COCO+, and Ref-COCOg datasets. All datasets are further divided into 3 subsets: training set, validation set, and test set. The training set is used to train the model, the validation set is used to verify the convergence of the model locally, and the test set is used for final model performance evaluation.

[0026] For the images in the above multi-modal dataset, a pre-trained Faster R-CNN object detection model on the Visual Genome dataset is used to extract the region features of the images where m is the number of region candidate boxes of the image, D od is the region feature dimension. Then, a learnable linear transformation is used to further process the image features extracted by the object detection model, map the feature dimension to the D-dimensional space, and obtain the final image region feature The specific formula is as follows:

[0027] X image = Linear(X od ) (1)

[0028] For the text in the above multi-modal dataset, a word embedding vector is used to extract the semantic features of the text where n is the number of words in the text, and D is the semantic feature dimension, which is the same as the final image region feature dimension.

[0029] Then, the extracted image and text features are concatenated to obtain the final input feature The specific formula is as follows:

[0030] X input = [Ximage , X text ] (2)

[0031] where num = m + n is the total number of image and text features.

[0032] Further, the step (2) of constructing the deep self-attention network model is specifically as follows:

[0033] The deep self-attention network is stacked by multiple layers with the same structure Layer, and each layer is composed of a multi-head attention module MHA and a feed-forward layer FFN. The deep self-attention network model is constructed to deeply understand and process the input features to obtain more rich multi-modal features

[0034] 2-1. Multi-head attention module MHA;

[0035] For a given query feature key feature and value feature D is the dimension of the feature, the multi-head attention module contains H parallel attention heads, and the feature is calculated. The specific formula is as follows:

[0036] F mha = MHA (Q, K, V) = [head1, head2, …, head H ]W 0 (3)

[0037]

[0038] where represents the projection matrix of the hth attention head, D H represents the feature dimension of each attention head, which is calculated by D H = D / H. At the same time, the matrix further maps the feature calculated by the multi-head attention. ATT represents the attention calculation method, which performs scaled dot-product operation on the processed query feature Q and key feature K to obtain the attention matrix, and then performs weighted summation with the processed value feature V. The specific formula is as follows:

[0039]

[0040] 2-2. Feed-forward layer FFN;

[0041] The feed-forward layer contains two fully connected layers and an activation function, which maps the output feature As input features, first project them into a high-dimensional space, and then map them back to the original dimension to obtain output features The specific formula is as follows:

[0042]

[0043] Wherein is a linear transformation projection matrix, and Nonlinear is a nonlinear activation function.

[0044] 2-3. Self-attention layer Layer;

[0045] Each self-attention layer Layer includes the multi-head attention module MHA and the feedforward layer FFN described above. For a given input feature The self-attention layer Layer processes the output feature The specific formula is as follows:

[0046]

[0047]

[0048] Wherein, LN represents layer normalization.

[0049] 2-4. Deep self-attention network model Model;

[0050] Considering that the feature dimension D does not change after each self-attention layer Layer, multiple self-attention layers Layer can be stacked to form a deep self-attention network model, denoted as Model, which processes the input feature to understand and process in depth, so as to obtain more rich multi-modal features The specific formula is as follows:

[0051] Model=[Layer (1) , Layer (2) , …, Layer (L) ] (9)

[0052] X output =Model(X input ) (10)

[0053] Wherein, L is the number of self-attention layers. Then, the model is initialized using the pre-trained weights W pretrain on a large-scale image-text pair corpus. The specific formula is as follows:

[0054] Model←W pretrain (11)

[0055] Further, the pre-training weight pruning of step (3) is specifically as follows:

[0056] To reduce the pre-training model size to improve the model inference speed, the pre-training weight is pruned in the present application. According to the deep self-attention network model structure of step (2), the pre-training weight pruning is divided into two types of cutting multi-head attention module MHA and cutting feedforward layer FFN.

[0057] 3-1. Cutting multi-head attention module MHA;

[0058] The multi-head attention module is aimed at cutting the number of attention heads H without changing the input and output feature dimensions D, so as to reduce the module parameter quantity. Specifically, for a given input feature The calculation method of the cut multi-head attention module is as follows:

[0059] F p_mha = MHA (Q, K, V) = [head1, head2, …, head H-t ]W o (12)

[0060]

[0061] Wherein, H is the original number of attention heads, and t is the number of cut attention heads. represents the parameter matrix of the hth attention head, D H = D / H is the dimension of each attention head. ATT is the attention calculation method, as shown in formula (5). In order to match the cut attention feature dimension (H-t)·D H , the corresponding cutting is made. is the output feature of the finally cut attention module, and the dimension is consistent with the input feature, indicating that the MHA module after attention head cutting will not change the feature dimension.

[0062] 3-2. Cutting feedforward layer FFN;

[0063] The cutting feedforward layer is aimed at cutting the parameter matrices W1 and W2 contained therein without changing the input and output feature dimensions D. Specifically, for a given input feature The calculation method of the cut feedforward layer FFN is as follows:

[0064]

[0065] Wherein, is the parameter matrix of the cut feedforward layer FFN module, s is the set cutting dimension, and Nonlinear is the activation function. For output features, the dimension is consistent with input features , i.e. the dimension of features does not change after being sliced by FFN.

[0066] Further, the step (4) of constructing the task adapter is specifically as follows:

[0067] The task adapter refers to a small amount of learnable parameters inserted into each layer of the deep self-attention network model. The model only trains these parameters when fine-tuning the downstream task while keeping the original pre-training weights of the model unchanged. Finally, the model still has the learning ability of the downstream task while retaining the pre-training general knowledge. A lightweight task adapter contains two learnable parameter matrices W down and W up , and a nonlinear activation function in between. For input features F adp_in and output features F adp_out , the calculation of each task adapter is as follows:

[0068] F adp_mid = Nonlinear(F adp_in W down ) (15)

[0069] F adp_out = F adp_in + F adp_mid W up (16)

[0070] wherein, Nonlinear is a nonlinear activation function, D is the dimension of input and output features, r is the adapter size parameter, the larger the r is, the more the learnable parameters are, and usually the stronger the learning ability of the adapter is. The feature dimension D of the input feature does not change after being processed by the task adapter, so it can be conveniently inserted into the deep self-attention network model. In order to make the model training stable, each task adapter contains a residual connection, and the input feature F adp_in is connected to the output feature F adp_out through a bypass connection.

[0071] Further, the step (5) of adapting the pre-training model is specifically as follows:

[0072] By combining the sliced pre-training model of step (3) with the task adapter of step (4), the adapted pre-training model is obtained. For different downstream tasks, they share the sliced pre-training weights and have different task adapters. Similarly, according to the structure of the deep self-attention network model, the adapted pre-training model is adapted to the multi-head attention module and the feedforward layer.

[0073] 5-1. Adapted Multi-Head Attention Module MHA;

[0074] The adapted multi-head attention module MHA aims to insert a small number of trainable task adapters as new attention heads for the split pre-trained MHA. Specifically, for a given downstream task input feature The calculation of the adapted multi-head attention module is as follows:

[0075]

[0076]

[0077] wherein, represents the h-th adapted attention head parameter matrix, D represents the input feature dimension, D H is consistent with the original attention head. ATT is the attention calculation method, ah is the number of settable adapted attention heads, is the task adapter attention head output feature, which is spliced with the original attention head output to obtain the adapted attention feature. In order to match the dimension of the adapted attention feature, is additionally introduced and spliced with the split to process the adapted attention feature together, and the final output feature is consistent with the input feature. The adapted multi-head attention module MHA only W adpQ , W adpK , W adpV and are trainable, and the remaining parameters remain unchanged in model training.

[0078] 5-2. Adapted Feed-Forward Layer FFN;

[0079] The adapted feed-forward layer FFN aims to make up and adapt W1 and W2 matrices in the split FFN module, so that it has the learning ability of downstream tasks. Specifically, for a given input feature The calculation of the adapted feed-forward layer FFN is as follows:

[0080]

[0081] wherein, af is the settable feed-forward layer adaptation size, which is spliced with the split pre-trained matrices W1 and W2, so that the adapted feed-forward layer FFN can not only retain general pre-training knowledge but also have the learning ability of downstream tasks. is the final output feature, which is consistent with the input feature in dimension. Similarly, in the adapted feed-forward layer FFN, only and The remaining parameters remain unchanged in the model training.

[0082] Further, the step (6) is designed to gradually guide the distillation training and train the model, specifically as follows:

[0083] The present application proposes a progressive guided distillation training algorithm, which adopts a full fine-tuning model under the traditional "pre-training-fine-tuning" paradigm as a teacher model, and the student model is the adapter model obtained in step (5). In each training iteration, the teacher model and the student model are trained together, and the knowledge learned by themselves is gradually distilled to the student adapter model in a step-by-step guided manner. In order to better align the parameter distribution of each layer of the student model, in addition to the output features of the distillation model, the training algorithm synchronously distills the output features of each layer of the teacher model.

[0084] Let the teacher model be M tea , and the model architecture is the original deep self-attention network model without segmentation adaptation; while the student model is M stu , and the model architecture is the adapter model obtained in step (5) after segmentation adaptation. In each training iteration, a batch of input data X and target Y are randomly sampled and input into the teacher model M tea (X), to obtain the predicted label Y tea and the output of each layer Layer tea , and calculate the teacher model loss Update the gradient Then freeze the feature gradient Y tea .detach(), Layer tea .detach() of the teacher model. In the current training iteration, the input data X is synchronously input into the student model M stu (X), to obtain the predicted label Y stu and the output of each layer Layer stu , and calculate the loss:

[0085]

[0086]

[0087]

[0088] Wherein, is the output loss, is the layer loss, is the final loss of the student model. λ1 is the output loss adjustment ratio, λ2 is the layer loss adjustment ratio, and its value can be set. After obtaining the total loss, calculate the gradient and update

[0089] The beneficial effects of this invention are as follows:

[0090] This invention analyzes the representational ability of weights in different parts of a pre-trained model, prunes some pre-trained weights, and designs an efficient adapter to adapt to them. It proposes a pre-trained model adapter architecture of pruning and then filling in the pruning, which makes the adapted model lighter than the original model.

[0091] This invention addresses the scenario of multi-task deployment of pre-trained models. First, it prunes some weights of the pre-trained model, reducing computational overhead by decreasing the model size. Simultaneously, it utilizes an efficient task adapter structure, employing adapter fine-tuning to reduce trainable parameters, thereby lowering storage overhead during model deployment. Finally, it proposes a progressive guided distillation training algorithm to better bridge the gap between pre-training tasks and downstream tasks, ensuring model performance on downstream tasks. This invention can be combined with any existing pre-trained model based on deep self-attention networks to obtain an adapter model that exhibits superior performance in downstream tasks, lower total storage overhead during model deployment, lower computational overhead during model inference, and greater model configuration flexibility. Attached Figure Description

[0092] Figure 1 This is a schematic diagram of pre-trained weight pruning in an embodiment of the present invention.

[0093] Figure 2 This is a schematic diagram of the task adapter structure in an embodiment of the present invention.

[0094] Figure 3 This is a schematic diagram of the adapted pre-trained model in an embodiment of the present invention. Detailed Implementation

[0095] The detailed parameters of the present invention will be further described below with reference to the accompanying drawings.

[0096] like Figure 1 As shown in Figures 2 and 3, this invention proposes a lightweight adaptive network learning method for multimodal and multitask learning.

[0097] The construction of the downstream task dataset in step (1) is as follows:

[0098] The final datasets used include VQA-v2, NLVR2, SNLI-VE, Ref-COCO, Ref-COCO+, and Ref-COCOg, covering four multimodal downstream tasks: visual question answering, natural language visual reasoning, visual entailment reasoning, and visual object localization. All datasets were divided into three subsets: training set, validation set, and test set. The training set was used to train the model, the validation set was used to locally verify the model's convergence, and the test set was used for the final model performance evaluation.

[0099] Further, for the images in the above multi-modal dataset, the region features of the images are extracted using a Faster R-CNN object detection model pre-trained on the Visual Genome dataset where m is the number of region candidate boxes of the image, D od is the dimension of the region feature. Subsequently, a learnable linear transformation is used to further process the image features extracted by the object detection model, map the feature dimension to D-dimensional space, and obtain the final image region feature The specific formula is as follows:

[0100] X image = Linear(X od ) (1)

[0101] For the text in the above multi-modal dataset, the semantic features of the text are extracted using word embedding vectors where n is the number of words in the text, and D is the dimension of the semantic feature, which is the same as the dimension of the final image region feature.

[0102] Subsequently, the extracted image and text features are concatenated to obtain the final input feature The specific formula is as follows:

[0103] X input = [X image , X text ] (2)

[0104] where num = m + n is the total number of image and text features.

[0105] Specifically, in this embodiment, for the image region feature, the maximum number of candidate boxes m = 36, the region feature dimension D od = 2048; for the text semantic feature, the maximum number of words n = 14; and the image region feature dimension and the semantic feature dimension D = 768. The final input feature

[0106] The deep self-attention network model constructed in step (2) is stacked by multiple layers with the same structure, each layer being composed of a multi-head attention module and a feedforward layer. The deep self-attention network model is constructed for deep understanding and processing of the input features to obtain multi-modal features with richer meanings.

[0107] Specifically, the deep self-attention network is stacked by multiple layers with the same structure, each layer being composed of a multi-head attention module MHA and a feedforward layer FFN. The deep self-attention network model is constructed for deep understanding and processing of the input features To obtain more meaningful multimodal features through in-depth understanding and processing.

[0108] 2-1. Multi-head attention module (MHA);

[0109] For a given query feature Key features and value characteristics D represents the dimension of the features. The multi-head attention module contains H parallel attention heads, which calculate the features. The specific formula is as follows:

[0110] F mha =MHA(Q,K,V)=[head1,head2,...,head H W 0 (3)

[0111]

[0112] in, D represents the projection matrix of the h-th attention head. H The feature dimension of each attention head is represented by D. H =D / H is calculated. Simultaneously, the matrix... The features obtained from multi-head attention computation undergo further mapping processing. ATT stands for Attention Calculation Method, which performs a scaled dot product operation on the processed inter-feedback features Q and key features K to obtain the attention matrix, and then weights and sums it with the processed value features V, as shown in the following formula:

[0113]

[0114] 2-2. Feedforward layer FFN;

[0115] The feedforward layer consists of two fully connected layers and an activation function, which incorporates the output features of the MHA module. As input features, they are first projected into a high-dimensional space, then mapped back to the original dimension to obtain the output features. The specific formula is as follows:

[0116]

[0117] in is the linear transformation projection matrix, and Nonlinear is the nonlinear activation function.

[0118] 2-3. Self-attention layer;

[0119] Each self-attention layer Layer comprises the multi-head attention module MHA and the feed-forward layer FFN described above, and for a given input feature Layer processes the output feature The specific formula is as follows:

[0120]

[0121]

[0122] where LN represents layer normalization.

[0123] 2-4. Deep self-attention network model Model;

[0124] It is considered that the feature dimension D does not change after each self-attention layer Layer, so multiple self-attention layers Layer can be stacked to form a deep self-attention network model, denoted as Model, which processes the input feature to obtain multi-modal features with richer meanings The specific formula is as follows:

[0125] Model = [Layer (1) , Layer (2) , …, Layer (L) ] (9)

[0126] X output = Model(X input ) (10)

[0127] where L is the number of self-attention layers. Then, the model is initialized using the pre-trained weights W pretrai n on a large-scale image-text pair corpus, and the specific formula is as follows:

[0128] Model ← W pretrain (11)

[0129] In this embodiment, by setting D = 768 and H = 12, the feature dimension D H of each attention head is 64. The input feature is processed by MHA to obtain the feature is processed by FFN to obtain the feature The final input feature is processed by the deep self-attention network model to obtain the output feature

[0130] The pre-training weight pruning in step (3) is performed to reduce the pre-training model size and improve the model inference speed. According to the deep self-attention network model structure, the weight pruning is divided into two types of cutting multi-head attention module and cutting feedforward layer.

[0131] Specifically, to reduce the pre-training model size and improve the model inference speed, the pre-training weight is pruned. According to the deep self-attention network model structure in step (2), the pre-training weight pruning is divided into two types of cutting multi-head attention module MHA and cutting feedforward layer FFN.

[0132] 3-1. Cutting multi-head attention module MHA;

[0133] The multi-head attention module is designed to cut the number of attention heads H without changing the input and output feature dimensions D, so as to reduce the number of module parameters. Specifically, for a given input feature The calculation method of the cut multi-head attention module is as follows:

[0134] F p_mha = MHA (Q, K, V) = [head1, head2, …, head H-t ]W 0 (12)

[0135]

[0136] Wherein, H is the original number of attention heads, and t is the number of cut attention heads. represents the parameter matrix of the hth attention head, D H = D / H is the dimension of each attention head. ATT is the attention calculation method, as shown in formula (5). In order to match the attention feature dimension (H-t)·D H After cutting, the corresponding cutting is made. is the output feature of the finally cut attention module, and the dimension is consistent with the input feature, which indicates that the MHA module after cutting the attention head will not change the feature dimension.

[0137] 3-2. Cutting feedforward layer FFN;

[0138] The cutting feedforward layer is designed to cut the parameter matrices W1 and W2 contained therein without changing the input and output feature dimensions D. Specifically, for a given input feature The calculation method of the cut feedforward layer FFN is as follows:

[0139]

[0140] wherein, is the parameter matrix of the split feed-forward layer FFN module, s is the set split dimension, and Nonlinear is the activation function. is the output feature, the dimension of which is consistent with the input feature , that is, the feature dimension does not change after the split feed-forward layer FFN.

[0141] Further, in the embodiment, the split multi-head attention module MHA needs to change the W Q , W K , W V and W 0 parameter matrices at the same time, so that the parameter amount of a single MHA module is reduced by 4xtxDxD H , t is the set number of attention head splitting, that is, t e {0, 1,..., H}, where H = 12.

[0142] The split feed-forward layer FFN needs to change the W1 and W2 parameter matrices at the same time, so that the parameter amount of a single FFN module is reduced by 2xsxD, s e {0, 1,..., 4D>}, which is the split dimension, and D = 768.

[0143] Step (4) constructs a task adapter. The task adapter refers to a small amount of learnable parameters inserted into each layer of the deep self-attention network model. The model only trains these parameters when fine-tuning the downstream task, while keeping the original pre-training weights of the model unchanged. Finally, the model still has the learning ability of the downstream task while retaining the pre-training general knowledge. A lightweight task adapter contains two learnable parameter matrices, and contains a nonlinear activation function in between.

[0144] Specifically, the task adapter refers to a small amount of learnable parameters inserted into each layer of the deep self-attention network model. The model only trains these parameters when fine-tuning the downstream task, while keeping the original pre-training weights of the model unchanged. Finally, the model still has the learning ability of the downstream task while retaining the pre-training general knowledge. A lightweight task adapter contains two learnable parameter matrices W down and W up , and contains a nonlinear activation function in between. For input feature F adp_in and output feature F adp_out , the calculation of each task adapter is as follows:

[0145] F adp_mid = Nonlinear(F adp_in W down ) (15)

[0146] F adp_out = F adp_in + F adp_midW up (16)

[0147] in, Nonlinear is a non-linear activation function, D is the input and output feature dimension, and r is the adapter size parameter. A larger r indicates a larger number of learnable parameters, and generally, a stronger learning ability for the adapter. After processing by the task adapter, the input features' feature dimension D remains unchanged, allowing them to be easily inserted into a deep self-attention network model. Furthermore, to ensure model training stability, each task adapter includes residual connections. The input features F... adp_in By bypass connection to the output feature F adp_out superior.

[0148] It should be noted that in this embodiment, the input feature F adp_in and output features F adp_out This simply represents the inputs and outputs of each adapter module, which facilitates the description of how each adapter performs its calculations. It is equivalent to the input characteristics of the Layer module described in point 5 above. and output features

[0149] Furthermore, in this embodiment, the number of learnable parameters for each task adapter is 2×D×r, where r is the size of the task adapter, and r = 64 can be set.

[0150] The adapted pre-trained model described in step (5) is combined with the pre-trained model segmented in step (3) and the task adapter in step (4) to obtain the adapted pre-trained model. For different downstream tasks, they share the segmented pre-trained weights but have different task adapters. Similarly, according to the deep self-attention network model structure, the adapted pre-trained model is divided into two categories: adapted multi-head attention module and adapted feedforward layer.

[0151] Specifically, by combining the pre-trained model segmented in step (3) with the task adapter in step (4), the adapted pre-trained model is obtained. For different downstream tasks, they share the segmented pre-trained weights but have different task adapters. Similarly, based on the structure of the deep self-attention network model, the adapted pre-trained model is divided into two categories: adapted multi-head attention modules and adapted feedforward layers.

[0152] 5-1. Adapted to the Multi-Head Attention Module (MHA);

[0153] The Adapted Multi-Head Attention Module (MHA) aims to insert a small number of trainable task adapters as new attention heads into the segmented pre-trained MHA. Specifically, for a given downstream task input features... The adapted multi-head attention module calculates it as follows:

[0154]

[0155]

[0156] wherein, represents the h-th adapted attention head parameter matrix, D represents the input feature dimension, D H is consistent with the original attention head. ATT is the attention calculation method, ah is the settable number of adapted attention heads, is the task adapter attention head output feature, which is spliced with the original attention head output to obtain the adapted attention feature. In order to match the dimension of the adapted attention feature, is additionally introduced and spliced with the segmented to process the adapted attention feature together, and the final output feature has the same dimension as the input feature. The adapted multi-head attention module MHA only W adpQ , W adpK , W adpV and are trainable, and the remaining parameters remain unchanged in model training.

[0157] 5-2. Adapted feedforward layer FFN;

[0158] The adapted feedforward layer FFN aims to fill and adapt the W1 and W2 matrices in the segmented FFN module, so that it has the learning ability of the downstream task. Specifically, for a given input feature The calculation method of the adapted feedforward layer FFN is as follows:

[0159]

[0160] wherein, ah is the settable size of the feedforward layer, which is spliced with the segmented pre-training matrices W1 and W2, so that the adapted feedforward layer FFN can not only retain the general pre-training knowledge but also have the learning ability of the downstream task. is the final output feature, which has the same dimension as the input feature . Similarly, only and in the adapted feedforward layer FFN are trainable, and the remaining parameters remain unchanged in model training.

[0161] Further, in this embodiment, the single module of the adapted MHA module has a learnable parameter amount of 4xahxD HX D, ah is the number of settable adaptive attention heads; and the adaptive FFN module has a single module learnable parameter amount of 2 X af X D, and af is the settable feedforward layer adaptation size. Wherein, ah and af are integers greater than 0.

[0162] Step (6) designs the progressive guided distillation training and trains the model.

[0163] Specifically, the application proposes a progressive guided distillation training algorithm, which adopts a full fine-tuning model under the traditional 'pre-training-fine-tuning' paradigm as a teacher model, and the student model is the adapter model obtained in step (5). In each training iteration, the teacher model and the student model are trained together, and the knowledge learned by themselves is progressively distilled to the student adapter model in a step-by-step guided manner. In order to better align the parameter distribution of each layer of the student model, in addition to the output features of the distilled model, the training algorithm synchronously distills the output features of each layer of the teacher model.

[0164] Let the teacher model be M tea , and the model architecture is the original deep self-attention network model without splitting adaptation; and the student model is M stu , and the model architecture is the adapter model obtained in step (5) after splitting and adaptation. In each training iteration, a batch of input data X and target Y are randomly sampled, which are input into the teacher model M tea (X), to obtain the predicted label Y tea and the output of each layer Layer tea , and calculate the teacher model loss Update gradient Then freeze the feature gradient Y tea .detach(), Layer tea .detach() of the teacher model. In the current training iteration, the input data X is synchronously input into the student model M stu (X), to obtain the predicted label Y stu and the output of each layer Layer stu , and calculate the loss:

[0165]

[0166]

[0167]

[0168] Wherein, is the output loss, is the layer loss, is the final loss of the student model. λ1 is the output loss adjustment ratio, and λ2 is the layer loss adjustment ratio, which can be set. After obtaining the total loss, the gradient is calculated and updated

[0169] Further, in the progressive guided distillation training algorithm in this embodiment, λ1 and λ2 are used to adjust the proportion between the different losses of the student model, which can be set as λ1 = λ2 = 0.5. MSE is the mean square error loss function.

Claims

1. A lightweight adaptive network learning method for multimodal, multi-task learning, characterized in that, Includes the following steps: Step (1): Construct a downstream task dataset and divide it into a training set, a validation set and a test set. The downstream task dataset includes visual question answering, natural language visual reasoning, visual implication reasoning and visual target localization. Step (2): For the image data in the dataset, use the existing trained Faster R-CNN object detection network to extract region image features. For the text data in the dataset, use word embedding vectors to extract its semantic features. The extracted image and text features are then concatenated to obtain the final input features; Step (3): Construct a deep self-attention network model. The deep self-attention network is composed of multiple layers with the same structure stacked together. Each layer consists of a multi-head attention module and a feedforward layer. The deep self-attention network model is used to deeply understand and process the input features to obtain multimodal features with richer meanings. Step (4): Pre-training weight pruning of deep self-attention network model. The weight pruning is divided into two categories: splitting multi-head attention modules and splitting feedforward layers. Step (5): Build the task adapter The task adapter consists of learnable parameters inserted into each layer of a deep self-attention network model. A lightweight task adapter contains two learnable parameter matrices W. dow n and W up And it contains a non-linear activation function in the middle; The method for building a task adapter is as follows: For each task adapter, the input and output features are denoted as input feature F, respectively. adp_in and output features F adp_out The calculation method for each task adapter is as follows: F adp_mid =Nonlinear(F adp_in W down ) F dp_out =F adp_in +F adp_mid W up in, Nonlinear is a nonlinear activation function, D is the input and output feature dimensions, r is the adapter size parameter, each task adapter contains residual connections, and the input features F adp_in By bypass connection to the output feature F adp_out superior; Step (6): Adapt the pre-trained model and combine the segmented pre-trained model with the task adapter to obtain the adapter model; Adapting to pre-trained models, as detailed below: Adaptation to the Multi-Head Attention Module (MHA): For a given downstream task input feature The adapted multi-head attention module calculates it as follows: in, This represents the attention head parameter matrix for the h-th adaptation, where D represents the input feature dimension. H For each attention head dimension, maintain consistency with the original attention heads; ATT represents the attention calculation method, and ah represents the configurable number of adaptable attention heads. The attention features are obtained by concatenating the output features of the task adapter's attention head with the original attention head output; to match the dimensions of the adapted attention features... Additional introduction and segmentation The features are concatenated and processed together to adapt the attention features, and finally output features. Its dimensions are consistent with the input features, and the adapted multi-head attention module MHA only has W adpQ W adpK W adpV and It is trainable, and the remaining parameters remain unchanged during model training; Adapting the feedforward layer FFN: For a given input feature The adapted feedforward layer FFN is calculated as follows: in, af is a configurable feedforward layer adaptation size. It is concatenated with the split pre-trained matrices W1 and W2, so that the adapted feedforward layer FFN can retain general pre-trained knowledge and have the ability to learn from downstream tasks. The final output feature has the same dimensions as the input feature. To maintain consistency; similarly, only in the adapted feedforward FFN and It is trainable, and the remaining parameters remain unchanged during model training; Step (7): Design progressive guided distillation training and train the model. The traditional "pre-training-fine-tuning" paradigm uses a full-scale fine-tuning model as the teacher model and an adapter model as the student model. In each training iteration, the teacher model and the student model train together. The teacher model gradually distills the knowledge it has learned to the student adapter model through a step-by-step guidance method. In addition to distilling the overall output features of the adapter model, the training algorithm simultaneously distills the output features of each layer of the teacher model.

2. The lightweight adaptive network learning method for multimodal, multi-task learning according to claim 1, characterized in that, The visual question answering task uses the VQA-v2 dataset, the natural language visual reasoning task uses the NLVR2 dataset, the visual entailment reasoning task uses the SNLI-VE dataset, and the visual object localization task uses the Ref-COCO, Ref-COCO+, and Ref-COCOg datasets.

3. The lightweight adaptive network learning method for multimodal, multi-task learning according to claim 1, characterized in that, In step (2), for the image data in the multimodal dataset mentioned above, the Faster R-CNN object detection model pre-trained on the Visual Genome dataset is used to extract the region features of the image. Where m is the number of region candidate boxes in the image, and D od For regional feature dimensions; Subsequently, a learnable linear transformation is used. Region features extracted from images by object detection models Further processing is performed to map its feature dimensions onto a D-dimensional space, yielding the final image region features. The specific formula is as follows: X image =Linear(X od ) (1) For the text in the aforementioned multimodal dataset, semantic features of the text are extracted using word embedding vectors. Where n is the number of words in the text, and D is the semantic feature dimension, which is the same as the final image region feature dimension; The extracted image and text features are then concatenated to obtain the final input features. The specific formula is as follows: X input =[X image ,X text ] (2) Where num = m + n is the total number of image and text features.

4. The lightweight adaptive network learning method for multimodal, multi-task learning according to claim 3, characterized in that, The construction of the deep self-attention network model is described in detail below: Deep self-attention network models are composed of multiple stacked self-attention layers with identical structures. Each layer consists of a multi-head attention module (MHA) and a feedforward layer (FFN), with input features... To obtain more meaningful multimodal features through deeper understanding and processing. The multi-head attention module (MHA), for a given query feature... Key features and value characteristics D represents the dimension of the features. The multi-head attention module contains H parallel attention heads, which calculate the features. The specific formula is as follows: F mha =MHA(Q,K,V)=[head1,head2,...,head H ]W 0 (3) in, D represents the projection matrix of the h-th attention head. H The feature dimension of each attention head is represented by D. H =D / H is calculated, and at the same time, the matrix The features obtained from multi-head attention computation are further mapped. ATT stands for Attention Calculation Method, which performs a scaled dot product operation on the processed query features Q and key features K to obtain the attention matrix, and then sums it with the processed value features V using a weighted sum. The specific formula is as follows: The feedforward layer (FFN) comprises two fully connected layers and an activation function, which incorporates the output features of the MHA module. As input features, they are first projected into a high-dimensional space, then mapped back to the original dimension to obtain the output features. The specific formula is as follows: in is the linear transformation projection matrix, and Nonlinear is the nonlinear activation function.

5. A lightweight adaptive network learning method for multimodal, multi-task learning according to claim 4, characterized in that, Each of the self-attention layers contains the multi-head attention module (MHA) and feedforward layer (FFN) described above, for a given input feature. The layer is processed to output features. The specific formula is as follows: Where LN represents layer normalization.

6. A lightweight adaptive network learning method for multimodal, multi-task learning according to any one of claims 1-5, characterized in that, The deep self-attention network model, denoted as Model, responds to input features To obtain more meaningful multimodal features through deeper understanding and processing. The specific formula is as follows: Model=[Later (1) ,Layer (2) ,...,Layer (L) ] (9) X output =Model(X) input (10) where L is the number of self-attention layers. Subsequently, the weights W pre-trained on a large-scale image-text corpus are used. pretrain The model is initialized using the following formula: Model←W pretrain (11)。 7. A lightweight adaptive network learning method for multimodal, multi-task learning according to claim 5, characterized in that, In step (4), the pre-trained weights are pruned as follows: The segmentation of the Multi-Head Attention Module (MHA): For a given input feature The multi-head attention module after segmentation is calculated as follows: F p_mha =MHA(Q,K,V)=[head1,head2,...,head H-t ]W0 (12) Where H represents the original number of attention heads, and t represents the number of attention heads that are split. D represents the parameter matrix of the h-th attention head. H =D / H represents the dimension of each attention head, and ATT represents the attention calculation method. In order to match the attention feature dimension (Ht)·D after segmentation H At the same time, make the corresponding division. The attention module outputs features after final segmentation; The feedforward layer FFN is divided as follows: For a given input feature The calculation method for the split feedforward layer FFN is as follows: in, Here is the parameter matrix of the feedforward FFN module after segmentation, where s is the set segmentation dimension and Nonlinear is the activation function. For output features.

8. A lightweight adaptive network learning method for multimodal, multi-task learning according to claim 7, characterized in that, In step (7), a progressive guided distillation training method is designed and the model is trained, as follows: Let the teacher model be M. tea Its model architecture is a raw deep self-attention network model without segmentation and adaptation; while the student model is M. stu Its model architecture is the segmented and adapted adapter model obtained in step (6). In each training iteration, a batch of input data X and target Y are randomly sampled and fed into the teacher model M. tea (X), to obtain the predicted label Y tea and each output layer tea And calculate the teacher model loss. Update gradient Subsequently, the gradient Y of the teacher model feature was frozen. tea .detach(), Layer tea `.detach()` synchronously transfers the input data X into the student model M during the current training iteration. stu (X), to obtain the predicted label Y stu and each output layer stu And calculate the loss: in, For output loss, For layer loss, The final loss for the student model is given by λ1, where λ1 is the output loss adjustment ratio and λ2 is the layer loss adjustment ratio. After obtaining the total loss, the gradient is calculated and the model is updated.

Images