Large-scale pre-trained model migration method based on multi-head attention redundancy evaluation

By filtering and replacing redundant multi-head attention modules in large-scale pre-trained models, using a propagated information adapter to realize information exchange, solving the problem of model redundant parameters and high computing overhead, and improving model migration efficiency and performance.

CN118153645BActive Publication Date: 2024-11-12XIAMEN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410569057.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-09
Publication Date
2024-11-12
Estimated Expiration
2044-05-09

AI Technical Summary

Technical Problem

Large-scale pre-trained models have redundant parameters and high computational overhead in downstream tasks, resulting in increased training costs and storage overhead and low inference efficiency.

Method used

By inserting a propagation information adapter before each multi-head attention module, calculate and filter out the redundant module, use the propagation information adapter to replace the redundant module, implement information exchange and modal routing, and re-parameterization after training to reduce model parameters and reasoning time.

Benefits of technology

Significantly reduces the number of parameters and inference time of the model, improves model migration efficiency, maintains or improves model performance, while reducing additional training and storage overhead.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118153645B_ABST
    Figure CN118153645B_ABST
Patent Text Reader

Abstract

The present invention provides a large-scale pre-trained model migration method based on multi-head attention redundancy evaluation, the method comprising: obtaining a large-scale pre-trained model, inserting a propagation information adapter before each multi-head attention module; calculating the preference score of each multi-head attention module; sorting the preference scores from high to low, screening out a certain number of multi-head attention modules with the lowest preference scores as skipped modules; for each skipped module, replacing it with a propagation information adapter of the same layer, performing information exchange and modal routing, and obtaining a compact network model; training the compact network model; after the training is completed, re-parameterizing the propagation information adapter and the network to obtain a trained model. The method of the present invention can significantly reduce the number of parameters and reasoning time of the model while maintaining or improving the performance of the model, thereby improving the efficiency of model migration.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning natural language and image modality processing, and in particular to a large-scale pre-training model migration method with efficient parameters. Background Art

[0002] Compared with traditional methods, large-scale pre-training models based on the Transformer structure have the characteristics of high accuracy, wide application scenarios, and large number of parameters. Due to certain differences between pre-training tasks and actual application tasks, when applying large-scale pre-training models to specific downstream tasks, fine-tuning is often required on specific downstream tasks.

[0003] While large-scale pre-trained models improve performance, they also bring two difficulties: during the fine-tuning process, the large number of model parameters contained in large-scale pre-trained models significantly increases the training cost of fine-tuning; during the deployment process, each downstream task needs to independently save a complete set of model parameters, which significantly increases storage overhead.

[0004] In order to solve the above problems, the current method mainly uses parameter efficient transfer learning (PETL) to migrate large-scale pre-trained models to downstream tasks. Its main methods include: prompt learning (PromptTuning) method, adapter (Adapter) method and low-rank adaptation (Low-RankAdaptation, LoRA) method. Among them, the prompt learning method adapts the downstream task to the pre-training task by adding additional vectors to the input or intermediate state sequence of the large-scale pre-trained model; the adapter method realizes the migration of large-scale pre-trained models by mapping the intermediate state to the subspace of the downstream task; the low-rank adaptation method updates the low-rank part of the pre-trained model parameters by training additional low-rank matrices to adapt the model to downstream tasks.

[0005] The above method achieves efficient migration of large-scale pre-trained models by adding a small amount of additional parameters, thereby reducing the video memory overhead and storage overhead of large-scale pre-trained models in the process of applying them to downstream tasks. However, due to the addition of additional parameters, the model obtained by the above method has lower inference efficiency than the model obtained directly through fine-tuning in actual deployment.

[0006] Generally speaking, the complexity of the model is proportional to the complexity of the task to be completed. When the complexity of the pre-training task far exceeds that of the downstream task, applying the large-scale pre-training model completely to the downstream task will bring unnecessary computational overhead. An ideal large-scale pre-training model migration method should achieve migration at low cost while removing the redundant parts of the pre-training model and reducing the computational overhead as much as possible. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a large-scale pre-trained model migration method based on multi-head attention redundancy evaluation, which can significantly reduce the number of model parameters and inference time while maintaining or improving model performance, thereby improving model migration efficiency. Each layer of the large-scale pre-trained model includes a multi-head attention module and a feedforward neural network. The method includes:

[0008] Step 1: Obtain a large-scale pre-trained model and insert a propagation information adapter before each multi-head attention module;

[0009] Step 2: Calculate the preference score of each multi-head attention module;

[0010] Step 3: Sort the preference scores from high to low, and select a certain number of multi-head attention modules with the lowest preference scores as the skipped modules;

[0011] Step 4: For each skipped module, replace it with the propagation information adapter of the same layer, perform information exchange and modal routing, and obtain a compact network model;

[0012] Step 5: training the compact network model;

[0013] Step 6: After the training is completed, the propagation information adapter and the network are reparameterized to obtain the trained model.

[0014] Furthermore, the structure of the propagation information adapter in step 1 is specifically as follows:

[0015]

[0016]

[0017] in, represents the output obtained by downsampling the i-th layer of the model, is the input of the i-th layer of the model, represents the output of the upsampled model layer i, f d1 and f d2 They represent the first down-sampled linear projection function and the second down-sampled linear projection function, respectively, and f u1 and f u2 They represent the first upsampling linear projection function and the second upsampling linear projection function respectively. These four linear projection functions each have their own parameter matrix and the parameters are not shared. avg represents the average pooling operation. and Respectively represent the first weight value and the second weight value, and the weight values and By vector The calculation results are:

[0018]

[0019] in, , W r represents the weight matrix, b r represents the bias term, represents the temperature coefficient of the softmax operation, router() represents the path routing function, and softmax() represents the function that transforms the input vector into a probability distribution so that each element in the output vector is between 0 and 1 and the sum of all elements is 1.

[0020] Furthermore, the step 2 is specifically as follows: for each multi-head attention module in the model, a multi-armed bandit algorithm is used to randomly sample and train the sub-network, and the reward value of the current module is calculated according to the loss value of the sub-network to reflect the contribution of the current module to the task performance, and then the preference score of the module is updated according to the reward value to indicate the probability of its retention. The specific calculation formula is as follows:

[0021]

[0022]

[0023] in, is the behavior strategy of the model's layer i at step t, used to decide which multi-head attention modules to skip. represents a uniform distribution between a and b, is the preference score of the model’s layer i at step t, which indicates the degree to which each multi-head attention module is considered redundant. is the loss value of the h-th sub-network at the t-th step, It represents the loss value of the sub-network for the j-th module that is skipped at the t-th step, m is the preset number of skipped modules, and n is the total number of layers of the model.

[0024] Furthermore, the certain number in step 3 is preset according to actual needs.

[0025] Furthermore, step 4 is specifically as follows: for each skipped multi-head attention module, its input and output are replaced by the propagation information adapter of the same layer to realize information exchange and modal routing, that is,

[0026]

[0027] in, is the input of the i-th layer of the model, It is the output of the i-th layer of the model. FFN is a feedforward network, and PIA is a propagation-of-information adapter.

[0028] Furthermore, the step 6 specifically includes:

[0029] After training is completed, the propagation information adapter and the network are reparameterized to obtain the trained model, where the propagation information adapter is directly reparameterized with the module immediately following it. If the multi-head attention module in this layer is skipped, the propagation information adapter in this layer is reparameterized into a feedforward network. The specific implementation method is as follows:

[0030]

[0031] in, represents the output of the reparameterized propagation information adapter, represents the input of the i-th layer of the model, and are the reparameterized weight matrices and bias terms.

[0032] The present invention has the following advantages: by selecting the m most redundant multi-head attention modules from a large-scale multimodal pre-training model with n pre-training modules, and using a propagation information adapter (PIA) to replace the redundant modules to obtain a compact network model structure, the computational overhead of the large-scale pre-training model migrated to downstream tasks is significantly reduced. The inference efficiency is improved while maintaining a very small amount of additional training and storage parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0034] Figure 1 This is an execution flow chart of a large-scale pre-trained model migration method based on multi-head attention redundancy evaluation of the present invention.

[0035] Figure 2 It is a schematic diagram of the structure of the propagation information adapter (PIA) of the present invention.

[0036] Figure 3 Schematic diagram of skipping and re-parameterization logic of the multi-head attention mechanism of the present invention.

[0037] Figure 4 Schematic diagram of the principle of the large-scale pre-trained model migration method based on multi-head attention redundancy evaluation of the present invention. DETAILED DESCRIPTION

[0038] The overall design idea of ​​the present invention: The present invention is mainly aimed at optimizing the parameters and computational efficiency of large-scale cross-modal pre-trained models. These models have the characteristics of large number of parameters and strong performance. At the same time, there are often large differences between the pre-training tasks and downstream tasks of these models. For multi-modal large-scale pre-trained models, the main idea of ​​the algorithm of this scheme is to evaluate the redundancy of the multi-head attention (MHA) module in downstream tasks through the efficient attention skipping method (abbreviated as EAS). Finally, the redundant MHA modules are skipped, and a propagation information adapter (PIA) is constructed to replace the redundant MHA modules. The information propagation adapter only needs to implement information exchange and modal routing, and after training is completed, the information propagation adapter is re-parameterized into a feedforward network to achieve zero additional overhead jump connection. This method can significantly reduce the number of parameters and inference time of the model while maintaining or improving the performance of the model, thereby improving the efficiency of model migration.

[0039] In order to better understand the technical solution of the present invention, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0040] like Figures 1 to 4 As shown, the present invention provides a large-scale pre-training model migration method based on multi-head attention redundancy evaluation, each layer of the large-scale pre-training model includes a multi-head attention module and a feedforward neural network, and the method includes:

[0041] Step 1: Obtain a large-scale pre-trained model and insert a propagation information adapter before each multi-head attention module;

[0042] Step 2: Calculate the preference score of each multi-head attention module;

[0043] Step 3: Sort the preference scores from high to low, and select a certain number of multi-head attention modules with the lowest preference scores as skipped modules; The purpose of this step is to reduce the computational complexity of the model and improve the reasoning speed of the model.

[0044] Step 4: For each skipped module, replace it with the propagation information adapter of the same layer, perform information exchange and modal routing, and obtain a compact network model;

[0045] Step 5: training the compact network model;

[0046] Step 6: After the training is completed, the propagation information adapter and the network are reparameterized to obtain the trained model.

[0047] In a preferred embodiment, for example, given a large-scale cross-modal pre-trained model ,in represents all the parameters of the model, and n represents the number of layers of the model. Assuming that each layer contains a multi-head attention (MHA) module and a feed-forward network (FFN) module, and Represent the parameters of the MHA module and FFN module of the i-th layer respectively. In order to find an optimal subset , so that after skipping the MHA modules in K, the performance loss of the model is minimized, and the number of model parameters and inference time are minimized. The present invention adopts an automatic redundancy evaluation algorithm based on a multi-armed bandit to learn the importance of each MHA module by continuously sampling and comparing different sub-networks, thereby determining which modules can be skipped. Specifically, first, after obtaining a large-scale pre-trained model, a propagation information adapter is inserted before each multi-head attention module, such as Figure 2 As shown, the structure of the PIA can be summarized into two parts: downsampling and upsampling, and the formula is as follows:

[0048]

[0049]

[0050] The PIA module first performs the input feature of the i-th layer of the model Downsample to obtain a low-dimensional hidden state , where d is the dimension of the input features, N is the number of input features, and r is the dimension of the hidden state, satisfying The downsampling process mainly consists of two linear projections (i.e., f d1 and f d2 ). Afterwards, the PIA module performs Upsample to get an output feature of the same dimension as the input feature The upsampling process also consists of two paths (i.e., f u1 and f u2 , where f u1 and f u2 With the downsampled f d1 and f d2 They are all linear projection functions, and each of these four linear projection functions has its own parameter matrix, and the parameters are not shared). The output of the two paths is based on a dynamic routing weight. and Perform a weighted summation, where the weights are calculated as follows:

[0051]

[0052] in , avg is the average pooling operation, W r is the weight matrix, b ris the bias term, is the temperature coefficient of the softmax operation, router() represents the path routing function, softmax() represents the function that converts the input vector into a probability distribution, so that each element in the output vector is between 0 and 1, and the sum of all elements is 1. The general form of the softmax function is as follows:

[0053]

[0054] in, is the i-th element in the input vector x, and L is the length of vector x. The Softmax function converts the input vector x into a probability distribution.

[0055] Weight value and Through the above vector Calculated.

[0056] Then, for each multi-head attention module in the model, a multi-armed bandit algorithm is used to randomly sample and train sub-networks, and the loss value of the sub-network is calculated. To calculate the reward value of the current module , reflecting the contribution of the current module to the task performance, and then according to the reward value To update the module's preference score , which indicates the probability of being retained. The specific calculation formula is as follows:

[0057]

[0058]

[0059] in, is the behavior strategy of the model's layer i at step t, used to decide which multi-head attention modules to skip. represents a uniform distribution between a and b, is the preference score of the model’s layer i at step t, which indicates the degree to which each multi-head attention module is considered redundant. is the loss value of the h-th sub-network at the t-th step, It represents the loss value of the sub-network of the j-th module that is skipped at the t-th step. m is the preset number of skipped modules, and n is the total number of layers of the model. After multiple rounds of iterations, each module can be processed according to its importance. The specific iteration termination condition is to stop the iteration when the preference score converges or reaches the specified number of iteration steps.

[0060] The multi-armed bandit algorithm is an algorithm for solving decision-making problems under uncertainty. It simulates a scenario where a person faces multiple bandits (each representing a decision) and how to maximize his expected return through limited attempts. In this scenario, the decision is whether to keep each multi-head attention module.

[0061] The score of each multi-head attention module is calculated by the above formula to evaluate the importance of the multi-head attention module in downstream tasks and decide which modules can be skipped.

[0062] In a preferred embodiment, the certain number of step 3 is preset according to actual needs.

[0063] In a preferred embodiment, in order to skip redundant MHA modules, the present invention introduces a novel propagation information adapter (PIA module) to replace the skipped MHA module. The function of the PIA module is to achieve information exchange and propagation while maintaining parameter efficiency, similar to the function of the MHA module. Specifically, step 4 is as follows: for each skipped multi-head attention module, its input and output are replaced by its same-layer propagation information adapter to achieve information exchange and modal routing, i.e.

[0064]

[0065] in, is the input of the i-th layer of the model, is the output of the i-th layer of the model, FFN is a feed-forward network, and PIA is a propagation information adapter.

[0066] In a preferred embodiment, the step 6 specifically includes:

[0067] After training is completed, the propagation information adapter and the network are reparameterized to obtain the trained model, where the propagation information adapter is directly reparameterized with the module immediately following it. If the multi-head attention module in this layer is skipped, the propagation information adapter in this layer is reparameterized into a feedforward network. The specific implementation method is as follows:

[0068]

[0069] in, represents the output of the reparameterized propagation information adapter, represents the input of the i-th layer of the model, and is the re-parameterized weight matrix and bias term, represents the weight matrix used for downsampling (dimensionality reduction), represents the weight matrix used for upsampling (dimensionality increase), represents the bias term after downsampling, Represents the bias term after upsampling.

[0070] This step is to merge the parameters of PIA into the parameters of adjacent modules in order to eliminate the extra calculation of PIA during inference. In this way, PIA can be transformed into a linear layer, thus achieving zero extra overhead skip connection, further improving the efficiency of the model.

[0071] To verify the effectiveness of the method, we compared it with other methods on the ScienceQA dataset. We found that compared with the general parameter efficient fine-tuning method, as shown in Table 1 below, the method proposed in this paper significantly improved the model performance and inference speed under the premise of using a similar total amount of updated parameters. 12 They represent the schemes of adopting the method of the present invention and choosing to skip 2 and 12 modules, respectively, and the best results are achieved in all sub-datasets. By replacing the redundant multi-head attention module with the propagation information adapter (PIA), the performance exceeds the baseline method LaVIN while the inference speed is reduced by half. In summary, the efficient attention skip method (EAS) and the propagation information adapter (PIA) proposed in this paper can obtain a model with the best performance under the premise of skipping m modules (even if 12 modules are skipped, the basic performance of the model can be maintained), and enable the model to perform well in downstream tasks.

[0072] Table 1: Comparison of this method (EAS) with other methods based on large language models

[0073]

[0074] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A large-scale pre-trained model migration method based on multi-head attention redundancy evaluation, which is applied to natural language and image modality processing in deep learning. Each layer of the large-scale pre-trained model includes a multi-head attention module and a feedforward neural network, and is characterized by: The method comprises: Step 1: Obtain a large-scale pre-trained model and insert a propagation information adapter before each multi-head attention module; Step 2: Calculate the preference score of each multi-head attention module; Step 3: Sort the preference scores from high to low, and select a certain number of multi-head attention modules with the lowest preference scores as the skipped modules; Step 4: For each skipped module, replace it with the propagation information adapter of the same layer, perform information exchange and modal routing, and obtain a compact network model; Step 5: training the compact network model; Step 6: After training, the propagation information adapter and the network are reparameterized to obtain the trained model. The trained model is then tested on the ScienceQA dataset to verify its problem reasoning efficiency. The structure of the propagation information adapter in step 1 is specifically as follows: in, represents the output obtained by downsampling the i-th layer of the model, is the input of the i-th layer of the model, represents the output of the upsampled model layer i, f d1 and f d2 They represent the first down-sampled linear projection function and the second down-sampled linear projection function, respectively, and f u1 and f u2 They represent the first upsampling linear projection function and the second upsampling linear projection function respectively, avg represents the average pooling operation, and Respectively represent the first weight value and the second weight value, and the weight values and By vector The calculation results are: in, , W r represents the weight matrix, b r represents the bias term, represents the temperature coefficient of the softmax operation, router() represents the path routing function, and softmax() represents the function that converts the input vector into a probability distribution so that each element in the output vector is between 0 and 1 and the sum of all elements is 1; The step 2 is specifically as follows: for each multi-head attention module in the model, a multi-armed bandit algorithm is used to randomly sample and train the sub-network, and the reward value of the current module is calculated according to the loss value of the sub-network to reflect the contribution of the current module to the task performance, and then the preference score of the module is updated according to the reward value to indicate the probability of its retention. The specific calculation formula is as follows: in, is the behavior strategy of the model’s layer i at step t, represents a uniform distribution between a and b, is the preference score of the model's layer i at step t, is the loss value of the h-th sub-network at the t-th step, It represents the loss value of the sub-network for the j-th module that is skipped at the t-th step, m is the preset number of skipped modules, and n is the total number of layers of the model.

2. The large-scale pre-trained model migration method based on multi-head attention redundancy evaluation according to claim 1 is characterized in that: The certain number in step 3 is preset according to actual needs.

3. The large-scale pre-training model migration method based on multi-head attention redundancy evaluation according to claim 1 is characterized in that: The step 4 is specifically as follows: for each skipped multi-head attention module, its input and output are replaced by the propagation information adapter of the same layer to realize information exchange and modal routing, that is, in, is the input of the i-th layer of the model, is the output of the i-th layer of the model, FFN is a feed-forward network, and PIA is a propagation information adapter.

4. The large-scale pre-training model migration method based on multi-head attention redundancy evaluation according to claim 1 is characterized in that: The specific steps of step 6 are as follows: After training is completed, the propagation information adapter and the network are reparameterized to obtain the trained model, where the propagation information adapter is directly reparameterized with the module immediately following it. If the multi-head attention module in the layer is skipped, the propagation information adapter in the layer is reparameterized into a feedforward network. The specific implementation method is as follows: in, represents the output of the reparameterized propagation information adapter, represents the input of the i-th layer of the model, and are the reparameterized weight matrices and bias terms.