Lightweight pre-training model pruning method based on task matching similarity

Abstract

The invention discloses a lightweight pre-training model pruning method based on task matching similarity, and the method comprises the steps: carrying out the pre-training fine tuning of a model, quantifying the matching similarity score between the general task knowledge of an upstream task and the specific task knowledge of a downstream task, executing a model dynamic pruning strategy, and performing joint fine-tuning training optimization on the dynamically pruned model. According to the method, the pruning strategy is optimized by fusing upstream and downstream task knowledge, the performance of the pruning model is remarkably improved, meanwhile, model parameters and calculation overhead are greatly reduced, the generalization ability of the model is enhanced, the pruning process is more flexible through the dynamic pruning strategy, personalized adjustment can be conducted according to task requirements, and the pruning efficiency is improved. The technical scheme of the invention is not only suitable for pre-training model pruning in the field of computer vision, but also can be expanded to deep learning model pruning in other fields, and has wide application prospects and potential economic values.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning model pruning optimization, and in particular to a lightweight pre-training model pruning method based on task matching similarity. Background Art

[0002] With the continuous advancement of deep learning technology, pretraining language models (PLM), especially models such as Vision Transformer (ViT), have achieved remarkable achievements in the field of computer vision (CV). PLM follows a self-supervised learning paradigm, including a pretraining phase and a fine-tuning phase. In the pretraining phase, the model learns general feature representations and semantic information by processing a large amount of unlabeled data; in the fine-tuning phase, the model uses a small amount of labeled data to train for specific tasks, thereby achieving higher task accuracy. However, although PLM has achieved significant performance advantages, its large parameter size and computational requirements limit its application on resource-constrained edge devices.

[0003] To address this problem, model pruning, as an effective model compression method, has been widely used to reduce the number of parameters and computational overhead of PLM. The pruning process usually includes three steps: first, training a complete dense model; second, pruning unimportant parameters in the model according to certain rules or standards to obtain a sparse model; finally, fine-tuning the sparse model to restore its performance. However, most traditional pruning methods use fixed pruning strategies and rules, which do not fully consider the relationship between the general task knowledge of the upstream task and the specific task knowledge of the downstream task.

[0004] Specifically, focusing only on the specific task knowledge of downstream tasks and adopting simple pruning methods (such as vanillapruning) may cause the pruned model to lose the original PLM task-irrelevant knowledge, thereby causing catastrophic forgetting and damaging the generalization ability of the model. On the other hand, although only maintaining the general knowledge of the original PLM can provide some general feature representations and semantic information, these features may not be fully applicable to downstream tasks, resulting in the performance degradation of the pruned model on specific tasks.

[0005] To overcome these challenges, researchers are constantly exploring new pruning strategies and methods. The current mainstream methods can be divided into two categories: one is to use the statistical information of the attention weights in the model for pruning, aiming to retain the parameters that are important to the upstream task; the other is to selectively delete the network structure based on the specific knowledge of the downstream task. However, these methods still have certain limitations because they are only dominated by upstream tasks or downstream tasks and do not fully utilize the correlation between general task knowledge and specific task knowledge. Therefore, how to effectively integrate and utilize these two kinds of knowledge in the pruning process to improve the performance and generalization ability of the pruned model is still a hot and difficult issue in current research.

[0006] For example, the invention application with application number 202211239440.3 discloses a visual Transformer model pruning method based on an attention graph. Using this application solution, the ViT model is finally lightweight with limited loss of ViT model performance. However, its solution also has the following problems: it does not comprehensively consider the general task knowledge of upstream tasks and the specific task knowledge of downstream tasks, and does not optimize the parameter selection and deletion in the pruning process, which is not conducive to significantly reducing the number of model parameters while maintaining model performance. Summary of the invention

[0007] In view of the above-mentioned problems, the purpose of the present invention is to provide a lightweight pre-trained model pruning method based on task matching similarity, which can comprehensively consider the general task knowledge of upstream tasks and the specific task knowledge of downstream tasks to optimize parameter selection and deletion in the pruning process, thereby significantly reducing the number of model parameters and computational overhead while ensuring the performance of the pruned model.

[0008] The embodiment of the present invention provides a lightweight pre-training model pruning method based on task matching similarity, comprising the steps of:

[0009] Pre-train and fine-tune the model, quantify the matching similarity between the general task knowledge of the upstream task and the specific task knowledge of the downstream task, execute the model dynamic pruning strategy based on the matching similarity score, and perform joint fine-tuning training optimization on the dynamically pruned model. Through optimization, the pruned model can better adapt to the specific task requirements of the downstream task while maintaining the general task knowledge of the upstream task.

[0010] Optionally, the model encoder block includes a multi-head attention layer MASH and a multi-layer perceptron layer MLP;

[0011] Optionally, the matching similarity calculation basis includes: KL divergence and Wasserstein distance.

[0012] The fully connected layers of MASH and MLP adopt pruned linear layers PLL.

[0013] Optionally, the MASH and MLP process the input data, and the formula is expressed as:

[0014]

[0015] Y=X+FC out (Attention(FC q (X),FC k (X),FC v (X)))

[0016] Where X∈R N×d is the input data feature vector, Q h , K h 、V h are the Q, K, V matrices of the model encoder block, is the weight matrix, h is the number of attention heads, is the output projection, FC out It is a linear projection layer.

[0017] Optionally, the KL divergence is expressed as:

[0018]

[0019] in, and They are the upstream task feature vectors X (A) ∈R n×d and downstream task feature vector X (B) ∈R n×d The mean vector of ,∈ is a small positive number.

[0020] Optionally, the Wasserstein distance is expressed as:

[0021] d E (μ A ,μ B )=||μ A -μ B ||2

[0022] Among them, μ A and μ B For X (A) ∈R n×d and X (B) ∈R n×d Unnormalized mean vector.

[0023] Optionally, the matching similarity score is expressed as:

[0024]

[0025] Where T is the matching similarity measurement value obtained by KL divergence and Wasserstein distance; threshold lower is the lower limit threshold of pruning, threshold upper is the upper pruning threshold.

[0026] Optionally, the execution model dynamic pruning strategy is expressed as:

[0027]

[0028] X * =PrunedLinearLayer(X)

[0029] in, represents the feature weight matrix after pruning, W ij represents the feature weight matrix before pruning, X * is the model feature vector after pruning.

[0030] Optionally, the model after dynamic pruning is jointly fine-tuned and optimized, and the formula is expressed as:

[0031] Q,K,V=PPL q (X),PPL k (X),PPL v (X)

[0032] Y=X+PPL out (Attention(Q,K,V)))

[0033] Z=Y+PPL2(PPL1(Q,K,V))

[0034] Among them, PPL q , PPL k , PPL v , PPL out , PPL2, PPL1 are the results of linear projection operation of the pruned feature vector;

[0035] Among them, the pruned feature weight matrix is ​​used in the joint fine-tuning training optimization process of the dynamically pruned model

[0036] Optionally, when executing the dynamic pruning strategy of the model, the pruning strategy is not executed in the last round of pruning training, and the model is trained again based on the sparse weight matrix to restore the model accuracy.

[0037] Beneficial effects of the present invention:

[0038] The present invention introduces a task matching similarity calculation method to quantify the correlation between the general task knowledge of the upstream task and the specific task knowledge of the downstream task, obtains the matching degree between the tasks, and provides guidance for the subsequent pruning process; based on the task matching similarity, the present invention adopts a dynamic pruning strategy, which can dynamically adjust the intensity and range of pruning according to the correlation of the upstream and downstream task knowledge, and give higher retention priority to parameters that are highly correlated with both upstream and downstream tasks; while parameters with low correlation with upstream and downstream tasks are more likely to be pruned; the present invention also adopts a joint fine-tuning method to train the pruned model. The method combines the training data of the upstream task and the downstream task, and through joint optimization, enables the pruned model to better adapt to the specific requirements of the downstream task while maintaining the general knowledge of the upstream task.

[0039] The method of the present invention optimizes the pruning strategy by integrating upstream and downstream task knowledge, significantly improving the performance of the pruning model, while greatly reducing model parameters and computational overhead, and enhancing the generalization ability of the model; the dynamic pruning strategy makes the pruning process more flexible and can be adjusted individually according to task requirements. The technical solution is not only applicable to the pruning of pre-trained models in the field of computer vision, but can also be extended to the pruning of deep learning models in other fields. This makes the technical solution have broad application prospects and potential economic value. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a principle structure diagram of the encoder block of the pruning method of the present invention;

[0041] Figure 2 It is a principle structure diagram of encoder block refinement of the pruning method of the present invention;

[0042] Figure 3 It is a schematic structural diagram of the electronic device of the present invention. DETAILED DESCRIPTION

[0043] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar symbols throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0044] Currently, in the model pruning process, the general task knowledge of upstream tasks and the specific task knowledge of downstream tasks are not effectively integrated, resulting in poor performance and generalization ability of model pruning.

[0045] In view of the above problems, the present invention provides a lightweight pre-training model pruning method based on task matching similarity. Figure 1The principle structure diagram of the lightweight pre-training model pruning method based on task matching similarity provided by an embodiment of the present invention, the method includes:

[0046] Pre-train and fine-tune the model, quantify the matching similarity between the general task knowledge of the upstream task and the specific task knowledge of the downstream task, execute the model dynamic pruning strategy based on the matching similarity score, and perform joint fine-tuning training optimization on the dynamically pruned model. Through optimization, the pruned model can better adapt to the specific task requirements of the downstream task while maintaining the general task knowledge of the upstream task.

[0047] In this paper, we mainly focus on the large model Vision Transformer (ViT) in the field of computer vision (CV). The specific architecture is as follows Figure 1 shown.

[0048] First, the model is pre-trained. Pre-training can make use of diverse and cost-effective training data to extract the common features of these data. After training, these extracted common features are transferred to the model designed for specific tasks.

[0049] On this basis, the model is fine-tuned using labeled datasets in this specific field. This process enables the model to focus on learning the unique features required for this specific task, starting from the general features it has mastered.

[0050] The present invention relates to a model structure such as Figure 1 As shown, the model includes multiple encoder blocks (TransformerEncoder Block), and the Transformer Encoder Block includes a multi-head attention layer (MASH) module and a multi-layer perceptron layer (MLP) module.

[0051] During model fine-tuning, a feature extractor is used to process samples to obtain personalized and unique features for each sample. These features are then converted into a feature vector input format that the model can understand and process and input into the encoder layer.

[0052] like Figure 1 As shown in the figure, the feature vector input enters the model encoder block (Transformer Encoder Block). Each encoder block (Transformer Encoder Block) contains: Multi-Head Self-Attention (MHSA) layer, Multilayer Perceptron (MLP) layer, Normalization layer (Layer Norm), Activation function layer and Random Loss layer (Dropout), etc.

[0053] like Figure 2 As shown, the present invention optimizes MASH and MLP. Specifically, the fully connected layer in MASH and MLP is replaced by a PLL (Pruned Linear Layer) layer. The PLL layer can more effectively extract the feature vector of the input data.

[0054] Through the above layer settings, the model can achieve complex representation and processing capabilities for input data. Among them, the MASH mechanism allows the model to capture long-distance dependencies, the MLP mechanism can solve nonlinear problems that single-layer perceptrons cannot solve, and layer normalization helps stabilize and accelerate the training process of deep networks.

[0055] MASH and MLP process the input data, and the formula is expressed as:

[0056]

[0057] Y=X+FC out (Attention(FC q (X),FC k (X),FC v (X)))

[0058] Where X∈R N×d Input data feature vector for MHSA, where N is the sequence length, d is the dimension, Q h , K h 、V h are the Q, K, V matrices of the model encoder block, is the encoder block weight matrix, h is the number of attention heads, is the output projection, FC out It is a linear projection layer.

[0059] The present invention uses a dynamic pruning strategy, the main purpose of which is to reduce the FLOPs (floating point operations) of the MASH and MLP modules in each encoder block (Transformer EncoderBlock).

[0060] For the feature vector X∈R extracted from the input data n×d , whose feature corresponding weight matrix is ​​W∈R n×d , where n represents the number of features to be pruned and d represents the dimension of the feature. The task similarity between feature vectors under different tasks is calculated to obtain the task similarity score. According to the task similarity score, important features are retained and relatively redundant features are eliminated from the linear projection.

[0061] Define the feature vector extracted from the upstream task dataset and the corresponding weight matrix as X (A)∈R n×d , W (A) ∈R n×d ; The feature vector extracted from the downstream task dataset and the corresponding weight matrix are X (B) ∈R n×d , W (B) ∈R n×d , the task similarity score is Score∈{0,1} d , that is, the generated important feature vectors and their corresponding scores are 1, while the corresponding scores of redundant feature vectors are 0.

[0062] For example: Features Respectively represent the m-th layer feature vector extracted from the upstream task and downstream task data, Represents the weight matrix corresponding to the m-th layer feature. There are two pruning thresholds in the setting: pruning lower limit threshold lower and pruning upper threshold threshold upper The weight of the feature gap below the pruning lower limit threshold can be regarded as a redundant feature, and its importance score is set to 0; the weight of the feature gap above the pruning upper limit threshold can be regarded as an important feature, and its importance score is set to 1.

[0063] For matching similarity calculation, KL divergence and Wasserstein distance measurement methods can be used to calculate the similarity between feature vectors under different tasks.

[0064] For KL divergence, let X (A) ∈R n×d and X (B) ∈R n×d are the feature vectors in the upstream task and the downstream task respectively. In order to calculate the cross entropy (as an approximation of the KL divergence), the feature vectors are normalized to the form of probability distribution through the softmax function. For the normalized feature vectors, the cross entropy between them can be calculated to approximate the KL divergence.

[0065] Since we are comparing a set of feature vectors of two tasks rather than a single vector, we need to perform some form of aggregation on the vectors in the set. This can be done by calculating the mean vector of the feature vector of each task and calculating the cross entropy between the two mean vectors.

[0066] Based on the above principle, the KL divergence is calculated as follows:

[0067]

[0068] in, and They are the upstream task feature vectors X (A) ∈R n×d and downstream task feature vector X(B) ∈R n×d The mean vector of , ∈ is a small positive number used to avoid numerical problems in logarithmic operations.

[0069] For the Wasserstein distance metric, the Euclidean distance between feature vectors can be used to approximate the Wasserstein distance. Similarly, since the feature vector sets of the two tasks are compared, it is necessary to calculate a certain average distance between the feature vectors in the set, and the Euclidean distance between the centroids (mean vectors) of the feature vector sets can be used as an approximation.

[0070] Based on the above principle, the Wasserstein distance is expressed as:

[0071] d E (μ A ,μ B )=||μ A -μ B ||2

[0072] Among them, μ A and μ B For X (A) ∈R n×d and X (B) ∈R n×d The mean vector is not normalized, since the Euclidean distance is calculated in the original feature space.

[0073] The KL divergence and Wasserstein distance calculated by the above-mentioned measurement method are numerically processed to obtain the matching similarity measurement value T, and then the matching similarity score is obtained. The formula is expressed as follows:

[0074]

[0075] Where T is the matching similarity measurement value obtained by KL divergence and Wasserstein distance; threshold lower is the lower limit threshold of pruning, threshold upper is the upper pruning threshold.

[0076] The algorithm for matching similarity scores can be implemented using the following code:

[0077]

[0078]

[0079] According to the matching similarity score, the model trained by the pruned weights is fine-tuned to obtain the weight matrix corresponding to the pruned features and reduce the accuracy drop. The above pruning process is expressed as:

[0080]

[0081] X * =PrunedLinearLayer(X)

[0082] in, represents the pruned feature matrix, W ij Represents the feature matrix before pruning.

[0083] The model after dynamic pruning is jointly fine-tuned and optimized. The formula is expressed as:

[0084] Q,K,V=PPL q (X),PPL k (X),PPL v (X)

[0085] Y=X+PPL out (Attention(Q,K,V)))

[0086] Z=Y+PPL2(PPL1(Q,K,V))

[0087] Among them, PPL q , PPL k , PPL v , PPL out , PPL2, PPL1 means that the linear projection operation corresponds to the linear projection result obtained by the pruned features.

[0088] During the joint fine-tuning training optimization process of the dynamically pruned model, the pruned feature weight matrix is ​​used.

[0089] Based on the above dynamic pruning strategy, the model is trained for multiple epochs, for example, 9 epochs. In the last epoch of pruning training, the dynamic pruning strategy is not executed, and the model after the sparse weight matrix can be trained again to restore the model accuracy. According to the model performance of the retained validation set, 5%, 10%, and 20% of the weights can be pruned to select the best pruned model.

[0090] The present invention also provides an electronic device, Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 3As shown, the electronic device may include: a processor, a communications interface, a memory, and a communications bus, wherein the processor, the communications interface, and the memory communicate with each other via the communications bus. The processor may call the logic instructions in the memory, for example, the following method may be executed:

[0091] Pre-train and fine-tune the model, quantify the matching similarity between the general task knowledge of the upstream task and the specific task knowledge of the downstream task, execute the model dynamic pruning strategy based on the matching similarity score, and perform joint fine-tuning training optimization on the dynamically pruned model. Through optimization, the pruned model can better adapt to the specific task requirements of the downstream task while maintaining the general task knowledge of the upstream task.

[0092] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0093] An embodiment of the present invention further provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method provided in each of the above embodiments is implemented, for example, including:

[0094] Pre-train and fine-tune the model, quantify the matching similarity between the general task knowledge of the upstream task and the specific task knowledge of the downstream task, execute the model dynamic pruning strategy based on the matching similarity score, and perform joint fine-tuning training optimization on the dynamically pruned model. Through optimization, the pruned model can better adapt to the specific task requirements of the downstream task while maintaining the general task knowledge of the upstream task.

[0095] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0096] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A lightweight pre-training model pruning method based on task matching similarity, characterized in that: include: Pre-train and fine-tune the model to quantify the matching similarity between the general task knowledge of the upstream task and the specific task knowledge of the downstream task. Based on the matching similarity score, execute the model dynamic pruning strategy, and jointly fine-tune the model after dynamic pruning.

2. The pruning method according to claim 1, characterized in that: The model encoder block includes a multi-head attention layer MASH and a multi-layer perceptron layer MLP; the fully connected layers of the MASH and MLP use a pruned linear layer PLL.

3. The pruning method according to claim 2, characterized in that: The matching similarity calculation basis includes: KL divergence and Wasserstein distance.

4. The pruning method according to claim 2, characterized in that: The MASH and MLP process the input data, and the formula is expressed as: Y=X+FC out (Attention(FC q (X),FC k (X),FC v (X))) Where X∈R N×d is the input data feature vector, Q h , K h 、V h are the Q, K, V matrices of the model encoder block, is the weight matrix, h is the number of attention heads, is the output projection, FC out It is a linear projection layer.

5. The pruning method according to claim 3, characterized in that: The KL divergence is expressed as: in, and They are the upstream task feature vectors X (A) ∈R n×d and downstream task feature vector X (B) ∈R n×d The mean vector of ,∈ is a small positive number.

6. The pruning method according to claim 3, characterized in that: The Wasserstein distance is expressed as: d E (m A ,m B )=||μ A -m B ||2 Among them, μ A and μ B For X (A) ∈R n×d and X (B) ∈R n×d Unnormalized mean vector.

7. The pruning method according to claim 5 or 6, characterized in that: The matching similarity score is expressed as: Where T is the matching similarity measurement value obtained by KL divergence and Wasserstein distance; threshold lower is the lower limit threshold of pruning, threshold upper is the upper pruning threshold.

8. The pruning method according to claim 7, characterized in that: The dynamic pruning strategy of the execution model is expressed as follows: X * =PrunedLinearLayer(X) in, represents the feature weight matrix after pruning, W ij represents the feature weight matrix before pruning, X * is the model feature vector after pruning.

9. The pruning method according to claim 8, characterized in that: The model after dynamic pruning is jointly fine-tuned and optimized. The formula is expressed as: Q,K,V=PPL q (X),PPL k (X),PPL v (X) Y=X+PPL out (Attention(Q,K,V))) Z=Y+PPL2(PPL1(Q,K,V)) Among them, PPL q , PPL k , PPL v , PPL out , PPL2, PPL1 are the results of linear projection operation of the pruned feature vector; Among them, the pruned feature weight matrix is ​​used in the joint fine-tuning training optimization process of the dynamically pruned model 10. The pruning method according to claim 1, characterized in that: When executing the dynamic pruning strategy of the model, the pruning strategy is not executed in the last round of pruning training. The model is trained again based on the sparse weight matrix to restore the model accuracy.

Images