Transformer adaptive compression method and system in data-limited scenarios

Abstract

The present invention discloses a method and system for adaptive compression of a Transformer in a data-constrained scenario. The method comprises the following steps: obtaining a small sample training set comprising multiple image samples; obtaining a pre-trained Transformer model; adaptively allocating pruning rates to the pruning rates using a search method based on an alternating direction multiplier method and Bayesian optimization to obtain an optimal pruning rate combination; compressing the pre-trained Transformer model using the optimal pruning rate combination in the small sample training set, and fine-tuning the compressed model for multiple rounds using a knowledge distillation framework based on feature alignment to obtain the final Transformer model. The present invention can be flexibly applied to image classification tasks under data-constrained conditions, and is of great significance for the practical application of visual Transformers in related tasks.

Description

Technical Field

[0001] The present invention belongs to the field of model compression, and in particular relates to a Transformer adaptive compression method and system in data-constrained scenarios. Background Art

[0002] In recent years, Transformer-based deep learning models have achieved breakthroughs in natural language processing, computer vision, and cross-modal tasks. Their core self-attention mechanism is capable of capturing long-range dependencies and, through stacking multiple layers, modeling complex data distributions. However, as model size has grown exponentially, Transformers face significant challenges in practical deployment. On the one hand, the massive computational resource requirements (such as graphics memory usage and inference latency) limit their application in resource-constrained scenarios such as mobile devices and edge computing. On the other hand, traditional compression methods rely heavily on large-scale training data for fine-tuning or knowledge distillation. However, in data-sensitive domains, the scarcity of high-quality training data makes existing compression techniques difficult to effectively implement.

[0003] In data-constrained scenarios, some researchers have conducted research on compression of deep learning models. However, current research is usually based on manually designed standard compression schemes, which cannot achieve more refined compression and usually do not consider global correlation, resulting in falling into local optimal strategies. For example, the Practice algorithm proposed by Guo-Hua Wang et al. mainly compresses convolutional neural networks in visual tasks under data-limited scenarios. The strategy for model compression is to directly discard selected block structures, which cannot achieve more dense and accurate compression (Wang G H, Wu J. Practical network acceleration with tiny sets[C] / / Proceedings ofthe IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023:20331-20340.); the Cross Distillation algorithm proposed by Haoli Bai et al. also compresses convolutional neural networks under data-limited scenarios. The strategy for model compression is to directly prune all layers of the model at the same ratio. This method does not take into account the differences between different layers of the model (Bai H, Wu J, King I,et al. Few shot network compression via cross distillation[C] / / Proceedings ofthe AAAI Conference on Artificial Intelligence. 2020, 34(04): 3203-3210.); The DC-ViT algorithm proposed by Hanxiao Zhang et al. mainly compresses the visual transformer in data-limited scenarios. The model compression strategy simply distributes the parameters that need to be pruned evenly to each block that needs to be compressed, without considering the sensitivity differences of different blocks of the model, which may lead to suboptimal compression and ultimately fail to achieve optimal performance (Zhang H, Zhou Y, Wang G H. Dense vision transformer compression with fewsamples[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2024: 15825-15834.). Summary of the Invention

[0004] In order to solve at least one of the problems existing in the prior art, the present invention provides a Transformer adaptive compression method and system in a data-constrained scenario, which can distribute the pruning ratio more reasonably, thereby improving the performance of the model after compression.

[0005] The purpose of the present invention is achieved by at least one of the following technical solutions.

[0006] A Transformer adaptive compression method and system in a data-limited scenario includes the following steps:

[0007] Obtain a small sample training set including multiple image samples;

[0008] Get the pre-trained Transformer model;

[0009] The optimal pruning rate combination is obtained by adaptively allocating pruning rates based on the alternating direction multiplier method and Bayesian optimization search method.

[0010] In a small sample training set, the pre-trained Transformer model is compressed by a joint pruning strategy and the optimal pruning rate combination. The compressed model is fine-tuned for multiple rounds using a knowledge distillation framework based on feature alignment to obtain the final Transformer model.

[0011] Furthermore, obtaining a small sample training set includes:

[0012] Under data-limited conditions, a small number of image samples required for the task are collected to obtain a small sample dataset;

[0013] The small sample data set is preprocessed to obtain a small sample training set.

[0014] Furthermore, the preprocessing includes:

[0015] The samples in the small sample dataset are scaled and center-cropped to ensure uniform input size. The images are then converted into processable tensors and the channel order is adjusted. Finally, the RGB channels of the images are normalized to normalize the input data and obtain the processed small sample training set. .

[0016] Furthermore, the pre-trained Transformer model is obtained, and a pre-trained Transformer model is obtained on a large-scale general image dataset. By analyzing the specific task, after comprehensively considering the types and number of categories of task images and the size of computing resources required, a visual Transformer model of appropriate size pre-trained on a suitable dataset is selected.

[0017] Furthermore, a block in the pre-trained Transformer model includes an Attention layer and an FFN layer, and each layer is preceded by a corresponding LayerNorm layer. The joint pruning strategy combines block-level pruning and channel pruning. Block-level pruning is to prune the Attention layer in the selected block in the pre-trained Transformer model as a whole, removing the entire Attention layer and the corresponding LayerNorm layer. Channel pruning is to prune the FFN layer in the block of the pre-trained Transformer model.

[0018] Furthermore, the structure of the FFN layer is , and then perform a dimensional transformation that first increases and then decreases, which can be expressed as ,in, is the activation function, is the size of a single dimension. In channel pruning, it refers to The channel of dimension is pruned, and the pruning rate is set to , then the channel dimension after pruning changes to , where the channels are selected randomly.

[0019] Furthermore, a backpropagation strategy is used to update the pre-trained Transformer model. Instead of backpropagating across all blocks in the pre-trained Transformer model, only the pruned block and one adjacent block are updated. This strategy is more conducive to fine-tuning with a small number of samples, ultimately resulting in a fine-tuned and updated student model.

[0020] Furthermore, the adaptive pruning rate allocation is performed based on the alternating direction multiplier method and the Bayesian optimization search method to obtain the optimal pruning rate combination, including:

[0021] The alternating direction multiplier method is used to transform the pruning rate allocation constraint into a differentiable optimization problem, and auxiliary variables are introduced to facilitate the solution.

[0022] The differentiable optimization problem is divided into the optimality subproblem and the feasibility subproblem. The update of the pruning rate is to solve the optimality subproblem. The update of is the solution to the feasibility subproblem;

[0023] Iteratively solve the optimality subproblem and feasibility subproblem. The iteration includes outer loop iteration and inner loop iteration. In the inner loop iteration, the optimality subproblem is solved iteratively through Bayesian optimization.

[0024] Through iterative solution, the optimal pruning rate combination is obtained.

[0025] Furthermore, in the search process of adaptive pruning rate allocation using the search method, the pre-trained Transformer model is compressed through the joint pruning strategy, and the compressed model is fine-tuned for a few rounds using a small sample training set and the performance is evaluated using the loss function.

[0026] Furthermore, the knowledge distillation framework based on feature alignment fine-tunes the compressed model to restore accuracy using only a small amount of data. The knowledge distillation framework based on feature alignment uses feature alignment to achieve the loss of teacher-student model distillation. The distillation loss Expressed as:

[0027] ;

[0028] in, and Represent the output features of the pruned student model and the original teacher model before the classification head, represents the Frobenius norm, refers to the compressed model, that is, the student model, refers to the pre-trained Transformer model, that is, the teacher model, Represents a small sample training set The samples in .

[0029] The Transformer adaptive compression system for data-constrained scenarios provided by the present invention includes the following modules:

[0030] Sample acquisition module, used to obtain small sample training sets;

[0031] The original model acquisition module is used to obtain the pre-trained Transformer model;

[0032] An optimal pruning rate acquisition module is used to perform adaptive pruning rate allocation based on an alternating direction multiplier method and a Bayesian optimization search method to obtain an optimal pruning rate combination;

[0033] The compression fine-tuning module is used to compress the pre-trained Transformer model through a joint pruning strategy and the optimal pruning rate combination under a small sample training set, and to perform multiple rounds of fine-tuning on the compressed model using a feature alignment-based knowledge distillation framework to obtain the final Transformer model.

[0034] The present invention also provides a computer device.

[0035] The present invention also provides a computer-readable storage medium.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] (1) Compared with traditional methods, the present invention focuses on compression and distillation with a small number of samples, which can effectively utilize the limited knowledge contained in a small number of samples, and will not cause problems such as overfitting due to too few samples. It significantly reduces the dependence on large-scale training data, achieves better model compression effects, and improves the applicability and reliability of the method in real data-limited scenarios.

[0038] (2) This paper adopts a joint pruning strategy to efficiently compress the model. The joint pruning strategy not only retains the global attention characteristics of the Transformer model, but also dynamically eliminates local redundant features through the coordination of block-level attention head pruning and channel feature pruning, which can achieve more comprehensive model compression.

[0039] (3) This paper dynamically allocates the pruning rate of model layers through a search method based on the alternating direction multiplier method and Bayesian optimization, achieving dense and compact high-precision model compression. Through knowledge distillation, the compressed model is fine-tuned to restore accuracy using only a small amount of data, ultimately achieving an optimal balance between model accuracy loss and computational resource consumption.

[0040] (4) The present invention innovatively constructs a two-layer optimization framework, uses the alternating direction multiplier method to transform the pruning rate allocation problem into a differentiable optimization problem, and uses Bayesian optimization to solve the sub-problems therein. This framework breaks through the bottleneck of traditional fixed pruning rate allocation and can realize the automatic search for the global optimal compression configuration, thereby achieving a compression effect with better performance than the fixed allocation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 4 is a flowchart of a Transformer adaptive compression method in a data-constrained scenario according to an embodiment of the present invention.

[0042] Figure 2 This is a pruning flow chart that integrates the dual strategies of block-level pruning and channel pruning in an embodiment of the present invention.

[0043] Figure 3 Flowchart of a two-layer optimization framework in an embodiment of the present invention.

[0044] Figure 4 This is a brief flowchart of the Transformer adaptive compression method in a data-constrained scenario in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0046] like Figure 1 As shown, an embodiment of the present invention provides a Transformer adaptive compression method for data-limited scenarios, including the following steps:

[0047] S1. Under data-limited conditions, a small number of image samples required for the task are collected to obtain a small sample data set. After preprocessing the small sample data set, a small sample training set is obtained.

[0048] In one embodiment of the present invention, ImageNet-1K, an image classification dataset covering a large number of common categories, is selected as the original dataset, and a small number of image samples are randomly extracted from the original dataset as a small sample dataset in a data-limited scenario.

[0049] The small sample data set is represented as ,in, Indicates the length of the small sample data set, that is, the total amount of data, Indicates the first image samples, In one embodiment of the present invention, .

[0050] Preprocess the small sample data set. The preprocessing steps are as follows:

[0051] The collected small sample data set The image samples in the image are scaled and center-cropped to ensure uniform input size. The image samples are then converted into processable tensors and the channel order is adjusted. Finally, the RGB channels of the image samples are normalized to normalize the input data and obtain the processed small sample training set. In one embodiment of the present invention, scaling is to uniformly process image samples into Size, center cropping is to crop a piece in the center area of ​​the image sample based on scaling Size of the image.

[0052] S2. Obtain a pre-trained Transformer model on a large-scale general image dataset.

[0053] In one embodiment of the present invention, a ViT-Base model pre-trained on the image classification dataset ImageNet-1K is obtained as a benchmark model, and the benchmark model is the pre-trained Transformer model.

[0054] S3. Construct a joint pruning strategy that integrates block-level pruning and channel pruning for pruning blocks in pre-trained Transformer models.

[0055] See also Figure 2 , in the joint pruning strategy:

[0056] For block-level pruning, a block in the pre-trained Transformer model includes an Attention layer and an FFN layer, and each layer is preceded by a corresponding LayerNorm layer. Block-level pruning refers to pruning the entire Attention layer in the selected block, removing the entire Attention layer and the corresponding LayerNorm layer.

[0057] Channel pruning refers to pruning the FFN layer in the selected Block in the pre-trained Transformer model. Specifically, the structure of the FFN layer includes the first linear layer, the activation function layer, and the second linear layer, which is usually , and then perform a dimensional transformation that first increases and then decreases, which can be expressed as ,in, is the activation function, is the channel dimension size. In channel pruning, Prune the channel of dimension and set the pruning rate = , then the dimension of the pruned channel changes to , pruning rate is the variable found by the search method of S4, wherein the channel is selected randomly. In one embodiment of the present invention, the channel dimension .

[0058] When selecting a block, the number of blocks to be pruned (A) and the number of parameters to be pruned (B) for each block are determined based on the number of parameters to be pruned (determined by the known total number of model parameters and the preset compression target) and the number of parameters in the block. After compressing this parameter B for all blocks that may participate in pruning and performing a full round of fine-tuning, the performance gap with the original model is measured, and the top A blocks with the smallest gap are selected based on the ranking of this gap.

[0059] S4. Adaptively allocate the pruning rate through an efficient search method based on alternating direction multipliers and Bayesian optimization. During the search process, a joint pruning strategy is used to compress the pre-trained Transformer model and a small sample training set is used for the compressed model. Perform a few rounds of fine-tuning and evaluate the performance to obtain the optimal pruning rate combination.

[0060] See also Figure 3 , this step includes:

[0061] S4.1. Use the alternating direction multiplier method to transform the pruning rate allocation constraint into a differentiable optimization problem (ALF), which is easy to solve under certain constraints.

[0062] Among them, the differentiable optimization problem can be defined as:

[0063]

[0064] ;

[0065] in, is the task function that needs to be minimized, is the loss function, is the pruning rate, is the set of upper and lower bounds for each pruning rate component, is the total pruning rate. Indicates the pruning rate The sum of each pruning rate component in must be equal to Only then can the overall compression ratio be met.

[0066] Loss Function Expressed as:

[0067] ;

[0068] in, represents the Cross-Entropy function, 、 They are pre-trained Transformer model and use pruning rate The pruned and fine-tuned Transformer model, It is a synthetic image dataset, which is a dataset artificially synthesized based on the original pre-trained Transformer model information. Represents training samples Used pruning rate The pruned and fine-tuned model is classified as The probability of a class, Is to use pruning rate The pruned and fine-tuned Transformer model is used in the training samples The prediction results on Indicates the pruning rate used for the pre-trained Transformer model To prune, Represents training samples Classified by the pre-trained Transformer model as The probability of a class, represents the set of image sample categories, Indicates that the pre-trained transformer model is used in the training sample The prediction results on .

[0069] This step will generate a large number of points that need to be evaluated during the iterative solution in the search process, that is, the pruning rate. After using these pruning rates for compression and a few rounds of fine-tuning, the loss function is used. The performance gap between the compressed and fine-tuned model and the pre-trained transformer model is calculated as the evaluation value.

[0070] The embodiment of the present invention is based on the data-limited scenario. When evaluating the performance of the pruned and fine-tuned model, it is not possible to use a large-scale test data set to directly measure the accuracy. Therefore, it is necessary to find an alternative evaluation method. If the training loss during fine-tuning is directly used as the evaluation standard, it will not reflect the model performance well due to overfitting. Therefore, the embodiment of the present invention uses the loss function To reflect the gap between the two models (i.e., the loss function value), the performance ranking of the pruned and fine-tuned models at different pruning rates can be obtained based on the loss function value, thereby reflecting the relative performance between the models. Due to data limitations, there is too little available data. This embodiment of the present invention is based on the idea of ​​data enhancement and assumes that the pre-trained transformer model can fully train and retain the key information in the dataset. A synthetic image dataset is generated using the information of the original pre-trained transformer model. As an evaluation dataset, a synthetic image dataset is used Performing an evaluation can provide a good reflection of the performance trend of the model.

[0071] In one embodiment of the present invention, for a synthetic image dataset The synthesis process first generates an image dataset randomly initialized by Gaussian noise and an arbitrary set of target labels , by minimizing the following objective on the synthetic image dataset To update:

[0072]

[0073] in, In the image dataset The cross entropy of the above metric, is the cross entropy between the predicted probability and the target label, is a regularization term, Indicates the target labels (label), Represents an image dataset The samples, , Represents an image dataset There are In one embodiment of the present invention, the synthetic image dataset The number of image samples in .

[0074] The regularization term is calculated using the following formula:

[0075] ;

[0076] in, and are two hyperparameters, To improve image stability, Used to control the sharpness of the synthesized data; Refers to image samples In position ( ) pixel point; Refers to image samples In position ( ) pixel point; Refers to image samples In position ( ) pixel point; It is a control hyperparameter; It is an item that improves stability; is a term that controls the sharpness of the data.

[0077] Furthermore, is the variable to be optimized, that is, a set of pruning rates, which can be expressed as:

[0078] ;

[0079] in, The pre-trained Transformer model The pruning ratio of a block (Block), the constraint condition is a sum term, that is, the pruning rate Each component of the sum must equal a value ; is the number of pruning ratios.

[0080] Afterwards, this differentiable optimization problem can be rewritten into the following form that is easy to optimize:

[0081]

[0082] ;

[0083] in, is a hyperparameter used for balancing, is an auxiliary variable introduced for solving the problem. is an indicator function defined as:

[0084] ;

[0085] In one embodiment of the present invention, The lower limit is 0.01, and the upper limit is the minimum value between 3.99 and the upper limit calculated according to the pruning rate. Set to 50.

[0086] S4.2. In order to further solve this differentiable optimization problem, Lagrange multipliers are introduced for iterative solution, which can be expressed as:

[0087] ;

[0088] in, is the augmented Lagrangian function, is the Lagrange multiplier, is a penalty parameter. In one embodiment of the present invention, The initial value of is 0.1.

[0089] After obtaining the augmented Lagrangian function, first initialize the unknown variables and Lagrangian multipliers, and use the alternating direction multiplication method to solve them in rounds. When solving in rounds, the update rule is expressed as:

[0090]

[0091]

[0092] ;

[0093] Where, It means in The pruning rate combination solved in each round, It means in Auxiliary variables solved in rounds, It means in The Lagrange multiplier variables solved in each round, It means in The Lagrange multiplier variables solved in each round, It means in Auxiliary variables solved in rounds.

[0094] In the update, the pruning rate The update is defined as the solution to the optimality subproblem, which is to minimize the loss function , that is, maximize the accuracy of the pruned model and achieve higher accuracy in classification tasks. The update of is defined as the solution of the feasibility subproblem, which means finding a feasible point under the constraints, which is also close to the unconstrained optimal point in the optimality subproblem. In one embodiment of the present invention, Is a loss function used to evaluate the accuracy of the model. The constraint is a sum constraint, which can be expressed as , It is a value calculated under a total pruning rate, which is used to ensure that the total pruning rate of the model meets the set value.

[0095] S4.3. Since optimality subproblems usually involve unknown objectives, Bayesian optimization is introduced to solve optimality subproblems. For feasibility subproblems, they can usually be solved directly.

[0096] The iteration in this invention is divided into two layers, including alternating direction multiplier method iteration (outer loop iteration) and Bayesian optimization iteration (inner loop iteration). Bayesian optimization is used to iteratively solve the optimality subproblem in each inner loop. In the optimization of each optimality subproblem, the objective function of the optimality subproblem is modeled using Gaussian process (GP), and EI is used as the objective selection function. In the iterative solution process, since the entire algorithm generates a large number of different points that need to be evaluated when searching for the optimal solution, it is necessary to calculate the loss function a large number of times. , in traditional methods, due to the loss function It is usually expressed as using a test data set to test the model accuracy. This evaluation method will bring a lot of extra time and is therefore basically not feasible. However, this problem does not exist in the embodiment of the present invention. Under data-limited conditions, the embodiment of the present invention uses the loss function mentioned above. This method is quite fast to evaluate the model performance and does not significantly affect the time in large-scale evaluation, so it is feasible under this condition.

[0097] Regarding the pruning rate The optimality subproblem of In the main loop, for convenience of representation, the objective function is optimized Abbreviated as .

[0098] Use Bayesian optimization to optimize the loss function To solve, assume Following the GP prior. Since it is a step-by-step solution, the second term can usually be regarded as a constant, so it can still be Modeled as GP. Therefore, based on the randomly generated initial observation data , which can be calculated , is randomly generated The initial pruning rate of the group, is the amount of data generated, Is to use the loss function The calculated value is used to measure the performance of the model. It refers to the set of pruning rate and optimization target value at this time. Refers to the use The function calculates the optimization target value required at this time. This is to calculate the following improvement and then solve the optimal pruning rate.

[0099] Then, we use the standard EI (Expected Improvement, EI) to calculate the improvement here:

[0100] ;

[0101] in, It is a standard expected improvement function, a mathematical method for optimization problems, which selects the next evaluation point by calculating the expected improvement on the current optimal solution. In a known set Under the premise of The expected value of the potential improvement ( ) is used to estimate the expected value of the positive part of , which is convenient for selecting the evaluation point. Refers to the optimal optimization target value in the current round; 、 The posterior distributions are The mean and variance of the distribution;

[0102] Indicates that under the current model prediction, the target value Less than or equal to the current optimal optimization target value The probability that the predicted mean far below When , the value of this item approaches 0 (low improvement possibility), when the predicted mean Close to or exceed When , the value of this term approaches 1 (the possibility of improvement is high), Indicates that under the current model prediction, the target value lie in The probability density around this degree of deviation is and When the difference is close to 0, the value of this item reaches its peak; the larger the difference, the smaller the value of this item.

[0103] Among them, the Sobol sequence is used to generate any set of pruning rates , calculate the pruning rate of this group The EI value of the Sobol sequence can be used to select the optimal pruning rate for each round based on the EI value. The Sobol sequence is a low-discrepancy sequence that can generate a highly uniformly distributed deterministic point set for use in sampling and other scenarios.

[0104] For a feasibility subproblem, it is usually defined as a problem consisting of one or more simple inequalities or equations. For a single problem, the problem to be solved in a single round is defined as:

[0105] ;

[0106] in, is the auxiliary variable that needs to be solved No. A quantity, 、 are the lower and upper bounds of each component respectively. In one embodiment of the present invention, the lower and upper bounds are the same as those in the previous The range remains consistent.

[0107] When solving the problem, first, initialize a basic vector , and then the basic vector is initially clipped, the purpose is to make the basic vector Each component in is restricted to the interval After trimming, we get the variable Here we divide the problem into two cases to solve and obtain the auxiliary variables :Case 1, use forced solution, for variables Perform projection so that the projected variable Forced satisfaction ; Case 2, use free solution ,and .

[0108] For case 1, we can Total pruning rate The relationship is adjusted by weight. It is a variable The sum of all components is used to determine the current Whether the constraints are met. , then it is proved that the constraint has been satisfied, let ;like , then the goal is to find the adjustment amount , making , we can first calculate the initial reduction , and try to subtract the initial reduction from all components , if some of the components touch the bottom, that is, less than or equal to the lower bound of the component , then fix it to , the remaining reduction will be redistributed to other components that have not reached the bottom, and the above process is repeated until the sum converges to ;like , then the goal is to find the adjustment amount , making , Representing variables The components, we can first calculate the initial increase , and try to increase the initial increment for all components , if some of the components touch the top, that is, greater than or equal to the upper bound of the component , then fix it to , the remaining increase will be redistributed to other components that have not reached the top, and the above process will be repeated until the sum converges to .in, Indicates the number of components that can be adjusted in the current component. In this embodiment, is the number of adjustable variables in a set of pruning rates.

[0109] After the above adjustments, the adjusted forced solution is obtained. For case 1, the corresponding loss is ; For case 2, the corresponding loss is The final solution to this problem is Expressed as:

[0110]

[0111] After the above solution, that is, this step can finally obtain a set of optimal pruning rates through the search process , which is the optimal pruning rate combination.

[0112] S5. Use the optimal pruning rate found above under the condition of using only a small amount of data, that is, a small sample training set The pre-trained Transformer model obtained in step S2 is compressed, and the compressed model is fine-tuned using the feature alignment-based knowledge distillation framework to restore accuracy to obtain the final Transformer model.

[0113] The knowledge distillation framework based on feature alignment is used to perform knowledge distillation on the compressed model. The knowledge distillation framework based on feature alignment is to use feature alignment to achieve the loss of teacher-student model distillation. This distillation loss It can be expressed as:

[0114] ;

[0115] in, and Represent the output features of the pruned student model and the original teacher model before the classification head, Represents the Frobenius norm, which is a norm for matrices or tensors. It means taking the square root of the sum of the squares of each element. refers to the compressed model (student model), refers to the pre-trained Transformer model (teacher model), Represents a small sample training set The samples in .

[0116] In one embodiment of the present invention, a backpropagation strategy is used to update the pre-trained Transformer model. The backpropagation strategy is not to perform backpropagation on all blocks in the pre-trained Transformer model, but only to update the pruned block and an adjacent block. This strategy is more conducive to fine-tuning with a small number of samples, and ultimately obtains a fine-tuned and updated student model.

[0117] In one embodiment of the present invention, progressive fine-tuning is used as a training strategy. That is, each time a specified block is pruned, the pre-trained Transformer model is fine-tuned through knowledge distillation for multiple rounds (e.g., 2000 epochs) using a small sample training set according to the back-propagation strategy until all blocks are pruned to obtain the student model after the final accuracy is restored.

[0118] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. Transformer adaptive compression method in data-limited scenarios, characterized by: The following steps are involved: Obtain a small sample training set including multiple image samples; Get the pre-trained Transformer model; By adaptively allocating pruning rates based on the alternating direction multiplier method and Bayesian optimization, the optimal pruning rate combination is obtained, including: The alternating direction multiplier method is used to transform the pruning rate allocation constraint into a differentiable optimization problem, and auxiliary variables are introduced to facilitate the solution. The differentiable optimization problem is divided into optimality subproblems and feasibility subproblems. The update of the pruning rate is the solution to the optimality subproblem, and the update of the auxiliary variables is the solution to the feasibility subproblem. Iteratively solve the optimality subproblem and feasibility subproblem. The iteration includes outer loop iteration and inner loop iteration. In the inner loop iteration, the optimality subproblem is solved iteratively through Bayesian optimization. Through iterative solution, the optimal pruning rate combination is obtained; Under a small sample training set, the pre-trained Transformer model is compressed using a joint pruning strategy and the optimal pruning rate combination. The compressed model is fine-tuned for multiple rounds using a knowledge distillation framework based on feature alignment to obtain the final Transformer model. A block in the pre-trained Transformer model includes an Attention layer and an FFN layer, and each layer is preceded by a corresponding LayerNorm layer. The joint pruning strategy combines block-level pruning and channel pruning. Block-level pruning is to prune the Attention layer of the selected block in the pre-trained Transformer model as a whole, removing the Attention layer and the corresponding LayerNorm layer. Channel pruning is to prune the FFN layer of the selected block in the pre-trained Transformer model. The pre-trained Transformer model is updated using a backpropagation strategy. The backpropagation strategy is to not perform backpropagation on all blocks in the pre-trained Transformer model, but only update the pruned block and an adjacent block.

2. The Transformer adaptive compression method in data-constrained scenarios according to claim 1, characterized in that: The obtaining of a small sample training set including a plurality of image samples comprises: Under data-limited conditions, a small number of image samples required for the task are collected to obtain a small sample dataset; The small sample data set is preprocessed to obtain a small sample training set.

3. The Transformer adaptive compression method in data-constrained scenarios according to claim 1, characterized in that In the search process of adaptive pruning rate allocation using the search method, the pre-trained Transformer model is compressed through the joint pruning strategy, and the compressed model is fine-tuned for a few rounds using a small sample training set and the performance is evaluated by the loss function.

4. The Transformer adaptive compression method in a data-constrained scenario according to any one of claims 1 to 3, characterized in that: The knowledge distillation framework based on feature alignment is to use feature alignment to achieve the loss of teacher-student model distillation. Expressed as: ； in, and Represent the output features of the pruned student model and the original teacher model before the classification head, represents the Frobenius norm, refers to the compressed model, that is, the student model, Refers to the pre-trained Transformer model, that is, the teacher model, Represents a small sample training set The samples in .

5. A Transformer adaptive compression system for data-constrained scenarios, configured to implement the Transformer adaptive compression method for data-constrained scenarios according to any one of claims 1 to 4, characterized in that: Includes the following modules: A sample acquisition module, used to acquire a small sample training set including multiple image samples; The original model acquisition module is used to obtain the pre-trained Transformer model; An optimal pruning rate acquisition module is used to perform adaptive pruning rate allocation based on an alternating direction multiplier method and a Bayesian optimization search method to obtain an optimal pruning rate combination; The compression fine-tuning module is used to compress the pre-trained Transformer model through a joint pruning strategy and the optimal pruning rate combination under a small sample training set, and to perform multiple rounds of fine-tuning on the compressed model using a knowledge distillation framework based on feature alignment to obtain the final Transformer model.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images