Transform adaptive compression method and system in data limited scene

Abstract

The invention discloses a Transform self-adaptive compression method and a Transform self-adaptive compression system in a data limited scene. The method comprises the following steps: acquiring a small sample training set comprising a plurality of image samples; the method comprises the following steps: acquiring a pre-trained Transform model; performing adaptive pruning rate distribution on the pruning rate through a search method based on an alternating direction multiplier method and Bayesian optimization to obtain an optimal pruning rate combination; and compressing the pre-trained Transform model by using the optimal pruning rate combination under a small sample training set, and performing multi-round fine tuning on the compressed model by using a knowledge distillation framework based on feature alignment to obtain a final Transform model. The method can be flexibly applied to an image classification task under the condition that data is limited, and is of great significance to the practical application of the visual Transform in related tasks.

Description

Technical Field

[0001] The present invention belongs to the field of model compression, and particularly relates to a Transformer adaptive compression method and system in data-constrained scenarios. Background Art

[0002] In recent years, deep learning models based on Transformer have achieved breakthrough progress in natural language processing, computer vision, and cross-modal tasks. Its core self-attention mechanism can capture long-range dependencies and model complex data distributions by stacking multiple layers. However, with the exponential growth of model size, Transformer faces significant challenges in practical deployment: on the one hand, the huge computational resource requirements (such as video memory occupancy, inference latency) limit its application in resource-constrained scenarios such as mobile devices and edge computing; on the other hand, traditional compression methods highly rely on large-scale training data for fine-tuning or knowledge distillation, and in data-sensitive fields, the scarcity of high-quality training data makes it difficult to effectively implement existing compression technologies.

[0003] In data-constrained scenarios, some researchers have conducted research on the compression of deep learning models. However, current research usually relies on manually designed standard compression schemes, which cannot achieve refined compression and often do not consider global correlations, leading to being trapped in local optimal strategies. For example, the Practise algorithm proposed by Guo-Hua Wang et al. mainly compresses convolutional neural networks in visual tasks in data-constrained scenarios. The strategy for model compression is to directly discard selected partial block structures, unable to achieve dense and precise compression (Wang G H, Wu J. Practical network acceleration with tiny sets[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023: 20331-20340.); the Cross Distillation algorithm proposed by Haoli Bai et al. also compresses convolutional neural networks in data-constrained scenarios. The strategy for model compression is to directly set the same ratio for pruning all layers of the model, without considering the differences between different layers of the model (Bai H, Wu J, King I, et al. Few shot network compression via cross distillation[C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34(04): 3203-3210.); the DC-ViT algorithm proposed by Hanxiao Zhang et al. mainly compresses vision transformers in data-constrained scenarios. The strategy for model compression is to simply evenly distribute the parameters to be pruned to each block to be compressed, without considering the sensitivity differences of different blocks of the model, which may lead to suboptimal compression and ultimately fail to obtain the optimal performance (Zhang H, Zhou Y, Wang G H. Dense vision transformer compression with few samples[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2024: 15825-15834.). Summary of the Invention

[0004] To at least solve one of the problems existing in the prior art, the present invention provides a Transformer adaptive compression method and system in a data-constrained scenario, which can allocate the pruning ratio more reasonably, thereby improving the performance of the model after compression.

[0005] The object of the present invention is achieved by at least one of the following technical solutions.

[0006] A Transformer adaptive compression method and system in a data-constrained scenario, comprising the following steps: Obtain a small-sample training set including a plurality of image samples; Obtain a pre-trained Transformer model; Perform adaptive pruning rate allocation through a search method based on the alternating direction method of multipliers and Bayesian optimization to obtain an optimal pruning rate combination; Under the small-sample training set, compress the pre-trained Transformer model through a joint pruning strategy and use the optimal pruning rate combination, and perform multiple rounds of fine-tuning on the compressed model using a knowledge distillation framework based on feature alignment to obtain the final Transformer model.

[0007] Further, the obtaining of the small-sample training set includes: Collect a small amount of image samples required for the task under data-constrained conditions to obtain a small-sample data set; Preprocess the small-sample data set to obtain a small-sample training set.

[0008] Further, the preprocessing includes: Scale and center-crop the samples in the small-sample data set to ensure uniform input size, then convert the image into a processable tensor and adjust the channel order, and finally standardize the RGB channels of the image to normalize the input data and obtain the processed small-sample training set. .

[0009] Further, for the obtaining of the pre-trained Transformer model, obtain a pre-trained Transformer model on a large-scale general image data set, and through the analysis of specific tasks, after comprehensively considering the types and numbers of categories required for the task images and the size of computing resources, select a visual Transformer model of an appropriate scale pre-trained on an appropriate data set.

[0010] Furthermore, a Block in the pre-trained Transformer model includes an Attention layer and an FFN layer, and a corresponding LayerNorm layer is included in front of each layer. The joint pruning strategy combines block-level pruning and channel pruning. Block-level pruning is to perform whole-block pruning on the Attention layer in the selected Blocks in the pre-trained Transformer model, removing the entire Attention layer and the corresponding LayerNorm layer. Channel pruning is to prune the FFN layer in the Blocks of the pre-trained Transformer model.

[0011] Furthermore, the structure of the FFN layer is , performing a dimension transformation that first increases and then decreases, which can be expressed as , where is the activation function, is the size of a single dimension. In channel pruning, it refers to pruning the channels of the dimension. Let the pruning rate be , then the channel dimension after pruning changes to , where the selection of channels is randomly chosen.

[0012] Furthermore, the pre-trained Transformer model is updated using the backpropagation strategy. The backpropagation strategy is as follows: Instead of performing backpropagation on all Blocks in the pre-trained Transformer model, only the pruned Blocks and one adjacent Block are updated. This strategy is more conducive to fine-tuning with a small number of samples, and finally a fine-tuned and updated student model is obtained.

[0013] Furthermore, the optimal pruning rate combination is obtained by performing adaptive pruning rate allocation through a search method based on the alternating direction method of multipliers and Bayesian optimization, including: Using the alternating direction method of multipliers to transform the pruning rate allocation constraint conditions into a differentiable optimization problem, and introducing auxiliary variables for solution; The differentiable optimization problem is split into an optimality subproblem and a feasibility subproblem. The update of the pruning rate is to solve the optimality subproblem, and the update of the auxiliary variable is to solve the feasibility subproblem; Iteratively solve the optimality subproblem and the feasibility subproblem. The iteration includes outer-loop iteration and inner-loop iteration. In the inner-loop iteration, the optimality subproblem is solved iteratively through Bayesian optimization; Through iterative solution, the optimal pruning rate combination is obtained.

[0014] Further, in the search process of using the search method for adaptive pruning rate allocation, after compressing the pre-trained Transformer model through the joint pruning strategy, the compressed model is fine-tuned with a small sample training set for a small number of rounds and the performance is evaluated through the loss function.

[0015] Further, based on the feature alignment-based knowledge distillation framework, the compressed model is fine-tuned to recover the accuracy under the condition of only using a small amount of data. The feature alignment-based knowledge distillation framework is to use the feature alignment method to implement the loss of teacher-student model distillation, and the distillation loss is expressed as: ; where and respectively represent the output features of the pruned student model and the original teacher model before the classification head. represents the Frobenius norm. refers to the compressed model, that is, the student model. refers to the pre-trained Transformer model, that is, the teacher model. represents the small sample training set in the samples.

[0016] The Transformer adaptive compression system in the data-constrained scenario provided by the present invention includes the following modules: A sample acquisition module for acquiring a small sample training set; An original model acquisition module for acquiring a pre-trained Transformer model; An optimal pruning rate acquisition module for performing adaptive pruning rate allocation through a search method based on the alternating direction multiplier method and Bayesian optimization to obtain an optimal pruning rate combination; A compression and fine-tuning module for compressing the pre-trained Transformer model through the joint pruning strategy and using the optimal pruning rate combination under the small sample training set, and performing multiple rounds of fine-tuning on the compressed model using the feature alignment-based knowledge distillation framework to obtain the final Transformer model.

[0017] The present invention also provides a computer device.

[0018] The present invention also provides a computer-readable storage medium.

[0019] Compared with the prior art, the present invention includes the following beneficial effects: Compared with traditional methods, the present invention focuses on compression and distillation with a small number of samples, can effectively utilize the limited knowledge contained in a small number of samples, does not produce problems such as overfitting caused by too few samples, significantly reduces the dependence on large-scale training data, achieves better model compression effects, and improves the applicability and reliability of the method in scenarios where real data is limited.

[0020] (2)The present invention adopts a joint pruning strategy to efficiently compress the model. The joint pruning strategy coordinates block-level attention head pruning and channel feature pruning, not only retains the global attention characteristics of the Transformer model, but also dynamically eliminates local redundant features, and can achieve more comprehensive model compression.

[0021] (3)The present invention dynamically allocates the pruning rate of the model layer through a search method based on the alternating direction method of multipliers and Bayesian optimization, achieving dense and compact high-precision model compression. Through knowledge distillation, the compressed model is fine-tuned to restore the accuracy under the condition of only using a small amount of data, and finally achieves the optimal balance between model accuracy loss and computational resource consumption.

[0022] (4)The present invention innovatively constructs a two-layer optimization framework, uses the alternating direction method of multipliers to convert the pruning rate allocation problem into a differentiable optimization problem, and uses Bayesian optimization to solve the sub-problems therein. This framework breaks through the bottleneck of traditional fixed pruning rate allocation, can achieve automatic search for the global optimal compression configuration, and thus can obtain a compression effect with better performance compared to the fixed allocation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flowchart of the Transformer adaptive compression method in a data-limited scenario in an embodiment of the present invention.

[0024] Figure 2 is a pruning flowchart integrating dual strategies of block-level pruning and channel pruning in an embodiment of the present invention.

[0025] Figure 3 is a flowchart of the two-layer optimization framework in an embodiment of the present invention.

[0026] Figure 4 is a brief flowchart of the Transformer adaptive compression method in a data-limited scenario in an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts are within the scope of protection of the present invention.

[0028] As Figure 1 shown, an adaptive compression method for Transformer in data-constrained scenarios provided by an embodiment of the present invention includes the following steps: S1. Under data-constrained conditions, collect a small number of image samples required for the task to obtain a small-sample dataset. After preprocessing the small-sample dataset, obtain a small-sample training set.

[0029] In one embodiment of the present invention, the image classification dataset ImageNet-1K covering a large number of common categories is selected as the original dataset, and a small number of image samples are randomly selected from the original dataset as the small-sample dataset in the data-constrained scenario.

[0030] The small-sample dataset is denoted as , where represents the length of the small-sample dataset, i.e., the total amount of data, represents the -th image sample in the small-sample dataset, ; in one embodiment of the present invention, .

[0031] Preprocess the small-sample dataset. The specific steps of the preprocessing are as follows: Scale and center-crop the image samples in the collected small-sample dataset to ensure a unified input size. Subsequently, convert the image samples into processable tensors and adjust the channel order. Finally, standardize the RGB channels of the image samples to normalize the input data and obtain the processed small-sample training set ; in one embodiment of the present invention, the scaling is to uniformly process the image samples into in size, and the center-crop is to crop a piece of in size from the central region of the image sample on the basis of the scaling.

[0032] S2. Obtain a pre-trained Transformer model on a large-scale general image dataset.

[0033] In one embodiment of the present invention, a pre-trained ViT-Base model on the ImageNet-1K image classification dataset is obtained as the baseline model, and the baseline model is the pre-trained Transformer model.

[0034] S3. Construct a joint pruning strategy that combines block-level pruning and channel pruning for pruning the Blocks in the pre-trained Transformer model.

[0035] Please refer to Figure 2 , in the joint pruning strategy: For block-level pruning, a Block in the pre-trained Transformer model includes an Attention layer and an FFN layer, and a corresponding LayerNorm layer is included in front of each layer. Block-level pruning refers to pruning the entire Attention layer in the selected Block, removing the entire Attention layer and the corresponding LayerNorm layer.

[0036] For channel pruning, it refers to pruning the FFN layer in the selected Block in the pre-trained Transformer model. Specifically, the structure of the FFN layer includes a first linear layer, an activation function layer, and a second linear layer, usually , a dimension transformation of first increasing and then decreasing is performed, which can be expressed as , where is the activation function, is the channel dimension size. In channel pruning, the channels of the dimension are pruned. Let the pruning rate = , then the dimension change of the pruned channels is , and the pruning rate is a variable searched by the search method of S4. Among them, the selection of channels is randomly selected. In one embodiment of the present invention, the channel dimension .

[0037] When selecting Blocks, the number A of Blocks to be pruned and the number B of parameters to be pruned for each Block are determined according to the number of parameters to be pruned (determined by the total number of parameters of the known model and the preset compression target) and the number of parameters of the Block. After compressing the number B of parameters for all possible Blocks to be pruned and performing a full round of fine-tuning, the performance gap with the original model is measured, and the top A Blocks with the smallest gap are selected according to the ranking of this gap.

[0038] S4. Adaptive pruning rate allocation for the pruning rate is performed through an efficient search method based on the alternating direction method of multipliers and Bayesian optimization. During the search process, a joint pruning strategy is used to compress the pre-trained Transformer model, and the compressed model is fine-tuned using a small sample training set for a small number of rounds and the performance is evaluated to obtain the optimal pruning rate combination.

[0039] Please refer to Figure 3 , this step includes: S4.1. Use the alternating direction method of multipliers to transform the pruning rate allocation constraint conditions into a differentiable optimization problem (ALF), which is easy to solve under certain constraint conditions.

[0040] Among them, the differentiable optimization problem can be defined as:

[0041] ; Among them, is the task function to be minimized, is the loss function, is the pruning rate, is the set of upper and lower bounds of each pruning rate component, is the total pruning rate. Indicates that the sum of each pruning rate component in the pruning rate should be equal to to meet the overall compression ratio requirement.

[0042] The loss function is expressed as: ; Among them, represents the Cross-Entropy function, , are the pre-trained Transformer model and the Transformer model pruned and fine-tuned using the pruning rate respectively, is the synthetic image dataset, which is a dataset artificially synthesized according to the information of the original pre-trained Transformer model, represents the training sample is classified as the th class by the model pruned and fine-tuned using the pruning rate probability, is the prediction result of the Transformer model pruned and fine-tuned using the pruning rate on the training sample , Indicates the pruning rate for the pre-trained Transformer model to perform pruning Indicates the training samples being classified by the pre-trained Transformer model as the probability of the k-th class Indicates the set of image sample classes Indicates the prediction result of the pre-trained transformer model on the training samples above

[0043] In this step, a large number of points to be evaluated, that is, pruning rates, will be generated during the iterative solution in the search process. After compressing using these pruning rates and performing a small number of rounds of fine-tuning, the loss function is used to calculate the performance gap between the compressed and fine-tuned model and the pre-trained transformer model as the evaluation value

[0044] In the embodiment of the present invention, in a scenario where data is limited, a large-scale test data set cannot be directly used to measure the accuracy when evaluating the performance of the pruned and fine-tuned model. Therefore, an alternative evaluation scheme needs to be found. If the training loss during fine-tuning is directly used as the evaluation criterion, it will not be able to well reflect the model performance due to overfitting. Therefore, the embodiment of the present invention uses the loss function to reflect the gap between the two models (i.e., the loss function value). The performance ranking of the pruned and fine-tuned models under different pruning rates can be obtained according to the loss function value, so as to reflect the relative performance between the models. Due to limited data resulting in too little available data, the embodiment of the present invention is based on the idea of data augmentation and assumes that the pre-trained transformer model can fully train and retain the key information in the data set. A synthetic image data set generated using the information of the original pre-trained transformer model is used as the evaluation data set. Using the synthetic image data set

[0045] In one embodiment of the present invention, for the synthesis process of the synthetic image data set , first generate an image data set randomly initialized with Gaussian noise and a set of arbitrary target labels . The synthetic image data set is updated by minimizing the following objective

[0046] where refers to the cross-entropy measured on the image data set and is the cross-entropy between the predicted probability and the target label, is a regularization term, denotes the th target label, denotes the th sample in the image dataset, , denotes the image dataset contains image samples. In one embodiment of the present invention, the number of image samples in the synthetic image dataset is .

[0047] The regularization term is calculated using the following formula: ; where, and are two hyperparameters, used to improve image stability, used to control the sharpness of the synthetic data; refers to the pixel at the position ( ) of the image sample ; refers to the pixel at the position ( ) of the image sample ; refers to the pixel at the position ( ) of the image sample ; is a hyperparameter for control; is a term to improve stability; is a term to control data sharpness.

[0048] Furthermore, is the variable to be optimized, i.e., a set of pruning ratios, which can be expressed as: ; where, is the pruning ratio of the th block of the pre-trained Transformer model. The constraint condition is a summation term, i.e., the sum of each component of the pruning ratio should be equal to a value ; is the number of pruning ratios.

[0049] After that, this differentiable optimization problem can be rewritten in the following form for easy optimization:

[0050] ; Among them, is a hyperparameter for balancing, is an auxiliary variable introduced for solving, is an indicator function, defined as: ; In one embodiment of the present invention, The lower limit of is set to 50, and the upper limit is the minimum value between 3.99 and the upper limit value calculated according to the pruning rate.

[0051] S4.2. To further solve this differentiable optimization problem, the Lagrange multiplier is introduced for iterative solution, expressed as: ; Among them, is the augmented Lagrangian function, is the Lagrange multiplier, is a penalty parameter. In one embodiment of the present invention, The initial value of

[0052] After obtaining this augmented Lagrangian function, first initialize the unknown variables and the Lagrange multiplier, and use the alternating direction method of multipliers for round-by-round solution. In the th round of solution, the update rule is expressed as:

[0053]

[0054] ; In the formula, refers to the pruning rate combination solved in the th round, refers to the auxiliary variable solved in the th round, refers to the Lagrange multiplier variable solved in the th round, refers to the Lagrange multiplier variable solved in the th round, refers to the th round of the solved auxiliary variable. In the update, the update of the pruning rate is defined as the solution of the optimal subproblem, which is to minimize the loss function , that is, to maximize the accuracy of the pruned model and achieve higher accuracy in the classification task. For the auxiliary variable The update of is a loss function for evaluating the model accuracy, and the constraint condition is a summation constraint, which can be expressed as is a value calculated under a total pruning rate to satisfy that the total pruning rate of the model meets the set value.

[0055] S4.3. Since the optimal sub - problem usually involves unknown objectives, Bayesian optimization is introduced to solve the optimal sub - problem. For the feasibility sub - problem, it can usually be directly solved.

[0056] The iteration in the present invention has two layers, including the alternating direction multiplier method iteration (outer - loop iteration) and the Bayesian optimization iteration (inner - loop iteration). The Bayesian optimization iteration is used to solve the optimal sub - problem in each inner loop. In the optimization of each optimal sub - problem, the Gaussian process (GP) is used to model the objective function of the optimal sub - problem, and EI is used as the selection function of the objective. During the iterative solution process, since the whole algorithm will generate a large number of different points to be evaluated when seeking the optimal solution, a large number of loss function evaluations are required , in the traditional method, since the loss function usually represents using the test data set to test the model accuracy, this evaluation method will bring a large amount of extra time, so it is basically infeasible. However, this problem does not exist in the embodiments of the present invention. Under the condition of data limitation, the embodiments of the present invention use the loss function mentioned above to evaluate the model performance. This evaluation method is quite fast and will not have a significant impact on time in large - scale evaluations, so it is feasible under this condition.

[0057] For the optimal sub - problem regarding the pruning rate , in the th main loop, for the sake of convenient representation, the optimization objective function is abbreviated as .

[0058] Use Bayesian optimization to solve the loss function . Assume that follows the GP prior. Since it is solved step by step, the second term can usually be regarded as a constant. Therefore, can still be modeled as GP. Therefore, based on the randomly generated initial observation data , can be calculated. refers to the randomly generated ​Group initial pruning rate, is the number of generated data, is the value calculated using the loss function and is used to measure the model performance, refers to the set of the pruning rate and the optimization objective value at this time, refers to using function to calculate the required optimization objective value at this time. This is to calculate the following improvement amount and then solve for the optimal pruning rate.

[0059] After that, the standard EI (Expected Improvement, EI) is adopted to calculate the improvement amount here: ; where, is the standard expected improvement function, which is a mathematical method for optimization problems. By calculating on the basis of the current optimal solution, it selects the next evaluation point by estimating the improvement degree that can be expected to be obtained; refers to calculating the expected value of the optimization objective under the premise of the known set , that is, estimating the positive part of the expected value of the potential improvement amount ( ) to facilitate the selection of the evaluation point, refers to the optimal optimization objective value in the current round; , are the mean and variance distributions of the posterior distribution respectively; represents the probability that under the current model prediction, the target value is less than or equal to the current optimal optimization objective value . When the predicted mean is much lower than , the value of this term approaches 0 (low probability of improvement). When the predicted mean is close to or exceeds , the value of this term approaches 1 (high probability of improvement), represents the probability density that under the current model prediction, the target value is located near the deviation degree of . When the difference between the predicted mean and is close to 0, the value of this term reaches the peak; the larger the difference, the smaller the value of this term.

[0060] Among them, the Sobol sequence is used to generate any set of pruning rates , and calculate this set of pruning rates The EI value. Further, the optimal pruning rate for each round can be selected according to this EI value. The Sobol sequence is a low-discrepancy sequence that can generate a highly uniformly distributed deterministic point set for use scenarios such as sampling.

[0061] For a feasibility sub-problem, it is usually defined as a problem composed of one or more simple inequalities or equations. For a single problem, the problem to be solved in a single round is defined as: ; Where is the auxiliary variable to be solved is the th component of , and are the lower and upper bounds of each component respectively. In one embodiment of the present invention, this lower and upper bounds are consistent with the range of the previous

[0062] When solving the problem, first, initialize a basic vector , and then perform an initial cut on this basic vector. The purpose is to limit each component in the basic vector to the interval . After cutting, the variable is obtained. Here, the problem is solved in two cases to obtain the auxiliary variable : Case 1: Use the forced solution to project the variable so that the projected variable is forced to satisfy ; Case 2: Use the free solution , and .

[0063] For Case 1, the component adjustment can be performed according to the relationship between and the total pruning rate . is the sum of all components of the variable , which is used to determine whether the current satisfies the constraint condition. If , it proves that the constraint has been satisfied, and let ; if , the goal is to find the adjustment amount such that . First, the initial reduction amount can be calculated, and all components are tried to be subtracted by the initial reduction amount . If some of these components reach the bottom, that is, less than or equal to the lower bound of the component , then fix it to , the remaining reduction amount will be redistributed to other components that have not reached the bottom, and the above process will be repeated until the sum converges to ; if , then the goal is to find the adjustment amount such that , represents the -th component in the variable . First, the initial increase amount can be calculated, and an attempt is made to increase all components by the initial increase amount . If some of these components reach the top, that is, they are greater than or equal to the upper bound of the component, then they will be fixed at , and the remaining increase amount will be redistributed to other components that have not reached the top. The above process will be repeated until the sum converges to . Among them, represents the number of adjustable components in the current component. In this embodiment, is the number of adjustable variables in a set of pruning rates.

[0064] After the above adjustment, the adjusted forced solution is obtained. For case 1, the corresponding loss is ; for case 2, the corresponding loss is . Then, the final solution selection for this problem is expressed as:

[0065] After the above solution, that is, in this step, a set of optimal pruning rates can be finally obtained through the search process, that is, the optimal pruning rate combination.

[0066] S5. Under the condition of using only a small amount of data, that is, in a small sample training set, use the above-mentioned set of optimal pruning rates to compress the pre-trained Transformer model obtained in step S2, and use the knowledge distillation framework based on feature alignment to fine-tune the compressed model to restore the accuracy, and obtain the final Transformer model.

[0067] Knowledge distillation is performed on the compressed model through the knowledge distillation framework based on feature alignment. The knowledge distillation framework based on feature alignment is to use the method of feature alignment to implement the loss of teacher-student model distillation. This distillation loss can be expressed as: ; Among them, and respectively represent the output features before the classification head of the pruned student model and the original teacher model. denotes the Frobenius norm, which is a norm for matrices or tensors and means taking the square root after summing the squares of each element. refers to the compressed model (student model). refers to the pre-trained Transformer model (teacher model). denotes the few-shot training set samples in it.

[0068] In one embodiment of the present invention, a backpropagation strategy is adopted to update the pre-trained Transformer model. The backpropagation strategy is not to perform backpropagation on all Blocks in the pre-trained Transformer model, but only to update the pruned Blocks and one adjacent Block. This strategy is more conducive to fine-tuning with a small number of samples, and finally the fine-tuned and updated student model is obtained.

[0069] In one embodiment of the present invention, progressive fine-tuning is used as the training strategy, that is, for each pruned specified Block, according to the backpropagation strategy, the pre-trained Transformer model is fine-tuned through knowledge distillation with the few-shot training set for multiple rounds (such as 2000 epochs) until all Blocks are pruned, and the student model with the final restored accuracy is obtained.

[0070] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and are all included in the protection scope of the present invention.

Claims

1. Transformer adaptive compression method in data-limited scenarios, characterized in that, It includes the following steps: Obtain a small-sample training set including multiple image samples; Obtain a pre-trained Transformer model; Perform adaptive pruning rate allocation through a search method based on the alternating direction method of multipliers and Bayesian optimization to obtain an optimal pruning rate combination; Under the small-sample training set, compress the pre-trained Transformer model through a joint pruning strategy and use the optimal pruning rate combination, and perform multiple rounds of fine-tuning on the compressed model using a knowledge distillation framework based on feature alignment to obtain the final Transformer model.

2. The Transformer adaptive compression method in the data-constrained scenario according to claim 1, wherein The obtaining of the small-sample training set includes: Collect a small number of image samples required for the task under data-limited conditions to obtain a small-sample data set; Preprocess the small-sample data set to obtain a small-sample training set.

3. The Transformer adaptive compression method in the data-limited scenario according to claim 1, wherein One Block in the pre-trained Transformer model includes an Attention layer and an FFN layer, and a corresponding LayerNorm layer is included in front of each layer. The joint pruning strategy combines block-level pruning and channel pruning. Block-level pruning is to perform whole-block pruning on the Attention layer of the selected Block in the pre-trained Transformer model, removing the Attention layer and the corresponding LayerNorm layer. Channel pruning is to perform pruning on the FFN layer of the selected Block in the pre-trained Transformer model.

4. The Transformer adaptive compression method in the data-constrained scenario according to claim 3, characterized in that, Adopt a backpropagation strategy to update the pre-trained Transformer model. The backpropagation strategy is to not perform backpropagation on all Blocks in the pre-trained Transformer model, but only update the pruned Blocks and one adjacent Block.

5. The Transformer adaptive compression method in the data-constrained scenario according to claim 1, wherein The performing of adaptive pruning rate allocation through a search method based on the alternating direction method of multipliers and Bayesian optimization to obtain an optimal pruning rate combination includes: Use the alternating direction method of multipliers to transform the pruning rate allocation constraint condition into a differentiable optimization problem and introduce auxiliary variables for solution; Split the differentiable optimization problem into an optimality sub-problem and a feasibility sub-problem. The update of the pruning rate is to solve the optimality sub-problem, and the update of the auxiliary variable is to solve the feasibility sub-problem; Iteratively solve the optimality sub-problem and the feasibility sub-problem. The iteration includes an outer-loop iteration and an inner-loop iteration. In the inner-loop iteration, solve the optimality sub-problem through Bayesian optimization iteration; Obtain an optimal pruning rate combination through iterative solution.

6. The Transformer adaptive compression method in the data-limited scenario according to claim 1, wherein During the search process of performing adaptive pruning rate allocation using the search method, after compressing the pre-trained Transformer model through the joint pruning strategy, perform a small number of rounds of fine-tuning on the compressed model using the small-sample training set and evaluate the performance through a loss function.

7. The Transformer adaptive compression method in the data-limited scenario according to any one of claims 1-6, characterized in that The knowledge distillation framework based on feature alignment is to use the feature alignment method to implement the loss of teacher-student model distillation, and the distillation loss is expressed as: ； wherein, and respectively represent the output features of the pruned student model and the original teacher model before the classification head, represents the Frobenius norm, refers to the compressed model, i.e., the student model, refers to the pre-trained Transformer model, i.e., the teacher model, represents the few-shot training set and the samples in it.

8. Transformer adaptive compression system in data-constrained scenarios, characterized in that It includes the following modules: A sample acquisition module for obtaining a small-sample training set; An original model acquisition module for obtaining a pre-trained Transformer model; The optimal pruning rate acquisition module is used to obtain the optimal pruning rate combination by performing adaptive pruning rate allocation through a search method based on the alternating direction method of multipliers and Bayesian optimization; The compression and fine-tuning module is used to compress the pre-trained Transformer model by means of a joint pruning strategy and use the optimal pruning rate combination under a small-sample training set, and perform multiple rounds of fine-tuning on the compressed model using a knowledge distillation framework based on feature alignment to obtain the final Transformer model.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the method according to any one of claims 1 to 7 is implemented.

Images