A hierarchical pruning method and application for large models based on pruning impact rate

Abstract

The present invention relates to a large model layered pruning method and application based on pruning impact ratio. This method measures the impact of pruning on the model's mid-layers by calculating the similarity of layer outputs obtained after pruning at different pruning rates. Based on the varying impacts of pruning on different layers in the large model, different pruning rates are assigned to different layers, effectively improving the accuracy of the pruned model. Furthermore, the present invention considers the impact of pruning on the model's mid-layers and uses the pruning impact ratio to assign different sparsity rates to different layers of the model. This method can be combined with unstructured pruning methods to reduce the performance loss caused by pruning.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a large model hierarchical pruning method based on pruning influence rate and its application. Background Art

[0002] Today's large models are mainly based on the Transformer structure. The Transformer structure is a sequence-to-sequence architecture based on the self-attention mechanism, which processes sequences through self-attention layers and feedforward layers.

[0003] However, one of the problems lies in the sheer size of large models and their billions of parameters. However, the large number of parameters also means that these models require enormous hardware resources and take a significant amount of time to use, which greatly complicates their actual deployment. To address this issue, researchers have proposed a variety of model compression methods, including model pruning, model quantization, and knowledge distillation. Model pruning involves removing redundant parts of a model using different pruning granularities and criteria to obtain different sub-models, which can reduce the number of model parameters while maintaining model performance. Model quantization involves converting model parameters or intermediate variables from floating-point numbers to low-precision integer representations, which can reduce the model's storage and computational requirements. Knowledge distillation transfers the knowledge of a large, complex teacher model to a smaller, simpler student model. While maintaining a small model size, it can inherit the generalization capabilities of the complex teacher model and achieve better model performance.

[0004] Current pruning methods simply assume that each layer in a large model plays the same role in the model, and that pruning has the same impact on each layer. Therefore, the same pruning rate is used for each layer in the large model, resulting in a large performance loss due to pruning.

[0005] Therefore, how to formulate appropriate pruning strategies based on the different effects of pruning on different layers in large language models and improve the accuracy of the model is an urgent problem to be solved. Summary of the Invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a large model hierarchical pruning method and application based on pruning influence rate, so as to solve or partially solve the problems of large hardware resources and time consumption when using large models in downstream applications of large language models such as language modeling.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] One aspect of the present invention provides a large model hierarchical pruning method based on pruning influence rate, comprising the following steps:

[0009] Obtain batches of calibration data and input them into a target large model consisting of multiple layers. When the first i-1 layers are configured as the first reference sparsity rate α, the calibration data X of the tth batch is obtained. t The output data X after passing through the first i-1 layers t,i-1 ;

[0010] Based on the output data X t,i-1 , calculate the output X of the i-th layer when it is configured as the second reference sparsity rate β t,i,β ;

[0011] Based on the output data X t,i-1 , calculate the output X when the i-th layer is configured as the third reference sparsity rate γ t,i,γ , where β≠γ;

[0012] Based on the second reference sparsity rate β and the output ∑ under the test third reference sparsity rate γ t X t,i,β and ∑ t X t,i,γ , calculate the pruning impact rate of the i-th layer;

[0013] Based on the pruning impact rate, the actual sparsity rate of each layer is adjusted in layers to achieve layered pruning.

[0014] As a preferred technical solution, the first reference thinning rate α=45%-55%, the second reference thinning rate β=35%-45%, and the third reference thinning rate γ=45%-55%.

[0015] As a preferred technical solution, the first reference thinning rate α=50%, the second reference thinning rate β=40%, and the third reference thinning rate γ=50%.

[0016] As a preferred technical solution, the pruning impact rate of the i-th layer is calculated using the following formula:

[0017]

[0018] Among them, ||||2 represents the modulo operation, T represents transpose, and the upper horizontal line represents the average value.

[0019] As a preferred technical solution, the process of implementing layered pruning based on the pruning impact rate includes the following steps:

[0020] Sort by the pruning impact rate of each layer;

[0021] A top-ranked layer is configured to have a first actual sparsity rate, and a bottom-ranked layer is configured to have a second actual sparsity rate, to implement layered pruning, wherein the first actual sparsity rate is greater than the second actual sparsity rate.

[0022] As a preferred technical solution, the first actual thinning rate is 50%-60%, and the second actual thinning rate is 40%-50%.

[0023] As a preferred technical solution, the first actual thinning rate is 54%, and the second actual thinning rate is 46%.

[0024] As a preferred technical solution, when the actual sparsity rate of each layer is adjusted in layers, the target layers for adjustment include all layers except the input layer and the final output of the target large model.

[0025] As a preferred technical solution, the target large model is LLaMA or LLaMA-2.

[0026] Another aspect of the present invention provides an application of a large model hierarchical pruning method based on pruning influence rate, comprising:

[0027] The input text sequence information is obtained as the input of the target large model, and the completed text information is output according to the text context through language modeling. In the inference optimization process of the target large model, the aforementioned large model hierarchical pruning method based on pruning influence rate is used for hierarchical pruning.

[0028] Compared with the prior art, the present invention has at least the following beneficial effects:

[0029] Improve pruning accuracy and reduce performance loss caused by pruning on large models: The present invention measures the impact of pruning on the middle layers of the model by calculating the similarity of layer outputs obtained by pruning at different pruning rates. Different pruning rates are arranged for different layers according to the different effects of pruning on different layers in the large model, which effectively improves the accuracy of the pruned model. Considering the impact of pruning on the middle layers of the model, different sparse rates are assigned to different layers of the model using the pruning impact rate. This can be combined with unstructured pruning methods to reduce the performance loss caused by pruning to the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Schematic diagram of the process of a large model hierarchical pruning method based on pruning influence rate in an embodiment;

[0031] Figure 2 Schematic diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0032] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0033] Example 1

[0034] In response to the problems existing in the aforementioned existing technologies, this embodiment provides a large model hierarchical pruning method based on pruning influence rate, so as to perform hierarchical pruning of the model in a pre-trained large model through pruning influence rate, thereby reducing the number of parameters in the model and reducing the accuracy loss of the model.

[0035] See also Figure 1 , this method comprises the following steps:

[0036] Step S1: Obtain batches of calibration data and input them into a target large model consisting of multiple layers. When the first i-1 layers are configured as the first reference sparsity rate α, the calibration data X of the tth batch is obtained. t The output data X after passing through the first i-1 layers t,i-1 .

[0037] Specifically, this step uses part of the training set of the existing C4 dataset as calibration data, which can effectively calculate the weight scores in the model and perform pruning.

[0038] Step S2, based on the output data X t,i-1 , calculate the output X of the i-th layer when it is configured as the second reference sparsity rate β t,i,β .

[0039] Step S3, based on the output data X t,i-1 , calculate the output X when the i-th layer is configured as the third reference sparsity rate γ t,i,γ , where β≠γ. Steps S2 and S3 can be swapped in order or performed in parallel.

[0040] Step S4: Based on the second reference sparsity rate β and the output X under the third reference sparsity rate γ t,i,β and X t,i,γ , calculate the pruning impact rate of the i-th layer.

[0041] Step S5: Based on the pruning impact rate, the actual sparsity rate of each layer is adjusted hierarchically to achieve hierarchical pruning.

[0042] A specific process is as follows:

[0043] Step 1: Use the pruning impact rate to measure the specific impact of pruning on the layers in the model. To calculate the pruning impact rate of the i-th layer, first set the sparsity rate of the first i-1 layers in the large model to 50%, and set the sparsity rate of the i-th layer to 40%.

[0044] Step 2: The calibration data is fed into the model in batches. The input data X t The output after the first i-1 layers is X t,i-1 Next, X t,1-- The corresponding output X is obtained by removing 40% of the parameters of the i-th layer t,i,40 .

[0045] Step 3, set the sparse rate of layer i to 50%, and then let X t,i-1 Re-pass the i-th layer with 50% of the parameters removed to obtain the new output X t,i,50 .

[0046] Step 4: The pruning impact rate of the i-th layer can be calculated using the following formula:

[0047]

[0048] The pruning impact rate mainly considers the similarity between the output before and after pruning using different pruning rates. This is mainly based on the following considerations: the smaller the impact of pruning on the model, the more similar the output obtained at different pruning rates should be.

[0049] In Step 5, perform layer-by-layer pruning based on the pruning sparsity ratio. Different sparsity ratios are used for different layers within the model without changing the overall sparsity ratio. Consider that increasing the sparsity ratio during pruning will increase the impact on layers with a greater impact, thereby affecting the model's final performance. On the other hand, increasing the sparsity ratio for layers with a less significant impact will not significantly affect the model's final performance.

[0050] Specifically, the layers are sorted according to the calculated pruning sparsity rate for each layer. For those layers ranked in the top 50%, we increase their sparsity rate to 54% during pruning, while for the remaining layers, we use a sparsity rate of 46%. In other words, the top 50% of layers will remove 4% more parameters during pruning, while the remaining layers will retain 4% more parameters during pruning. In addition, the first and last layers in the model are the input and output layers of the model, respectively. Considering that they also play a more important role in the model than other layers, their sparsity rates are not adjusted.

[0051] In summary, compared to existing post-training pruning methods for large models, this method takes into account the impact of pruning on different layers, using the pruning impact ratio to convert the abstract impact ratio into a concrete value. Furthermore, during pruning, the pruning impact ratio is used to assign different pruning rates to different layers. This reduces the impact of pruning on model layers while maintaining the overall sparsity ratio.

[0052] To verify the effectiveness of this method, see Table 1. The experiment uses perplexity as a measure of model performance. The smaller the perplexity, the better the model performance. Experiments on the LLaMA and LLaMA-2 family models show that this application has excellent performance, indicating that layered pruning can effectively improve the performance of the pruned model.

[0053] Table 1 Experimental results of different pruning methods on LLaMA and LLaMA-2 family models

[0054]

[0055] This method has the following characteristics:

[0056] (1) Large model layered pruning based on pruning impact rate: This method arranges different pruning rates for different layers according to the different pruning impacts on different layers in the large model, effectively improving the accuracy of the pruned model.

[0057] (2) Pruning impact rate: The similarity of layer outputs obtained by pruning at different pruning rates is calculated to measure the impact of pruning on the layers in the model.

[0058] (3) Hierarchical pruning method: Considering the impact of pruning on the model's middle layers, the pruning impact rate is used to assign different sparsity rates to different layers of the model. This method can be combined with unstructured pruning methods to reduce the performance loss caused by pruning.

[0059] (4) Layer-wise adjustment of sparsity: The sparsity rate of different layers is a hyperparameter that needs to be adjusted. Among the parameters between 1% and 10%, 4% is preferred as the most appropriate parameter value.

[0060] Example 2

[0061] Language modeling is an important application of large language models. Based on Example 1, this example provides an application of the large model hierarchical pruning method based on pruning influence rate of Example 1, including:

[0062] The input text sequence information is obtained as the input of the target large model, and the completed text information is output according to the text context through language modeling. In the reasoning optimization process of the target large model, the large model hierarchical pruning method based on pruning influence rate of Example 1 is used to perform hierarchical pruning.

[0063] Specifically, take text generation as an example: the user inputs a text sequence into the model, and the model will internally complete the input text based on the input text context, and finally feedback the completed text to the user as output. Many large models can perform language modeling tasks well, such as the LLaMA-2 model launched by Meta in 2023. The LLaMA-2 13B model has 13 billion parameters and has demonstrated excellent performance in language modeling tasks. However, users who want to be able to run the LLaMA-2 13B model need to use four 3090 GPUs. This technology can compress the number of parameters of large models to half of the original, which can effectively reduce the hardware resources and time required when users use large models.

[0064] In a specific example, the input to the large language model is: Luoyang, the region of his birthplace, was recovered by government forces in the winter of 762, and in the spring of 765 Du Fu and his family sailed down the Yangtze, apparently with the intention of making their way there. They traveled

[0065] The completed output of the large language model is: Luoyang, the region of his birthplace, was recovered by government forces in the winter of 762, and in the spring of 765 Du Fu and his family sailed down the Yangtze, apparently with the intention of making their way there. They traveled as far as Huangpi, near modern Wuhan, before turning back. They then spent around two years in Chengdu. In 767, Du Fuwas granted a post as Prefect of Chang'an.He moved there in the spring of768.He was soon dismissed from the post,and spent the next five years inSichuan.

[0066] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A large model hierarchical pruning method based on pruning impact rate, characterized in that: The steps include: Obtain batches of calibration data and input them into a target large model consisting of multiple layers. When the first i-1 layers are configured as the first reference sparsity rate α, the calibration data X of the tth batch is obtained. t The output data X after passing through the first i-1 layers t,i-1 ; Based on the output data X t,i-1 , calculate the output X of the i-th layer when it is configured as the second reference sparsity rate β t,i,β ; Based on the output data X t,i-1 , calculate the output X when the i-th layer is configured as the third reference sparsity rate γ t,i,γ , where β≠γ; Based on the output ∑ under the second reference sparsity rate β under all batches of calibration data and the third reference sparsity rate γ under testing t X t,i,β and ∑ t X t,i,γ , calculate the pruning impact rate of the i-th layer; Based on the pruning impact rate, the actual sparsity rate of each layer is adjusted in layers to achieve layered pruning.

2. A large model hierarchical pruning method based on pruning influence rate according to claim 1, characterized in that: The first reference thinning rate α=45%-55%, the second reference thinning rate β=35%-45%, and the third reference thinning rate γ=45%-55%.

3. A large model hierarchical pruning method based on pruning influence rate according to claim 2, characterized in that: The first reference thinning rate α=50%, the second reference thinning rate β=40%, and the third reference thinning rate γ=50%.

4. A large model hierarchical pruning method based on pruning influence rate according to claim 1, characterized in that: The pruning impact rate of the i-th layer is calculated using the following formula: Among them, ||||2 represents the modulo operation, T represents transpose, and the upper horizontal line represents the average value.

5. The large model hierarchical pruning method based on pruning influence rate according to claim 1 is characterized in that: The process of implementing layered pruning based on the pruning impact rate includes the following steps: Sort by the pruning impact rate of each layer; A top-ranked layer is configured to have a first actual sparsity rate, and a bottom-ranked layer is configured to have a second actual sparsity rate, to implement layered pruning, wherein the first actual sparsity rate is greater than the second actual sparsity rate.

6. A large model hierarchical pruning method based on pruning influence rate according to claim 5, characterized in that: The first actual thinning rate is 50%-60%, and the second actual thinning rate is 40%-50%.

7. A large model hierarchical pruning method based on pruning impact rate according to claim 6, characterized in that: The first actual thinning rate is 54%, and the second actual thinning rate is 46%.

8. The large model hierarchical pruning method based on pruning influence rate according to claim 1 is characterized in that: When the actual sparsity rate of each layer is adjusted in layers, the target layers for adjustment include all layers except the input layer and the final output of the target large model.

9. The large model hierarchical pruning method based on pruning influence rate according to claim 1, characterized in that: The target macromodel is LLaMA or LLaMA-2.

10. An application of a large model hierarchical pruning method based on pruning impact rate, characterized in that: include: The input text sequence information is obtained as the input of the target large model, and the completed text information is output according to the text context through language modeling. In the reasoning optimization of the target large model, the large model hierarchical pruning method based on the pruning influence rate as described in any one of claims 1 to 9 is used for hierarchical pruning.

Images