Large model hierarchical pruning method based on pruning influence rate and application

Abstract

The invention relates to a large model layered pruning method based on pruning influence rate and application, the method measures the influence of pruning on the middle layer of the model by calculating the similarity of layer output obtained by pruning at different pruning rates, and arranges different pruning rates for different layers according to different pruning influences on different layers in the large model, so as to improve the pruning effect of the large model. And the accuracy of the pruning model is effectively improved. Besides, the influence of pruning on the middle layer of the model is considered, different sparse rates are allocated to different layers of the model by utilizing the pruning influence rate, the method can be combined with an unstructured pruning method, and the performance loss of the model caused by pruning is reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a large model hierarchical pruning method based on pruning influence rate and an application thereof. Background Art

[0002] Today's large models are mainly based on the Transformer structure. The Transformer structure is a sequence-to-sequence architecture based on the self-attention mechanism, which processes sequences through self-attention layers and feed-forward layers.

[0003] However, one of the problems is the large scale of large models and their billions of parameters. However, a large number of parameters also means that these models require huge hardware resources to support them, and they take a lot of time to use, which also brings great difficulties to the actual deployment of these models. In order to solve this problem, many model compression methods have been proposed by researchers, including model pruning, model quantization, and knowledge distillation. Model pruning refers to removing redundant parts of the model through different pruning granularities and pruning standards to obtain different sub-models, which can reduce the number of model parameters while maintaining the performance of the model. Model quantization refers to converting the parameters or intermediate variables of the model from floating point numbers to low-precision integer representations, which can reduce the storage and computing requirements of the model. Knowledge distillation transfers the knowledge of large and complex teacher models to smaller and simpler student models. While maintaining a small model size, it can inherit the generalization ability of the complex teacher model and have better model performance.

[0004] The current pruning method simply assumes that each layer in the large model plays the same role in the model, and the impact of pruning on each layer is also the same. Therefore, the same pruning rate is used for each layer in the large model, resulting in a large performance loss due to pruning.

[0005] Therefore, how to formulate appropriate pruning strategies based on the different effects of pruning on different layers in a large language model and improve the accuracy of the model is an urgent problem to be solved. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a large model hierarchical pruning method and application based on pruning influence rate, so as to solve or partially solve the problem of large hardware resources and time consumption when using large models in downstream applications of large language models such as language modeling.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] One aspect of the present invention provides a large model hierarchical pruning method based on pruning influence rate, comprising the following steps:

[0009] The calibration data in batches are obtained and input into a target large model including multiple layers. When the first i-1 layers are configured as the first reference sparsity rate α, the calibration data X of the tth batch is obtained. t The output data X after passing through the first i-1 layers t,i-1 ;

[0010] Based on the output data X t,i-1 , calculate the output X of the i-th layer when it is configured as the second reference sparsity rate β t,i,β ;

[0011] Based on the output data X t,i-1 , calculate the output X when the i-th layer is configured as the third reference sparsity rate γ t,i,γ , where β≠γ;

[0012] Based on the second reference sparse rate β and the output ∑ under the test third reference sparse rate γ t X t,i,β and∑ t X t,i,γ , calculate the pruning impact rate of the i-th layer;

[0013] Based on the pruning impact rate, the actual sparse rate of each layer is adjusted in layers to achieve layered pruning.

[0014] As a preferred technical solution, the first reference thinning rate α=45%-55%, the second reference thinning rate β=35%-45%, and the third reference thinning rate γ=45%-55%.

[0015] As a preferred technical solution, the first reference thinning rate α=50%, the second reference thinning rate β=40%, and the third reference thinning rate γ=50%.

[0016] As a preferred technical solution, the pruning impact rate of the i-th layer is calculated using the following formula:

[0017]

[0018] Among them, ||||2 represents the modulo operation, T represents transpose, and the upper horizontal line represents the average value.

[0019] As a preferred technical solution, the process of implementing hierarchical pruning based on the pruning impact rate includes the following steps:

[0020] Sort by the pruning impact rate of each layer;

[0021] The top-ranked layers are configured to have a first actual sparse rate, and the bottom-ranked layers are configured to have a second actual sparse rate, to implement layered pruning, wherein the first actual sparse rate is greater than the second actual sparse rate.

[0022] As a preferred technical solution, the first actual thinning rate is 50%-60%, and the second actual thinning rate is 40%-50%.

[0023] As a preferred technical solution, the first actual thinning rate is 54%, and the first actual thinning rate is 46%.

[0024] As a preferred technical solution, when the actual sparsity rate of each layer is adjusted in layers, the target layers for adjustment include all layers except the input layer and the final output of the target large model.

[0025] As a preferred technical solution, the target macromodel is LLaMA or LLaMA-2.

[0026] Another aspect of the present invention provides an application of a large model hierarchical pruning method based on pruning influence rate, comprising:

[0027] The input text sequence information is obtained as the input of the target large model, and the completed text information is output according to the text context through language modeling. During the reasoning optimization process of the target large model, the aforementioned large model hierarchical pruning method based on pruning influence rate is used for hierarchical pruning.

[0028] Compared with the prior art, the present invention has at least the following beneficial effects:

[0029] Improve pruning accuracy and reduce performance loss caused by pruning on large models: The present invention measures the impact of pruning on the middle layers of the model by calculating the similarity of layer outputs obtained by pruning at different pruning rates. Different pruning rates are arranged for different layers according to the different effects of pruning on different layers in the large model, which effectively improves the accuracy of the pruned model. Considering the impact of pruning on the middle layers of the model, different sparse rates are allocated to different layers of the model using the pruning impact rate. This can be combined with unstructured pruning methods to reduce performance loss caused by pruning to the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flowchart of a large model hierarchical pruning method based on pruning influence rate in an embodiment;

[0031] Figure 2 Schematic diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0033] Example 1

[0034] In response to the problems existing in the aforementioned prior art, the present embodiment provides a large model hierarchical pruning method based on pruning influence rate, so as to perform hierarchical pruning of the model in a pre-trained large model through pruning influence rate, thereby reducing the amount of parameters in the model and reducing the accuracy loss of the model.

[0035] See also Figure 1 , the method comprises the following steps:

[0036] Step S1, obtain batches of calibration data and input them into a target large model including multiple layers. When the first i-1 layers are configured as the first reference sparsity rate α, the calibration data X of the tth batch is obtained. t The output data X after passing through the first i-1 layers t,i-1 .

[0037] Specifically, this step uses part of the training set of the existing C4 data set as calibration data, which can effectively calculate the weight scores in the model and perform pruning.

[0038] Step S2, based on the output data X t,i-1 , calculate the output X of the i-th layer when it is configured as the second reference sparsity rate β t,i,β .

[0039] Step S3, based on the output data X t,i-1 , calculate the output X when the i-th layer is configured as the third reference sparsity rate γ t,i,γ , where β≠γ. Step S2 and step S3 can be switched in order or executed in parallel.

[0040] Step S4, based on the second reference sparsity rate β and the output X under the test third reference sparsity rate γ t,i,β and X t,i,γ , calculate the pruning impact rate of the i-th layer.

[0041] Step S5, based on the pruning influence rate, the actual sparse rate of each layer is adjusted hierarchically to achieve hierarchical pruning.

[0042] A specific process is as follows:

[0043] Step 1: Use the pruning impact rate to measure the specific impact of pruning on the layers in the model. In order to calculate the pruning impact rate of the i-th layer, first set the sparsity rate of the first i-1 layers in the large model to 50%, and set the sparsity rate of the i-th layer to 40%.

[0044] Step 2: The calibration data is fed into the model in batches. The input data X of the tth batch is t The output after the first i-1 layers is X t,i-1 Next, X t,1-- By removing 40% of the parameters of the i-th layer, the corresponding output X is obtained t,i,40 .

[0045] Step 3, set the sparseness rate of the i-th layer to 50%, and then let X t,i-1 Re-pass the i-th layer with 50% of the parameters removed to get the new output X t,i,50 .

[0046] Step 4, the pruning impact rate of the i-th layer can be calculated by the following formula:

[0047]

[0048] The pruning impact rate mainly considers the similarity between the outputs before and after pruning using different pruning rates. This is mainly based on the following considerations: the smaller the impact of pruning on the model, the more similar the outputs obtained at different pruning rates should be.

[0049] Step 5: Perform layered pruning of the model based on the pruning sparsity rate. Different sparsity rates are used for different layers in the model without changing the overall sparsity rate. Consider that for layers with a greater impact from pruning, increasing the sparsity rate during pruning will increase the impact on them, thereby affecting the final performance of the model. For layers with less impact, even if the sparsity rate is increased to a certain extent, it will not have a significant impact on the final performance of the model.

[0050] Specifically, the layers are sorted according to the calculated pruning sparsity rate of each layer. For those layers ranked in the top 50%, we will increase their sparsity rate to 54% during pruning, and for the remaining layers, use a sparsity rate of 46% for pruning. That is, the first 50% of the layers will remove 4% more parameters during pruning, while the remaining layers will retain 4% more parameters during pruning. In addition, the first and last layers in the model are the input and output layers of the model, respectively. Considering that they also play a more important role in the model than other layers, their sparsity rates are not adjusted.

[0051] In summary, compared with the existing post-training pruning methods for large models, this method takes into account the impact of pruning on different layers and uses the pruning impact rate to convert the abstract impact rate into a specific value. At the same time, different pruning rates are assigned to different layers through the pruning impact rate during pruning. Under the premise that the overall sparsity rate does not change, the impact of pruning on the model's middle layers is reduced.

[0052] To verify the effectiveness of this method, see Table 1. The experiment uses perplexity as a measure of model performance. The smaller the perplexity, the better the model performance. Experiments on the LLaMA and LLaMA-2 family models show that this application has excellent performance, indicating that layered pruning can effectively improve the performance of the pruned model.

[0053] Table 1 Experimental results of different pruning methods on LLaMA and LLaMA-2 family models

[0054]

[0055] This method has the following characteristics:

[0056] (1) Large model layered pruning based on pruning impact rate: This method arranges different pruning rates for different layers according to the different impacts of pruning on different layers in the large model, effectively improving the accuracy of the pruned model.

[0057] (2) Pruning impact rate: The similarity of layer outputs obtained after pruning at different pruning rates is calculated to measure the impact of pruning on the layers in the model.

[0058] (3) Hierarchical pruning method: Considering the impact of pruning on the middle layers of the model, different sparsity rates are assigned to different layers of the model using the pruning impact rate. This method can be combined with the unstructured pruning method to reduce the performance loss caused by pruning to the model.

[0059] (4) Layer-wise adjustment of sparsity ratio: The sparsity ratio of different layers that increases or decreases is a hyperparameter that needs to be adjusted. Among the parameters ranging from 1% to 10%, 4% is preferred as the most appropriate parameter value.

[0060] Example 2

[0061] The language modeling task is an important application of a large language model. Based on Example 1, this embodiment provides an application of the large model hierarchical pruning method based on pruning influence rate of Example 1, including:

[0062] The input text sequence information is obtained as the input of the target large model, and the completed text information is output according to the text context through language modeling. During the reasoning optimization process of the target large model, the large model hierarchical pruning method based on pruning influence rate of Example 1 is used to perform hierarchical pruning.

[0063] Specifically, take text generation as an example: the user inputs a text sequence into the model, and the model will complete the input text internally according to the input text context, and finally feed back the completed text to the user as output. Many large models can perform language modeling tasks well, such as the LLaMA-2 model launched by Meta in 2023. The LLaMA-2 13B model has 13 billion parameters and has shown excellent performance in language modeling tasks. However, users need to use 4 3090 GPUs to run the LLaMA-2 13B model. This technology can compress the number of parameters of large models to half of the original, which can effectively reduce the hardware resources and time required for users to use large models.

[0064] In a specific example, the input of the large language model is: Luoyang, the region of his birthplace, was recovered by government forces in the winter of 762, and in the spring of 765 Du Fu and his family sailed down the Yangtze, apparently with the intention of making their way there. They traveled

[0065] The output of the large language model after completion is: Luoyang, the region of his birthplace, was recovered by government forces in the winter of 762, and in the spring of 765 Du Fu and his family sailed down the Yangtze, apparently with the intention of making their way there. They traveled as far as Huangpi, near modern Wuhan, before turning back. They then spent around two years in Chengdu. In 767, Du Fuwas granted a post as Prefect of Chang'an.He moved there in the spring of768.He was soon dismissed from the post,and spent the next five years inSichuan.

[0066] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A large model hierarchical pruning method based on pruning influence rate, characterized in that: The steps include: The calibration data in batches are obtained and input into a target large model including multiple layers. When the first i-1 layers are configured as the first reference sparsity rate α, the calibration data X of the tth batch is obtained. t The output data X after passing through the first i-1 layers t,i-1 ; Based on the output data X t,i-1 , calculate the output X of the i-th layer when it is configured as the second reference sparsity rate β t,i,β ; Based on the output data X t,i-1 , calculate the output X when the i-th layer is configured as the third reference sparsity rate γ t,i,γ , where β≠γ; Based on the output ∑ under the second reference sparsity rate β under all batches of calibration data and the third reference sparsity rate γ under testing t X t,i,β and∑ t X t,i,γ , calculate the pruning impact rate of the i-th layer; Based on the pruning impact rate, the actual sparse rate of each layer is adjusted in layers to achieve layered pruning.

2. A large model hierarchical pruning method based on pruning influence rate according to claim 1, characterized in that: The first reference thinning rate α=45%-55%, the second reference thinning rate β=35%-45%, and the third reference thinning rate γ=45%-55%.

3. A large model hierarchical pruning method based on pruning influence rate according to claim 2, characterized in that: The first reference thinning rate α=50%, the second reference thinning rate β=40%, and the third reference thinning rate γ=50%.

4. The large model hierarchical pruning method based on pruning influence rate according to claim 1 is characterized in that: The pruning impact rate of the i-th layer is calculated using the following formula: Among them, ||||2 represents the modulo operation, T represents transpose, and the upper horizontal line represents the average value.

5. The large model hierarchical pruning method based on pruning influence rate according to claim 1 is characterized in that: The process of implementing hierarchical pruning based on the pruning impact rate comprises the following steps: Sort by the pruning impact rate of each layer; The top-ranked layers are configured to have a first actual sparse rate, and the bottom-ranked layers are configured to have a second actual sparse rate, to implement layered pruning, wherein the first actual sparse rate is greater than the second actual sparse rate.

6. A large model hierarchical pruning method based on pruning influence rate according to claim 5, characterized in that: The first actual thinning rate is 50%-60%, and the second actual thinning rate is 40%-50%.

7. A large model hierarchical pruning method based on pruning influence rate according to claim 6, characterized in that: The first actual thinning rate is 54%, and the first actual thinning rate is 46%.

8. The large model hierarchical pruning method based on pruning influence rate according to claim 1 is characterized in that: When the actual sparsity rate of each layer is adjusted in layers, the target layers for adjustment include all layers except the input layer and the final output of the target large model.

9. The large model hierarchical pruning method based on pruning influence rate according to claim 1, characterized in that: The target macromodel is LLaMA or LLaMA-2.

10. An application of a large model hierarchical pruning method based on pruning influence rate, characterized in that: include: The input text sequence information is obtained as the input of the target large model, and the completed text information is output according to the text context through language modeling, wherein in the reasoning optimization of the target large model, the large model hierarchical pruning method based on the pruning influence rate as described in any one of claims 1 to 9 is used for hierarchical pruning.

Images