Method and related equipment for efficient compensation after pruning of large language model layer

Abstract

The invention relates to the technical field of large language model lightweight, in particular to a method and related equipment for efficient compensation after pruning of a large language model layer. The method comprises the following steps: determining a to-be-pruned layer and a pruning surface through layer importance evaluation indexes; recording a calibration set input lower pruning surface and a to-be-pruned layer output hidden state pair; calculating the variance of each channel between hidden states and screening key channels; training a key channel linear change relation; and performing linear compensation on the output of the pruning surface after pruning. According to the method, the performance of the pruned model can be remarkably improved, only 0.5% of channels are modified, extra parameters are hardly introduced, training is efficient, the performance of cross-layer pruning and the performance of continuous-layer pruning on multiple models are improved by 3.69% and 28.02% respectively, and the method is suitable for multiple open-source large language models.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of large language model lightening, and particularly relates to a method for efficient compensation after layer pruning of a large language model and related equipment. BACKGROUND

[0002] As an important technical achievement in the field of artificial intelligence, large language models have shown excellent performance in natural language processing tasks. However, with the continuous expansion of model size, its storage requirements, computing resource consumption, and inference speed have gradually become bottlenecks in practical applications. In order to solve these problems, large language model lightening technology has emerged, aiming to reduce model size and storage requirements while minimizing the significant decline in model performance, thereby improving the deployment efficiency and inference speed of the model. Pruning technology is one of the core methods for realizing model lightening, which can significantly improve the compression ratio by deleting redundant parameters. According to whether the model structure is changed after pruning, pruning technology can be divided into structured pruning and unstructured pruning. Among them, structured pruning does not change the model main framework, which is convenient for direct deployment on various terminal devices, so it has attracted widespread attention. Layer pruning, as a form of structured pruning, achieves a very high compression ratio by directly deleting intermediate layers of the model, and since large language models are usually composed of multiple layers of identical intermediate layers, layer pruning has become an easy-to-deploy pruning method.

[0003] However, existing layer pruning techniques mainly focus on how to determine the most suitable redundant layer for pruning, and pay less attention to the compensation of the performance decline of the pruned model. Directly concatenating layers that were originally not adjacent after layer pruning can cause a jump in the semantic dependence of the hidden state between layers, resulting in a significant decline in model performance. Although some existing technologies attempt to compensate for the pruned model through parameter efficient fine-tuning or layer replacement, these methods still have many shortcomings. For example, the retraining time is too long, too many additional compensation parameters are introduced, and there is a lack of theoretical basis, which limits its practical application effect. In addition, existing compensation techniques often require complex network structures to learn the change law of hidden states, resulting in high training costs and difficulty in rapid deployment. Therefore, developing a reasonable, efficient, and effective compensation method for large language model layer pruning with few introduced parameters has become a technical problem to be solved.

[0004] The present application proposes an accurate and efficient compensation method to address the above problems, which realizes the effective recovery of the performance of the pruned model by linearly compensating the key channels with significant variance changes in the hidden state. This method not only significantly reduces the number of parameters required for compensation, but also significantly shortens the training time, providing a new solution for the lightening application of large language models. SUMMARY

[0005] The application aims at the problem of great performance decline of a large language model after layer pruning in the prior art, and provides a method for post-pruning compensation of a large language model and related equipment. The method solves the technical problem of performance decline of a model after pruning through a key channel identification mechanism based on variance changes of hidden states and a linear compensation strategy.

[0006] The application provides a method for efficient post-pruning compensation of a large language model and related equipment, characterized by comprising the following steps:

[0007] S1: obtaining an unimportant layer to be pruned and a previous layer of the layer to be pruned as a pruning surface by using a preset layer importance evaluation index of a large language model. The layer importance evaluation index includes a cosine similarity, an Euclidean distance or a perplexity between input and output hidden states.

[0008] S2: inputting sentences in a calibration set into a selected large language model one by one, recording hidden states output by the pruning surface and the last layer to be pruned under each input sentence, and forming output state pairs. Each output state pair corresponds to a calibration set data. In particular, the output state pair is represented in the form of a tuple in Python, and the format is (pruning surface layer number, pruning surface output hidden state matrix, last layer to be pruned output hidden state matrix).

[0009] S3: calculating the variance of each channel between the output hidden states according to the channel as the smallest evaluation unit for each recorded output state pair.

[0010] S4: selecting the top 0.5% channels with the largest variance changes in each output state pair as channels to be compensated, and calculating the frequency of these channels in all selected channels with the most significant changes. Channels with a frequency higher than a threshold value are reserved as key channels. The threshold value is dynamically determined according to different models, and is usually selected as 0.6.

[0011] S5: using the key channel values in the output state pairs as a training set to train a linear compensation of the key channels from the pruning surface to the output hidden states of the last layer to be pruned.

[0012] S6: deleting the layer to be pruned to obtain a pruned large language model, and adding the change relationship trained in S5 to the output of the pruning surface in the pruned model.

[0013] Further, in the step S1, if the layer to be pruned is a continuous layer, the previous layer of the first layer in the continuous layers is recorded as the pruning surface; if the layer to be pruned is a non-continuous layer, the pruning surface of each part of the non-continuous layer is recorded respectively. The layer importance evaluation process determines redundant layers by calculating the above-mentioned indexes layer by layer and sorting.

[0014] Furthermore, in step S3, for models with fixed channels in the hidden state of the intermediate layer output, such as the 4096 channels of the Llama2-7B model, the variance value is calculated channel by channel.

[0015] Furthermore, in step S4, the number of key channels accounts for 0.5% of the total number of output channels of the model, and statistical analysis is used to ensure that these channels have high reproducibility in multiple output state pairs.

[0016] Furthermore, in step S5, during the training process, only the hidden states corresponding to the key channels in each output state pair are selected as training data, and linear transformation is used to fit the variation patterns of these channels. In particular, since a high Pearson correlation coefficient is observed between the hidden states on these key channels, linear transformation is chosen instead of a complex network for modeling.

[0017] Furthermore, in step S6, during each inference process, when the pruned surface is output, the hidden state of the key channel is multiplied by a linear change coefficient, thereby compensating for the model performance. Through this method, the LLaMA2-7B, LLaMA2-13B, and LLaMA3-8B models achieve performance improvements of 3.69% and 28.02% respectively on the PPL evaluation metric for cross-layer pruning and continuous-layer pruning.

[0018] Furthermore, the present invention also provides an efficient compensation device for pruning large language model layers, the device comprising the following modules:

[0019] The layer importance assessment module is used to determine redundant layers based on preset layer importance evaluation indicators and record the layers to be pruned and their corresponding pruning surfaces. In particular, this module supports processing continuous and discontinuous layers separately to ensure the flexibility of the pruning structure.

[0020] The variance processing module calculates the channel-by-channel variance between the output hidden state of the pruned surface and the output hidden state of the last layer to be pruned, and retains the top 0.5% of channels with the largest variance changes. Furthermore, this module uses a channel-by-channel comparison and sorting algorithm to filter out channels with significant variance changes.

[0021] The threshold processing module calculates the frequency of the channels retained by the variance processing module among all retained channels, and retains only channels whose frequency exceeds a threshold. Specifically, this module uses statistical analysis to ensure that the selected key channels have high reproducibility and representativeness.

[0022] The compensation training module is used to specifically train the key channels retained in the thresholding module. Furthermore, this module employs a linear regression algorithm to learn the variation pattern of the key channels from the pruning plane to the last layer to be pruned, and generates corresponding linear variation parameters.

[0023] The compensation module performs linear compensation on the critical channels of the pruned surface output in the pruned model. Specifically, this module automatically adjusts the critical channels of the pruned surface output during each inference process to compensate for the performance degradation caused by pruning.

[0024] According to another aspect of the present invention, a computer device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the above-described method. Furthermore, the computer device supports a distributed computing architecture, enabling efficient processing of training tasks on large-scale datasets.

[0025] According to another aspect of the present invention, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the above-described method. In particular, the storage medium supports fast data read and write operations, making it suitable for real-time inference scenarios.

[0026] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the above-described method. Furthermore, the computer program product supports cross-platform operation and is suitable for various hardware environments.

[0027] This invention proposes a precise and efficient compensation method for layer pruning in large language models. It obtains the layers to be pruned based on layer importance evaluation metrics and obtains the hidden states of the output of these layers using a small calibration set as input. By observing that the variance variation of a very small number of channels in the hidden states far exceeds that of other channels, and noting the high Pearson correlation coefficients between these channels, linear compensation is used to compensate for the key channels of these hidden states in each inference, thus compensating for the performance of the pruned model. Furthermore, this invention only requires modifying 0.5% of the total output channels of the model, introducing almost no additional parameters. Linear compensation training for these 0.5% channels can be completed within minutes on a consumer-grade 4090 graphics card. For language modeling tasks, it achieves performance improvements of 3.69% and 28.02% for cross-layer pruning and continuous layer pruning, respectively, across multiple models. In particular, this invention is applicable to various open-source large language models with the same intermediate layer structure, such as Llama2-7B.

[0028] Compared with existing technologies, this invention has the following advantages: First, it requires fewer new parameters, modifying only 0.5% of the total output channels of the model, introducing almost no additional parameters; second, it is highly efficient in training, with linear compensation training for these 0.5% channels completed within minutes on a consumer-grade 4090 graphics card; third, it has a wide compensation scope, effectively compensating for cross-layer and continuous-layer pruning as a plug-and-play compensation component; fourth, it significantly improves performance on language modeling and zero-shot tasks, achieving performance improvements of 3.69% and 28.02% respectively on the PPL evaluation metric for cross-layer and continuous-layer pruning in LLaMA2-7B, LLaMA2-13B, and LLaMA3-8B models for language modeling tasks, and improving model performance on multiple datasets for zero-shot tasks; finally, it has a wide applicability, suitable for various open-source large language models with the same intermediate layer structure.

[0029] In summary, this invention provides a reasonable, efficient, and effective compensation method for pruning large language model layers with a small number of introduced parameters, solving the problems of excessive retraining time, excessive introduction of compensation parameters, and lack of theoretical basis in the prior art. Attached Figure Description

[0030] Figure 1 A flowchart illustrating a precise and efficient compensation method for layer-oriented pruning provided in this embodiment of the invention;

[0031] Figure 2 This is a schematic diagram of the structure of the large language model compensation device provided in an embodiment of the present invention;

[0032] Figure 3 This is a functional diagram of the layer importance determination module in an embodiment of the present invention;

[0033] Figure 4 This is a functional diagram of the variance processing module in an embodiment of the present invention;

[0034] Figure 5 This is a functional diagram of the threshold processing module in an embodiment of the present invention;

[0035] Figure 6 This is a functional diagram of the compensation training module and the compensation module in an embodiment of the present invention. Detailed Implementation

[0036] This invention provides a precise and efficient compensation method and related equipment for layer pruning in large language models. The technical solution, by combining layer importance evaluation indicators, hidden state variance analysis, and a linear compensation strategy, achieves a significant improvement in the performance of the pruned model. The following will be discussed in conjunction with the appendix... Figure 1 To be continued Figure 6 The specific implementation process of the present invention will be described in detail with reference to specific embodiments.

[0037] like Figure 1 As shown, step S1 is executed first, which determines the layers to be pruned and their corresponding pruning surfaces based on preset layer importance evaluation metrics. In practice, the layer importance judgment module is used to accomplish this task. For example, when processing the Llama2-7B model, the layer importance evaluation metrics can be quantitative indicators such as cosine similarity between the input and output hidden states, Euclidean distance, or perplexity. These metrics are calculated and sorted layer by layer for the hidden states of each layer, selecting the layers with the highest redundancy as the layers to be pruned. At the same time, the layer preceding the layer to be pruned is recorded as the pruning surface. If the layer to be pruned is a continuous layer, only the layer preceding the first layer of these continuous layer indices needs to be recorded as the pruning surface; if the layer to be pruned is a discontinuous layer, the pruning surface of each part of the discontinuous layer is recorded separately. This process ensures the flexibility and applicability of the pruning structure.

[0038] Next, step S2 is executed, in which the statements in the calibration set are input into the selected large language model one by one, and the pruning surface and the hidden state of the last layer output to be pruned are recorded for each input statement. Figure 1 The diagram illustrates the process: calibration set data is sequentially input into the model, while the variance processing module records the hidden state matrices of the pruned surfaces and the last layer to be pruned. Each output state pair corresponds to one calibration set data point and is represented as a tuple in the format (pruning surface layer number, pruned surface output hidden state matrix, last layer to be pruned output hidden state matrix). For example, in the Llama2-7B model, the intermediate layer output hidden states contain 4096 channels, therefore each row of the hidden state matrix contains 4096 channel values.

[0039] Then proceed to step S3, where for each output state pair of a record, the variance of each channel between output hidden states is calculated, using the channel as the smallest evaluation unit. Figure 4 The diagram illustrates the functionality of the variance processing module, where the channel variance calculation unit calculates the variance of the hidden state matrix channel by channel. Specifically, the channel variance calculation unit extracts the values ​​of the corresponding channels from the hidden state matrix of the pruning surface and the last layer to be pruned, and calculates the variance between the two. This process can identify the variation patterns of the hidden state across different channels, laying the foundation for subsequent screening of key channels.

[0040] In step S4, the top 0.5% of channels with the largest variance changes for each output state are selected as channels to be compensated, and the frequency of these channels in all selected channels with the largest changes is calculated. Channels with a frequency higher than the threshold are retained and called key channels. Figure 5The diagram illustrates the function of the threshold processing module, where the frequency statistics unit is responsible for calculating the reproducibility of each channel across multiple output state pairs. For example, in the Llama2-7B model, the threshold is typically chosen to be 0.6, meaning that only channels that appear more than 60% of the time across multiple output state pairs are retained as critical channels. Furthermore, the number of critical channels accounts for 0.5% of the total number of output channels in the model; this proportion ensures high reproducibility and representativeness of these channels through statistical analysis.

[0041] Next, step S5 is executed, using the recorded output state pairs as the training set to train and obtain the relationship between the key channels and the output of the last layer to be pruned from the pruning surface. Figure 6 The diagram illustrates the functionality of the compensation training module, where the linear compensation unit is responsible for learning the variation patterns of key channels. Specifically, during training, only the hidden states corresponding to the key channels in each output state pair are selected as training data, and a linear regression algorithm is used to fit the variation trends of these channels. Because a high Pearson correlation coefficient was observed between the hidden states on these key channels, linear variation was chosen instead of a complex network for modeling. This linear variation accurately describes the variation patterns of the key channels from the pruning surface to the last layer to be pruned.

[0042] Finally, step S6 is executed to delete the layers to be pruned, obtain the pruned large language model, and add the change relationship obtained in training in S5 to the output of the pruned surface in the pruned model. Figure 6 The diagram shows the functional schematic of the compensation module, where the linear compensation unit automatically adjusts the critical channels of the pruned surface output during each inference process. Specifically, when the pruned model performs inference, the hidden states of the pruned surface output are multiplied by a linear change coefficient on the critical channels, thereby compensating for model performance. Using the above method, in the language modeling task, the performance of LLaMA2-7B, LLaMA2-13B, and LLaMA3-8B models on the PPL evaluation index was improved by 3.69% and 28.02%, respectively, for cross-layer pruning and continuous-layer pruning.

[0043] Furthermore, the present invention also provides a device for layer pruning of a large language model and immediate compensation after layer pruning, the structure of which is as follows: Figure 2As shown, the device includes a layer importance judgment module, a variance processing module, a threshold processing module, a compensation training module, and a compensation module. The layer importance judgment module determines redundant layers based on preset layer importance evaluation indicators and records the layers to be pruned and their corresponding pruning surfaces. The variance processing module calculates the channel-by-channel variance between the output hidden state of the pruning surface and the output hidden state of the last layer to be pruned, and retains the top 0.5% of channels with the largest variance changes. The threshold processing module calculates the frequency of the channels retained by the variance processing module among all retained channels and retains only channels with a frequency higher than the threshold. The compensation training module performs targeted training on the key channels retained by the threshold processing module to generate linear variation relationships. The compensation module performs linear compensation on the key channels output by the pruning surface in the pruned model, thereby compensating for the performance degradation caused by pruning.

[0044] The technical solution of this invention is applicable to various open-source large language models with the same intermediate layer structure, such as Llama2-7B. Experimental results show that this method only modifies 0.5% of the total output channels of the model, introduces almost no additional parameters, and the linear compensation training of these 0.5% channels can be completed within minutes on a consumer-grade 4090 graphics card. In language modeling tasks, it can improve the performance of LLaMA2-7B, LLaMA2-13B, and LLaMA3-8B models by 3.69% and 28.02% respectively on the PPL evaluation index for cross-layer pruning and continuous layer pruning.

[0045] Furthermore, this method addresses the problems of excessive retraining time, excessive introduction of compensation parameters, and lack of theoretical basis in existing technologies, and provides a reasonable, efficient, and effective compensation method for pruning large language model layers with a small number of introduced parameters.

Claims

1. A method and related equipment for efficient compensation after pruning in large language model layers, characterized in that: The method for efficient compensation after pruning in large language model layers includes the following steps: S1: Using a pre-defined layer importance evaluation index, the unimportant layers to be pruned and the layer before the layer to be pruned are obtained from the large language model, which are called the pruning surface; S2: Input the statements in the calibration set into the selected large language model one by one, record the hidden state of the pruning surface and the last layer output to be pruned under each input statement, and form them into output state pairs; S3: For each output state pair of a record, calculate the variance of each channel between output hidden states, using the channel as the smallest evaluation unit. S4: Select the top 0.5% channels with the most significant variance changes for each output state, and calculate the frequency of these channels among all the selected channels with the most significant changes. Keep the channels with a frequency higher than the threshold, which are called key channels. S5: Use the recorded output state values ​​of the key channels as the training set to train and obtain the linear change relationship of the key channels from the pruning surface to the output of the last layer to be pruned. S6: Delete the layers to be pruned, obtain the pruned large language model, and add the change relationship obtained in S5 to the output of the pruned surface in the pruned model.

2. The method for efficient compensation after pruning in a large language model layer according to claim 1, characterized in that: The layer importance evaluation metrics include cosine similarity between the input and output hidden states, Euclidean distance, or a perplexity metric.

3. The method for efficient compensation after pruning in a large language model layer according to claim 2, characterized in that: If the layer to be pruned is a continuous layer, then record the layer before the first layer of the index of these continuous layers as the pruning surface; if the layer to be pruned is a non-continuous layer, then record the pruning surface of each part of the non-continuous layer.

4. The method for efficient compensation after pruning in a large language model layer according to claim 1, characterized in that: The output state pairs are represented in tuple form, with the format (pruning surface layer number, pruning surface output hidden state matrix, and the output hidden state matrix of the last layer to be pruned).

5. The method for efficient compensation after pruning in a large language model layer according to claim 1, characterized in that: The number of key channels accounts for 0.5% of the total output channels of the model, and statistical analysis ensures that these channels have high reproducibility in multiple output state pairs.

6. The method for efficient compensation after pruning in a large language model layer according to claim 1, characterized in that: The changing relationship is learned through a linear regression algorithm to study the changing pattern of the key channel from the pruning surface to the last layer to be pruned, and corresponding linear change parameters are generated.

7. A high-efficiency compensation device for pruning large language model layers, characterized in that... Includes the following modules: The layer importance judgment module is used to determine redundant layers based on preset layer importance evaluation indicators and record the layers to be pruned and their corresponding pruning surfaces. The variance processing module is used to calculate the channel-by-channel variance between the output hidden state of the pruning surface and the output hidden state of the last layer to be pruned, and retain the first 0.5% of channels with the largest variance changes. The threshold processing module is used to calculate the frequency of the channels that the variance processing module retains among all retained channels, and only retains the channels whose frequency of occurrence is higher than the threshold. The compensation training module is used to perform targeted training on the key channels retained in the threshold processing module. The compensation module is used to perform linear compensation on the key channels of the pruned surface output in the model after pruning.

8. The efficient compensation device for pruning large language model layers according to claim 7, characterized in that... The compensation training module uses a linear regression algorithm to learn the changing patterns of key channels and generates corresponding linear change parameters. The compensation module automatically adjusts the key channels output by the pruning surface during each inference process.

Images