A large language model heterogeneous perception global pruning method for maintaining reasoning ability

Abstract

The application discloses a large language model heterogeneous perception global pruning method for maintaining reasoning ability, and relates to the technical field of artificial intelligence model compression. In view of the problems that the existing pruning method is difficult to adapt to the heterogeneous structure of a large language model, is easy to damage the reasoning ability, and has low pruning precision, the application calculates the importance of model parameters through a calibration data set and aggregates them according to the structure group to obtain group-level importance. The application divides a Transformer layer into two types of functional blocks, namely an attention block and an MLP block, calculates a sensitivity coefficient through an identity bypass experiment, normalizes and eliminates the heterogeneous bias of the structure group, combines the sensitivity coefficient to obtain a final score, determines a pruning scheme through global greedy selection, removes the structure group to be pruned, and fine-tunes to obtain a compressed model. The application realizes heterogeneous perception global pruning, accurately protects key reasoning structures, effectively balances the model compression rate and reasoning ability, is suitable for mainstream large language models, and significantly improves the compression efficiency and reasoning stability.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a global pruning method for heterogeneous perception of large language models that preserves reasoning ability. Background Technology

[0002] Large language models possess superior reasoning capabilities in fields such as natural language processing, intelligent question answering, and code generation, and have become the core cognitive foundation of intelligent agent systems, widely applied in scenarios such as adaptive network optimization, semantic communication, and autonomous decision-making. However, the scale of model parameters has expanded dramatically; for example, GPT-3 has 175 billion parameters, and the Llama-3-8B model has 8 billion parameters. This massive size places enormous storage and computing power pressure on mobile terminals, edge computing platforms, and real-time application deployments.

[0003] Structured pruning, as a mainstream model compression method, achieves model lightweighting by removing structured parameter sets such as MLP neurons and attention heads. However, existing structured pruning methods have two major technical drawbacks: First, the heterogeneity of model structures leads to biases in importance assessment. Taking Llama-3-8B as an example, the MLP module accounts for 70.2% of the total parameters, while the attention module accounts for only 16.7%. At the fine-grained structure group level, the MLP neuron group has approximately 12,000 parameters, while the attention group has approximately 5.2 million parameters, a difference of up to 400 times. Existing methods directly aggregate parameter-level importance scores, which easily overestimates the importance of large structure groups, leading to pruning biases that preserve attention structures and mistakenly delete key MLP neurons. Although LLM-Pruner avoids this problem by fixing the pruning ratio within a module, it cannot achieve global resource optimization across modules.

[0004] Secondly, there is a lack of a mechanism to protect reasoning ability. Systematic validation showed that after model fine-tuning, the perplexity representing fluency recovered by an average of 65.8%, while the zero-sample accuracy representing reasoning ability only recovered to 15.7%, a difference of 50.1%. In 16 experiments (4 pruning methods × 4 7B / 8B models), this asymmetry occurred in 95.8% of the experiments. This indicates that reasoning ability is difficult to repair once damaged. Existing pruning methods do not distinguish between critical and non-critical reasoning structures, resulting in a significant decrease in reasoning performance after pruning, making it difficult to balance compression effects and reasoning ability.

[0005] In summary, existing structured pruning methods cannot adapt to the heterogeneous characteristics of large language models, are difficult to achieve globally optimal pruning, and cannot effectively protect reasoning capabilities. There is an urgent need for a pruning scheme that takes into account heterogeneous perception, global optimization, and reasoning protection. Summary of the Invention

[0006] In view of the above-mentioned deficiencies of the prior art, the present invention at least solves the following technical problems: 1. Large language models exhibit severe structural heterogeneity, with the number of parameters in the MLP neuron group and attention group differing by as much as 400 times. Existing pruning methods directly sum and aggregate parameter-level importance, systematically overestimating the importance of large structures and underestimating the importance of small structures. This leads to pruning biases that retain redundant attention structures and mistakenly delete key MLP structures. Existing methods such as LLM-Pruner circumvent this problem by applying a fixed pruning ratio independently to each module type, but they cannot achieve global resource optimization across modules. ShortGPT operates at the whole-layer granularity and cannot achieve optimal allocation of parameter budgets across different structures, resulting in a lack of global optimality in pruning.

[0007] Existing pruning methods lack a mechanism to protect reasoning ability. Systematic validation showed that after model fine-tuning, the perplexity representing fluency recovered by an average of 65.8%, while the zero-sample accuracy representing reasoning ability only recovered to 15.7%, a difference of 50.1 percentage points. This asymmetric recovery occurred in 95.8% of the experiments, indicating that damaged reasoning ability is difficult to repair. Current techniques cannot distinguish between critical and redundant reasoning structures, resulting in a significant degradation in reasoning performance after pruning.

[0008] To achieve the above objectives, this invention discloses a heterogeneous perceptual global pruning method for large language models that preserves reasoning ability. This method includes the following steps: S1: Obtain the calibration dataset, perform forward and backward propagation on the target large language model based on the Transformer architecture, calculate the importance of each parameter, and aggregate them by structure group to obtain the group-level importance of each structure group. S2: Divide each Transformer layer of the target large language model into two functional blocks: an attention block and an MLP block; perform an identity bypass experiment on each functional block, calculate the loss difference of the model before and after bypass, and obtain the sensitivity coefficient corresponding to each functional block; S3: Perform cost-aware normalization on the group-level importance to eliminate heterogeneity bias caused by differences in parameter quantities between different structural groups; combine the sensitivity coefficient modulation normalization result to obtain the final score corresponding to each structural group. S4: Determine the pruning scheme through a global greedy selection process; S5: Remove the structure group to be pruned according to the pruning scheme, fine-tune the pruned model, and obtain a compressed model that maintains inference ability.

[0009] Furthermore, the importance of parameters in S1 is calculated using a first-order Taylor expansion, with the following formula:

[0010] in, The importance of parameter w, For the loss function with respect to parameters gradient, For parameter values.

[0011] Furthermore, the calibration dataset mentioned in S1 is the validation set of the target large language model, with a sample size of 128 to 512.

[0012] Furthermore, the formula for calculating the sensitivity coefficient mentioned in S2 is as follows:

[0013] in, For the first The sensitivity coefficient of each functional block. The model loss after the function block is bypassed. The original loss of the target large language model is given.

[0014] Furthermore, the higher the sensitivity coefficient mentioned in S2, the more likely the corresponding functional block is to be a key inference block; the lower the sensitivity coefficient, the more likely the corresponding functional block is to be a redundant block.

[0015] Furthermore, the normalization formula described in S3 is:

[0016] in, Given the importance of normalization, For group-level importance, For structural group parameters; Furthermore, the modulation formula described in S3 is:

[0017] in, For the final score, This is the sensitivity coefficient.

[0018] Furthermore, in S4, the global greedy selection takes the final score as the value, the number of parameters of the structure group as the weight, and the parameter budget corresponding to the preset pruning rate as constraints, and selects the structure group to be pruned in ascending order of the final score.

[0019] Furthermore, the fine-tuning described in S5 uses LoRA fine-tuning.

[0020] Furthermore, the LoRA fine-tuning parameters are: rank r = 8, scaling factor α = 16, and dropout probability = 0.05.

[0021] This invention achieves at least the following beneficial technical effects: 1. Cost-aware normalization eliminates the systematic scoring advantage of the attention group over the MLP group, and the score distributions of the two structural groups change from separation to overlap; unnormalized methods such as Magnitude and Wanda exhibit severe degradation with perplexity exceeding 1000, which this invention effectively avoids. The global optimization framework enables cross-module resource allocation, achieving the highest inference accuracy on four 7B / 8B models with a 20% pruning rate. After fine-tuning, the average zero-shot accuracy is improved by 2-4% compared to LLM-Pruner, with more significant advantages at 30% and 50% pruning rates. The entire pruning process, including calibration, sensitivity analysis, sorting, and pruning, takes only about 10 minutes on a single RTX4090 GPU, with manageable computational overhead.

[0022] 2. Block-sensitive modulation directly protects the inference structure. On four 7B / 8B models, the inference capability retention rate before fine-tuning reaches 87.3%, which is significantly better than the 75.2% retention rate of ShortGPT, and avoids perplexity anomalies. The global parameter budget is redistributed from redundant structures to critical structures, effectively solving the problem of asymmetric inference capability recovery.

[0023] At the same time, this invention has strong practicality and industrial application prospects: This invention is the first structured pruning framework that actively protects reasoning ability during the pruning stage. Based on the systematic discovery design that fluency can be recovered but reasoning ability cannot, it only requires two steps, normalization and sensitivity analysis, on the basis of first-order Taylor pruning. The method is simple and efficient. It has been fully validated on five mainstream models from two families with parameter sizes ranging from 7B to 14B. It is suitable for large language models with mainstream Transformer architectures such as Llama and Qwen. It does not depend on specific hardware or training paradigms and can be seamlessly integrated with the LoRA fine-tuning process. The calibration data only requires 128 to 512 samples and is not sensitive to the data source, making it highly practical for engineering. This invention enables large language models to reduce parameters by 20% to 50% while maintaining reasoning capabilities, significantly reducing inference computing power and storage costs. It is suitable for low-computing-power deployment scenarios such as intelligent assistants, autonomous driving, and edge computing. This technology is orthogonal to compression methods such as quantization and knowledge distillation, and can be used in combination to obtain higher compression gains. It has a wide range of applications and good prospects for industrial application. Attached Figure Description

[0024] Figure 1 A flowchart illustrating the heterogeneous perception global pruning method for large language models that preserves reasoning ability in this invention; Figure 2 This is a schematic diagram of the block sensitivity bypass analysis of the present invention; Figure 3This is a schematic diagram comparing the structural heterogeneity and cost-aware normalization of the present invention. Detailed Implementation

[0025] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0026] In the accompanying drawings, components with the same structure are indicated by the same numerical designation, and components with similar structures or functions are indicated by similar numerical designations. The dimensions and thicknesses of each component shown in the drawings are arbitrary, and the present invention does not limit the dimensions and thicknesses of each component. To make the illustrations clearer, the thickness of some components has been appropriately exaggerated in the drawings.

[0027] This invention proposes a heterogeneous perceptual global pruning method for large language models that preserves reasoning ability. The overall process is as follows: Figure 1 As shown, Figure 1 The complete process of this invention framework is demonstrated, comprising five steps: Step 1 calculates Taylor importance scores on calibration data; Step 2 performs sensitivity analysis on each functional block using identity bypass; Step 3 generates a global score through cost-aware normalization and sensitivity modulation; Step 4 makes a global greedy pruning decision; and Step 5 performs LoRA fine-tuning recovery. Steps 1 and 2 are preprocessing steps, steps 3 and 4 are the core pruning mechanisms, and step 5 is post-processing.

[0028] The Llama-3-8B-Instruct model is used as a specific example for detailed explanation. This model contains 32 Transformer layers and a total of 64 functional blocks. The method of this invention is executed with a 20% pruning rate. The specific steps are as follows: Step 1: Calibration and Importance Calculation. Sample 128 to 512 text sequences from the C4 validation set to form a calibration dataset. Input the calibration data into the target large language model, perform forward and backward propagation, and calculate the importance of each parameter using the first-order Taylor expansion formula:

[0029] in, The importance of parameter w, For the loss function with respect to parameters gradient, For parameter values.

[0030] Based on the model dependency graph, the parameters are divided into structural groups: each MLP neuron group contains approximately 12,000 parameters related to gate projection, up projection, and down projection; each attention group contains approximately 5.2 million parameters related to Q, K, V, and O projection. The importance of all parameters within each structural group is summed to obtain the group-level aggregated importance; this group-level aggregated importance is then normalized to obtain the unit parameter importance (i.e., ROI score).

[0031] Step 2: Block Sensitivity Analysis. Each Transformer layer is divided into two functional blocks: an attention block and an MLP block, as follows: Figure 2 As shown, Figure 2 This demonstrates the complete process of block sensitivity analysis: Figure 2 (a) shows the Transformer layer structure of the original model; Figure 2 (b) Demonstrate the identity bypass experiment and calculate the loss change after replacing a specific function block with an identity mapping; Figure 2 (c) The sensitivity results and pruning strategies for each block are presented. Structures in high-sensitivity blocks are given increased importance and prioritized for protection, while structures in low-sensitivity blocks are suppressed and actively pruned. An identity bypass experiment is performed on each functional block in sequence: the block is replaced with an identity mapping, the input is directly passed as the output, the calculation of the block is skipped, and the model loss is calculated on the calibration data.

[0032] The formula for calculating the functional sensitivity coefficient of this block is:

[0033] in, For the first The sensitivity coefficient of each functional block. The model loss after the function block is bypassed. This represents the original loss of the target large language model.

[0034] Based on this, the following are determined: functional blocks that show a significant increase in loss and high sensitivity after being bypassed are key inference blocks; functional blocks that show a small change in loss and low sensitivity are redundant blocks. The analysis process is as follows: Figure 2 As shown.

[0035] Step 3: Cost-Aware Global Ranking. Cost-aware normalization is applied to the importance of group-level aggregations, using the following formula:

[0036] in, Given the importance of normalization, For group-level importance, These are the parameters of the structural group.

[0037] Figure 3This demonstrates the difference in the number of parameters between the attention module and the MLP module in the Transformer layer structure, as well as the effect of cost-aware normalization: Figure 3 (a) shows the Transformer layer structure, with annotations showing a 400-fold difference between the attention group (approximately 5.2 million parameters per group) and the MLP group (approximately 12,000 parameters per group). Figure 3 (b) Comparison of aggregate scores before normalization: the attention group scored 5200 and the MLP group scored 12, showing a difference of 433 times. Figure 3 (c) is the normalized ROI score, which is 0.001 for both groups, to achieve a fair comparison.

[0038] By combining the sensitivity coefficient modulation, the final score is obtained:

[0039] in, For the final score, This is the sensitivity coefficient. A logarithmic transformation and an offset of 2 are used to ensure numerical stability. All structure groups are uniformly sorted according to the final score.

[0040] Using the Llama-3-8B model as an example, we can illustrate the necessity of normalization: Assuming there are two structural groups, each with the same average importance of 0.001 for each parameter, the aggregate importance of the attention group is 5.2 million multiplied by 0.001, which equals 5200, while the aggregate importance of the MLP group is 12,000 multiplied by 0.001, which equals 12. Without normalization, the attention group's score is 433 times that of the MLP group, even though the actual contribution of each parameter is the same for both. After cost-aware normalization, both have an ROI score of 0.001, allowing for a fair comparison on the same scale and avoiding systematic bias caused by differences in structural scale.

[0041] Step 4: Global Greedy Pruning Decision. The pruning task is transformed into a global greedy selection problem: using the final score as value, the number of structural parameters as weight, and the parameter budget corresponding to the preset pruning rate as constraints, a greedy approach is used to determine the pruning scheme.

[0042] Taking a 20% pruning rate as an example, the parameter budget is 80% of the total number of parameters. All structure groups are sorted in ascending order of their final scores. Starting with the structure group with the lowest score, they are marked as pruning objects. The number of parameters pruned is accumulated sequentially until the cumulative number of pruned parameters reaches the target. At the same time, dependency constraints are enforced to ensure the integrity of the model structure.

[0043] Step 5: LoRA Fine-tuning Recovery. The pruned model is fine-tuned using LoRA with the following parameters: rank r = 8, scaling factor α = 16, dropout probability = 0.05; the AdamW optimizer is used with a learning rate of 10. -4The batch size is 64, the cosine learning rate is used for scheduling, and the warm-up ratio is 3%. The instruction fine-tuning model is trained for 1 epoch using 52,000 samples from the Alpaca dataset. The fine-tuning mainly restores fluency-related capabilities, and the inference capability has been protected by block-sensitive modulation during the pruning stage.

[0044] The method of this invention can be applied to different pruning rate settings. At a 20% pruning rate, the Llama-3-8B-Instruct model achieves a zero-shot inference accuracy of 65.40%. At higher pruning rates such as 30% and 50%, the advantages of cost-aware normalization and block-sensitive modulation become more significant due to tighter parameter budgets. This method is also applicable to larger-scale models, including the Qwen2.5-14B-Instruct model.

[0045] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0046] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A heterogeneous perceptual global pruning method for large language models that preserves reasoning ability, characterized in that, The method includes the following steps: S1: Obtain the calibration dataset, perform forward and backward propagation on the target large language model based on the Transformer architecture, calculate the importance of each parameter, and aggregate them by structure group to obtain the group-level importance of each structure group. S2: Divide each Transformer layer of the target large language model into two functional blocks: an attention block and an MLP block; perform an identity bypass experiment on each functional block, calculate the loss difference of the model before and after bypass, and obtain the sensitivity coefficient corresponding to each functional block; S3: Normalize the group-level importance to eliminate heterogeneity bias caused by differences in parameter quantities between different structural groups; combine the sensitivity coefficient modulation normalization result to obtain the final score corresponding to each structural group. S4: Determine the pruning scheme through a global greedy selection process; S5: Remove the structure group to be pruned according to the pruning scheme, fine-tune the pruned model, and obtain a compressed model that maintains inference ability.

2. In the heterogeneous perceptual global pruning method for large language models that preserves reasoning ability as described in claim 1, the parameter importance in S1 is calculated using a first-order Taylor expansion formula, and the calculation formula is as follows: in, The importance of parameter w, For the loss function with respect to parameters gradient, For parameter values.

3. In the heterogeneous perceptual global pruning method for maintaining reasoning ability of a large language model according to claim 1, the calibration dataset in S1 is the validation set of the target large language model, and the number of sampled samples is 128 to 512.

4. In the heterogeneous perceptual global pruning method for large language models that preserves reasoning ability according to claim 1, the formula for calculating the sensitivity coefficient in S2 is as follows: in, For the first The sensitivity coefficient of each functional block. The model loss after the function block is bypassed. The original loss of the target large language model is given.

5. In the heterogeneous perception global pruning method for maintaining reasoning ability in a large language model according to claim 4, the higher the sensitivity coefficient in S2, the more critical the corresponding functional block is determined to be a reasoning key block; the lower the sensitivity coefficient, the more redundant the corresponding functional block is determined to be a redundant block.

6. In the heterogeneous perceptual global pruning method for large language models that preserves reasoning ability according to claim 1, the normalization formula in S3 is: in, Given the importance of normalization, For group-level importance, These are the parameters of the structural group.

7. The heterogeneous perceptual global pruning method for large language models that preserves reasoning ability according to claim 1, wherein the modulation formula in S3 is: in, For the final score, This is the sensitivity coefficient.

8. In the heterogeneous perception global pruning method for maintaining reasoning ability of large language models according to claim 1, the global greedy selection in S4 is constrained by the final score as the value, the number of parameters of the structure group as the weight, and the parameter budget corresponding to the preset pruning rate, and selects the structure group to be pruned in ascending order of the final score.

9. In the heterogeneous perceptual global pruning method for maintaining reasoning ability in large language models according to claim 1, the fine-tuning in S5 adopts LoRA fine-tuning.

10. The heterogeneous perceptual global pruning method for large language models that preserves reasoning ability according to claim 9, wherein the LoRA fine-tuning parameters are: rank r = 8, scaling factor α = 16, and dropout probability = 0.05.

Images