Lightweight pruning method and system based on channel dominant modeling

Abstract

The invention belongs to the technical field of model compression, and discloses a lightweight pruning method and system based on channel dominant modeling, and the method comprises the steps: executing one-time forward reasoning under the condition of not modifying the model weight, and extracting each linear layer input activation matrix; calculating a basic scoring matrix according to the mechanism; channel dominance indexes are calculated, and dominant channels are sorted and identified; selecting channels of a front proportion to form a dominant set, applying a scaling factor to a dominant channel column, and generating a correction scoring matrix; independently sorting according to output channel dimensions, and cutting low-score weights to a target sparse rate; and storing the sparse model and directly performing reasoning evaluation. According to the method, on the premise of not increasing extra training and calculation overhead, the pruning quality and the model robustness under the high sparse rate are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of model compression, and particularly relates to a lightweight pruning method and system based on channel dominance modeling. BACKGROUND

[0002] With the wide application of large language models (LLM) in natural language processing, the parameter size of the models is continuously expanding. The parameter size of models represented by GPT and LLaMA series has reached tens of billions or even hundreds of billions. The large parameter size significantly increases the inference delay and storage cost, which seriously restricts the deployment of the models in edge computing and resource-constrained environments. Therefore, model compression technology has become a research hotspot. Pruning, as a classic compression method, can effectively reduce the model size and improve the inference efficiency by setting some weights to zero to achieve sparsification. In recent years, one-shot pruning methods such as SparseGPT and Wanda have shown good results on large language models. SparseGPT uses a second-order approximation compensation mechanism to maintain high accuracy, but the computational overhead is large. Wanda constructs a score index through the product of weights and activations, and can complete pruning in one forward pass, which is highly efficient. However, in the high sparsity (e.g., ≥60%) scenario, existing methods still have problems such as structural imbalance and performance degradation. How to improve the pruning stability and expressiveness in the high sparsity scenario while ensuring the efficiency of one forward pass has become a research focus in this field.

[0003] In US20180114114A1 (“Systems and methods for pruning neural networks”), a method for constructing a pruning criterion based on gradient or first-order derivative information is disclosed. Specifically, the method receives the first-order gradient information of the cost function of the trained model, then calculates a “pruning criterion” for each parameter or neuron, and removes the neuron or connection with the smallest pruning criterion, thereby gradually compressing the network.

[0004] In addition, in the field of deep learning literature, the Wanda (Pruning by Weights and Activations) method proposed by Sun et al. is also a lightweight pruning method that does not require retraining. The pruning criterion is based on the product of the absolute value of the weight and the input activation norm, so that the weights are sorted and set to zero in the output channel dimension.

[0005] These two technologies are the closest prior art to the present application: the former represents the pruning idea based on gradient or sensitivity, and the latter represents the lightweight activation + weight combination scoring pruning idea.

[0006] The prior art has the following technical problems: (1) Weak identification ability of dominant channels In the method of US20180114114A1, the pruning criteria mainly rely on the gradient or first derivative change estimation of the parameters, which focuses on the sensitivity of a single parameter or the local cost variation, and lacks the "dominance" analysis of the input channels as a whole on the output. This leads to difficulty in effectively identifying the "channel dominance" connection paths that have a great impact on the output in actual high-dimensional networks, especially in Transformer-type models, and thus it is easy to prune the channels that play a key role in model performance but have small gradients.

[0007] (2) Lack of soft constraint suppression mechanism, and significant decline in risk accuracy Although the Wanda method has the advantage of not needing to be retrained for pruning, it directly uses the same criteria for all connections without distinguishing between dominant channels and ordinary channels, so it may "over-prune" important channels at high sparsity rates, resulting in a significant decline in model performance. It lacks a mechanism to explicitly protect dominant channels (such as soft suppression, scaling control, etc.), making it difficult to maintain good performance in extreme pruning scenarios.

[0008] Therefore, the prior art has weaknesses in both "channel dominance identification" and "differential treatment of important channels", and the solution of the present application is to define a dominance index and use a soft suppression mechanism to distinguish and treat it, thereby achieving more robust high-sparsity pruning without retraining. SUMMARY

[0009] In view of the problems existing in the prior art, the present application provides a lightweight pruning method based on channel dominance modeling and a system.

[0010] The present application is implemented as follows: a lightweight pruning method based on channel dominance modeling, characterized in that the lightweight pruning method based on channel dominance modeling comprises the following steps: S1: forward sampling and data acquisition, performing a forward inference without modifying the model weights, and extracting the input activation matrix of each linear layer; S2: basic score matrix construction, calculating the basic score matrix according to the Wanda mechanism; S3: calculation of channel dominance index, calculating the channel dominance index, sorting and identifying the dominant channels; S4: soft suppression of dominant channels, selecting the top k proportion of channels to form a dominant set, applying a scaling factor to the dominant channel column, and generating a revised score matrix; S5: pruning execution and sparsification, independently sorting according to the output channel dimension, and pruning low-score weights to the target sparsity rate; S6: Model export and verification, save the sparse model and directly evaluate the inference without retraining.

[0011] Further, S1, perform a forward inference on the target model, input a small amount of calibration data, and record the input activation matrix X and the corresponding weight matrix W of each linear layer to be pruned in the model during the inference process. The linear layers here include weight matrices such as q_proj, k_proj, v_proj, o_proj, up_proj, and down_proj in the Transformer model, while the embedding layer and the output layer are not pruned due to their special structure.

[0012] Further, S2, for each linear layer, calculate its pruning score matrix S according to the scoring method of Wanda:

[0013] where i represents the output channel and j represents the input channel, is the two-norm of the input channel activation.

[0014] Further, S3, define the channel dominance index :

[0015] where is the two-norm of the weight column corresponding to the jth input channel.

[0016] Further, S4, according to value size, select the top k proportion of channels to form the dominant set D, usually 0.5%-2% of the total number of input channels, and recommend 1%. In the pruning score stage, apply scaling operations to the score columns corresponding to these dominant channels:

[0017] where is the suppression factor, usually 0.3.

[0018] Further, S5, after completing the suppression, independently sort each output channel (each row), delete the connections with smaller score values, and reach the set sparse rate. The pruning method is unstructured sparsification, i.e., setting individual weight elements to zero.

[0019] Further, S6, after pruning is completed, export the sparse weight matrix, which can be directly used for inference without any retraining or weight reconstruction. In the verification stage, evaluate the pruning effect through language modeling perplexity (PPL) and zero-shot task accuracy.

[0020] Another objective of this invention is to provide a lightweight pruning system based on channel dominance modeling, the system specifically comprising: The activation acquisition module is used to extract the input activation matrix of each linear layer; The pruning score calculation module is used to calculate the basic score matrix based on the Wanda mechanism; The dominance modeling module is used to calculate the channel dominance index ranking and identify the dominant channel; The soft suppression execution module is used to apply a scaling factor to the dominant channel column and generate a corrected score matrix; The channel-level pruning module is used to independently sort by the output channel dimension and prune low-scoring weights to the target sparsity. The model export and validation module is used to save sparse models and perform inference evaluation directly.

[0021] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows: This invention performs unstructured pruning of the linear layers of a large language model using only a single forward inference, without relying on backpropagation and weight reconstruction. Based on Wanda's "weight × activation norm" scoring framework, a "channel dominance index" is introduced and softly suppressed to weaken the systematic advantage of channels dominated by activated outliers in global pruning ranking, thereby improving structural balance and performance stability under high sparsity. It has the following advantages: By introducing channel-dominant indicators This invention can accurately identify and suppress channels dominated by activation outliers, fundamentally reducing the scoring bias caused by activation anomalies in the traditional Wanda method. At a high sparsity of 60%–70%, the model perplexity (PPL) is reduced by an average of approximately 40%–60% compared to the original Wanda method, demonstrating stronger high sparsity stability.

[0022] The dominant inhibition mechanism applies a soft scaling factor to high-biased channel columns. This avoids some channels being repeatedly retained in multiple outputs, resulting in a more even distribution of connections after pruning. This structural balance allows the model to distribute information across different channels, improving the overall generalization ability and robustness of the model.

[0023] The entire pruning process requires only one forward inference step, without relying on gradient, backpropagation, or weighted regression calculations, nor involving computationally intensive steps such as second-order approximation or matrix inversion. Therefore, it maintains the lightweight and efficient characteristics of the Wanda system. Under the same hardware and calibration scale, index calculations are typically completed in seconds, offering significant time and memory advantages compared to reconstruction methods that require second-order approximation / matrix inversion, thus effectively reducing additional computational burden and resource consumption.

[0024] The present application realizes the double improvement of pruning performance and speed without increasing the computational complexity: on the LLaMA-2-13B model, the perplexity at 70% sparsity is reduced from 44.95 to 18.71, the average accuracy of the zero-shot task is improved by 20%, and the pruning index calculation time is only 3.84 seconds, which is excellent in performance-efficiency balance.

[0025] The method does not need to change the original model structure and can be directly applied to various Transformer large language models and their sub-modules. At the same time, it can be combined with quantization, distillation, and structured sparsity to form a multi-level compression framework, which is suitable for cloud inference acceleration and edge deployment.

[0026] The obtained pruning model can be inferred without retraining, the structural sparsity can be flexibly adjusted, and it is compatible with existing GPU / TPU sparse inference engines (such as CUTLASS, torchao, TensorRT-Sparsity). While ensuring model accuracy, it significantly reduces memory usage and inference delay, making it suitable for practical engineering deployment and power-limited scenarios.

[0027] (1) The expected income and business value of the technical solution of the present application after transformation are: Significant cost reduction and efficiency improvement: While maintaining the pruning completed by one forward pass without the need for regression / tuning, the perplexity and zero-shot accuracy can still be improved at high sparsity (50%-70%), and the performance is better than Wanda on LLaMA / LLaMA-2 series in multiple indicators. Therefore, without changing the inference service architecture, it can directly support existing business traffic with smaller computing power, reducing GPU / cloud inference costs. For example, at 70% sparsity of LLaMA-2-13B, DAWanda significantly outperforms Wanda and approaches the accuracy range of SparseGPT, with engineering landing flexibility and replaceability.

[0028] Faster compression process and shorter delivery cycle: The stage of calculating pruning scores is similar to Wanda, much faster than SparseGPT which needs weight reconstruction (the speed difference between 13B levels on the same hardware can reach an order of magnitude), shortening the model compression online period and improving the iteration efficiency of enterprise multi-model and multi-version.

[0029] Wider deployment map and energy saving and emission reduction: better performance at high sparsity, facilitating the migration of medium and large LLMs to edge / local environments (including resource-limited GPUs / low-power acceleration cards), reducing inference latency and bandwidth dependence, and having energy consumption and carbon cost advantages.

[0030] Application scenario coverage: enterprise knowledge Q&A, customer service / quality inspection, AIGC assisted creation, end-side intelligent agent, etc. Scenarios can carry more concurrency with less memory and lower latency under the same service level, forming a quantifiable TCO advantage.

[0031] (2) The technical solution of the application fills the gap in the industry at home and abroad: The existing one-time post-training pruning (such as Wanda) is easily affected by the activation outlier dominant bias at high sparsity, and has the defect of "a few high activation channels are systematically retained, and the structure is single". The scheme that needs to be reconstructed (such as SparseGPT) has high computational and memory overhead, and high engineering threshold. DAWanda first explicitly models "channel dominance" in the Wanda framework and performs soft suppression, systematically alleviates the ordering bias, improves the structure balance and robustness, and forms a "high sparsity + one-time + stable performance" combination that can be implemented. This position has been blank for a long time.

[0032] Give an operable dominance metric and suppression strategy ( , Suppression factor), and verify the effectiveness of the design principle and default hyperparameters through ablation / sensitivity experiments, providing a reusable engineering paradigm for subsequent related methods.

[0033] (3) The technical solution of the application solves the technical problems that people have been eager to solve but have always failed to succeed: Problem A: Ordering distortion of one-time pruning at high sparsity. Outlier activation leads to a "domination" of the scoring matrix by a few channels, and high sparsity coexists with mispruning / repeated retention. DAWanda breaks this domination effect through "channel dominance scoring + soft suppression", making pruning more balanced and expression more stable. Experiments show that the indicators are significantly better than Wanda at 60%-70% sparsity.

[0034] Problem B: Approximate reconstruction method performance without weight reconstruction / tuning. It is generally believed in practice that "one-time methods without regression inevitably drop significantly at extreme compression". DAWanda's language modeling and zero-shot average accuracy at LLaMA-2-13B, 70% sparsity, approximate SparseGPT, verify that this proposition is not insurmountable, significantly reducing the engineering threshold of high-performance compression.

[0035] (4) The technical solution of the application overcomes technical bias:

[0036] Bias 1: High-activated channels must be prioritized without reservation. The industry often takes "the greater the activation, the more important" as an approximate axiom, leading to ranking being swayed by outliers. The present invention proves that "limited weakening of dominance" can improve overall structural diversity and task performance in the form of "dominance identification + moderate suppression (rather than hard pruning)".

[0037] Bias 2: In order to maintain usable performance at an extremely high sparsity of 70%, second-order reconstruction or retraining must be performed. DAWanda exhibits near-reconstruction stability at a multi-model scale while maintaining one-time and lightweight, breaking the established mindset of "high performance ≡ high complexity".

[0038] Bias 3: One-time methods cannot form a controllable engineering closed loop. The present invention proves that one-time pruning can also have an explainable, adjustable, and verifiable engineering system through sensitivity analysis of the proportion of Top-k (selecting the top k input channels (columns of the weight matrix) in each linear layer according to the "channel dominance" score from high to low to form a set D) and the suppression factor . BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is a flowchart of a lightweight pruning method based on channel dominance modeling provided by an embodiment of the present invention; Figure 2 is a lightweight pruning system module diagram based on channel dominance modeling provided by an embodiment of the present invention; Figure 3 is a bias diagram of the existing Wanda method when there are outliers in activation provided by an embodiment of the present invention; Figure 4 is a schematic diagram of the dominant channel suppression mechanism provided by an embodiment of the present invention; Figure 5 is a parameter sensitivity analysis diagram of the dominant suppression mechanism provided by an embodiment of the present invention: (a) fixing , changing ; (b) fixing , changing Top-k proportion. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical scheme and advantages of the present invention clearer and more apparent, the present invention will be further described in detail below in combination with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the present invention.

[0041] As shown in Figure 1 , an embodiment of the present invention provides a lightweight pruning method based on channel dominance modeling, which specifically includes: S1: Forward sampling and data acquisition. Perform forward inference once without modifying the model weights to extract the input activation matrix X of each linear layer. S2: Construction of the basic scoring matrix, calculating the basic scoring matrix based on the Wanda mechanism; S3: Calculate channel dominance indicators, sort and identify dominant channels; S4: Dominant channel soft suppression. Select the top k proportion of channels to form a dominant set, apply a scaling factor to the dominant channel column, and generate a modified score matrix. S5: Pruning and sparsification are performed, and the output channel dimensions are sorted independently. Low-scoring weights are pruned to the target sparsity. S6: Model export and validation, save the sparse model and perform inference evaluation directly without retraining.

[0042] The theoretical basis of channel dominance modeling. Traditional pruning methods typically rely on gradients or L1 norms to measure the importance of weights. This invention, based on the concept of channel dominance modeling, extracts the linear layer input activation matrix X through a single forward sampling and, combined with the Wanda (Wight and Activation Normalized Distribution Analysis) mechanism, establishes a mapping relationship between weights and activation responses. This process does not involve parameter updates; it only utilizes the dynamic response information of the original model to reflect the energy contribution of channels in feature propagation, laying the foundation for accurate identification of dominant channels.

[0043] The basic scoring matrix and channel dominance index are calculated. After obtaining the input activation matrix X, the weighted response amplitudes of each channel's output are standardized to form the basic scoring matrix. This scoring matrix reflects the average influence of each channel in feature propagation. Subsequently, the channel dominance index D is calculated, defined as the ratio of the channel response variance to the mean or the weighted energy percentage, used to quantitatively characterize the degree of channel dominance in the global feature flow. A higher channel dominance index indicates a stronger sensitivity of that channel to the model output.

[0044] Dominant Channel Soft Suppression Mechanism. Traditional pruning can easily lead to model degradation by directly removing high-weight channels. This invention adopts a "soft suppression" strategy, which selects the top k proportions of high-dominance channels to form a dominant set and applies a scaling factor to the corresponding score matrix columns. This mechanism suppresses the influence of channels without completely eliminating them. It rebalances the contributions between channels, allowing pruning decisions to focus more on non-dominant but potentially redundant channels, thus maintaining the stability of global information transmission in the network.

[0045] Sparsity and pruning strategies are implemented. Based on the modified scoring matrix, the output channel weights of each linear layer are independently sorted and then adjusted according to the target sparsity. Layer-wise pruning of low-scored weights. This process adopts output channel dimension as unit for sparsification, ensuring the integrity of feature mapping and the proportionality between layers after pruning. Since the score matrix has already considered the channel activation distribution and dominance indicators in the early stage, the pruning results of the present application achieve a dynamic balance between compression rate and accuracy loss.

[0046] Model export and availability without retraining. After pruning is completed, the model structure and parameters are directly exported to form a sparse model. Since the channel dominance repair of the present application has achieved feature redistribution in the forward inference stage, no additional retraining or fine-tuning is required to perform inference verification on the original data set. This feature significantly reduces the deployment cost of pruning, while maintaining the dual advantages of inference speed improvement and reduction in computing resource occupancy, embodying the core values of lightweight and portability of the present application.

[0047] S1, perform a forward inference on the target model, input a small amount of calibration data, for example, randomly extract 128 pieces of text from C4 or WikiText corpus, each piece of text has 2048 tokens. During the inference process, record the input activation matrix X and the corresponding weight matrix W of each linear layer to be pruned in the model. The linear layers here include the weight matrices q_proj, k_proj, v_proj, o_proj, up_proj, down_proj, etc. in the Transformer model; and the embedding layer and the output layer are not pruned due to their special structure.

[0048] S2, for each linear layer, calculate its pruning score matrix S according to the scoring method of Wanda:

[0049] where i represents the output channel, j represents the input channel, is the two-norm of the input channel activation. This matrix is used to measure the importance of different weight elements, and the larger the value represents the more critical the connection.

[0050] S3, in order to identify the "dominant channel" that repeatedly obtains high scores in multiple output channels, the present application defines the channel dominance indicator :

[0051] where represents the two-norm of the weight column corresponding to the jth input channel. This indicator reflects the overall dominance of the input channel in the full-layer score matrix, and the larger the value represents that the channel is continuously high-scored in multiple outputs, which may be dominated by outlier activation.

[0052] Through statistical analysis, it can be found that these high Channels usually correspond to a few extreme activation columns, whose norms are much higher than the average level, thus being repeatedly reserved in the original Wanda, leading to structural imbalance.

[0053] S4, according to The value size selects the top k proportion of channels to form the dominant set D, usually 0.5%-2% of the total number of input channels, and 1% is recommended. In the pruning score stage, scaling operation is performed on the score columns corresponding to these dominant channels:

[0054] Wherein is an inhibition factor, generally taken as 0.3. This step does not delete weights, but weakens their advantage in row ordering by reducing the score value, so that ordinary channels that are originally easy to be covered can get a more fair reservation opportunity, thereby improving the balance and diversity of the pruning structure.

[0055] S5, after the inhibition is completed, each output channel is independently sorted, and the connections with smaller score values are deleted until the set sparsity rate, such as 50%, 60%, 70%, is reached. The pruning method is unstructured sparsification, that is, the single weight element is set to zero.

[0056] S6, after the pruning is completed, the sparsified weight matrix is derived, which can be directly used for inference. No retraining or weight reconstruction is required. In the verification stage, the pruning effect is evaluated by language modeling perplexity (PPL) and zero-shot task accuracy. Experimental results show that the method of the present application reduces the perplexity by 58% and improves the zero-shot task accuracy by about 20% when the sparsity rate is 70% on LLaMA and LLaMA-2 models, which is significantly better than the Wanda method.

[0057] As Figure 2 shown, the lightweight pruning system based on channel dominance modeling provided by the embodiment of the present application specifically comprises: An activation collection module for extracting input activation matrices of each linear layer; A pruning score calculation module for calculating a basic score matrix according to the Wanda mechanism; A dominance modeling module for calculating channel dominance index ranking and identifying dominant channels; A soft inhibition execution module for applying a scaling factor to the dominant channel columns to generate a revised score matrix; A channel-level pruning module for independently sorting according to the output channel dimension and pruning low-score weights to a target sparsity rate; A model export and verification module for saving the sparse model and directly performing inference evaluation.

[0058] I. The specific application field or related product of the present application.

[0059] (I) Enterprise knowledge Q&A / RAG private deployment Q&A and summary for internal knowledge base such as legal, operation and manufacturing documents. The present application prunes q / k / v / o and feedforward up / down linear layers without changing the embedding and output head, relies on a small amount of field calibration, and can be landed, reduces memory and concurrent costs, and is convenient to access in mainstream frameworks such as vLLM / Triton / TensorRT-LLM / ONNXRuntime.

[0060] (II) Multi-tenant customer service and quality inspection (SaaS) For different tenants, quickly generate "tenant-specific sparse weight copies" according to their corpus, without fine-tuning. Through the Top-k dominant soft inhibition control structure, the balance is improved, the cross-domain Q&A stability is improved, the single-tenant memory occupation and service cost are reduced, and the gray release and version management are supported.

[0061] (III) Edge and end-side devices (such as RK3588, Jetson, PC-NPU, etc.) Provide local dialogue, offline summary, and patrol records in memory / power-constrained environments. The sparse weights output by the present application can be stacked with low-bit quantization and exported to the weight format supported by the end-side, and if the hardware has a sparse operator, further time delay / throughput gains are obtained.

[0062] (IV) Teaching and research platform Provide "second-level compression" reproducible experimental baseline for courses, competitions and laboratories for research on sparse degree-accuracy trade-off and deployment economy.

[0063] II. Related evidence of the technical effects obtained by the embodiments of the present application.

[0064] (I) Implementation environment and experimental setup The method of the present application is implemented in an NVIDIA L20 GPU (48GB memory) environment, using LLaMA and LLaMA-2 two series models as experimental objects, and testing 7B and 13B parameter scales. The pruning target layers include six types of linear layers in the Transformer module: q_proj, k_proj, v_proj, o_proj, up_proj, and down_proj; the embedding layer and output head are not involved in pruning. The sparse rate is set to 50%, 60%, and 70%, and the pruning method is unstructured sparse, element-wise zeroing, and maintaining the same sparse rate for each layer. The calibration sample is selected from the C4 corpus, containing 128 segments of 2048 tokens of text.

[0065] (II) Experimental results and effect description 1. Language modeling performance Table 1 WikiText perplexity of LLaMA and LLaMA-2 models

[0066] From the experimental results, DAWanda almost achieved the optimal or suboptimal perplexity performance in all model and sparsity settings, especially at high sparsity rates (such as 60% and 70%). For example, on LLaMA-13B, DAWanda reduced the perplexity to 41.18 at a 70% sparsity rate, significantly better than Wanda's 55.21, indicating that it is more stable and retains expression ability than Wanda in high compression ratio scenarios. However, it still has a certain gap compared to SparseGPT's 23.88, indicating that weight updating is necessary in extreme compression scenarios.

[0067] In addition, DAWanda is particularly stable on large models (such as LLaMA2-13B), maintaining a perplexity of 18.71 even at a 70% extremely high sparsity rate, which is significantly lower than Wanda's 44.95, indicating that its pruning strategy effectively preserves the model's language modeling ability while maintaining structural sparsity.

[0068] In summary, DAWanda improves the instability of the Wanda method at high sparsity rates without introducing additional structural reconstruction or parameter reconstruction, and exhibits better pruning performance and robustness in multiple models and sparsity settings.

[0069] 2. Zero-shot task performance Table 2 Zero-shot task accuracy of LLaMA-2 at 50% sparsity

[0070] Table 3 Zero-shot task accuracy of LLaMA-2 at 60% sparsity

[0071] Table 4 Zero-shot task accuracy of LLaMA-2 at 70% sparsity

[0072] In the zero-shot task accuracy test, Tables 2 and 3 show the zero-shot task performance of LLaMA at 50% and 60% pruning rates, respectively. The results show that traditional amplitude pruning significantly reduces accuracy at high sparsity rates, while SparseGPT, Wanda, and other methods maintain high sparsity while still maintaining certain performance. In contrast, DAWanda achieves better results on multiple tasks (such as RTE, HellaSwag, and WinoGrande), with overall average accuracy reaching or exceeding the optimal baseline (Wanda) at different model sizes, especially at 60% sparsity, showing stronger robustness. This shows that DAWanda can effectively alleviate the performance degradation caused by high sparsity and has obvious advantages in complex reasoning tasks, further verifying its migration ability and generalization performance under extreme compression conditions, providing a better pruning choice for actual deployment.

[0073] Table 4 shows that at 70% sparsity, the performance of Wanda drops significantly, while SparseGPT benefits from its dense weight update mechanism to achieve the best accuracy under this extreme condition. In contrast, the average accuracy of DAWanda is slightly lower than that of SparseGPT, but still better than that of Wanda, indicating that at extremely high sparsity, additional weight compensation can bring some accuracy improvement, but also needs to be balanced between efficiency and complexity.

[0074] Table 5 shows the pruning speed of LLaMA-2 at 50% sparsity

[0075] Table 5 shows the pruning time of LLaMA-2 at 50% sparsity. Specifically, we measured the cumulative time of each layer's pruning indicators on an NVIDIA L20 (48GB) GPU (excluding the forward propagation part common to all methods). The results show that the pruning speed of Wanda and DAWanda is much faster than that of SparseGPT: on the 7B model, Wanda and DAWanda only take 0.92s and 1.32s, respectively, while SparseGPT takes 159.21s; on the 13B model, the gap is still obvious, and the time taken by SparseGPT is 78 times that of DAWanda. It is worth noting that although DAWanda has slightly higher overhead than Wanda, it remains on the same order of magnitude and still exhibits better performance in the evaluation of perplexity and accuracy (Tables 1, 2, and 3). This shows that DAWanda further improves pruning results without sacrificing speed, making it more suitable for practical application scenarios that require both efficiency and accuracy.

[0076] (Three) Drawings Wanda's pruning scoring mechanism relies on the product of the activation norm and the corresponding weight for each input channel. It's worth noting that the activation norm, as column-level information for the input channel, is shared across all output channels. While this shared structure is computationally efficient, it introduces a potential ranking bias. Specifically, in actual inputs, activation values ​​often contain a few outliers. When an input channel exhibits abnormally high activation values ​​in some samples, its corresponding norm will also be abnormally high, resulting in repeated high scores across multiple output channels. This phenomenon leads to the systematic retention of certain channel columns, creating ranking dominance in pruning, such as... Figure 3 As shown, when outliers appear in the activation, the third column of Figure (b) is dominated by the activation values, which are systematically preserved after pruning.

[0077] This bias is amplified under high sparsity settings, causing the pruned structure to concentrate in a few channels, increasing channel redundancy, decreasing structural diversity, and ultimately limiting the model's expressive power.

[0078] To address this, the present invention proposes a lightweight perturbation mechanism to identify and suppress such channel columns dominated by outlier activation, as a complement to the Wanda framework, aiming to improve the balance of the pruning structure and the performance stability under high sparsity.

[0079] To reduce the interference of dominant channels on the pruning structure, we apply softmasking to the top k dominant input channel columns. In the original Wanda scoring matrix, the scores of the corresponding columns of these channels are multiplied by a scaling factor. The specific adjustments are as follows:

[0080] Where Top-k(D) represents the set D consisting of the top k weighted columns of the dominant score. It is a scaling factor used to control the intensity of the disturbance. The value is 0.3. For example... Figure 4 As shown, this mechanism does not directly prune these channels, but rather guides the pruning order to be more balanced by reducing their relative advantage in the scoring matrix. This allows channels with stable activation medians and reasonable weight distributions to be retained, thereby improving the diversity and expressiveness of the pruning structure.

[0081] To further investigate the impact of hyperparameter selection on model performance in DAWanda's scoring suppression mechanism, we fixed one parameter and then tested the effect of changing the other parameter on the model's pruning performance. The two hyperparameters we focused on were: the dominant channel selection ratio and the suppression intensity factor α.

[0082] Figure 5 (a) shows the change trend of model perplexity under different inhibition intensity factors when Top-k is fixed at 0.01. It can be observed that when the inhibition intensity is 1.0, i.e., no inhibition, the PPL is 72.26, which is consistent with the original Wanda performance; while moderate inhibition is introduced, such as 0.3-0.5, the model performance is significantly improved, and the lowest PPL is reduced to 49.12, indicating that the inhibition mechanism guided by dominant scoring effectively alleviates the adverse effects of outlier activation on the pruning structure; but when the inhibition factor is too small, such as 0.1, the model performance decreases sharply, and the PPL is as high as 2316.33, indicating that excessive inhibition may destroy the expression ability of important channels, which also shows that highly activated channels are important to the model, but under high sparsity, such channels will be over-retained, playing the opposite role.

[0083] Figure 5 (b) shows the influence of different Top-k dominant channel selection ratios on the model perplexity under the condition of fixing the inhibition intensity at 0.3. It can be seen that a smaller ratio, such as 0.01, brings the best performance, with PPL reduced to 49.12; as Top-k increases, the number of inhibited channels increases, and the model performance gradually decreases, indicating that "guiding a small number of important channels" is more effective than "roughly suppressing a large number of channels". This also confirms that the design principle of "low proportion of dominant channel identification + moderate inhibition" in the DAWanda method is reasonable and effective.

[0084] The experimental results show that DAWanda has good robustness to hyperparameter settings. When the proportion of dominant channels is 1%, the inhibition factor is 0.3, which can achieve a better trade-off between perplexity and accuracy, and become the recommended configuration. Too small proportion or inhibition intensity will weaken the mechanism, while too large proportion or inhibition may misinjure important channels, leading to performance degradation.

[0085] Embodiment 1 In a preferred embodiment, the proposed lightweight pruning method based on channel dominance modeling is applied to a pre-trained large language model. The system first inputs a calibration dataset (e.g., 100 text samples) to perform a forward inference, and records the input activation matrix of each linear layer through the hook mechanism. Without changing the model weights, the pruning score matrix is calculated by the product of the linear layer input activation tensor and the weight matrix, thereby realizing the rapid estimation of the importance of different weight connections.

[0086] By imposing a dominance modeling and soft suppression mechanism on the score matrix, explicit channel control of pruning decisions is achieved. The final derived sparse model can be directly loaded to perform inference tasks without retraining, and its perplexity on the Wikitext-2 dataset rises by no more than 3%, significantly better than the traditional Wanda pruning method. This embodiment shows that the method can achieve a sparsity of more than 50% while maintaining model performance.

[0087] Embodiment 2 In another implementation, the internal linear layers of the Transformer structure perform activation sampling and one-time pruning: covering q_proj, k_proj, v_proj, o_proj of multi-head attention, and up_proj, down_proj of the feedforward network; the embedding layer and the output classification head are not involved in pruning, so as to avoid model degradation caused by weight disturbance of word embedding and output layer.

[0088] The sampling stage can obtain stable activation distribution statistics with a small amount of calibration data (e.g. 100 sentences). Tests have shown that when the size of the sampling set is greater than 64 texts, the stability of the ranking results of the dominant channel index reaches more than 95%, thereby proving that activation sampling is still statistically effective without relying on a large amount of training data.

[0089] Embodiment 3 In a specific implementation, the method implements one-time post-training pruning for the internal linear layers of the Transformer, covering q_proj, k_proj, v_proj, o_proj in multi-head attention, and up_proj, down_proj in the feedforward network; the word embedding layer and the output layer are not involved in pruning, so as to avoid disturbance to the word vector space and the output distribution. In the scoring stage, a small amount of calibration data is used to extract the input activation of each layer, and the L2 norm of each "input channel" (i.e. the column of the weight matrix) is calculated; then the weight matrix is scaled column by column with the corresponding activation norm to obtain the score matrix, and the higher the score, the more significant the contribution of the connection to the output. The entire scoring process is based on parallel matrix operations and is completed on a single GPU, usually with a second-level overhead, without any back propagation or weight reconstruction. To reduce the dominant effect of a small number of high activation channels on the ranking, a "channel dominance suppression" can be optionally introduced to proportionally down-regulate the input channels that are dominantly ranked in the top (rather than directly pruning them), making the pruning ranking more balanced and robust. This scheme can effectively characterize the importance of weights without relying on gradient or Hessian information, avoiding the additional cost and instability brought by traditional gradient-based re-computation, and still maintaining stable model performance at a high sparsity rate.

[0090] Embodiment 4 In an embodiment, the system multiplies the scores corresponding to each channel in the dominant channel set D by a suppression factor of 0.3 to weaken its dominant role in ranking. The proportion is determined by grid search in experiments, and is dynamically adjusted on different model layers to maintain pruning stability.

[0091] Through the soft suppression operation, the model pruning reduces the redundancy of the dominant channel while effectively preserving the non-dominant channel. This strategy has better robustness and generalization ability than the traditional hard mask method.

[0092] Embodiment 5 In this embodiment, the pruning execution stage adopts unstructured sparsification. For the internal linear layers (q_proj, k_proj, v_proj, o_proj, up_proj, down_proj) of the Transformer, the system first ranks according to the score matrix, and then cuts the weight elements with lower scores from bottom to top in each output channel (row by row) until the layer reaches the preset sparsity (such as 50% / 60% / 70%). This "in-row ranking + in-layer target proportion" approach is consistent with the Wanda baseline, which can avoid the mutual influence between different output channels, maintain the relative balance of in-layer pruning, and provide a stable ranking basis for subsequent "dominance suppression".

[0093] It should be noted that the memory and speed benefits brought by unstructured sparsification depend on whether sparse storage format and sparse computation kernel are used: without introducing a dedicated sparse kernel, the zeroing of parameters can reduce the "nominal amount" of multiply-add operations, but the memory occupancy and end-to-end inference speed do not promise a fixed multiple; therefore, this embodiment does not advocate giving a unified "memory reduction xx% / speedup x.x times" value, but rather emphasizes the accuracy and stability improvement under the same sparsity.

[0094] Under this strategy, combined with the score correction mechanism of "channel dominance identification + soft suppression" (proportionally adjusting the scores of a small number of input columns with high dominance, rather than hard pruning), this embodiment can still maintain excellent language modeling and zero-shot performance under high sparsity: for example, under the 70% sparsity setting of LLaMA-2-13B, the perplexity is significantly reduced from 44.95 of Wanda to 18.71, close to the level of SparseGPT (17.77), verifying the performance retention ability and robustness of "unstructured sparsification + dominance modeling".

[0095] Embodiment 6 In another embodiment, the sparse model is directly verified on the language modeling task. The system calculates the perplexity on the Wikitext-2 validation set and performs zero-shot classification test on the BoolQ and PIQA tasks. The results show that the perplexity of the sparse model only increases by 3.5%, while the zero-shot accuracy decreases by less than 1%.

[0096] The verification mechanism shows that the method can maintain the task performance without retraining, and the channel dominance inhibition has obvious effect in preventing over-pruning.

[0097] Embodiment 7 In one system implementation, the pruning system is deployed in the PyTorch framework and adopts a modular design. The activation collection module is mounted to each linear layer through a hook function to capture the input activation; the pruning score calculation module calls the matrix operation interface for parallel calculation; and the dominance modeling module calculates the channel index and completes the sorting according to the input activation and weight norm.

[0098] The soft inhibition module and the channel-level pruning module run cooperatively, the former adjusts the score matrix, and the latter performs the weight zeroing operation according to the correction result. This system structure realizes the full-process automation from activation collection to inference verification.

[0099] Embodiment 8 In one computer implementation, the method is packaged as an executable software module and stored in a computer-readable medium. The program is written in Python language and relies on the PyTorch underlying API to realize the whole process of activation collection, score calculation, dominance modeling, sparse execution and model export. When the processor executes the program, the whole process operation from the calibration data input to the sparse model generation is automatically completed.

[0100] Through testing on different hardware platforms (such as A100 and 3090 GPUs), the execution results of the program remain consistent, indicating that the medium implementation has cross-platform applicability and portability, providing reliable software support for the industrial deployment of the method.

[0101] It should be noted that embodiments of the present application can be realized by hardware, software, or a combination of software and hardware. The hardware portion can be realized by a special logic; the software portion can be stored in a memory and executed by a proper instruction execution system, such as a microprocessor or a specially designed hardware. A person of ordinary skill in the art can understand that the above-mentioned apparatus and method can be realized by computer executable instructions and / or included in processor control codes, such as a carrier medium, such as a magnetic disk, CD or DVD-ROM, a programmable memory, such as a read-only memory (firmware), or a data carrier, such as an optical or electronic signal carrier. The apparatus of the present application and its modules can be realized by a hardware circuit, such as a very large scale integrated circuit or a gate array, a semiconductor, such as a logic chip, a transistor, or a programmable hardware device, such as a field programmable gate array, a programmable logic device, or the like, by software executed by various types of processors, or by a combination of the above-mentioned hardware circuit and software, such as firmware.

[0102] The above description is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any modification, equivalent replacement, and improvement within the technical range disclosed by the present application, and within the spirit and principle of the present application, should be covered within the protection scope of the present application.

Claims

1. A lightweight pruning method based on channel dominance modeling, used to realize deep model weight sparsification without retraining, characterized in that, The method comprises the following steps: (1) performing forward reasoning to collect input activation matrices of each linear layer; (2) calculating a pruning score matrix based on the activation matrices and corresponding weight matrices; (3) calculating a channel dominance index according to the dominant degree of input channels on output weights and sorting the index; (4) selecting a top-ranked channel set as a dominant channel set and applying a scaling factor to the score column corresponding to the dominant channel set to generate a revised score matrix; (5) independently sorting according to the output channel dimension and deleting low-score weights to achieve a target sparsity rate; (6) exporting a sparse model and directly performing reasoning verification.

2. The method of claim 1, wherein, During the forward reasoning process, a small amount of calibration data set is input, and the input activation matrix and the corresponding weight matrix of each linear layer are recorded, wherein the linear layer includes the query projection layer, the key projection layer, the value projection layer, the output projection layer, the up-sampling layer and the down-sampling layer in the Transformer model.

3. The method of claim 1, wherein, The calculation of the pruning score matrix is based on the absolute value weighted average of the product of the channel norm of the input activation matrix and the corresponding weight element, so as to reflect the relative importance of each connection to the output channel.

4. The method of claim 1, wherein, The channel dominance index is the product of the norm of the weight column corresponding to each input channel and the activation intensity of the channel, which is used to measure the comprehensive dominant ability of the input channel to the model output.

5. The method of claim 1, wherein, The size of the dominant channel set accounts for 1% of the total number of input channels, and a scaling factor less than 1 is applied to the score column corresponding to the channels in the set to reduce the pruning priority, thereby balancing the weight importance distribution.

6. The method of claim 1, wherein, The pruning operation is a non-structured sparse method, which independently sorts the score matrix of each output channel and deletes the connection weight element with the lowest score value until the preset sparsity rate is reached.

7. The method of claim 1, wherein, The performance of the pruned model is evaluated by language modeling perplexity and zero-shot classification task accuracy to verify the preservation of the generalization ability of the model during the sparse process.

8. A lightweight pruning system based on channel dominance modeling, characterized in that, The method comprises the following steps: An activation collection module is configured to extract input activation matrices of each linear layer during the forward reasoning stage of the model; A pruning score calculation module is configured to calculate a pruning score matrix based on the activation matrices and weight matrices; A dominance modeling module is configured to calculate a channel dominance index and complete dominant channel sorting; A soft inhibition execution module is configured to apply a scaling factor to the score column corresponding to the dominant channels to generate a revised score matrix; A channel-level pruning module is configured to independently sort according to the output channel dimension and delete low-score weights; A model exporting and verifying module is configured to save the sparse model and perform reasoning evaluation.

9. The system of claim 8, wherein, The modules interact with each other through a unified model intermediate layer interface, and the activation collection module and the pruning score calculation module realize zero-copy data transmission through a tensor shared buffer.

10. A computer-readable storage medium having a computer program stored thereon, the program implementing all steps of the lightweight pruning method based on channel dominance modeling according to any one of claims 1 to 7 when executed by a processor.

Images