A pruning optimization method for large language models
By employing pruning optimization methods based on inter-layer feature spectra and gradient calibration, the problems of insufficient GPU memory and inference latency for large language models on edge devices are solved, achieving fair resource allocation and improved model accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI UNIV
- Filing Date
- 2026-04-30
- Publication Date
- 2026-06-05
Smart Images
Figure CN122154806A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and machine learning technology, and in particular to a pruning optimization method for large language models. Background Technology
[0002] In recent years, the number of parameters in large language models based on the Transformer architecture has expanded rapidly, ranging from billions to hundreds of billions of parameters. Although large models perform well in many benchmark tests, in real-world applications, especially on edge computing devices, mobile terminals, or servers with low GPU memory, directly deploying the original large models often faces problems such as insufficient GPU memory, high inference latency, and high energy consumption. To overcome these limitations, model pruning, as an effective model compression technique, is widely used to reduce redundant parameters in the model.
[0003] Existing model pruning methods are mainly divided into two categories: uniform pruning and non-uniform pruning. Uniform pruning methods apply the same pruning ratio to all layers or all types of weights, which is simple to implement but ignores the differences in importance between different layers or parameters within the model. For example, some shallow layers may carry more basic feature extraction functions, while deeper layers are more concerned with semantic abstraction. Simply applying a uniform pruning ratio can easily lead to the loss of key information, resulting in a significant deterioration in model performance.
[0004] Non-uniform pruning methods attempt to differentiate pruning based on parameter importance. Some methods measure parameter importance based on gradient magnitude; a larger gradient indicates a greater impact on the loss function and that the parameter should be retained. However, in real-world large language models, the distribution of original gradients often exhibits a long-tail distribution, meaning that a very small number of layers have extremely large gradients, while the vast majority of layers have small gradients. Directly using original gradients as the allocation criterion leads to a severe imbalance in resource allocation, with a few high-gradient layers occupying the majority of the retained dimensions, while other layers are overcompressed. This not only disrupts the heterogeneity between layers in the model but may also impair overall inference performance.
[0005] Furthermore, existing gradient-based pruning methods typically lack a global perspective on resource allocation optimization. They often make pruning decisions independently for each layer, failing to adequately consider how to optimally allocate the limited total retained dimensions across layers to maximize global performance. Therefore, combining feature spectrum information, calibrating gradient sensitivity, and introducing a global non-uniform sparsity allocation mechanism are crucial for achieving efficient and accurate model compression. Summary of the Invention
[0006] To address the aforementioned problems in existing technologies, this invention provides a pruning optimization method for large language models. This invention aims to solve the problems of traditional pruning methods neglecting inter-layer heterogeneity, the long-tail effect of gradient distribution leading to resource allocation imbalance, and the lack of a globally optimal dimension allocation strategy.
[0007] This invention provides a pruning optimization method for large language models, including: Determine the large language model to be pruned and prepare the calibration sample set; Based on the calibration sample set, the principal component spectra of each layer of the large language model are obtained, and an information retention function is constructed based on the principal component spectra. The average gradient sensitivity of parameters in each layer is calculated and then calibrated into layer weights using a logarithmic mapping. Based on the information preservation function and layer weights, the optimal sparsity of each layer is output; Based on the optimal sparsity of each layer, principal component slicing and weight transformation are performed on each layer to output the pruned model.
[0008] In one embodiment of the present invention, the pruned model is exported as ONNX and deployed to an edge platform, and verified using Tokens / s, TTFT, TPOT and LoadTime metrics.
[0009] In one embodiment of the present invention, obtaining the principal component spectra of each layer based on the calibration sample set and constructing the information retention function includes: Perform model forward propagation on the calibration sample set, and simultaneously collect input features from each layer. Obtain the complete feature distribution of this layer; Based on the collected input features from each layer, a covariance matrix is constructed. , where i is the layer index and t is the calibration sample index; Eigenvalue decomposition of the covariance matrix yields the principal component spectra of each layer arranged in descending order. Among them, the larger the eigenvalue, the more information the principal component carries; Normalize the eigenvalues in the principal component spectrum to obtain ; Construct an information preservation function based on the normalized eigenvalues. Used to achieve a given target sparsity When that happens, directly output the percentage of information retained after pruning at that layer.
[0010] In one embodiment of the present invention, the step of statistically calculating the average gradient sensitivity of each layer parameter and calibrating it into layer weights using a logarithmic mapping includes: Mean gradient sensitivity of parameters in each layer of the statistical model ; The average gradient sensitivity of all layers is corrected, and then... ; Taking the logarithm of the corrected gradient sensitivity and then performing linear normalization yields the layer weights. .
[0011] In one embodiment of the present invention, the step of outputting the optimal sparsity of each layer based on the information preservation function and the layer weights includes: Initialize the sparsity of each layer to the maximum allowable value under the constraints of global parameters; Based on the layer weights obtained from the information retention function and gradient sensitivity calibration, a marginal return index is constructed. ; By using a greedy allocation strategy, the sparsity of the layer with the highest marginal benefit is reduced, and the sparsity of each layer is iteratively updated until the preset global sparsity constraint is met. Output the optimal sparsity configuration for each layer. .
[0012] In one embodiment of the present invention, the global parameters include the target sparsity. Maximum sparsity Minimum sparsity Weight dynamic range Hidden Dimensions Hardware alignment step size and sparsity step size .
[0013] In one embodiment of the present invention, the step of performing principal component slicing and weight transformation on each layer based on the optimal sparsity of each layer, and outputting the pruned model, includes: According to the retention dimensions of each layer Perform principal component slicing and weight transformation on each layer; use , Output the pruned model in the following way; in The input side principal component orthogonal matrix, The output side principal component orthogonal matrix, For the input linear layer weights, This is for outputting the linear layer weights.
[0014] This invention also provides a model compression system based on a gradient-guided adaptive non-uniform pruning method, characterized in that it includes: A processor is used to execute instructions; Memory, used to store instructions; The storage module is used to store the weights and calibration dataset of the large language model; The pruning module is used to execute the above methods and steps to generate a sparse model.
[0015] In one embodiment of the present invention, the pruning processing module includes an interlayer feature spectrum acquisition unit, a gradient sensitivity acquisition and logarithmic mapping calibration unit, and a global non-uniform sparsity allocation unit.
[0016] In one embodiment of the present invention, a model export interface is also included, which is used to export the pruned model as an ONNX model format.
[0017] The present invention has the following beneficial effects: (1) By collecting interlayer feature spectra and using PCA or eigenvalue decomposition techniques, the amount of information under different retained dimensions can be accurately estimated, ensuring that the first few principal components are retained during the compression process, and maintaining the expressive power of the original model as much as possible under fewer dimensions, thus avoiding excessive loss of information caused by uniform pruning.
[0018] (2) The introduction of a logarithmic mapping calibration mechanism effectively addresses the long-tail distribution problem of the original gradient. By compressing the maxima and widening the differences between the minima, the imbalance in resource allocation is reduced, making resource competition among layers more equitable and avoiding situations where critical layers are squeezed out by edge layers or vice versa.
[0019] (3) A global non-uniform sparsity allocation strategy is adopted, and the dimensions are allocated according to the principle of maximizing marginal benefits. This method can dynamically adjust the retention ratio of each layer, so that limited computing resources are given priority to the layers that contribute more to performance, thereby achieving higher model accuracy at the same compression ratio, or achieving a higher compression rate at the same accuracy.
[0020] This invention does not rely solely on abstract mathematical algorithms, logical rules, or purely intellectual deduction to achieve theoretical calculations. Instead, it combines the hardware deployment characteristics of large language models, the differences in physical structure between model layers, and the real training gradient feedback and weight entity distribution characteristics to construct a complete and feasible engineering processing flow. Each step is carried out based on the actual multi-layer weight entity parameters of the large language model and the real operational data of forward inference and backpropagation. Through concrete operations such as matrix analysis, numerical mapping operations, and quantitative benefit iterative allocation that can be executed by computer hardware, the structured pruning optimization of the model is completed. The overall method closely integrates the real operating mechanism and physical layer structure attributes of the large model, has clear entity processing objects, hardware-implementable execution steps, and practical engineering implementation scenarios. It does not belong to purely mathematical algorithm deduction, human intellectual rule setting, or abstract theoretical logic schemes, and can effectively solve practical technical problems such as parameter redundancy, excessive computing power consumption, and poor uniform pruning effect in the deployment and implementation of large language models. Attached Figure Description
[0021] Figure 1A flowchart of a pruning optimization method for large language models according to an embodiment of the present invention is shown; Figure 2 A schematic diagram of the hardware deployment in one embodiment of the present invention is shown; Figure 3 This figure shows a comparison of the dialogue demo results of different pruning methods on Jetson Orin NX in one embodiment of the present invention; Figure 4 A comparison diagram of perplexity after structural pruning of the Llama model in one embodiment of the present invention is shown; Figure 5 This paper shows a comparison of the perplexity of the Qwen model structure after pruning in one embodiment of the present invention. Figure 6 A perplexity comparison chart is shown in one embodiment of the present invention when the sparsity is high. Detailed Implementation
[0022] In the following description, the invention is described with reference to various embodiments. However, those skilled in the art will recognize that the embodiments may be practiced without one or more specific details or with other alternatives and / or additional methods, materials, or components. In other instances, well-known structures, materials, or operations are not shown or described in detail so as not to obscure the inventive points of the invention. Similarly, for illustrative purposes, specific quantities, materials, and configurations are set forth to provide a comprehensive understanding of embodiments of the invention. However, the invention is not limited to these specific details.
[0023] In this invention, the various embodiments are merely intended to illustrate the solutions of the invention and should not be construed as limiting.
[0024] In this specification, references to "an embodiment" or "this embodiment" mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the invention. The phrase "in one embodiment" appearing throughout this specification does not necessarily refer to the same embodiment in all instances.
[0025] Furthermore, the numbering of the steps in the methods of the present invention does not limit the execution order of the method steps. Unless otherwise specified, the method steps may be executed in different orders.
[0026] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.
[0027] Figure 1 A flowchart of a pruning optimization method for large language models according to an embodiment of the present invention is shown.
[0028] like Figure 1 As shown, in this embodiment, the pruning optimization method for large language models includes: S1. Model and Data Preparation: In this step, the target deep learning model to be pruned is first obtained. This model can be a convolutional neural network (CNN), a Transformer model, or other deep neural networks with multi-layer structures. The target model is then structurally analyzed to clarify the type of each layer (e.g., convolutional layers, fully connected layers, normalized layers), parameter dimensions (e.g., the number of input / output channels and kernel size for convolutional layers, and the input / output dimensions for fully connected layers), and computational unit alignment constraints (e.g., the number of channels required by the hardware platform must be a multiple of 8, 16, etc.). This provides the basic constraints for hardware adaptation in subsequent structured pruning.
[0029] Subsequently, unlabeled samples covering typical input distributions in the actual application scenarios of the model are selected. The number of samples is determined according to the model size, preferably 500-2000 (or the corresponding number of sequence samples) to ensure that the sample distribution is consistent with the model training data distribution. The calibration samples are preprocessed, including normalization, resizing, format conversion and other operations, to adapt them to the input requirements of the target model.
[0030] After this step is completed, the original model structure parameters and standardized calibration sample set are output, and the global objective sparsity is set. Maximum sparsity Minimum sparsity Weight dynamic range Hidden Dimensions Hardware alignment step size and sparsity step size .
[0031] S2, Layer Input Feature Acquisition and PCA Decomposition: This step, based on the calibration sample set and the original model obtained in step S1, collects and performs principal component analysis (PCA) on the input features of each layer to quantify the information distribution of the features at each layer. The specific process is as follows: The calibration samples are input into the target model in batches, and the model forward propagation is performed. During the propagation process, the input features of each layer are collected simultaneously. .
[0032] For each layer, construct the covariance matrix of the input features. Where i is the layer index and t is the calibration sample index. Covariance matrix The correlation and distribution patterns of the input features of this layer among different samples were quantified, reflecting the degree of information association between feature dimensions.
[0033] For covariance matrix Performing eigenvalue decomposition yields a set of eigenvalues arranged in descending order. This refers to the principal component spectrum; where the larger the eigenvalue, the higher the proportion of feature information contained in the corresponding principal component, and the greater its contribution to the model task.
[0034] The eigenvalues are normalized to obtain the normalized eigenvalues. This eliminates the scale differences in feature values across different layers, allowing the information proportions of each layer to be compared under a unified dimension.
[0035] Information retention functions for each layer are constructed based on normalized eigenvalues. ,in The target sparsity (i.e., pruning ratio) for this layer, with a value ranging from [0,1]. This indicates that no pruning will be done. (This represents all parameters of pruning). The physical meaning of this function is: when the layer is sparse... When pruning, the proportion of information carried by the retained principal components relative to the total information of that layer can be directly used to quantify the degree of information loss under different pruning strategies.
[0036] This step ultimately outputs the principal component spectra of each layer. Normalized eigenvalues With information retention function This provides a quantitative basis for information loss in subsequent global sparsity allocation.
[0037] S3, Gradient sensitivity statistics and logarithmic calibration: This step quantifies the sensitivity of each layer to pruning operations by statistically analyzing the gradient sensitivity of the parameters of each layer, and then calibrates the sensitivity to layer weights through logarithmic mapping, providing a priority basis for non-uniform sparsity allocation. The specific process is as follows: The calibration samples are input into the target model, and forward and backward propagation are performed. During backward propagation, the gradients of the parameters in each layer are calculated; for convolutional layers, the gradient is the gradient tensor of the convolution kernel parameters; for fully connected layers, the gradient is the gradient tensor of the weight matrix. The mean sensitivity of the gradients of the parameters in each layer is then calculated. Gradient sensitivity mean The higher the value, the greater the impact of small changes in the parameters of that layer on the model output. In other words, the more sensitive that layer is to pruning operations, the more likely pruning will lead to a decrease in model performance.
[0038] To avoid problems in subsequent logarithmic calculations To address the underflow problem caused by values that are too small (e.g., approaching 0), the mean of the gradient sensitivity is truncated to obtain the corrected gradient sensitivity. This ensures that all gradient sensitivity values are greater than a very small positive number, guaranteeing the stability of subsequent calculations.
[0039] Sensitivity to the corrected gradient Take the logarithm, then perform linear normalization on the logarithmic gradient sensitivity of all layers, and map it to a preset weight interval to obtain the layer weights. The specific formula for calculating the layer weights is as follows: ; This formula linearly maps the logarithmic distribution of gradient sensitivity to... In a given interval, the layer with higher gradient sensitivity corresponds to higher layer weights. The larger the value, the lower the pruning priority of that layer in subsequent sparsity allocation, meaning a smaller pruning ratio, thereby protecting the parameters of the highly sensitive layer and reducing the impact of pruning on model performance.
[0040] This step ultimately outputs the calibration layer weights for each layer. This provides a quantitative basis for pruning priorities in subsequent global non-uniform sparsity allocation.
[0041] S4. Non-uniform sparsity distribution based on max-heap This step uses the information retention function obtained in step S2. The layer weights obtained in step S3 Based on this, and under global sparsity constraints and hardware alignment constraints, a greedy allocation strategy based on a max-heap is adopted to solve for the optimal sparsity configuration of each layer. The specific process is as follows: Initialize the sparsity of each layer to the maximum allowed value.
[0042] Under global sparsity constraints, construct pruning marginal benefit indicators for each layer. This metric represents the current sparsity of the i-th layer. Add another pruning step, in which When the preset sparsity adjustment step size is preferably a constant between 0.01 and 0.05, the weighted information loss caused by the increase in unit sparsity is considered.
[0043] In the greedy allocation, the sparsity of each layer with the largest marginal benefit is reduced (i.e., the retained dimension is increased) until the target global sparsity is reached.
[0044] This step ultimately outputs the optimal sparsity configuration for each layer. ,in This represents the final pruning ratio for the i-th layer, providing a direct basis for subsequent structured pruning.
[0045] S5. Perform structured pruning and weight output based on the allocation results: According to the retention dimensions of each layer Perform principal component slicing and weight transformation on each layer, where the following methods can be used: , The method outputs the pruned dense regular structure model.
[0046] This step is based on the optimal sparsity configuration of each layer obtained in step S4. The target model is subjected to structured pruning, and weight transformation and model structure adjustment are completed to obtain a lightweight pruned model. The specific process is as follows: Based on optimal sparsity Calculate the number of channels that need to be pruned in this layer; according to the sorting of principal component spectra in step S2, select the output channel with the lowest information ratio for pruning, and at the same time prune the corresponding channel parameters in the convolution kernel of this layer; in addition, if this layer is an intermediate convolutional layer of the model, it is also necessary to simultaneously prune the corresponding input channels in the next convolutional layer or fully connected layer to ensure the matching of inter-layer dimensions. Based on optimal sparsity Calculate the number of output dimensions that need to be pruned for this layer, select the corresponding column (or row) in the weight matrix of this layer for pruning, and at the same time prune the corresponding input dimensions in subsequent layers to ensure the continuity of the model structure.
[0047] Since structured pruning alters the model's parameter distribution, causing a shift in the model's output, it is necessary to fine-tune the pruned weights. Ideally, 1-5 rounds of fine-tuning training should be conducted using a small number of calibration samples to adjust the values of the pruned parameters, restoring the model's output distribution to its pre-pruning state. Furthermore, the pruned weight matrix should be rearranged to adapt its dimensions to the hardware's storage and computation format; for example, adjusting the number of channels to the alignment required by the hardware for easier subsequent deployment.
[0048] The pruned model structure and weights are saved to obtain a lightweight model with simplified structure, compressed parameter size, and controllable performance loss. All dimensions of this model satisfy hardware alignment constraints and can be directly used for subsequent deployment.
[0049] S6, Model Export, Deployment, and Performance Testing: This step exports the lightweight pruning model obtained in step S5 into a deployment format, and deploys and performs performance tests on the target hardware platform to verify the actual deployment effect of the model. The specific process is as follows: Depending on the requirements of the target deployment platform, the pruned model is exported to the corresponding format. For example, for PC GPU platforms, it can be exported to ONNX or TensorRT format; for mobile or edge devices, it can be exported to TFLite, CoreML, or ONNXRuntime format. During the export process, the model needs to be optimized before deployment, including operator fusion, constant folding, and precision conversion (such as FP32 to FP16 or INT8 quantization), to further improve the inference efficiency of the model.
[0050] Load the exported model onto the target hardware platform, configure the corresponding inference engine (such as TensorRT, TFLite, ONNXRuntime), and set parameters such as inference batches, number of threads, and computation backend to ensure that the model can run normally on the target platform.
[0051] The deployed model was comprehensively tested using a test dataset to verify the following core metrics: Accuracy metrics include task-related metrics such as model accuracy, recall, and mAP on the test set, ensuring that the accuracy drop of the pruned model is within an acceptable range (e.g., accuracy loss does not exceed 1%). Inference performance metrics include model single-frame inference latency, throughput (number of samples processed per second), CPU / GPU utilization, etc., to verify the model's inference efficiency on the target hardware; Resource usage metrics include model memory usage, GPU memory usage, and disk storage space, confirming that the lightweight effect of the model meets expectations.
[0052] Iterative optimization: If the test results meet the preset deployment requirements (i.e., accuracy, performance and resource consumption indicators all meet the standards), then the lightweight model is determined to be the final deployable version; if the test results do not meet the requirements (e.g., excessive accuracy loss, inference latency not meeting the standards), then return to steps S1-S4, adjust the calibration sample set, global sparsity constraints or layer weight parameters, re-execute the pruning process, and iteratively optimize the model until a lightweight model that meets the deployment requirements is obtained.
[0053] In one embodiment of the present invention, a specific operational process for hardware deployment is shown.
[0054] like Figure 2 The hardware shown is configured as shown in the table below: The specific deployment process is as follows: After pruning, the PyTorch model is exported to ONNX (Open Neural Network Exchange) format. ONNX is an open model exchange format that supports cross-platform deployment. During export, the input and output node names are explicitly specified, covering the token sequence, attention mask, positional encoding, and key-value caches for each layer. The batch size, sequence length, and key-value cache length are declared as dynamic dimensions to support variable-length inference. Model weights are stored as external data files, facilitating the management of large parameters on storage-constrained edge devices.
[0055] To address the hardware characteristics of the Jetson Orin NX platform, the exported ONNX models were optimized for format adaptation. A key focus was checking model operator compatibility, replacing unsupported higher-order operators with compatible ones to ensure the model could be correctly parsed by the ONNX Runtime. Simultaneously, the data types of the model's input and output were adjusted to match the computational accuracy requirements of edge devices, avoiding inference failures or performance degradation due to format incompatibility.
[0056] Enabling constant folding optimization during the export process allows the exporter to pre-compute fixed-value subgraphs in the graph during the tracing phase, eliminating redundant computations during inference. After deployment to Jetson devices, ONNX Runtime automatically performs runtime graph optimizations, including operator fusion and memory access merging, when loading the model, further reducing inference latency.
[0057] After transferring the adapted and optimized ONNX model files to the Jetson device, dependency checks, disk space verification, and model file integrity verification are required to ensure the deployment environment is ready before loading the model. Simultaneously, the ONNX Runtime runtime parameters are configured, specifying the model loading path and hardware inference backend, to prepare for subsequent inference testing.
[0058] The ONNX Runtime inference interface was used to run tests on both CPU and GPU backends, with a batch size of 1 (edge deployment scenario). Several rounds of warm-up were performed before inference to eliminate cold start errors, followed by multiple rounds of formal testing. Performance metrics such as throughput, first-word latency, and word-by-word latency were recorded to verify the deployment performance of the pruned model on the edge platform.
[0059] The performance inference results of the CPU (where GAP is the result of this invention) are shown in the table below.
[0060] The performance inference results of the GPU (where GAP is the result of this invention) are shown in the table below.
[0061] In one embodiment of the present invention, the perplexity of the Llama model after pruning is shown in the table below, where the optimal structure pruning result is marked in bold.
[0062] The perplexity of the pruned Qwen model is shown in the table below, where the optimal structure pruning results are highlighted in bold.
[0063] like Figure 4 and Figure 5 As shown, perplexity is a core metric for evaluating the predictive power of a language model, and a lower value is better. Structural pruning removes parts of the network structure (such as Transformer layers, attention heads, or FFN submodules), which is usually accompanied by a slight increase in perplexity. If the perplexity under the optimal pruning structure is close to that of the original model, it indicates that the strategy effectively preserves key information pathways.
[0064] Figure 3 These are the results of a dialogue demo on Jetson Orin NX using different pruning methods.
[0065] like Figure 3 As shown, the pruning method used in this invention has the best effect.
[0066] like Figure 6 As shown, at extremely high parameter pruning ratios (e.g., pruning rate > 50%), the divergence in perplexity curves between Llama and Qwen often reveals architectural differences. If a model exhibits a more gradual increase in perplexity under high sparsity, it indicates that its original structure has higher redundancy, or that its training method makes it more suitable for compression.
[0067] In a specific embodiment of the present invention, Llama-3.2-3B is used as the object to be pruned. 128 samples are randomly selected from WikiText2 as calibration data, with each sample having a length of 2048 words. A target sparsity is set. Maximum sparsity Minimum sparsity Weight dynamic range Hardware alignment step size After completing feature spectrum acquisition, gradient calibration, and non-uniform allocation according to the method of this invention, the ONNX model is exported and deployed to a Jetson OrinNX16GB device for edge dialogue inference. This state is suitable for real-time question-and-answer applications with explicit requirements for edge throughput and first-word latency.
[0068] In another specific embodiment of the present invention, Qwen2.5-1.5B or Qwen2.5-3B is used as the object to be pruned, and the same calibration process is maintained. The target sparsity is obtained in the Chinese question-and-answer or instruction generation scenario. After pruning is completed according to the method of this invention, it can be deployed in low-memory servers, embedded terminals, or all-in-one devices.
[0069] In a specific embodiment of the present invention, when the model to be processed is a Decoder-only Transformer, and adopts RMSNorm or an equivalent normalized structure that can be converted to RMSNorm, the present invention can more conveniently combine layer input principal component analysis to complete the regularized dimension pruning; when the model's hidden dimensions are aligned with the hardware step size. or When compatible, it can further improve the efficiency of operator matching and inference during actual deployment.
[0070] Although various embodiments of the invention have been described above, it should be understood that they are presented by way of example only and not as limitations. It will be apparent to those skilled in the art that various combinations, modifications, and alterations can be made without departing from the spirit and scope of the invention. Therefore, the breadth and scope of the invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined solely by the appended claims and their equivalents.
Claims
1. A pruning optimization method for large language models, characterized in that, include: Determine the large language model to be pruned and prepare the calibration sample set; Based on the calibration sample set, the principal component spectra of each layer of the large language model are obtained, and an information retention function is constructed based on the principal component spectra. The average gradient sensitivity of parameters in each layer is calculated and then calibrated into layer weights using a logarithmic mapping. Based on the information preservation function and layer weights, the optimal sparsity of each layer is output; and Based on the optimal sparsity of each layer, principal component slicing and weight transformation are performed on each layer to output the pruned model.
2. The method according to claim 1, characterized in that, It also includes exporting the pruned model to ONNX and deploying it to the edge platform, and validating it using metrics such as Tokens / s, TTFT, TPOT, and LoadTime.
3. The method according to claim 1, characterized in that, The process of obtaining principal component spectra at each level based on the calibration sample set and constructing information retention functions includes: Perform model forward propagation on the calibration sample set, and simultaneously collect input features from each layer. Obtain the complete feature distribution of this layer; Based on the collected input features from each layer, a covariance matrix is constructed. , where i is the layer index and t is the calibration sample index; Eigenvalue decomposition of the covariance matrix yields the principal component spectra of each layer arranged in descending order. Among them, the larger the eigenvalue, the more information the principal component carries; Normalize the eigenvalues in the principal component spectrum to obtain ; Construct an information preservation function based on the normalized eigenvalues. Used to achieve a given target sparsity When that happens, directly output the percentage of information retained after pruning at that layer.
4. The method according to claim 1, characterized in that, The process of statistically analyzing the average gradient sensitivity of parameters in each layer and calibrating it into layer weights using a logarithmic mapping includes: Mean gradient sensitivity of parameters in each layer of the statistical model ; The average gradient sensitivity of all layers is corrected, and then... ; Taking the logarithm of the corrected gradient sensitivity and then performing linear normalization yields the layer weights. .
5. The method according to claim 1, characterized in that, The optimal sparsity of each layer, based on the information preservation function and layer weights, includes: Initialize the sparsity of each layer to the maximum allowable value under the constraints of global parameters; Based on the layer weights obtained from the information retention function and gradient sensitivity calibration, a marginal return index is constructed. ; By using a greedy allocation strategy, the sparsity of the layer with the highest marginal benefit is reduced, and the sparsity of each layer is iteratively updated until the preset global sparsity constraint is met. Output the optimal sparsity configuration for each layer. .
6. The method according to claim 5, characterized in that, The global parameters include the target sparsity. Maximum sparsity Minimum sparsity Weight dynamic range Hidden Dimensions Hardware alignment step size and sparsity step size .
7. The method according to claim 1, characterized in that, The process of performing principal component slicing and weight transformation on each layer based on the optimal sparsity of each layer, and outputting the pruned model includes: According to the retention dimensions of each layer Perform principal component slicing and weight transformation on each layer; use , Output the pruned model in the following way; in The input side principal component orthogonal matrix, The output side principal component orthogonal matrix, For the input linear layer weights, This is for outputting the linear layer weights.
8. A model compression system based on a gradient-guided adaptive non-uniform pruning method, characterized in that, include: A processor is used to execute instructions; Memory, used to store instructions; The storage module is used to store the weights and calibration dataset of the large language model; A pruning module is used to perform the steps of the method according to any one of claims 1 to 7 to generate a sparse model.
9. The system according to claim 8, characterized in that, The pruning processing module includes an interlayer feature spectrum acquisition unit, a gradient sensitivity acquisition and logarithmic mapping calibration unit, and a global non-uniform sparsity allocation unit.
10. The system according to claim 8, characterized in that, It also includes a model export interface for exporting pruned models to the ONNX model format.