Large language model decoding acceleration method and system based on normalized sparse activation
By introducing a sparse activation method with dynamic thresholding and rotation matrix, the problems of sparsity inaccuracy and hardware efficiency in large language models are solved, achieving efficient decoding acceleration and adapting to various model architectures.
Patent Information
- Application Number
- CN202511621370.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-06
- Publication Date
- 2026-02-10
AI Technical Summary
Existing sparse activation methods fail to adequately consider the correlation between contextual information and dimensions in large language models, resulting in a significant performance drop under high sparsity and efficiency issues in hardware implementation.
We introduce a dynamic thresholding mechanism based on vector norm and a data-driven orthogonal rotation matrix. We optimize sparse activation through offline calibration and online inference stages to achieve context-aware sparsity. We then combine this with a hardware-aware sparse matrix multiplication kernel to accelerate computation.
It maintains near-original model performance under extremely high sparsity, significantly improves computational acceleration, reduces memory access frequency and energy consumption, and adapts to different model architectures without retraining.
Abstract
Description
Technical Field
[0001] This invention relates to the field of natural language processing, and in particular to a method and system for accelerating the decoding of large language models based on normalized sparse activation. Background Technology
[0002] With the widespread application of large language models (LLMs) in natural language processing tasks, their inference speed, especially the computational efficiency in the autoregressive decoding process, has become a key bottleneck in deployment. Existing acceleration techniques mainly include quantization, pruning, and speculative decoding. Among these, sparse activation, by selectively ignoring specific dimensions of the activation vector during forward propagation to reduce computation and memory access, has become an effective optimization method.
[0003] However, current sparse activation methods (such as TEAL and LaRoSA) are based on a strong assumption: the activation values of different dimensions of the hidden state follow an independent and identically distributed (i.i.d.) distribution. This assumption has significant limitations in real-world scenarios: 1. Causal dependency between tokens: In autoregressive decoding, the probability distribution of the current token depends on the historical tokens. The attention mechanism causes the hidden states of tokens to be correlated, which violates the independence assumption.
[0004] 2. Correlation between dimensions: There is a linear correlation between the dimensions of the hidden state (such as the semantic correlation of word embeddings), and the input received by the underlying model is a fixed word embedding rather than a random sampled value, which further challenges the iid hypothesis.
[0005] Existing methods suffer from significant performance degradation at high sparsity levels because they ignore the aforementioned correlations. For example, TEAL uses global threshold filtering, and LaRoSA selects the dimensions to retain through Top-K selection, but neither of them fully considers the contextual information and the correlation between dimensions, thus limiting the accuracy and generalization ability of sparsification.
[0006] Furthermore, existing methods suffer from efficiency issues in hardware implementation: LaRoSA's sorting operation introduces excessive overhead and cannot flexibly allocate computational resources. Therefore, a context-aware, computationally efficient sparsification method that supports dynamic parameter allocation is needed to minimize performance loss while maintaining inference speed.
[0007] The Normalized Sparse Activation (NorSA) method proposed in this patent effectively solves the above problems by introducing a normalized threshold and a rotation matrix, providing an innovative solution for efficient decoding of large language models. Summary of the Invention
[0008] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is how to efficiently decode large language models.
[0009] To achieve the above objectives, this invention provides a method for accelerating the decoding of large language models based on normalized sparse activation. The method is characterized by introducing a dynamic thresholding mechanism based on vector norms to replace traditional fixed thresholds or Top-K selection. To improve the accuracy of matrix sparsification, this method introduces a data-driven orthogonal rotation matrix to rotate and transform the input activation space, thereby minimizing the linear correlation between dimensions.
[0010] Furthermore, the complete implementation process of the method in the reasoning process of large-scale language models includes two stages: offline calibration and online reasoning.
[0011] Furthermore, the offline calibration phase is a preparatory step performed before model deployment. It involves analyzing the calibration dataset to determine the optimal parameter configuration. The specific steps are as follows: Step 1: Data preparation and configuration; Step 2: Statistical information collection and threshold determination; Step 3: Rotation matrix calculation and optimization; Step 4: Mixed sparsity optimization configuration.
[0012] Furthermore, in step 1, a representative text dataset is selected as the calibration data source, and several text sequences of length L are randomly sampled from it, with a total of N sequences. All text preprocessing procedures are completely consistent with the original training settings of the model, including the tokenization method and sequence truncation strategy.
[0013] Further, in step 2, the calibration data is input into the target model to perform a complete forward propagation process. During this process, the statistical characteristics of the input activation vector of each linear layer are recorded, including the numerical distribution of each dimension and the norm size information. Based on this statistical information, an iterative search algorithm is used to determine the optimal threshold scaling parameter τ for each linear layer. The goal of determining this parameter is to achieve the preset global sparsity target when using the threshold τ‖X‖ for sparsification, while minimizing the impact on the model output.
[0014] Furthermore, in step 3, the corresponding rotation matrices are calculated for different layer types in the model; for activations input to Q / K / V projection and Up / Gate projection, their composite covariance matrices are calculated, and eigenvectors are extracted using principal component analysis (PCA) to form the rotation matrices. For the output projection layer of the multi-head attention mechanism, a block diagonal matrix structure is adopted. The rotation submatrix is calculated independently for each attention head. PCA initialization is performed based on the covariance matrix of each attention head's output, and finally, the two matrices are combined into a rotation matrix. .
[0015] Furthermore, step 4 employs a hierarchical optimization strategy to assign different sparsity targets to different layers in the model; using a greedy algorithm or a sensitivity-based allocation method, the optimal sparsity configuration for each layer is found under the constraint of the total computational budget, in order to maximize overall performance.
[0016] Furthermore, the inference phase applies the parameters obtained in the calibration phase to the actual inference process, thereby accelerating computation; the specific process is as follows: Step S1: Model preparation and parameter injection; Step S2: Application of the rotation matrix; Step S3: Norm-based dynamic sparsity reduction; Step S4: Optimization of sparse matrix calculation.
[0017] Further, in step S1, the pre-trained model weights and structure configuration are loaded, and the calibrated rotation matrix is integrated into a specified position in the model architecture; the corresponding threshold parameter τ is set for each linear layer to complete the model transformation and parameter initialization; in step S2, a rotation operation is inserted at a specific position during the model forward propagation; specifically, the rotation matrix is applied before Q / K / V projection and Up / Gate projection. The input activation is transformed; a block diagonal rotation matrix is applied before the output projection. The input is transformed so that the structure of the matrix perfectly matches the multi-head attention mechanism. In step S3, for the input activation vector X of each linear layer, its L2 norm ‖X‖ is first calculated, and the dynamic threshold τ‖X‖ is calculated according to the preset threshold ratio parameter τ of the layer. Dimensions in the activation vector whose absolute value is lower than the dynamic threshold are set to zero, while important dimensions are retained for subsequent calculations. This process achieves context-based adaptive sparsity. In step S4, matrix multiplication is performed only on the non-zero dimensions retained after sparsification, avoiding unnecessary calculations and memory accesses on zero-value dimensions. Through a dedicated sparse computing kernel, a significant improvement in computational efficiency is achieved.
[0018] A large-scale language model decoding acceleration system based on normalized sparse activation is characterized in that the kernel architecture adopts a hierarchical optimization strategy to achieve efficient sparse matrix-vector multiplication calculation on GPU hardware; the weight matrix adopts a column-first storage format, which facilitates selective loading of the required weight columns according to the sparse mode; the calculation process adopts a block processing method, dividing the large matrix into multiple small blocks to make full use of the parallel computing capabilities of the GPU. In the computation, the three steps of sparsity determination, selective weight loading, and matrix multiplication are integrated into a single computational kernel, reducing memory transfer overhead and kernel startup latency. By making reasonable use of shared memory and register files, data reuse is maximized and the frequency of global memory access is reduced.
[0019] The present invention has the following technical effects: (1) Excellent performance retention: It can still approach the performance of the original model even under extremely high sparsity. Experiments show that when the activation sparsity reaches 50% on the LLaMA3-8B model, the perplexity is only 0.44 higher than that of the dense model, and the accuracy of the zero-shot task decreases by only 1.23%, which is significantly better than existing methods (LaRoSA and TEAL). (2) High-efficiency computation acceleration: Through hardware-aware customized kernels (such as the fusion of sparse GEMV kernels, norm calculation and activation function fusion), the end-to-end decoding speed on NVIDIA A100 is up to 1.73 times faster (70B model, 75% sparsity), and the computational overhead is only slightly higher than the simplest threshold method. (3) Strong generalization: It performs consistently well on models of different architectures and scales such as LLaMA, Mistral, and Qwen, and can be applied directly without retraining, thus lowering the deployment threshold.
[0020] The following will further explain the concept, specific structure, and technical effects of the present invention in order to fully understand the purpose, features, and effects of the present invention. Detailed Implementation
[0021] The following describes several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.
[0022] Components with identical structures are designated by the same numerical designation, and components with similar structures or functions are designated by similar numerical designations. The dimensions and thicknesses of each component shown are arbitrary, and the present invention does not limit the dimensions and thicknesses of each component. For clarity, the thickness of components has been appropriately exaggerated in some places.
[0023] This invention provides a method for accelerating the decoding of large language models based on Normalized Sparse Activation (NorSA). The following detailed implementation details are provided through multiple examples, focusing on the innovative principles and technical effects.
[0024] Example 1: Complete Implementation Process of the NorSA Method
[0025] This embodiment describes the complete implementation process of the NorSA method in the inference process of large-scale language models, including two stages: offline calibration and online inference, using the LLaMA series models as an example.
[0026] (a) Offline calibration phase
[0027] The calibration phase is a preparatory step performed before model deployment, determining the optimal parameter configuration by analyzing the calibration dataset. This phase does not participate in actual inference but provides key parameters for subsequent inference phases.
[0028] Step 1: Data Preparation and Configuration
[0029] Select representative text datasets as calibration data sources, such as standard corpora like WikiText-2, C4, or Alpaca. Randomly sample several text sequences of length L (e.g., L=2048), typically N (e.g., N=16), to ensure coverage of different language patterns and semantic scenarios. All text preprocessing procedures are completely consistent with the original model training settings, including tokenization methods and sequence truncation strategies, to ensure consistency in data distribution.
[0030] Step 2: Statistical Information Collection and Threshold Determination
[0031] The calibration data is input into the target model for a complete forward propagation process. During this process, the statistical characteristics of the input activation vectors for each linear layer are recorded, including the numerical distribution of each dimension and the norm. Based on this statistical information, an iterative search algorithm is used to determine the optimal threshold scaling parameter τ for each linear layer. The objective of determining this parameter is to achieve a preset global sparsity target (e.g., 50%) when using the threshold τ‖X‖ for sparsification, while minimizing the impact on the model output.
[0032] Step 3: Rotation Matrix Calculation and Optimization
[0033] For different layer types in the model, the corresponding rotation matrices are calculated. For activations input to Q / K / V projection and Up / Gate projection, their composite covariance matrices are calculated, and eigenvectors are extracted using principal component analysis (PCA) to form the rotation matrices. For the output projection layer of the multi-head attention mechanism, a block diagonal matrix structure is adopted. A rotation submatrix is independently calculated for each attention head, and PCA initialization is performed based on the covariance matrix of each attention head's output. Finally, these are combined into a rotation matrix. .
[0034] Step 4: Mixed sparsity optimization configuration
[0035] A hierarchical optimization strategy is adopted, assigning different sparsity targets to different layers in the model. Using a greedy algorithm or a sensitivity-based allocation method, the optimal sparsity configuration for each layer is found under the constraint of the total computational budget to maximize overall performance.
[0036] (ii) Online reasoning stage
[0037] The inference phase applies the parameters obtained in the calibration phase to the actual inference process, thereby accelerating computation.
[0038] Step 1: Model Preparation and Parameter Injection
[0039] Load the pre-trained model weights and structure configuration, and integrate the calibrated rotation matrix into the specified location in the model architecture. Set the corresponding threshold parameter τ for each linear layer to complete the model transformation and parameter initialization.
[0040] Step 2: Application of the rotation matrix
[0041] During the model's forward propagation, rotation operations are inserted at specific locations. Specifically, rotation matrices are applied before the Q / K / V projection and the Up / Gate projection. The input activation is transformed; a block diagonal rotation matrix is applied before the output projection. The input is transformed so that the structure of the matrix perfectly matches the multi-head attention mechanism.
[0042] Step 3: Norm-based dynamic sparsity
[0043] For each linear layer's input activation vector X, its L2 norm ‖X‖ is first calculated. Based on the layer's preset threshold scaling parameter τ, a dynamic threshold τ‖X‖ is calculated. Dimensions of the activation vector whose absolute value is below this dynamic threshold are set to zero, while important dimensions are retained for subsequent calculations. This process achieves context-based adaptive sparsity.
[0044] Step 4: Optimization of Sparse Matrix Computation
[0045] Matrix multiplication is performed only on the non-zero dimensions retained after sparsification, avoiding unnecessary calculations and memory accesses on zero-valued dimensions. A significant improvement in computational efficiency is achieved through a dedicated sparse computation kernel.
[0046] Example 2: Hardware Optimization of the Sparse Matrix Multiplication Kernel
[0047] This embodiment describes in detail the hardware-aware optimization implementation of NorSA, focusing on the design principles of the sparse matrix multiplication kernel.
[0048] Kernel architecture design: A hierarchical optimization strategy is employed to achieve efficient sparse matrix-vector multiplication (SparseGEMV) computation on GPU hardware. The weight matrix uses a column-major storage format, facilitating the selective loading of required weight columns based on the sparsity pattern. The computation process utilizes a block-based approach, dividing the large matrix into multiple smaller blocks to fully leverage the parallel computing capabilities of the GPU.
[0049] Optimization of computational process: By integrating the three steps of sparsity determination, selective weight loading, and matrix multiplication into a single computational kernel, memory transfer overhead and kernel startup latency are reduced. Through the rational utilization of shared memory and register files, data reuse is maximized, and the frequency of global memory access is reduced.
[0050] Performance optimization features: It supports variable sparsity modes, adapting to the sparsity characteristics of different layers. It employs asynchronous computation and data prefetching techniques to hide memory access latency. Through automatic tuning, it dynamically selects the optimal computation parameters (such as thread block size, memory access mode, etc.) for different matrix sizes and sparsity configurations.
[0051] Implementation results: On typical hardware platforms (such as NVIDIA A100), for common-sized matrix computations (such as 14336×4096), approximately 2x latency reduction can be achieved at 50% sparsity, and approximately 3.5x latency reduction can be achieved at 75% sparsity. Memory bandwidth requirements are reduced by 40-60% accordingly, and overall power consumption is reduced by 35-55%.
[0052] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.
Claims
1. A method for accelerating the decoding of large language models based on normalized sparse activation, characterized in that, This method introduces a dynamic thresholding mechanism based on vector norms to replace the traditional fixed threshold or Top-K selection. To improve the accuracy of matrix sparsification, this method introduces a data-driven orthogonal rotation matrix to rotate and transform the input activation space, thereby minimizing the linear correlation between dimensions.
2. The method for accelerating the decoding of large language models based on normalized sparse activation as described in claim 1, characterized in that, The complete implementation process of the method in the reasoning process of large-scale language models includes two stages: offline calibration and online reasoning.
3. The method for accelerating the decoding of large language models based on normalized sparse activation as described in claim 2, characterized in that, The offline calibration phase is a preparatory step performed before model deployment. It involves analyzing the calibration dataset to determine the optimal parameter configuration. The specific steps are as follows: Step 1: Data preparation and configuration; Step 2: Statistical information collection and threshold determination; Step 3: Rotation matrix calculation and optimization; Step 4: Mixed sparsity optimization configuration.
4. The method for accelerating the decoding of large language models based on normalized sparse activation as described in claim 3, characterized in that, Step 1 selects a representative text dataset as the calibration data source, and randomly samples several text sequences of length L, with a total of N sequences. All text preprocessing procedures are completely consistent with the original training settings of the model, including the tokenization method and sequence truncation strategy.
5. The method for accelerating the decoding of large language models based on normalized sparse activation as described in claim 4, characterized in that, Step 2 involves inputting calibration data into the target model and performing a complete forward propagation process. During this process, the statistical characteristics of the input activation vectors of each linear layer are recorded, including the numerical distribution and norm information of each dimension. Based on this statistical information, an iterative search algorithm is used to determine the optimal threshold scaling parameter τ for each linear layer. The objective of determining this parameter is to achieve the preset global sparsity target when using the threshold τ‖X‖ for sparsification, while minimizing the impact on the model output.
6. The method for accelerating the decoding of large language models based on normalized sparse activation as described in claim 5, characterized in that, Step 3 calculates the corresponding rotation matrix for different layer types in the model; for activations input to Q / K / V projection and Up / Gate projection, the composite covariance matrix is calculated, and eigenvectors are extracted using principal component analysis (PCA) to form the rotation matrix. For the output projection layer of the multi-head attention mechanism, a block diagonal matrix structure is adopted. The rotation submatrix is calculated independently for each attention head. PCA initialization is performed based on the covariance matrix of each attention head's output, and finally, the two matrices are combined into a rotation matrix. .
7. The method for accelerating the decoding of large language models based on normalized sparse activation as described in claim 6, characterized in that, Step 4 employs a hierarchical optimization strategy, assigning different sparsity targets to different layers in the model; using a greedy algorithm or a sensitivity-based allocation method, the optimal sparsity configuration for each layer is found under the constraint of the total computational budget, in order to maximize overall performance.
8. The method for accelerating the decoding of large language models based on normalized sparse activation as described in claim 7, characterized in that, The inference phase applies the parameters obtained in the calibration phase to the actual inference process, thereby accelerating computation; the specific process is as follows: Step S1: Model preparation and parameter injection; Step S2: Application of the rotation matrix; Step S3: Norm-based dynamic sparsity reduction; Step S4: Optimization of sparse matrix calculation.
9. The method for accelerating the decoding of large language models based on normalized sparse activation as described in claim 8, characterized in that, Step S1 loads the pre-trained model weights and structure configuration, and integrates the calibrated rotation matrix into a specified position in the model architecture; sets the corresponding threshold parameter τ for each linear layer, completing the model transformation and parameter initialization; Step S2 inserts rotation operations at specific positions during the model forward propagation process; specifically, before Q / K / V projection and Up / Gate projection, the rotation matrix is applied. The input activation is transformed; a block diagonal rotation matrix is applied before the output projection. The input is transformed so that the structure of the matrix perfectly matches the multi-head attention mechanism. In step S3, for the input activation vector X of each linear layer, its L2 norm ‖X‖ is first calculated, and the dynamic threshold τ‖X‖ is calculated according to the preset threshold ratio parameter τ of the layer. Dimensions in the activation vector whose absolute value is lower than the dynamic threshold are set to zero, and important dimensions are retained for subsequent calculations. This process realizes context-based adaptive sparsity. Step S4 only performs matrix multiplication on the non-zero dimensions retained after sparsification, avoiding unnecessary calculations and memory accesses on zero-value dimensions; through a dedicated sparse computing kernel, a significant improvement in computational efficiency is achieved.
10. A large-scale language model decoding acceleration system based on normalized sparse activation, characterized in that, The kernel architecture employs a layered optimization strategy to achieve efficient sparse matrix-vector multiplication calculations on GPU hardware; the weight matrix uses a column-major storage format, which facilitates selective loading of the required weight columns according to the sparse pattern. The calculation process adopts a block-based processing method, dividing the large matrix into multiple small blocks to make full use of the parallel computing capabilities of the GPU; In the computation, the three steps of sparsity determination, selective weight loading, and matrix multiplication are integrated into a single computational kernel, reducing memory transfer overhead and kernel startup latency. By making reasonable use of shared memory and register files, data reuse is maximized and the frequency of global memory access is reduced.