A large language model quantization method and system based on a global topology-aware grid

Abstract

The application relates to a large language model quantization method and system based on a global topology-aware grid, and belongs to the technical field of artificial intelligence model compression. The application calculates a structure sensitivity vector of each linear layer weight by performing forward-backward propagation on a calibration set; extracts diagonal elements of a local Hessian matrix of input activation; combines global topology information and local curvature through an exponential fusion mechanism to construct a topology Hessian matrix; performs K-means clustering optimization of a non-uniform quantization grid with the matrix as weight to generate quantization anchor points adaptive to weight distribution; and finally maps the weight to a target bit grid and completes index packing storage. The application can effectively maintain model reasoning accuracy under 3-bit ultra-low bit constraint without retraining, avoids mixed precision overhead through regular bit width design, and is suitable for efficient deployment of large models on end-side and edge computing devices.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence model compression technology, specifically relating to a method and system for quantizing large language models based on a global topological sensing grid. Background Technology

[0002] With the increasing parameter size of Large Language Models (LLMs), Post-Training Quantization (PTQ) has become the mainstream technique for deploying large models. Existing weight quantization methods (such as GPTQ and AWQ) and non-uniform grid optimization methods (such as LeanQuant and OSTQuant) mainly compensate for the accuracy loss caused by quantization by minimizing layer-wise local reconstruction errors or by utilizing the local Hessian matrix, achieving good results in 4-bit quantization.

[0003] Existing mesh optimization methods are based on the Local Independence Assumption, which means that optimization is performed in isolation for only a single network layer. When quantizing the model to ultra-low bit depths, this assumption leads to the following drawbacks:

[0004] 1. Ignoring error cascading effects: Existing techniques only optimize the quantization mesh based on the input distribution and local curvature of the current layer, without considering the global cascading amplification problem that occurs in deep networks when tiny local quantization noise propagates through residual connections and nonlinear activation functions.

[0005] 2. Lack of differentiation in global sensitivity across different projection layers: In the Transformer architecture, the Value (V) and Output (O) projection matrices directly inject information into the residual stream, resulting in significantly higher global sensitivity than the Query (Q) and Key (K) matrices affected by the Softmax function. Purely local optimization methods cannot detect this structural asymmetry, making it impossible to provide targeted precision protection for network layers with high global sensitivity. Summary of the Invention

[0006] The purpose of this invention is to solve the problems of local optimization leading to global error amplification and lack of differentiated protection for the global importance of different network layers in existing large model quantization techniques, and to provide a large language model quantization method and system based on a global topology-aware grid.

[0007] To achieve the above objectives, the technical solution of the present invention is: a large language model quantization method based on a global topology-aware grid, comprising:

[0008] Obtain the weights of the pre-trained large language model to be quantized and the calibration dataset;

[0009] Based on the calibration dataset, the structural sensitivity vector of the weights of each linear layer in the large language model is calculated through forward and backward propagation.

[0010] Extract the diagonal elements of the local Hessian matrix of the activation values ​​of each linear layer input;

[0011] A topological Hessian matrix is ​​generated by exponential fusion calculation based on the structural sensitivity vector and the diagonal elements of the local Hessian matrix.

[0012] Using the topological Hessian matrix as the weight coefficient, non-uniform quantization grid optimization is performed on the weights of the pre-trained large language model, and the optimized grid is mapped to the target bit width to complete the model quantization.

[0013] Furthermore, the calculation of the structure sensitivity vector for the weights of each linear layer specifically includes:

[0014] Perform a single forward-backward propagation on the calibration dataset to obtain the global gradient of the loss function with respect to the weight matrices of each linear layer;

[0015] The structural sensitivity matrix is ​​obtained by calculating the Hadamard product of the global gradient and the absolute weights.

[0016] Perform along the output dimension of the structure sensitivity matrix Norm aggregation yields a one-dimensional structure sensitivity vector.

[0017] Furthermore, the formula for calculating the structural sensitivity matrix is ​​as follows:

[0018]

[0019] in, Indicates the l-th layer Line 1 The structural sensitivity value of column weights This represents the global gradient of the loss function with respect to the weights. Indicates the l-th layer Line 1 The original weights of the columns, represents the loss function, and ⊙ represents the Hadamard product operator.

[0020] Furthermore, the formula for calculating the one-dimensional structural sensitivity vector is as follows:

[0021]

[0022] in, Indicates the l-th layer One-dimensional structure sensitivity value for each channel Indicates the output channel dimension.

[0023] Furthermore, the specific calculation formula for extracting the diagonal elements of the local Hessian matrix of the input activation values ​​of each linear layer is as follows:

[0024]

[0025] in, This represents the diagonal elements of the local Hessian matrix at layer l. Let represent the input activation matrix of the l-th layer, diag(⋅) represents the operation of taking the diagonal of the matrix, and T represents the matrix transpose operation.

[0026] Furthermore, the topological Hessian matrix is ​​generated through exponential fusion calculation, and the specific calculation formula is as follows:

[0027]

[0028] in, Indicates the l-th layer Topological Hessian value of each channel, Indicates the l-th layer Local Hessian diagonal elements of each channel, Indicates the l-th layer The structural sensitivity value of each channel, For all of this layer The resulting one-dimensional structural sensitivity vector. Here are the preset topology-aware scalar hyperparameters, and exp represents the natural exponential function. This indicates the operation of retrieving the maximum value.

[0029] Furthermore, the non-uniform quantized grid optimization employs a topology-weighted K-means clustering algorithm, with the objective function being:

[0030]

[0031] in, Indicates the quantization of the central cluster of the grid. Indicates the first There are K cluster centers, where K is the number of grids corresponding to the target bit width. This represents the value of the variable that minimizes the objective function. This represents the original weight values. These are the weighting coefficients of the topological Hessian matrix for the corresponding channel.

[0032] Furthermore, the target bit width is 3 bits, and the number K of the quantization grid center clusters is set to... indivual.

[0033] Furthermore, after model quantization is completed, the process also includes a weight deployment step: mapping the quantized weights to the corresponding grid index values; packaging and storing multiple low-bit indices into high-bit-width integer variables through bit shifting operations; during inference execution, mapping the indices back to the corresponding grid center floating-point values ​​through table lookup operations, and performing matrix multiplication operations with the input activation values.

[0034] This invention also provides a large language model quantization system based on a global topology-aware grid, which performs the method described above, including:

[0035] The data and model loading module is used to import pre-trained large language model weights and calibration datasets.

[0036] The global topology analysis module is used to perform forward-backward propagation based on the calibration dataset and calculate the global structural sensitivity vector of the weights of each layer.

[0037] The local feature extraction module is used to capture the input activation values ​​of each linear layer and calculate the diagonal elements of the local Hessian matrix;

[0038] The matrix exponential fusion module is used to receive the global structure sensitivity vector and the diagonal elements of the local Hessian matrix, and generate the topological Hessian matrix through exponential fusion calculation.

[0039] The topology-aware grid clustering module is used to perform K-means clustering iterations on non-uniformly quantized grids using the topological Hessian matrix as weight coefficients, output quantized grid anchor points, and complete bit-reduction compression replacement of model weights.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] First, existing Hessian-weighted K-means algorithms typically rely only on local second derivatives. A common approach is to use linear weighting to incorporate global sensitivity. However, in LLM (Low Bit Quantization), due to the significant difference in dimensions between local activation variance and global loss gradient, conventional linear weighting causes weight centers to drift during clustering, leading to accuracy collapse. The topology-aware scalar exponential fusion in this scheme is designed to overcome this technical bias. It leverages the monotonically increasing nature of the exponential function and boundary constraints to more accurately amplify weights at structural bottlenecks without altering the local Hessian dominance, representing an innovative design. Secondly, under extremely stringent 3-bit pure weight constraints, this scheme can recover approximately 96.0% of the original model's zero-sample inference capability, effectively overcoming the accuracy bottleneck of traditional quantization methods at ultra-low bit depths. Thirdly, through dynamic grid allocation in the underlying representation space, it mathematically simulates a higher-precision protection effect while strictly maintaining a single bit width in physical hardware, avoiding memory fragmentation and dynamic unpacking overhead caused by mixed precision, significantly improving the inference speedup. Finally, the algorithm design is lightweight and does not require expensive retraining. Topology analysis and quantization calibration of 7B and 13B ultra-large parameter models can be completed on a single consumer-grade GPU (such as an RTX 4090).

[0042] This invention is applicable to various scenarios requiring the deployment of ultra-large-scale language models (such as the LLaMA series) in environments with limited GPU memory and computing power. These include, but are not limited to: large model deployment on smartphones, in-vehicle edge computing nodes, inference acceleration using heterogeneous NPU chips, and efficient model hosting on low-cost single-card servers. Attached Figure Description

[0043] Figure 1 This is a flowchart of the method of the present invention.

[0044] Figure 2 This is a system block diagram of the present invention. Detailed Implementation

[0045] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0046] This invention provides a method for quantizing large language models based on a global topology-aware grid, comprising:

[0047] Obtain the weights of the pre-trained large language model to be quantized and the calibration dataset;

[0048] Based on the calibration dataset, the structural sensitivity vector of the weights of each linear layer in the large language model is calculated through forward and backward propagation.

[0049] Extract the diagonal elements of the local Hessian matrix of the activation values ​​of each linear layer input;

[0050] A topological Hessian matrix is ​​generated by exponential fusion calculation based on the structural sensitivity vector and the diagonal elements of the local Hessian matrix.

[0051] Using the topological Hessian matrix as the weight coefficient, non-uniform quantization grid optimization is performed on the weights of the pre-trained large language model, and the optimized grid is mapped to the target bit width to complete the model quantization.

[0052] After model quantization is completed, the process also includes a weight deployment step: mapping the quantized weights to the corresponding grid index values; packaging and storing multiple low-bit indices into high-bit-width integer variables through bit shifting operations; during inference execution, mapping the indices back to the corresponding grid center floating-point values ​​through table lookup operations, and performing matrix multiplication operations with the input activation values.

[0053] This invention also provides a large language model quantization system based on a global topology-aware grid, which performs the method described above, including:

[0054] The data and model loading module is used to import pre-trained large language model weights and calibration datasets.

[0055] The global topology analysis module is used to perform forward-backward propagation based on the calibration dataset and calculate the global structural sensitivity vector of the weights of each layer.

[0056] The local feature extraction module is used to capture the input activation values ​​of each linear layer and calculate the diagonal elements of the local Hessian matrix;

[0057] The matrix exponential fusion module is used to receive the global structure sensitivity vector and the diagonal elements of the local Hessian matrix, and generate the topological Hessian matrix through exponential fusion calculation.

[0058] The topology-aware grid clustering module is used to perform K-means clustering iterations on non-uniformly quantized grids using the topological Hessian matrix as weight coefficients, output quantized grid anchor points, and complete bit-reduction compression replacement of model weights.

[0059] The following is a detailed implementation process of the present invention.

[0060] like Figure 1 As shown, the present invention provides a large language model quantization method based on a global topology-aware grid, the implementation process of which includes the following steps:

[0061] S101: Global topology analysis and structural sensitivity calculation.

[0062] Obtain the pre-trained large model weights to be quantized and calibration dataset Perform a single forward-backward propagation on the calibration set. For the weight matrix of each linear layer... Calculate its global loss gradient. Define the structure sensitivity matrix. The Hadamard product of the global gradient and the absolute weights is given by the following formula:

[0063]

[0064] To align with channel-level metrics, use along the output dimension. By aggregating norms, a one-dimensional structure sensitivity vector is obtained. , where the vector's th element The calculation formula is:

[0065]

[0066] S102: Extraction of Hessian information for local activation.

[0067] For the current layer Obtain its input activation value Calculate the diagonal of the local activation Hessian matrix. As a metric for measuring the numerical variance within a layer, the calculation formula is:

[0068]

[0069] S103: Exponential fusion construction of the topological Hessian matrix.

[0070] To scale local curvature using global topology and combat the exponential cascading of errors at bottlenecks, this invention proposes fusing two metrics using exponential multiplication. The topological Hessian matrix is ​​then calculated. The formula is as follows:

[0071]

[0072] in, This is a topology-aware scalar (hyperparameter) used to adjust the balance between local curvature and global topology. Its exponential form establishes a very strong optimization penalty term at network information flow bottlenecks.

[0073] S104: Non-uniform quantization mesh optimization based on topological Hessian weights.

[0074] Given target bit width (e.g., 3-bit), initialization Each grid center cluster .by As weighting coefficients, a topology-weighted K-means clustering objective function is constructed:

[0075]

[0076] Cluster centers are updated iteratively using the Lloyd algorithm. The weighted channels of the score generate a stronger attraction, making the optimized grid centers denser in the structure-sensitive region (i.e., the quantization interval narrower).

[0077] S105: Model weight deployment and export.

[0078] Original weights Mapped and truncated to the optimized quantization grid In the middle, quantization is completed, and the compressed model weights are exported.

[0079] like Figure 2 As shown, the system of the present invention comprises five interconnected functional modules:

[0080] 1. Data and Model Loading Module (201): Used to import high-precision large language models and small batch calibration text data, and output the data to the topology analysis module.

[0081] 2. Global Topology Analysis Module (202): Contains a backpropagation derivative unit, which calculates the global structural sensitivity vector of each weight in the network based on the idea of ​​zero-cost proxy.

[0082] 3. Local feature extraction module (203): Calculate the local Hessian diagonal of each layer based on the activation value of forward propagation.

[0083] 4. Matrix index fusion module (204): Receives the output of modules (202) and (203), and uses the processor to perform the index fusion calculation in step S103 above to generate the topological Hessian matrix index.

[0084] 5. Topology-aware grid clustering module (205): contains a K-means clustering iterator, which takes the topological Hessian matrix as input weights, outputs non-uniform quantized grid anchors, and performs bit-reduction compression replacement on the original model.

[0085] The following provides a complete closed-loop implementation example of quantization for a large language model (such as LLaMA-3-8B) based on the PyTorch deep learning framework, specifically including the following operation process:

[0086] Phase 1: Calibration Data Preprocessing and Loading. High-precision weights from the open-source LLaMA-3-8B pre-trained model are obtained, and a calibration data stream is constructed: 128 text data entries are randomly extracted from a public corpus (such as WikiText-2). The text is converted into a token sequence using the model's corresponding token segmenter, and then truncated / padded to a uniform fixed sequence length to construct an input tensor batch.

[0087] The second stage involves inputting the calibration tensor into the large model and performing a complete forward and backward propagation. In PyTorch, hook functions are registered with the linear operators of each layer of the model to capture the global gradient matrix of the corresponding weight tensor. The one-dimensional structure sensitivity vector of each output channel is calculated using the Hadamard product formula, and a one-dimensional global structure sensitivity vector is generated using the Hadamard product operation, thus achieving physical perception of the model's global topological bottlenecks in the underlying data flow. Then, during the forward propagation, the input activation matrix data of each linear layer is captured, and the diagonal elements of the local activation Hessian matrix are calculated and extracted using matrix multiplication.

[0088] The third stage involves constructing a non-uniform mesh and a quantized mapping topological Hessian tensor at the code level. On the computation nodes, the extracted local Hessian diagonal and the normalized global sensitivity vector are used to generate a novel topological Hessian tensor in memory according to the fusion rules designed in this invention. This successfully endows the original local curvature features with global topological awareness attributes. To meet the target 3-bit quantization requirement, the K-means algorithm is used to initialize 2... 3 =8 floating-point cluster centers. The topological Hessian tensor with topological awareness is used as the sample weights and input into the clustering function for iterative updates, constructing a topologically aware quantized grid. After clustering convergence, each parameter of the original FP16 weights is mapped to the nearest cluster center, and the corresponding 3-bit index value is obtained. In memory, these index values ​​are flattened, and bitwise operations are used to compactly pack every 8 3-bit indices into three 8-bit integer variables (INT8) for physical storage, while simultaneously saving the FP16 scalar values ​​of the 8 cluster centers.

[0089] Phase 4: Model overloading and zero-shot inference evaluation with custom operator replacement. In the inference code, the standard nn.Linear layer of the original model is replaced with a custom quantized linear layer. This quantized linear layer loads the packaged INT8 index and quantization vocabulary during initialization. During the forward propagation of actual inference, the custom quantized linear layer quickly maps the 3-bit index of the input back to the corresponding FP16 cluster center value through a lookup table operation, and then performs standard matrix multiplication with the input activation value. The overloaded quantized model is then connected to the large model evaluation framework LM-Evaluation Harness to perform inference evaluation on multiple zero-shot commonsense reasoning datasets (such as PIQA and HellaSwag), outputting the final accuracy metric, completing the entire quantization and evaluation loop.

[0090] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for quantizing large language models based on a global topology-aware grid, characterized in that, include: Obtain the weights of the pre-trained large language model to be quantized and the calibration dataset; Based on the calibration dataset, the structural sensitivity vector of the weights of each linear layer in the large language model is calculated through forward and backward propagation. Extract the diagonal elements of the local Hessian matrix of the activation values ​​of each linear layer input; A topological Hessian matrix is ​​generated by exponential fusion calculation based on the structural sensitivity vector and the diagonal elements of the local Hessian matrix. Using the topological Hessian matrix as the weight coefficient, non-uniform quantization grid optimization is performed on the weights of the pre-trained large language model, and the optimized grid is mapped to the target bit width to complete the model quantization.

2. The method for quantizing large language models based on a global topology-aware grid according to claim 1, characterized in that, The calculation of the structure sensitivity vector for the weights of each linear layer specifically includes: Perform a single forward-backward propagation on the calibration dataset to obtain the global gradient of the loss function with respect to the weight matrices of each linear layer; The structural sensitivity matrix is ​​obtained by calculating the Hadamard product of the global gradient and the absolute weights. Perform along the output dimension of the structure sensitivity matrix Norm aggregation yields a one-dimensional structure sensitivity vector.

3. The method for quantizing large language models based on a global topology-aware grid according to claim 2, characterized in that, The formula for calculating the structural sensitivity matrix is ​​as follows: in, Indicates the l-th layer Line 1 The structural sensitivity value of column weights This represents the global gradient of the loss function with respect to the weights. Indicates the l-th layer Line 1 The original weights of the columns, represents the loss function, and ⊙ represents the Hadamard product operator.

4. The method for quantizing large language models based on a global topology-aware grid according to claim 3, characterized in that, The formula for calculating the one-dimensional structural sensitivity vector is as follows: in, Indicates the l-th layer One-dimensional structure sensitivity value for each channel Indicates the output channel dimension.

5. The method for quantizing large language models based on a global topology-aware grid according to claim 1, characterized in that, The specific formula for extracting the diagonal elements of the local Hessian matrix of the input activation values ​​of each linear layer is as follows: in, This represents the diagonal elements of the local Hessian matrix at layer l. Let represent the input activation matrix of the l-th layer, diag(⋅) represents the operation of taking the diagonal of the matrix, and T represents the matrix transpose operation.

6. The method for quantizing large language models based on a global topology-aware grid according to claim 1, characterized in that, The topological Hessian matrix is ​​generated through exponential fusion calculation, and the specific calculation formula is as follows: in, Indicates the l-th layer Topological Hessian value of each channel, Indicates the l-th layer Local Hessian diagonal elements of each channel, Indicates the l-th layer The structural sensitivity value of each channel, For all of this layer The resulting one-dimensional structural sensitivity vector. Here are the preset topology-aware scalar hyperparameters, and exp represents the natural exponential function. This indicates the operation of retrieving the maximum value.

7. The method for quantizing large language models based on a global topology-aware grid according to claim 1, characterized in that, The non-uniform quantized grid optimization employs a topology-weighted K-means clustering algorithm, with the objective function being: in, Indicates the quantization of the central cluster of the grid. Indicates the first There are K cluster centers, where K is the number of grids corresponding to the target bit width. This represents the value of the variable that minimizes the objective function. This represents the original weight values. These are the weighting coefficients of the topological Hessian matrix for the corresponding channel.

8. The method for quantizing large language models based on a global topology-aware grid according to claim 1, characterized in that, The target bit width is 3 bits, and the number K of the quantization grid center clusters is set to... indivual.

9. The method for quantizing large language models based on a global topology-aware grid according to claim 1, characterized in that, After model quantization is completed, the process also includes a weight deployment step: mapping the quantized weights to the corresponding grid index values; packaging and storing multiple low-bit indices into high-bit-width integer variables through bit shifting operations; during inference execution, mapping the indices back to the corresponding grid center floating-point values ​​through table lookup operations, and performing matrix multiplication operations with the input activation values.

10. A large language model quantization system based on a global topology-aware grid, performing the method as described in claims 1-9, characterized in that, include: The data and model loading module is used to import pre-trained large language model weights and calibration datasets. The global topology analysis module is used to perform forward-backward propagation based on the calibration dataset and calculate the global structural sensitivity vector of the weights of each layer. The local feature extraction module is used to capture the input activation values ​​of each linear layer and calculate the diagonal elements of the local Hessian matrix; The matrix exponential fusion module is used to receive the global structure sensitivity vector and the diagonal elements of the local Hessian matrix, and generate the topological Hessian matrix through exponential fusion calculation. The topology-aware grid clustering module is used to perform K-means clustering iterations on non-uniformly quantized grids using the topological Hessian matrix as weight coefficients, output quantized grid anchor points, and complete bit-reduction compression replacement of model weights.

Images