Large language model weight and activation combined quantification method and system

Abstract

The invention discloses a large language model weight and activation combined quantification method and system, and belongs to the technical field of model quantification. The method comprises the following steps: collecting and preprocessing a calibration set, inputting a large language model to execute forward propagation, and recording an activation matrix; for each embedding dimension, counting maximum activation absolute values of all lexical elements on the dimension; generating a global threshold value by combining a quantile statistical method with the global sensitivity coefficient, and determining that the dimension with the maximum activation absolute value exceeding the global threshold value in the dimensions is an outlier dimension; respectively designing scaling factors for the normal dimension and the outlier dimension, and generating a reconstruction weight matrix; using Bayesian-gradient joint optimization to reconstruct a truncation threshold value of the weight; calculating a scaling factor of the reconstructed weight matrix to obtain a reconstructed quantized weight matrix; quantizing the activation matrix of the current layer according to the embedding dimension by applying a scaling factor; and performing multiplication calculation with the reconstructed quantization weight matrix to obtain a multiplication output result in an integer domain, and then performing unified inverse quantization recovery.

Description

Technical Field

[0001] The present invention belongs to the technical field of model quantization, and specifically relates to a method and system for jointly quantifying the weights and activations of a large language model. Background Art

[0002] Large language models (such as GPT-3, PaLM, LLaMA, etc.) with their hundreds of billions of parameters and powerful semantic understanding capabilities have demonstrated excellent performance in tasks such as natural language generation and dialogue systems. However, the huge model scale results in extremely high computational resources and memory occupancy during the inference process. Especially in edge devices, low-latency scenarios, and resource-constrained environments, the high deployment cost severely restricts practical applications. To break through this bottleneck, quantization technology has become a key means to reduce the computational and storage overhead of the model. Its core goal is to map the model weights and activation values from high precision (such as FP32 / FP16) to low precision (such as INT8 / INT4), thereby improving the inference efficiency. Although quantization technology has made certain progress, it still faces significant challenges in the practical application of large language models.

[0003] In the Transformer architecture, the distribution of activation values often has a long-tail characteristic, and the activation values of some dimensions deviate significantly from the mean, forming outliers. The change patterns of different tokens (the smallest processing units in the input sequence, such as words, sub-words, or characters) in each embedding dimension are similar. Although these outliers are rare, they will dominate the result of matrix multiplication, leading to significant errors due to insufficient resolution during direct quantization. To address this problem, existing research has proposed methods such as mixed-precision quantization and mathematical equivalent transformation. The mixed-precision quantization method maintains performance by reserving high precision for outliers, but such methods are difficult to execute efficiently on hardware accelerators, resulting in limited actual acceleration effects; methods based on mathematical equivalent transformation (such as smooth factor allocation) attempt to balance the quantization difficulty of weights and activations. However, the expanded dynamic range after weight scaling may cause the quantization error to exceed the activation error compensation ability, instead exacerbating the performance loss; the per-channel quantization method can reduce the quantization error, but it has compatibility issues with the general matrix multiplication (GEMM) kernel widely used in hardware acceleration and is difficult to achieve efficient computation; in addition, when dealing with the uneven distribution of activation values in different embedding dimensions, existing methods often use static thresholds or fixed strategies and cannot dynamically adapt to the data characteristics of different model layers or tasks, resulting in missed detection or false detection problems and further affecting the quantization accuracy.

[0004] The distribution of the weight matrix is usually relatively concentrated. However, in large language models, the actual weight distribution often contains extreme values, which also restricts the quantization effect in the optimization problem of the dynamic range of the weight matrix. The traditional MinMax quantization method (minimum-maximum mapping quantization method) directly determines the dynamic range based on the weight extreme values. However, the widespread existence of extreme values in large language models leads to the scaling factor being dominated by extreme values, significantly reducing the quantization resolution of normal weight values and significantly increasing the cumulative error. Although some studies have tried to optimize the truncation threshold through the gradient descent method, such methods highly depend on the initial value selection, are prone to falling into local optimal solutions, and it is difficult to guarantee the global optimum. Moreover, the computational overhead introduced in the optimization process further limits its practicality. Summary of the Invention

[0005] Aiming at the deficiencies of the prior art, the purpose of the present invention is to provide a method and system for jointly quantifying the weights and activations of large language models, which solves the problems in the prior art.

[0006] The purpose of the present invention can be achieved through the following technical solutions: A method for jointly quantifying the weights and activations of a large language model, comprising the following steps: Collect a calibration set, preprocess the calibration set, and input it into the large language model to perform forward propagation, and record the activation matrices of each layer; Based on the activation matrices, for each embedding dimension, statistically calculate the maximum absolute value of the activations of all tokens in this dimension; by combining the quantile statistical method with the global sensitivity coefficient, dynamically generate a global threshold, and determine that the dimension with the maximum absolute value of the activation exceeding the global threshold is an outlier dimension; Based on the outlier dimension determination result, design different scaling factors for the normal dimension and the outlier dimension respectively, and apply them to the corresponding rows of the original weight matrix to generate a reconstructed weight matrix; Based on the reconstructed weight matrix, use Bayesian-gradient joint optimization to optimize the truncation threshold of the reconstructed weights; Based on the optimized truncation threshold, calculate the scaling factor of the reconstructed weight matrix, and quantize the reconstructed weights to obtain a reconstructed quantized weight matrix; Load the scaling factors of the normal dimension and the outlier dimension, and quantize the activation matrix of the current layer according to the embedding dimension by applying the scaling factor to obtain a quantized activation matrix; Perform a multiplication calculation on the quantized activation matrix and the reconstructed quantized weight matrix to obtain the multiplication output result in the integer domain, and based on the scaling factor of the reconstructed weight matrix, map the multiplication output result in the integer domain back to the floating-point approximate representation for unified dequantization and recovery.

[0007] Further, the global threshold The calculation formula of is: Among them, is the quantile statistical function; is the global sensitivity coefficient; is the set that activates the maximum absolute value; is the high quantile parameter.

[0008] Furthermore, the construction rule of the reconstructed weight matrix is as follows: Among them, is the unified scaling factor for all normal dimensions; is the independent scaling factor for each outlier dimension; j is the index of the activation matrix embedding dimension and also the index of each row of the original weight matrix and the reconstructed weight matrix; i is the index of each row of the activation matrix, that is, the index of each token; O is the set of outlier dimensions, storing the indices of the outlier dimensions; N is the quantization bit width; is the j th row vector of the reconstructed weight matrix; is the j th row vector of the original weight matrix; is the value of the i th token's j th embedding dimension in the activation matrix; is the maximum absolute value of the activation of all tokens in the normal dimension j .

[0009] Furthermore, the quantization process of the reconstructed weight is as follows: Among them, is the value at the position of the d th row and the k th column of the reconstructed quantized weight matrix; is the value at the position of the d th row and the k th column of the reconstructed weight matrix, which is a floating-point value; and are respectively the lower truncation threshold and the upper truncation threshold in the quantization process of the reconstructed weight matrix; is the scaling factor of the reconstructed weight matrix; N is the quantization bit width; Indicates the execution of a rounding operation; Is an interval truncation function.

[0010] Furthermore, the formula for quantizing the activation matrix of the current layer by applying a scaling factor according to the embedding dimension is: Where, i Is the row index of the activation matrix and the quantized activation matrix, representing the position of the token in the sequence; j Is the column index of the activation matrix and the quantized activation matrix, representing the index of the embedding dimension; Is the value of the i th token in the j th embedding dimension of the quantized activation matrix of the current layer, and this value is an integer; Is the value of the i th token in the j th embedding dimension of the activation matrix of the current layer, and it is a floating-point value; Indicates the execution of a rounding operation; Is the scaling factor of the j th embedding dimension of the activation matrix, O Is the outlier dimension set, which stores the indices of the outlier dimensions. If , that is, j is a normal dimension, then , representing the unified scaling factor of all normal dimensions of the activation matrix. If the dimension , that is, j is an outlier dimension, then , representing the independent scaling factor of each outlier dimension.

[0011] Furthermore, the expression for unified dequantization recovery is: Where, Is the output matrix after unified dequantization recovery; Is the multiplication output result in the integer domain; Is the scaling factor of the reconstructed weight matrix; Is the quantized activation matrix; Is the reconstructed quantized weight matrix.

[0012] A large language model weight and activation joint quantization system, including: Activation matrix acquisition module: Collect the calibration set, preprocess the calibration set, and input it into the large language model to perform forward propagation, and record the activation matrices of each layer; Outlier Dimension Determination Module: Based on the activation matrix, for each embedding dimension, statistically calculate the maximum absolute value of activation of all tokens in this dimension; dynamically generate a global threshold through the quantile statistical method combined with the global sensitivity coefficient, and determine that the dimension with the maximum absolute value of activation exceeding the global threshold is an outlier dimension; Weight Reconstruction Module: Based on the outlier dimension determination result, design different scaling factors for the normal dimensions and outlier dimensions respectively, and apply them to the corresponding rows of the original weight matrix to generate a reconstructed weight matrix; Truncation Threshold Optimization Module: Based on the reconstructed weight matrix, use the Bayesian-gradient joint optimization to optimize the truncation threshold of the reconstructed weight; Weight Quantization Module: Based on the optimized truncation threshold, calculate the scaling factor of the reconstructed weight matrix, and quantize the reconstructed weight to obtain a reconstructed quantized weight matrix; Activation Matrix Quantization Module: Load the scaling factors of the normal dimensions and outlier dimensions, and quantize the activation matrix of the current layer according to the embedding dimension by applying the scaling factor to obtain a quantized activation matrix; Unified Dequantization Module: Multiply the quantized activation matrix and the reconstructed quantized weight matrix to obtain the multiplication output result in the integer domain, and based on the scaling factor of the reconstructed weight matrix, map the multiplication output result in the integer domain back to the floating-point approximate representation for unified dequantization recovery.

[0013] A computer storage medium stores a readable program, which, when run by a processor, can execute a method for jointly quantizing the weights and activations of a large language model as described above.

[0014] An electronic device includes: a processor, a memory, a communication interface, and a communication bus, and the processor, the memory, and the communication interface complete mutual communication through the communication bus; The memory is used to store at least one executable instruction, and the executable instruction causes the processor to execute the operations corresponding to a method for jointly quantizing the weights and activations of a large language model as described above.

[0015] A computer program product includes computer instructions, and the computer instructions instruct a computing device to execute the operations corresponding to a method for jointly quantizing the weights and activations of a large language model as described above.

[0016] Advantages of the present invention: 1. The present invention designs an activation outlier determination mechanism based on dynamic quantile thresholds and a differential scaling factor calculation method, enabling fine-grained quantization of activation dimensions with long-tailed distributions in large language models, thereby effectively suppressing quantization errors on the activation side; further, by combining the joint optimization strategy of Bayesian global search and gradient descent local fine-tuning, the quantization truncation threshold of weights is optimized, enabling dynamic adaptation to the extreme value distribution problem on the weight side and achieving coordinated control of quantization accuracy and hardware execution efficiency.

[0017] 2. The present invention pre-multiplies the activation scaling factor to the weights and implements an offline quantization design, enabling the restoration of accuracy with only one dequantization operation during the inference process. Combined with hardware-friendly INT8 matrix multiplication instructions, it realizes the inference acceleration of large language models with high precision, high throughput, and low latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0019] Figure 1 is a schematic flowchart of the joint quantization method of the present invention; Figure 2 is a schematic flowchart of the activation outlier dimension determination and parameter optimization of the present invention; Figure 3 is a schematic flowchart of the weight quantization truncation threshold optimization of the present invention; Figure 4 is a schematic flowchart of the quantization execution and accuracy restoration in the online inference stage of the present invention; Figure 5 is a schematic diagram of quantization and inference inside a Transformer block of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0021] Embodiment 1 As Figure 1 shown, a method for joint quantization of weights and activations in a large language model, where S1 - S5 is the offline calibration stage and S6 - S7 is the online inference stage; specifically includes the following steps: S1, Collect the calibration set, preprocess the calibration set, and input it into the large language model for forward propagation, and record the activation matrices of each layer; Collect a representative input dataset (calibration set) that covers various input scenarios that the large language model may encounter, and preprocess the calibration set. Input the calibration set into the large language model to be quantized for forward propagation, and record the activation matrices of each layer. The large language model is a deep neural network model constructed based on the Transformer structure, with a stacked structure composed of multiple self-attention modules and feed-forward networks, and its number of parameters is usually in the hundreds of millions to hundreds of billions. It is suitable for tasks such as natural language understanding and text generation. The model can include a Decoder-only (encoder-only) structure (such as GPT-like models), an Encoder-only (decoder-only) structure (such as BERT-like models), or an Encoder-Decoder (encoder-decoder) structure (such as T5-like models), etc., with certain generality and expandability; The specific steps include: S11, Collect and preprocess the calibration set; According to the actual input features of the target task, select representative samples covering a wide range of scenarios from the public corpus or real business data to form the calibration set; perform standardization processing on the data in the calibration set, including operations such as word segmentation, encoding, normalization, and sequence padding length alignment, to ensure that the large language model can directly accept this data for forward propagation and improve sampling representativeness.

[0022] S12, Activation data acquisition; Input the preprocessed calibration set item by item into the original full-precision model (taking 16-bit floating-point precision as an example), and sequentially perform forward propagation calculations along the model structure, and record the current layer activation matrix , where T is the number of tokens, D is the number of embedding dimensions.

[0023] S2, Based on the activation matrix, for each embedding dimension, statistically calculate the maximum absolute value of the activation of all tokens in this dimension; dynamically generate a global threshold through the quantile statistical method combined with the global sensitivity coefficient, and determine that the dimension with the maximum absolute value of the activation exceeding the global threshold is an outlier dimension; As Figure 2 shown, the steps to determine the outlier dimension include: S21, Activation extreme value (maximum absolute value of activation) statistics; Let the current layer activation matrix be , where is the number of tokens, is the number of embedding dimensions. For each embedding dimension d, calculate the maximum absolute value of activation across all tokens: Obtain the set of maximum absolute values of activation , which is used to measure the dynamic amplitude of different embedding dimensions.

[0024] S22, Global threshold calculation; Set the high quantile parameter for automatically determining outlier dimensions and the global sensitivity coefficient , and generate the global threshold according to the following formula : where is the quantile statistical function, which is used to extract the quantile of the maximum absolute value of activation for all dimensions; is the high quantile parameter, which is used to determine the position of the quantile in the global threshold calculation of the activation value distribution. The closer its value is to 100%, the wider the data covered; is the global sensitivity coefficient, which is used to adjust the scaling amplitude of the global threshold and enhance the sensitivity to outliers. When its value is relatively high, it is more sensitive to extreme values.

[0025] S23, Outlier dimension determination; Compare the maximum absolute value of activation corresponding to each dimension with the global threshold to determine whether it is an outlier dimension. Define the outlier dimension set as follows: That is, if the maximum absolute value of activation in a certain dimension j exceeds the global threshold , then it is regarded as an outlier dimension of the activation distribution, and independent scaling processing will be implemented in the subsequent quantization process.

[0026] In this embodiment, considering that different model layers have different tolerances for outlier detection, in order to improve the quantization robustness and generalization performance, a grid search strategy is used to determine the optimal parameter combination in the candidate parameter set for hyperparameter search and optimization. Specifically, let: Enumerate all combinations, and define the optimization objective as: where is the high quantile parameter, and its value range is the discrete set , which is used to determine the position of the quantile in the global threshold calculation of the activation value distribution. The closer its value is to 100%, the wider the data covered; is the global sensitivity coefficient, and its value range is a continuous interval , which is used to adjust the scaling amplitude of the global threshold and enhance the sensitivity to outliers. When its value is relatively high, it is more sensitive to extreme values; is the quantization operation function that maps floating-point inputs to a target integer range (such as INT8); and are the activation matrix and the reconstruction weight matrix respectively; is the original weight matrix; is the mathematical expectation operator, representing the statistical average of all samples in the calibration dataset; is the square of the Frobenius norm of the matrix, defined as the sum of the squares of the matrix elements, and is used to measure the difference between the outputs before and after quantization; is the optimal parameter combination determined by the grid search strategy, with the goal of minimizing the mean squared error (MSE) between the quantized output and the original full-precision output of each layer under the outlier dimension determination.

[0027] S3. Based on the outlier dimension determination results, design different scaling factors for the normal dimension and the outlier dimension respectively, and apply them to the corresponding rows of the original weight matrix to generate the reconstruction weight matrix; The steps to generate the reconstruction weight matrix include: S31. Scaling factor calculation: 1) Scaling factor for the normal dimension: For all non-outlier dimensions, i.e., the normal dimension , first calculate the maximum absolute value of the activation for each dimension: where, represents the activation matrix of the current layer; T is the number of tokens; D is the number of embedding dimensions; is the row index of the activation matrix, representing the position of the token in the sequence; is the column index of the activation matrix, representing the index of the embedding dimension; is the value of the i th token of the j th embedding dimension in the activation matrix of the current layer; is the maximum absolute value of the activation of all tokens in the normal dimension j .

[0028] Take the maximum value among the maximum absolute values of the activation of all normal dimensions, and define the scaling factor for the normal dimension as: where, N is the quantization bit width (e.g., INT8 corresponds to N= 8), since symmetric quantization usually uses intervals for mapping; in all normal dimensions, the scaling factor is unified as ; 2) Scaling factor for outlier dimensions: For each outlier dimension , calculate its independent scaling factor respectively: where is the independent scaling factor for each outlier dimension, which is used to retain the dynamic range of the outlier dimension and avoid precision loss; the independent scaling factors of all outlier dimensions form the set ; S32, calculate the reconstructed weight matrix; To maintain the numerical equivalence of the scaling operation in the inference path, multiply the above scaling factor backward to the corresponding row of the original weight matrix to obtain the reconstructed weight matrix . Its construction rule is: where j is the index of the activation matrix embedding dimension, and also the index of each row of the original weight matrix and the reconstructed weight matrix; is the j th row vector of the reconstructed weight matrix; is the j th row vector of the original weight matrix; the reconstructed weight matrix will perform quantization operations in the offline stage.

[0029] To support the inference stage, in S3, it is necessary to save: (1) The unified scaling factor of all normal dimensions of the activation matrix ; (2) The set composed of the independent scaling factors of each outlier dimension of the activation matrix , where ; (3) The reconstructed weight matrix .

[0030] These parameters will jointly participate in quantization mapping and inference multiplication execution in subsequent steps. By fusing the activation scaling factor into the weights in the offline stage and performing a one-time weight quantization operation offline, only one-sided quantization of the activation is required in the inference stage to perform efficient integer matrix multiplication, further reducing the online repeated calculation overhead.

[0031] S4, based on the reconstructed weight matrix, use Bayesian-gradient joint optimization to optimize the truncation threshold of the reconstructed weights; Such as Figure 3As shown in the figure, to further improve the quantization accuracy of weights and avoid the dynamic range compression problem caused by extreme values, the present invention proposes a truncated threshold search strategy combining Bayesian optimization and gradient fine-tuning. This strategy is divided into two stages: first, use Bayesian optimization to explore the global parameter space to determine a better initial truncated threshold; then, based on the results of Bayesian optimization, perform gradient descent fine-tuning to further reduce the quantization error.

[0032] The specific steps for optimizing the truncated threshold of the reconstructed weights include: S41, initialize the search space; Let the reconstructed weight matrix be , where is the number of input channels, is the number of output channels; define its quantization truncation interval (truncated threshold) as , and are respectively the lower bound and upper bound of the truncated threshold in the quantization process of the reconstructed weight matrix; is the value at the d -th row and k -th column position of the reconstructed weight matrix, restricting each element to fall within this dynamic range: When initializing, set the search interval as: where is the standard deviation of the reconstructed weight matrix ; randomly select m groups of candidate truncated thresholds , respectively perform reconstructed weight quantization and calculate the KL divergence between the reconstructed quantization weight distribution and the reconstructed weight distribution : where is the KL divergence value corresponding to the i -th group of candidate truncated thresholds, used to evaluate the deviation between the quantization reconstructed weight distribution and the reconstructed weight distribution under this candidate truncated threshold; is the KL divergence function, measuring the relative entropy between two probability distributions P and Q .

[0033] The obtained data pairs constitute the initial training dataset : The reconstructed weight quantization adopts the following asymmetric linear mapping: Among them, is the value at the position of the d -th row and k -th column of the reconstructed quantization weight matrix, and this value is an integer falling within the set quantization integer interval, i.e., interval; is the value at the position of the d -th row and k -th column of the reconstructed weight matrix, which is a floating-point value and has incorporated the activation scaling factor; and are respectively the lower truncation threshold and the upper truncation threshold during the quantization of the reconstructed weight matrix; is the scaling factor of the reconstructed weight matrix; N is the quantization bit width (e.g., INT8 corresponds to N = 8); represents performing a rounding operation; is the interval truncation function, which represents performing an interval truncation operation on the quantization result to ensure that the result falls within the set quantization integer interval, i.e., interval.

[0034] S42, Gaussian process surrogate modeling; Construct a Gaussian process surrogate model with KL divergence as the output, and establish a function mapping for the input candidate truncation threshold : Among them, is the KL divergence value corresponding to the candidate truncation threshold , which is used to evaluate the deviation between the quantization reconstruction weight distribution and the reconstruction weight distribution under this candidate truncation threshold; is the Gaussian process modeler, indicating that the output and between any two candidate truncation thresholds satisfy a joint Gaussian distribution; is the mean function, representing the predicted mean of the candidate truncation threshold ; is the covariance function (kernel function), representing the correlation between two candidate truncation thresholds; is another candidate truncation threshold, which, together with , serves as an input pair for the kernel function to establish a covariance matrix.

[0035] The kernel function adopts a radial basis function (RBF kernel): Among them, is the signal variance, which is a hyperparameter of a Gaussian process and controls the amplitude of function fluctuations; is the length scale, which is a hyperparameter of a Gaussian process and adjusts the smoothness of the function; is the Euclidean distance, which is used to measure the similarity between two candidate truncation thresholds; a hyperparameter of the Gaussian process and are optimized by maximum likelihood estimation (MLE), and the objective function is: where, is a vector composed of KL divergence values corresponding to all candidate truncation thresholds, with a dimension of m ×1, where m is the number of candidate truncation thresholds; is the candidate truncation threshold sample set, which is m a set of candidate truncation thresholds used to train the Gaussian process surrogate model; is the transposed vector of; is the covariance matrix generated by the kernel function ; is the inverse of the covariance matrix and is used for likelihood calculation and prediction distribution solution of the Gaussian process; is the weighted quadratic form of the sample error term, representing the sum of squared fitting residuals of the observed KL divergence under the current Gaussian process hyperparameters and l ; is the determinant of the covariance matrix; const is a constant term that can be ignored when solving the maximum likelihood; is the log-likelihood function, representing the log probability density of observing under the condition of the given candidate truncation threshold sample set and l and the Gaussian process hyperparameters ;

[0036] S43, generation and iterative update of new candidate truncation thresholds; Based on the Gaussian process surrogate model, the "Expected Improvement (EI)" criterion is used as the sampling strategy to select the next set of candidate truncation thresholds that may improve the current optimal result: where, is the current candidate truncation threshold The expected improvement value corresponding to the Gaussian process surrogate model, that is, the potential improvement benefit of the current point compared to the historical optimal result; is the mathematical expectation operator, which represents the expectation of the average return under the uncertainty distribution (Gaussian process prediction distribution); is the current historical optimal KL divergence value; Candidate cutoff threshold for Gaussian process surrogate model T The predicted value of KL divergence at ; solved using the L-BFGS method (limited memory quasi-Newton method): Cutoff threshold for new candidates Perform reconstruction weight quantization and KL divergence calculation to update the data set , refit the Gaussian process surrogate model: Repeat the above process with the upper limit of iteration set to 50 times to obtain a new candidate cutoff threshold that converges.

[0037] S44, gradient descent fine-tuning; From the Bayesian search results (the final dataset obtained above ) select the first one with the lowest KL divergence value P interval , fine-tuning is performed with the mean square error between the reconstructed quantization weight matrix and the reconstructed weight matrix as the objective function: Compute the gradient: , Perform a gradient descent update: , in, is the learning rate, It is a partial derivative operation; at the candidate cutoff threshold Gradient descent is performed until the loss function converges or the maximum number of iterations is reached; finally, the Minimum cutoff threshold .

[0038] S5, based on the optimized truncation threshold, calculating the scaling factor of the reconstruction weight matrix, and quantizing the reconstruction weight to obtain a reconstruction quantized weight matrix; The steps of obtaining the reconstructed quantization weight matrix include: S51, based on the cutoff threshold determined in step 4 , calculate the reconstruction weight matrix Scaling factor is as follows: Among them, N represents the target quantization bit width (e.g., INT8 corresponds to N = 〖8)〗, and the scaling factor is used to linearly map the reconstructed weight value to interval. The specific weight quantization process is as follows: Among them, is the value at the position of the d -th row and the k -th column of the reconstructed quantization weight matrix. This value is an integer and falls within the set quantization integer interval, that is, the interval of [0, ; is the value at the position of the d -th row and the k -th column of the reconstructed weight matrix. It is a floating-point value and has incorporated the activation scaling factor; is the scaling factor of the reconstructed weight matrix, N is the target quantization bit width (e.g., INT8 corresponds to N = 〖8)〗; represents performing a rounding operation; is the interval truncation function, indicating to perform an interval truncation operation on the quantization result to ensure that the result falls within the integer range.

[0039] S52, Parameter saving: For supporting the decoding and dequantization operations in the online inference process, the following quantization meta-information needs to be persistently saved to the deployment file in S5: 1) Reconstructed quantization weight matrix: , stored in integer format (e.g., INT8); 2) Scaling factor of the reconstructed weight matrix : ; 3) Outlier dimension marking table: used to indicate whether independent scaling processing is used in the inference stage.

[0040] S6, Load the scaling factors of the normal dimension and the outlier dimension, and apply the scaling factor to the activation matrix of the current layer according to the embedding dimension to obtain the quantized activation matrix; Assume the activation matrix of the current layer is , among which, is the number of tokens, is the number of embedding dimensions. Quantize the activation matrix of the current layer according to the embedding dimension (column) by applying the scaling factor. The quantization formula is: Among them, i is the row index of the activation matrix and the quantized activation matrix, representing the position of the token in the sequence; j is the column index of the activation matrix and the quantized activation matrix, representing the index of the embedding dimension; is the value of the i -th token in the j -th embedding dimension of the quantized activation matrix of the current layer, and this value is an integer; is the value of the i -th token in the j -th embedding dimension of the activation matrix of the current layer, and it is a floating-point value; represents performing a rounding operation; is the scaling factor of the j -th embedding dimension of the activation matrix, O is the set of outlier dimensions, storing the indices of the outlier dimensions. If , that is, j is a normal dimension, then , representing the unified scaling factor of all normal dimensions of the activation matrix. If the dimension , that is, j is an outlier dimension, then , representing the independent scaling factor of each outlier dimension.

[0041] The obtained result is an integer tensor (quantized activation matrix) , which can be directly involved in the subsequent INT8 matrix multiplication calculation; this activation quantization process effectively adapts to the dynamic changes of the activation distribution through a dimension-aware scaling strategy, combines an outlier independent processing mechanism, further reduces the quantization error, and provides accuracy guarantee for the subsequent integer-domain matrix multiplication module.

[0042] S7. Multiply the quantized activation matrix and the reconstructed quantized weight matrix ( ) to obtain the multiplication output result in the integer domain, and based on the scaling factor ( ) of the reconstructed weight matrix, map the multiplication output result in the integer domain back to the floating-point approximate representation for unified dequantization recovery; To achieve high-throughput integer multiplication and controllable precision restoration during the inference process, the present invention adopts a unified dequantization mechanism. As Figure 4 shown, after completing the low-precision matrix multiplication calculation, the output tensor is restored to the floating-point approximate representation by fusing the scaling factor. The specific steps include: S71. Low-precision calculation; Invoke the integer matrix multiplication kernel supported by underlying hardware instructions (such as AVX-512, CUDA Tensor Core, or ARM NEON), and perform multiplication on the quantized activation matrix and the reconstructed quantized weight matrix using the INT8 data type: Wherein, is the quantized activation matrix; is the reconstructed quantized weight matrix; is the multiplication output result in the integer domain; this step is executed entirely at INT8 precision, with extremely high parallel efficiency and cache utilization.

[0043] S72, unified dequantization recovery; Map the multiplication output result in the integer domain back to the floating-point approximate representation, using the unified scaling recovery formula: Where is the output matrix after unified dequantization recovery; is the scaling factor of the reconstructed weight matrix (i.e., in S5). Since the present invention pre-multiplies the activation scaling factor into the original weight matrix, only the dequantization step on the weight side needs to be retained in the inference path, expressed as follows: Wherein, is the operation of fusing the activation scaling factor into the weight; is the diagonal matrix composed of the scaling factors of each column of the activation matrix; is the output matrix after unified dequantization recovery, for use by subsequent layers or task modules. This unified dequantization design simplifies the integer-domain inference path, effectively avoids multiple floating-point multiplications, and improves the online execution efficiency and system stability.

[0044] Figure 5 This is the quantization and inference schematic diagram inside a Transformer block in the present invention. The activation matrix X First, after normalization, it is split into three projection paths of Q, K, and V. The original weight matrix of each path (such as ) is first reconstructed through the activation scaling factor, and then offline quantization is performed (the part within the left dashed box in the figure is the weight reconstruction and quantization in the offline stage). The activation matrix is also quantized to , enabling subsequent activation and weight multiplication operations to be completed in the integer domain, and restoring the accuracy through the scaling factor of the reconstructed weight matrix. The attention output result (operations such as Softmax and residual connection are not shown in the figure, where Softmax is the normalized exponential function used to convert attention scores into probability distributions) is then normalized and passed into the linear layer of the feed-forward neural network part (only one linear layer is shown in the figure), and the quantized matrix multiplication and inverse quantization restoration are also performed.

[0045] Based on a similar inventive concept, an embodiment of the present invention also provides a computer storage medium storing a readable program that, when run by a processor, can execute the above-mentioned method for jointly quantizing weights and activations of a large language model.

[0046] Based on a similar inventive concept, an embodiment of the present invention provides an electronic device, including: a processor, a memory, a communication interface, and a communication bus, and the processor, the memory, and the communication interface complete communication with each other through the communication bus; The memory is used to store at least one executable instruction, and the executable instruction causes the processor to execute the operations corresponding to the above-mentioned method for jointly quantizing weights and activations of a large language model.

[0047] Based on a similar inventive concept, an embodiment of the present invention also provides a computer program product including computer instructions that direct a computing device to execute the operations corresponding to the above-mentioned method for jointly quantizing weights and activations of a large language model.

[0048] Embodiment 2 Based on the method for jointly quantizing weights and activations of a large language model proposed in Embodiment 1, in this embodiment, a system for jointly quantizing weights and activations of a large language model is proposed, as Figure 1 shown, specifically including: Activation matrix acquisition module: Collect a calibration set, preprocess the calibration set, and input it into the large language model to perform forward propagation, and record the activation matrices of each layer; Outlier dimension determination module: Based on the activation matrix, for each embedding dimension, statistically calculate the maximum absolute value of the activation of all tokens in this dimension; dynamically generate a global threshold through the quantile statistical method combined with the global sensitivity coefficient, and determine that the dimension with the maximum absolute value of the activation exceeding the global threshold is an outlier dimension; Weight reconstruction module: Based on the outlier dimension determination result, design different scaling factors for the normal dimension and the outlier dimension respectively, and apply them to the corresponding rows of the original weight matrix to generate a reconstructed weight matrix; Truncation threshold optimization module: Based on the reconstructed weight matrix, use Bayesian-gradient joint optimization to optimize the truncation threshold of the reconstructed weight; Weight quantization module: Based on the optimized truncation threshold, calculate the scaling factor of the reconstructed weight matrix, and quantize the reconstructed weights to obtain the reconstructed quantized weight matrix; Activation matrix quantization module: Load the scaling factors for the normal dimension and the outlier dimension, and quantize the activation matrix of the current layer by applying the scaling factors according to the embedding dimension to obtain the quantized activation matrix; Unified dequantization module: Multiply the quantized activation matrix and the reconstructed quantized weight matrix to obtain the multiplication output result in the integer domain, and based on the scaling factor of the reconstructed weight matrix, map the multiplication output result in the integer domain back to the floating-point approximate representation for unified dequantization recovery.

[0049] The method of the present invention can be implemented in hardware, firmware, or be implemented as software or computer code that can be stored in a recording medium (such as a CDROM, RAM, floppy disk, hard disk, or magneto-optical disk), or be implemented as computer code originally stored in a remote recording medium or a non-transitory machine-readable medium and downloaded through a network and to be stored in a local recording medium, so that the method described herein can be stored on such a software process on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or an FPGA). It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component (such as RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown herein.

[0050] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and what is described in the above embodiments and the specification only illustrates the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed.

Claims

1. A method for jointly quantizing large language model weights and activations, characterized in that, It includes the following steps: Collect a calibration set, preprocess the calibration set, and input it into the large language model to perform forward propagation, and record the activation matrices of each layer; Based on the activation matrices, for each embedding dimension, statistically calculate the maximum absolute value of the activation of all tokens in this dimension; by combining the quantile statistical method with the global sensitivity coefficient, dynamically generate a global threshold, and determine that the dimension with the maximum absolute value of the activation exceeding the global threshold is an outlier dimension; Based on the outlier dimension determination result, design different scaling factors for the normal dimension and the outlier dimension respectively, and apply them to the corresponding rows of the original weight matrix to generate a reconstructed weight matrix; Based on the reconstructed weight matrix, use Bayesian-gradient joint optimization to optimize the truncation threshold of the reconstructed weight; Based on the optimized truncation threshold, calculate the scaling factor of the reconstructed weight matrix, and quantize the reconstructed weight to obtain a reconstructed quantized weight matrix; Load the scaling factors of the normal dimension and the outlier dimension, and quantize the activation matrix of the current layer according to the embedding dimension by applying the scaling factor to obtain a quantized activation matrix; Perform a multiplication calculation on the quantized activation matrix and the reconstructed quantized weight matrix to obtain a multiplication output result in the integer domain, and based on the scaling factor of the reconstructed weight matrix, map the multiplication output result in the integer domain back to a floating-point approximate representation for unified dequantization recovery.

2. The method for jointly quantizing large language model weights and activations according to claim 1, characterized in that Global threshold The calculation formula is as follows: Among them, is the quantile statistical function; is the global sensitivity coefficient; is the activation maximum absolute value set; is the high quantile parameter.

3. A method for jointly quantizing large language model weights and activations according to claim 1, characterized in that, Reconstruct the weight matrix The construction rule is as follows: Among them, is the unified scaling factor for all normal dimensions; is the independent scaling factor for each outlier dimension; j is the index of the activation matrix embedding dimension, and also the index of each row of the original weight matrix and the reconstructed weight matrix; i is the index of each row of the activation matrix, that is, the index of each token; O is the set of outlier dimensions, storing the indices of the outlier dimensions; N is the quantization bit width; is the j th row vector of the reconstructed weight matrix; is the j th row vector of the original weight matrix; is the value of the i th token in the j th embedding dimension of the activation matrix; is the maximum absolute value of the activation of all tokens in the normal dimension j ; 4. A method for jointly quantizing large language model weights and activations according to claim 3, characterized in that, The quantization process of the reconstructed weight is: Wherein, is the value at the d -th row and k -th column positions of the reconstructed quantization weight matrix; is the value at the d -th row and k -th column positions of the reconstructed weight matrix, which is a floating-point value; and are respectively the lower truncation threshold and the upper truncation threshold during the quantization process of the reconstructed weight matrix; is the scaling factor of the reconstructed weight matrix; N is the quantization bit width; indicates performing a rounding operation; is the interval truncation function.

5. A method for jointly quantizing large language model weights and activations according to claim 4, characterized in that, The formula for quantizing the activation matrix of the current layer according to the embedding dimension by applying the scaling factor is: Among them, i is the row index of the activation matrix and the quantized activation matrix, representing the position of the token in the sequence; j is the column index of the activation matrix and the quantized activation matrix, representing the index of the embedding dimension; is the value of the i -th token in the j -th embedding dimension of the quantized activation matrix of the current layer, and this value is an integer; is the value of the i -th token in the j -th embedding dimension of the activation matrix of the current layer, and it is a floating-point value; represents performing a rounding operation; is the scaling factor of the j -th embedding dimension of the activation matrix, O is the set of outlier dimensions, storing the indices of the outlier dimensions. If , that is, j is a normal dimension, then , representing the unified scaling factor of all normal dimensions of the activation matrix. If the dimension , that is, j is an outlier dimension, then , representing the independent scaling factor of each outlier dimension.

6. A method for jointly quantizing large language model weights and activations according to claim 5, characterized in that, The expression for unified dequantization recovery is: Among them, is the output matrix after unified dequantization and restoration; is the multiplication output result in the integer domain; is the scaling factor of the reconstructed weight matrix; is the quantized activation matrix; is the reconstructed quantized weight matrix.

7. A large language model weight and activation joint quantization system, characterized in that, It includes: Activation matrix acquisition module: Collect a calibration set, preprocess the calibration set, and input it into the large language model to perform forward propagation, and record the activation matrices of each layer; Outlier dimension determination module: Based on the activation matrices, for each embedding dimension, statistically calculate the maximum absolute value of the activation of all tokens in this dimension; by combining the quantile statistical method with the global sensitivity coefficient, dynamically generate a global threshold, and determine that the dimension with the maximum absolute value of the activation exceeding the global threshold is an outlier dimension; Weight reconstruction module: Based on the outlier dimension determination result, design different scaling factors for the normal dimension and the outlier dimension respectively, and apply them to the corresponding rows of the original weight matrix to generate a reconstructed weight matrix; Truncation threshold optimization module: Based on the reconstructed weight matrix, use Bayesian-gradient joint optimization to optimize the truncation threshold of the reconstructed weight; Weight quantization module: Based on the optimized truncation threshold, calculate the scaling factor of the reconstructed weight matrix, and quantize the reconstructed weight to obtain a reconstructed quantized weight matrix; Activation matrix quantization module: Load the scaling factors of the normal dimension and the outlier dimension, and quantize the activation matrix of the current layer according to the embedding dimension by applying the scaling factor to obtain a quantized activation matrix; Unified dequantization module: Perform a multiplication calculation on the quantized activation matrix and the reconstructed quantized weight matrix to obtain a multiplication output result in the integer domain, and based on the scaling factor of the reconstructed weight matrix, map the multiplication output result in the integer domain back to a floating-point approximate representation for unified dequantization recovery.

8. A computer storage medium stores a readable program, characterized in that, When the program is run by a processor, it can execute a method for jointly quantizing the weights and activations of a large language model as described in any one of claims 1-6.

9. An electronic device, characterized in that, It includes: A processor, a memory, a communication interface, and a communication bus, where the processor, the memory, and the communication interface complete communication with each other through the communication bus; The memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to a method for jointly quantizing large language model weights and activations according to any one of claims 1-6.

10. A computer program product comprising computer instructions, characterized in that, The computer instruction instructs the computing device to perform operations corresponding to a method for jointly quantizing large language model weights and activations according to any one of claims 1-6.

Images