Low-privilege weight quantization method and system for neural network model inference deployment

Abstract

The application relates to the technical field of artificial intelligence model compression and low-bit inference deployment, and discloses a low-bit weight quantization method and system based on generalized Gaussian distribution and row-by-row Fisher correction and a storage medium. The method performs generalized Gaussian distribution fitting on a weight matrix of a target neural network layer, constructs a zero-inclusion quantization codebook and performs block-by-block quantization, calculates diagonal Fisher information based on calibration loss, solves a scalar scale factor row by row to correct the quantized weight, and outputs the codebook index and the scaling factor for inference deployment. Under the condition of a small number of calibration samples (N<=16), the scheme is beneficial to reducing low-bit quantization error, reducing calibration and correction overhead, and improving model deployment stability and inference availability.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence model compression and low-bit inference deployment technology, and in particular to a low-bit weight quantization technology for neural networks based on generalized Gaussian distribution fitting, zero-inclusion quantization codebook, and line-by-line Fisher information correction. Background Technology

[0002] As large language models, visual language models, and multimodal models are increasingly deployed in cloud inference, enterprise private deployments, and edge device applications, the storage footprint, GPU memory capacity, and memory access bandwidth of model weights are becoming crucial factors influencing inference costs and deployment scope. For example, in services targeting intelligent question answering, code generation, or multimodal retrieval, the same model often needs to handle a large number of concurrent requests on limited GPU resources. In enterprise local knowledge bases and edge intelligent devices, models also need to maintain usable inference quality under relatively limited storage and computing power conditions. Therefore, post-training low-bit weight quantization has become a common technique. Its task is to convert floating-point weights into low-bit weight representations, codebook indices, and scaling factors for subsequent inference deployment without retraining the model as much as possible.

[0003] Existing low-bit quantization methods typically employ fixed codebooks, nearest rounding, group scaling, activation-aware scaling, or Hessian-based compensation strategies. While these methods can reduce model storage and memory access overhead to some extent, they still have several limitations in the aforementioned practical deployment scenarios: Firstly, fixed distribution assumptions or fixed low-bit codebooks are difficult to adapt to the differences in the marginal distribution of weights in different neural network layers, easily leading to large initial quantization errors; secondly, complex Hessian or matrix compensation methods usually rely on more calibration statistics and higher computational overhead, resulting in insufficient stability in scenarios with few samples and rapid deployment; furthermore, simple block-by-block quantization often lacks lightweight post-processing capabilities for sensitive weight errors, making it difficult to balance quantization accuracy, deployment efficiency, and format compatibility. Therefore, there is an urgent need for a low-bit weight quantization technique for neural network model inference deployment, which can improve the matching degree between quantized weights and original weights with fewer calibration samples and lower correction complexity, and reduce the impact of error accumulation in low-bit deployment. Summary of the Invention

[0004] The purpose of this application is to provide a low-bit weight quantization method, system, electronic device, and computer-readable storage medium based on generalized Gaussian distribution and row-by-row Fisher correction, so as to improve the matching degree between low-bit quantized weights and original weights with fewer calibration samples and lower correction complexity, and reduce the impact of error accumulation in low-bit deployment.

[0005] This application provides a low-bit weight quantization method for deploying neural network model inference, including: Step S1: Fit a generalized Gaussian distribution to the weight matrix of the target neural network layer to obtain the shape parameters of the weight matrix. and scale parameters ; Step S2, based on the shape parameters and the scale parameters Construct a zero-inclusive quantization codebook, wherein the zero value is fixed as one codebook entry of the zero-inclusive quantization codebook, and the remaining codebook entries are symmetrically distributed in the positive and negative half-intervals; Step S3: Divide the weight matrix into multiple blocks by row. For each block, based on the block scale factor, map the weights within the block to the nearest codebook entry in the zero-inclusive quantization codebook to obtain a block-by-block quantization weight matrix. Step S4: Input the calibration sample into the neural network model to perform forward propagation, and calculate the diagonal Fisher information of the target neural network layer based on the calibration loss; Step S5: Using the diagonal Fisher information as weights, for each row of the weight matrix, solve for the scalar scaling factor that minimizes the Fisher weighted reconstruction error between the original weights of that row and the block quantization weights scaled by the scalar scaling factor, and scale the block quantization weights of that row using the scalar scaling factor to obtain the corrected quantization weight matrix. Specifically, the corrected quantization weight matrix, along with the corresponding codebook index and scaling factor, is output for low-bit inference deployment of the neural network model.

[0006] In a preferred embodiment, in step S1, the shape parameters are obtained by performing maximum likelihood estimation or moment estimation on random subsamples of the weight matrix. and the scale parameters The number of elements in the random subsample is no more than The probability density function of the generalized Gaussian distribution is: in, For weight values, for The absolute value, The scale parameter is... For the shape parameters, It is the Gamma function; In step S2, based on the shape parameters and the scale parameters The non-zero codebook entries of the zero-containing quantized codebook are determined through Lloyd-Max iteration, and for the tail region exceeding the standard deviation range determined based on the generalized Gaussian distribution parameters, the minimum spacing threshold is applied. Tail homogenization is performed, among which This is the lower limit of the spacing between adjacent codebook entries at the tail of the zero-containing codebook; In step S3, the block size of each block Preferably, the block scale factor is 64, 128, or 256. The block scale factor is determined based on the maximum absolute value of the weights within the block. It is used to scale the weights within the block to the value range covered by the zero-inclusive quantized codebook, and then map them to the nearest codebook entry.

[0007] In a preferred embodiment, in step S4: The number of calibration samples satisfy ; The calibration loss is at least one of the following: next token prediction cross-entropy loss, layer output reconstruction error, KL divergence, or teacher model output error; The diagonal Fisher information includes the diagonal Fisher information of each column of the target neural network layer. In the column-sharing statistical method, By number of valid tokens and the number of rows in the weight matrix On average: in, For the first Fisher information on the diagonal of the column, For the first The calibration loss corresponding to each valid token For the weight matrix of the first... Line 1 Column weights The number of valid tokens included in the statistics. The number of rows in the weight matrix; When the gradient corresponding to a certain token is NaN, Inf, or exceeds the gradient clipping threshold When this happens, the token is removed and the diagonal Fisher information is recalculated. .

[0008] In a preferred embodiment, in step S5, for each row of the weight matrix... Calculate row cross-correlation With effective signal energy : in, For the target neural network layer number 1 Fisher information on the diagonal of the column, For the weight matrix of the first... Line 1 The original weights of the columns, For the block-by-block quantization weight matrix, the first... Line 1 Quantitative weights of columns This indicates all columns in that row. Summation; when Greater than the lower limit threshold of the denominator , Greater than 0 and When the calculation result is not NaN or Inf, let the scalar scaling factor... Otherwise, let Without correcting the line, the "otherwise" includes , The block-by-block quantization weights mentioned in this line are all zero, or The calculation result is either NaN or Inf; and the scalar scaling factor Cut off to ,in and Preferably 0.90 and 1.10, or for the attention projection layer, MLP projection layer, or output layer. ;in, The lower threshold value is the effective signal energy of the row. , These are the lower and upper bounds of the scalar scaling factor, respectively.

[0009] In a preferred embodiment, the method further includes: merging the scalar scaling factor of each row after correction with the block scaling factor of each block in that row in step S3 to obtain a merged block scaling factor; the corrected quantization weight matrix is ​​packaged into INT4 format, NF4 format, custom 4-bit format or NVFP4 compatible format; when using NVFP4 compatible format, the merged block scaling factor is projected into a value that can be represented in E4M3 format.

[0010] In a preferred embodiment, step S6 is further included: according to the forward propagation order of the neural network model, steps S1 to S5 are executed sequentially for each neural network layer; after the quantization of the current neural network layer is completed, the forward propagation is re-executed using the corrected quantization weight matrix of the current neural network layer, and its output is used to update the calibration activation of the next neural network layer, so that the calibration activation used by the next neural network layer when executing steps S1 to S5 includes the quantization error of the current neural network layer.

[0011] In a preferred embodiment, for the NVFP4 format or a compatible low-precision floating-point format, during the codebook construction in step S2 and the mapping in step S3, a local candidate set is maintained for each weight. The candidate center of the local candidate set is obtained from rounding to the nearest, initialization based on quantization error compensation, format-aware adaptive rounding, or initialization based on multiple candidate code points, and the candidate radius is 1 or 2. Gumbel-Softmax is used to learn soft allocation on the local candidate set according to the following formula. : in, For the first Learnable assigned weights for each candidate key entry It is Gumbel noise. For temperature parameters, For the first One candidate FP4 codebook entry , This represents the summation of candidates within the local candidate set; after training, the soft allocation is performed using argmax. Harden the codebook index to a discrete format and make the block scale factor consistent with the deployment format in the training forward pass-through estimator using the E4M3 format.

[0012] This application provides a low-bit weight quantization system for deploying neural network model inference, comprising: The distribution fitting module is used to fit a generalized Gaussian distribution to the weight matrix of the target neural network layer to obtain the shape parameters of the weight matrix. and scale parameters ; The codebook construction module is used to construct the codebook based on the shape parameters. and the scale parameters Construct a zero-inclusive quantization codebook, wherein the zero value is fixed as one codebook entry of the zero-inclusive quantization codebook, and the remaining codebook entries are symmetrically distributed in the positive and negative half-intervals; The block-by-block quantization module is used to divide the weight matrix into multiple blocks by rows, and for each block, based on the block scale factor, map the weights within the block to the nearest codebook entry in the zero-inclusive quantization codebook to obtain the block-by-block quantization weight matrix. The Fisher information calculation module is used to input calibration samples into the neural network model to perform forward propagation, and calculate the diagonal Fisher information of the target neural network layer based on the calibration loss; The row-by-row scaling correction module is used to calculate, for each row of the weight matrix, a scalar scaling factor that minimizes the Fisher weighted reconstruction error between the original weight of that row and the block-by-block quantization weight after scaling by the scalar scaling factor, using the diagonal Fisher information as weights, and then scaling the block-by-block quantization weight of that row with the scalar scaling factor to obtain the corrected quantization weight matrix. The output module is used to output the corrected quantization weight matrix and the corresponding codebook index and scaling factor for low-bit inference deployment of the neural network model.

[0013] This application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the above-described method.

[0014] This application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0015] By adopting the above technical solution, a relatively complete technical effect chain can be formed around the input, processing and output of low bit weight quantization.

[0016] The GGD fitting in step S1 helps to: accurately capture the shape and scale characteristics of the weight distribution by parametrically modeling the true marginal distribution of the weight matrix, thus laying an accurate distribution foundation for subsequent codebook construction. Compared to traditional Gaussian or Laplace assumptions, GGD parameterization can more accurately characterize the peak-heavy-tailed characteristics of the weights in large language models, thereby reducing the initial distribution matching error during the codebook design stage.

[0017] The zero-inclusion Lloyd-Max codebook construction in step S2 helps to: design the codebook under GGD distribution constraints through information-theoretically optimal Lloyd-Max iterations, and fix zero values ​​as codebook entries, so that the density distribution of the codebook matches the true distribution of the weights. The non-uniform distribution characteristic of the codebook, which is dense in the center and sparse at the tail, can efficiently represent a large number of near-zero weights with fewer code points. This step also improves the numerical stability under high dynamic range through a tail homogenization strategy, reducing the initial error of block-by-block quantization compared to a uniform codebook or a fixed Gaussian quantization codebook.

[0018] The block-by-block scaling and mapping in step S3 helps to: adaptively handle the dynamic range differences between different blocks by dynamically scaling based on the maximum absolute value within the block, avoid the precision loss caused by over-quantization of some low-amplitude blocks, and maintain block-level storage and computation efficiency, so that the four-bit representation can achieve a good balance between precision and efficiency within a limited range.

[0019] The calculation of diagonal Fisher information in step S4 is helpful: through a small number of calibration samples ( This allows for the rapid acquisition of information about the weights' sensitivity to the loss. Compared to the full Hessian approach... Storage and The computational complexity is much lower than that of the diagonal Fisher, which only requires... This provides a measure of weight importance for subsequent row-by-row correction. In particular, in the high-dimensional space of large models, the diagonal Fisher information shared by this column fully captures the main contribution of the weights to the loss, while off-diagonal elements are easily affected by sampling noise in finite samples.

[0020] The row-by-row scalar scaling correction in step S5 helps to: solve for a scalar scaling factor that minimizes the objective function for each row using diagonal Fisher information as weights; perform lightweight post-processing on the block quantization results at the row-level granularity; thus, weights more sensitive to loss receive more attention, while avoiding over-correction of insensitive weights. The closed-form solution of this scaling factor ( This method is computationally efficient, and multiple protection mechanisms, including denominator protection, outlier removal, zero-row degradation, and scale truncation, collectively improve numerical stability under conditions of weak signal rows and very few samples, reducing the risk of noise Fisher statistics negatively impacting accuracy. Compared to full Hessian or SVD matrix compensation methods, this row-level scalar correction maintains accuracy while reducing deployment complexity.

[0021] The progressive update in step S6 helps to decompose the global optimization problem of multi-layer quantization into subproblems that can be solved step by step by processing layer by layer in the order of forward propagation. This allows subsequent layers to continue to perform distribution fitting, codebook construction, and correction on activations that include accumulated quantization errors from upstream layers. This layer-by-layer propagation mechanism enables the Fisher information calculation of subsequent layers to reflect the impact of upstream errors on the sensitivity of their weights, thereby improving the overall stability and accuracy consistency of multi-layer quantization and having a significant effect on suppressing error accumulation compared to independent quantization of each layer.

[0022] Row-scale merging and deployment format processing help to: merge row-by-row scalar scale factors into block scale factors, so that row-level correction can be completed on the deployment side without additional storage fields through a single block scale factor. In NVFP4 or other low-precision formats, the merged block scale is further projected into an E4M3 representable value, achieving a unified lightweight numerical correction and deployment format compatibility, and avoiding additional deployment overhead.

[0023] In summary, the overall technical solution defined in this application achieves improvements over existing post-training quantization methods in multiple dimensions, including weight marginal distribution matching, initial codebook design, few-sample sensitivity estimation, numerical stability of row-level correction, and merging of multi-layer error propagation and zero-overhead deployment. This helps to reduce low-bit weight quantization errors and improve inference stability under conditions of very few calibration samples.

[0024] The specification of this application contains numerous technical features distributed across various technical solutions. Listing all possible combinations of these technical features (i.e., technical solutions) would make the specification excessively lengthy. To avoid this problem, the various technical features disclosed in the above-described invention, the various technical features disclosed in the following embodiments and examples, and the various technical features disclosed in the accompanying drawings can be freely combined to form various new technical solutions (all of which are considered to have been described in this specification), unless such a combination of technical features is technically infeasible. For example, one example discloses feature A+B+C, and another example discloses feature A+B+D+E. Features C and D are equivalent technical means that serve the same function, and technically only one needs to be used; they cannot be used simultaneously. Feature E can technically be combined with feature C. Therefore, the solution A+B+C+D should not be considered as described because it is technically infeasible, while the solution A+B+C+E should be considered as described. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating a low-bit-weighted quantization method based on generalized Gaussian distribution fitting, zero-inclusion quantization codebook, block-by-block quantization, and line-by-line Fisher correction according to an embodiment of this application.

[0026] Figure 2 This is a schematic diagram of a low-bit weight quantization system for deploying neural network model inference according to an embodiment of this application. Detailed Implementation

[0027] In the following description, many technical details are presented to help the reader better understand this application. However, those skilled in the art will understand that the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments.

[0028] Explanation of some concepts Low-bit weight quantization refers to the process of converting floating-point weights in a neural network model into low-bit codebook indices, quantized weights, and scaling factors to reduce model storage usage and inference memory access overhead.

[0029] The target neural network layer refers to the neural network layer to which low-bit weight quantization is to be performed, and exemplary examples include linear layers, attention projection layers, MLP projection layers, word embedding layers, or output layers in the Transformer architecture model.

[0030] weight matrix , refers to the original floating-point weight matrix to be quantized in the target neural network layer, where Indicates the first Line 1 The original weights of the columns.

[0031] Generalized Gaussian distribution fitting refers to using the generalized Gaussian distribution to parametrically model the marginal distribution of weight values ​​in the weight matrix in order to obtain parameters that describe the shape of the distribution.

[0032] Shape parameters , refers to the parameter used in the generalized Gaussian distribution to describe the sharpness of the peaks and the thickness of the tails of the weighted distribution.

[0033] Scale parameters , refers to the parameter in the generalized Gaussian distribution used to describe the overall breadth or scale of the weight distribution.

[0034] Zero-inclusive quantization codebook refers to a low-bit quantization codebook that fixes the zero value as a codebook entry and sets the other codebook entries in the positive and negative half-ranges.

[0035] Lloyd-Max iteration refers to the codebook construction process that reduces quantization error by updating the quantization boundary and codebook core under a given weight distribution.

[0036] Block Scale Factor , refers to the scaling factor used when normalizing or dequantizing a weight block in the weight matrix, and is usually used to adapt the local dynamic range of different weight blocks.

[0037] Block-by-block quantization weight matrix This refers to the quantized weight matrix obtained by dividing the weight matrix into multiple blocks by rows and mapping the weights in each block to the quantized codebook.

[0038] Calibration samples refer to sample data input into the neural network model during the post-training quantization process, used to statistically analyze activation or gradient information.

[0039] Calibration loss , refers to the loss function calculated based on calibration samples, used to form diagonal Fisher information or other sensitivity statistics.

[0040] Diagonal Fisher information , refers to the layer used to represent the first layer in the target neural network. Fisher's information is a diagonal approximation of the sensitivity of column weights to calibration loss.

[0041] scalar scaling factor This refers to the weight matrix of the first... The row-level scaling factor obtained from the row-level solution is used for post-processing correction of the block-by-block quantization weights of that row.

[0042] Fisher weighted reconstruction error refers to the weighted error formed by using diagonal Fisher information as weights when calculating the error between the original weights and the quantized weights.

[0043] Layer-by-layer activation cache update refers to the process of completing the quantization of the current layer according to the forward propagation order of the model, then using the quantization weights of the current layer to propagate forward again, and using the output as the calibration activation of the subsequent layers.

[0044] NVFP4 compatible format refers to a low-precision floating-point deployment format or a compatible format that works with FP4 values, block-level scaling factors, and tensor-level scaling factors.

[0045] Gumbel-Softmax adaptive code point selection refers to a process that performs differentiable selection through Gumbel-Softmax soft allocation in a local candidate code point set, and then hardens it into a discrete codebook index.

[0046] Line cross-correlation , refers to the The cross-correlation between the original row weights and the block-by-block quantization weights under Fisher weights is calculated as follows: .

[0047] Effective signal energy , refers to the The effective signal energy of the row-by-block quantization weights under Fisher weights is calculated as follows: .

[0048] The following is a brief summary of some of the innovative aspects of this application. In summary, the innovation of this application does not lie in using a single known low-bit quantization tool, but in constructing a quantization processing chain that connects the preceding and following steps, addressing the specific contradictions in the deployment of large language model inference: "mismatched weight distribution, unstable calibration with few samples, and high deployment cost of complex matrix compensation".

[0049] Specifically, such as Figure 1 As shown, first, the weight matrix of the target neural network layer... By fitting a generalized Gaussian distribution, the shape parameters are obtained. and scale parameters Compared to existing methods that consistently assume a Gaussian distribution, this application precisely characterizes the peak and tail features of the weights in a large language model by parametrically modeling the true distribution characteristics of the weights. , ,like Figure 2 The constructed zero-containment quantization codebook is shown. This process fixes zero values ​​as codebook entries, while non-zero code points are distributed along the weight margins in the positive and negative half-intervals, forming a non-uniform configuration. This step is not a simple codebook replacement, but rather ensures that the spatial distribution of code points matches the actual weight values ​​from the outset, offering a distribution-driven advantage compared to schemes that only use general non-uniform codebooks.

[0050] Through block size factor Perform block-by-block quantization to obtain the block-by-block quantization weight matrix. The key innovation lies in the fact that the result is not directly used as the final deployment result, but rather further utilized using a small number of calibration samples ( Forming diagonal Fisher information The sensitivity of different column weights to calibration loss is incorporated into the row-by-row scaling process. Thus, the scalar scaling factor for each row... Not the usual global scaling parameter, but the one applied to the original weights. Block-by-block quantization weights With sensitivity weight The closed-form solution of the row-level post-processing result obtained under common constraints is: ,in , .

[0051] At the same time, by using the lower limit of the denominator and outlier statistics to degenerate into Mechanisms such as scale truncation prevent Fisher noise under few-sample conditions from being amplified into overcorrection. This differs from the issue of full Hessian correction, which suffers performance degradation due to limited-sample noise. The corrected row scale can be merged with the block scale to obtain a scaling factor for INT4, NF4, or NVFP4 compatible formats.

[0052] Furthermore, if the forward propagation of the model is performed layer by layer, the corrected weights of the current layer will participate in generating the calibration activation of the next layer, so that the subsequent layers can continue to complete Fisher statistics and scaling correction on the input that already includes the upstream quantization error, thus forming a progressive relationship between single-layer correction and full model error propagation.

[0053] The above features form an organic chain from "distributed modeling - codebook construction - block-by-block quantization - sensitivity correction - deployment merging": GGD zero-inclusion codebook reduces initial mapping bias Diagonal Fisher row-by-row scalar correction provides lightweight compensation for sensitive weight errors, offering a complexity advantage compared to full Hessian or SVD compensation. Scale merging maintains the feasibility of low-bit deployment formats. Layer-by-layer propagation reduces the accumulation of multi-layer quantization errors Therefore, the overall approach is not a simple parallel sequence of several known steps, but rather differs from conventional routes that rely on fixed Gaussian codebooks, nearest rounding, full Hessian, or matrix factorization compensation. This chain embodies a systematic design philosophy that starts with the characteristics of weight distribution, proceeds through sensitivity weighting, and achieves deployment format compatibility.

[0054] Example 1: Low-bit weighted quantization method based on generalized Gaussian distribution fitting and row-by-row Fisher correction This embodiment provides a low-bit weight quantization method for deploying neural network model inference; the overall process can be found in [reference needed]. Figure 1 The example includes steps 100 to 700. Steps 100 to 500 correspond to the aforementioned steps S1 to S5, and steps 600 and 700 are further processing steps.

[0055] Step 100: Fitting the generalized Gaussian distribution By fitting a generalized Gaussian distribution to the weight matrix of the target neural network layer, the shape parameters of the weight matrix can be obtained. and scale parameters .

[0056] This embodiment uses the Generalized Gaussian Distribution (GGD) to model the marginal distribution of each element in the weight matrix. The probability density function of the GGD distribution is exemplarily as follows: In Formula 1, Indicates the weight values. express The absolute value, The scale parameter is used to characterize the degree of broadening of the distribution. The shape parameters are used to characterize the morphology of the peaks and tails of the distribution. For the Gamma function, Indicates in the parameter Given time weight values The probability density. It should be noted that when When taking specific values, the generalized Gaussian distribution can degenerate into common distribution forms, such as... The time corresponds to a Gaussian distribution. This corresponds to a Laplace distribution. After analyzing the weight distributions of several large language models, the inventors of this application found that the weights are generally closer to a generalized Gaussian distribution, and the shape parameter is typically in the range of 1.2 to 1.5, smaller than the value corresponding to the Gaussian distribution (i.e., less than 2), resulting in a relatively sharper peak and a relatively heavier tail. Based on this finding, a quantization codebook matching the true marginal distribution is constructed, which helps reduce initial quantization errors. This is also the basis for using a generalized Gaussian distribution parameterized codebook in the subsequent step 200.

[0057] Considering that the number of elements in the weight matrix is ​​usually large, to balance fitting accuracy and computational cost, step 100 can be subdivided into the following sub-steps. All elements of the weight matrix can be statistically analyzed directly, or a subsample can be randomly selected from the weight matrix for statistical analysis; the number of elements in the subsample can be flexibly determined based on the size of the weight matrix, calibration time, and computational resources, and is exemplarily no more than 10,000 elements. For the selected subsample, the shape parameters are solved using maximum likelihood estimation, moment estimation, or other numerical estimation methods capable of fitting a generalized Gaussian distribution. and scale parameters When using maximum likelihood estimation, the subsamples can be substituted into the likelihood function corresponding to Formula 1 and solved through iterative optimization. The initial values, iteration methods, and convergence criteria can be set according to implementation needs. When using moment estimation, the absolute moments or variances of the samples can be estimated. and Optionally, for individual neural network layers that are difficult to fit stably or have abnormal parameter values, fallback methods such as using neighboring layer parameters, global default parameters, or degenerating to a Gaussian assumption can be used.

[0058] Step 200: Construction of the zero-inclusion quantization codebook Based on the shape parameters obtained in step 100 and scale parameters A zero-containing quantization codebook is constructed, where the zero value is fixed as one codebook entry (i.e., the codebook entry corresponding to the zero value is kept as 0), and the remaining codebook entries are symmetrically set in the positive and negative half-intervals according to their weights. This approach is because neural network weights typically contain a large number of weight values ​​close to zero. If the zero value cannot be accurately represented, it can easily introduce systematic biases in low-bit quantization. Fixing the zero value as a codebook entry reduces the offset of near-zero weights during dequantization. Taking a four-bit codebook as an example, it exemplarily includes at least one codebook entry fixed as zero and multiple non-zero codebook entries set according to a positive-negative correspondence. It should be noted that when the codebook level cannot ensure that all non-zero codebook entries are strictly paired due to the fixed zero value, this can be adapted to the specific encoding index by setting reserved code points, repeating zero code points or boundary code points, or making the number of non-zero codebook entries in the positive and negative half-intervals differ by one.

[0059] In one implementation, each non-zero codebook entry of the zero-containing quantized codebook can be obtained through Lloyd-Max iteration, with the goal of minimizing the mean square error between the weight values ​​and their quantized representations under a given weight distribution, which can be expressed as: In Formula 2, This represents the quantization mapping to be optimized. This represents the quantization mapping that minimizes the mean square error. Indicates the weight values The expected value when it follows the generalized Gaussian distribution defined by Formula 1. Indicates the weight values The mapped codebook entries. Formula 2 allows codebook entries to be distributed more extensively in regions with higher weight probability densities, while preserving necessary representational power in the tail region. During the Lloyd-Max iteration, quantization boundaries are determined based on adjacent codebook entries, and codebook entries are updated according to the distribution centroids within each quantization interval. For the ... Each codebook entry is updated in the following exemplary manner: (Formula 3) In formula 3, Indicates the first Each codebook entry (i.e., the condition centroid of the interval it belongs to). , Indicates the relationship with the first Each codebook entry corresponds to the lower and upper decision boundaries of the quantization interval. The probability density is defined in Formula 1. By repeatedly updating the decision boundary and codebook entries, and using the aforementioned symmetric constraint to make the codebook entries in the negative half-interval take the negative numbers of the corresponding entries in the positive half-interval, a non-uniform quantization codebook adapted to the current layer weight distribution can be obtained. Furthermore, to improve the stability of the tail region under high dynamic range, a minimum spacing threshold can be set for the tail region that exceeds the standard deviation range determined based on the generalized Gaussian distribution parameters (e.g., exceeding four times the standard deviation). This ensures that the distance between adjacent codebook entries at the tail is not less than the threshold, thereby reducing the risk of overfitting caused by excessive clustering of code points at the tail. The threshold can be a preset threshold or determined based on the maximum representation value, standard deviation, or tail interval width of the codebook. The zero-inclusion quantization codebook obtained after convergence exhibits a non-uniform distribution with a dense center and sparse tails, matching the peak-heavy-tailed characteristics of the aforementioned generalized Gaussian distribution. Optionally, the zero-inclusion quantization codebook can be generated layer by layer, generated per weight matrix, shared by layer type, or globally reused.

[0060] Step 300: Block-by-block quantization The weight matrix is ​​divided into multiple blocks by rows. For each block, the weights are mapped to the nearest codebook entry in the zero-containment quantization codebook based on the block scale factor, thus obtaining a block-by-block quantization weight matrix.

[0061] Specifically, for the first Row weights can be determined based on block size. Divide the row into several blocks, block size Examples of values ​​for weight ranges are 64, 128, and 256, which can also be determined by the target deployment format, hardware vector width, or inference engine requirements. Optionally, for weight ranges that do not end in a complete block, the remaining weight elements can be used to form a separate final block, or padding can be used in the implementation to align to the target hardware format.

[0062] For the Line 1 block For example, the block scale factor is determined based on the maximum absolute value of the weights within the block: In Formula 4, Indicates the first Line 1 The set of columns corresponding to each block This represents the maximum absolute value of the weights within that block. This indicates that the zero represents the value of the codebook entry with the largest absolute value in the quantized codebook. This represents the block scale factor. By using the block scale factor, weighted blocks with different dynamic ranges can be normalized to the same codebook coverage area, thus balancing local dynamic range with low bit representation efficiency.

[0063] In particular, when the first Line 1 The maximum absolute value of the weight within each block When all weights in the block are zero, all weights in the block are directly mapped to zero code points, and the block scale factor of the block is set to a preset security value or This avoids division by zero during the subsequent normalization process.

[0064] when Then, each weight within the block is normalized and mapped to the nearest codebook entry in the zero-container quantized codebook: In Formula 5, Indicates weight The mapped codebook index, Indicates the first Each codebook entry The block-by-block quantization weight matrix represents the first... Line 1 The quantization weights of the columns are determined. This yields a block-by-block quantization weight matrix, along with the corresponding codebook index and block scale factor. This block-by-block quantization weight matrix serves as the basis for subsequent row-by-row Fisher weighted scaling correction.

[0065] Step 400: Calculation of diagonal Fisher information The calibration samples are input into the neural network model to perform forward propagation, and the diagonal Fisher information of the target neural network layer is calculated based on the calibration loss.

[0066] This embodiment uses a small number of calibration samples to obtain the sensitivity information of the weights to the loss, in order to guide subsequent line-by-line correction. The calibration samples can be unlabeled text samples, publicly available corpus samples, user-authorized domain corpus samples, or other data used for post-training quantization calibration. (Number of calibration samples) The preferred embodiment can be determined based on the requirements for accuracy and efficiency. It can even be as low as This significantly reduces the deployment's reliance on calibration data. The calibration loss is denoted as... In the case of an autoregressive language model, the cross-entropy loss is exemplarily used to predict the next token. In other implementations, layer output reconstruction error, KL divergence, or teacher model output error can also be used.

[0067] After inputting calibration samples into the model and performing forward propagation, the diagonal Fisher information (i.e., a sensitivity index approximating the diagonal information of the Hessian matrix) of the target neural network layer is calculated based on the calibration loss. In one column-sharing implementation, the first... Fisher information on the diagonal of the column This is obtained by summing the squared gradients of the weights in each row of this column with respect to the calibration loss, exemplarily averaged by the number of valid tokens and the number of rows: In Formula 6, Indicates the first The diagonal Fisher information of the column is used to measure the sensitivity of the column weights to the calibration loss. The weight matrix represents the first... Line 1 Column weights Indicates the first The calibration loss corresponding to each valid token This indicates that the calibration loss affects the weight. gradient, This indicates the row number of the weight matrix. This indicates the number of valid tokens included in the statistics. , These are the valid token sequence number and the row sequence number, respectively. By summing the squared gradients of the weights in each row of the same column, the shared diagonal Fisher information is obtained. It shares the same sensitivity weight across different rows in the same column, thus having lower storage and computational overhead. It should be noted that diagonal Fisher information can also be averaged over the number of calibration samples, in which case the number of valid tokens can be... Replace with the number of calibration samples The summation is performed across sample dimensions; when only a single token is counted for each calibration sample, the two averaging methods yield consistent results. Optionally, an element-wise approach can also be used. This allows for a more granular reflection of the sensitivity differences between different output channels in the same column.

[0068] Furthermore, to ensure the numerical stability of Fisher statistics under conditions of few samples, when the gradient corresponding to a certain token is NaN, Inf, or exceeds the gradient clipping threshold... If this happens, the token can be removed and the diagonal Fisher information can be recalculated, where... The threshold is set based on the activation or gradient magnitude.

[0069] Step 500: Solving and correcting the row-by-row scalar scaling factor Using the diagonal Fisher information as weights, for each row of the weight matrix, the scalar scaling factor that minimizes the Fisher weighted reconstruction error between the original weights of that row and the block quantization weights scaled by the scalar scaling factor is calculated, and the block quantization weights of that row are scaled by the scalar scaling factor to obtain the corrected quantization weight matrix.

[0070] This embodiment performs lightweight post-processing on the block-by-block quantization results obtained in step 300, line by line. For the first... The goal is to find the scalar scaling factor that minimizes the Fisher-weighted reconstruction error for that row, i.e.: In Formula 7, Indicates the first The optimal scalar scaling factor for a row. This represents the scalar variable to be optimized. For the first Fisher information on the diagonal of the column, , The first row of the line is respectively The original weights and block-by-block quantization weights of the column. This indicates the column number of the row. The objective function implies that columns with greater Fisher information should be given higher weights, so that the weight errors, which are more sensitive to model loss, are given more consideration during scaling. Taking the derivative of Equation 7 and setting it to zero yields the closed-form solution to this univariate least squares problem. For ease of calculation, row cross-correlation is defined. With effective signal energy And when the denominator satisfies the preset conditions, the closed-form scaling factor is obtained: In Formula 8, Indicates the first Cross-correlation between row-based original weights and block-by-block quantization weights under Fisher weights Indicates the first Effective signal energy of row-by-block quantization weights under Fisher weights. This represents the scalar scaling factor obtained from the closed-form solution. More specifically, when... Greater than the lower limit threshold of the denominator , Greater than 0 and When the calculation result is not NaN or Inf, take Otherwise, including , The block-by-block quantization weights of this row are all zero, or If the calculation result is NaN or Inf, then let That is, it degenerates into not performing scale correction on the row, where This is the lower limit threshold for the effective signal energy of the row. This avoids anomalous Fisher statistics by releasing the quantization error of the row through closed loops, thereby maintaining numerical stability under conditions of weak signal rows and very few samples.

[0071] Furthermore, the scalar scaling factor is truncated to a preset range, and the corrected quantization weights are obtained accordingly: In Formula 9, This represents the scalar scaling factor after truncation. , These respectively truncate the lower and upper bounds. , This represents the operation of taking the smaller and larger values. Indicates the first Linearly corrected quantization weights Indicates the first Row-by-row quantization weights (vectors). For example, and The values ​​can preferably be 0.90 and 1.10; in other embodiments, a wider or narrower range can be used for the attention projection layer, MLP projection layer, or output layer, for example... The specific correspondence between layer types and truncation intervals can be determined through implementation. The purpose of truncation is to limit the amplification effect of few-sample Fisher information noise on the scalar scaling factor, keeping the row-by-row correction within a relatively stable range. Performing the above processing on each row of the weight matrix yields the corrected quantized weight matrix.

[0072] Optionally, as a preferred processing method in step 500, the scalar scaling factor can be continuously shrunk according to the row effective signal energy and sample number before truncation according to Formula 9. An exemplary form is as follows: In Formula 10, Indicates the first The reliability of the line, Represents the reference energy (exemplarily taken from each row within the layer). (median or preset quantile). The regularization coefficient is... Indicates the sample confidence level. The number of confidence saturation samples. This represents the scalar scaling factor after contraction. It can be seen that when a row is a weak signal row (i.e.,...) much smaller When the number of calibration samples is small, It smoothly approaches 1, thus constituting the aforementioned order. This is a continuous form of hard degradation. This continuous contraction method is optional and can be chosen or discarded depending on implementation needs.

[0073] Step 600: Scale factor merging and deployment format processing The scalar scaling factor of each row after correction is combined with the block scaling factor of each block in that row, and the corrected quantization weight matrix, along with the corresponding codebook index and scaling factor, is output for low-bit inference deployment.

[0074] Specifically, if the deployment format supports additional line scale fields, then... Stored together with the block scale factor; if the deployment format only supports the block scale factor, the row-level scalar scale factor can be merged into the block scale factor of each block in that row: In Formula 11, This represents the block size factor after merging. Indicates the first Scalar scaling factor after line truncation. Indicates the first Line number The original block scale factor of each block. Using Equation 11, row-level correction can be folded into the block-scale dequantization process, thereby reducing the number of additional deployment fields in this implementation. The corrected quantization weight matrix can be packaged in INT4, NF4, a custom four-bit format, or an NVFP4-compatible format; when using an NVFP4-compatible format, the merged block scale factor can be further projected into a value representable in E4M3 format to adapt to the target inference hardware or inference engine.

[0075] Step 700: Progressive Update and Activation Cache Refresh In a further embodiment, steps 100 to 600 are executed sequentially for each neural network layer according to the forward propagation order of the neural network model; after the quantization of the current neural network layer is completed, the forward propagation is re-executed using the quantized weight matrix corrected for the current neural network layer, and its output is used to update the calibration activation of the next neural network layer.

[0076] Specifically, after this processing, the calibration activation used by the next neural network layer when calculating diagonal Fisher information and performing row-by-row Fisher scaling already includes the quantization error introduced by the upstream quantization layer. The generalized Gaussian distribution fitting and block-by-block quantization of this next neural network layer are still performed based on the layer's own weight matrix. Therefore, the upstream quantization error mainly affects the calculation of diagonal Fisher information and row-by-row scaling through calibration activation, thus helping to reduce the impact of multi-layer quantization error accumulation on subsequent layer corrections. For example, for the Transformer architecture model, the forward propagation order of linear layers containing weight matrices, such as the word embedding layer, attention projection layer, MLP projection layer, and output layer, can be followed sequentially. The normalization layer can participate in activation updates as a node passed through in the forward propagation path, but it is generally not used as the target linear layer for low-bit weight quantization in this embodiment. For different model structures, the layer processing order can also be determined according to the actual forward computation graph order. It should be noted that layer-by-layer progressive processing is not necessarily limited; in some implementations, each layer can be processed independently. Using layer-by-layer progressive processing is beneficial to further improve the overall stability of multi-layer quantization.

[0077] It should be noted that the above steps are progressively connected and mutually supportive: the shape and scale parameters output from step 100 are used by step 200 to construct the zero-inclusion quantization codebook; the zero-inclusion quantization codebook output from step 200 is used by step 300 for block-by-block quantization; the block-by-block quantization weight matrix output from step 300 is used for correction in step 500; the diagonal Fisher information output from step 400 serves as the weighting basis for step 500; and the corrected quantization weight matrix output from step 500 is merged and packaged in step 600 as the final deployment object. Thus, the zero-inclusion codebook construction based on the generalized Gaussian distribution and the diagonal Fisher row-by-row scalar scaling correction under few-sample conditions are functionally mutually supportive—the former makes the initial block-by-block quantization more closely match the marginal distribution of the weights to reduce the initial quantization error, while the latter... The computational complexity is reduced by performing row-level post-processing on the quantization results to further reduce the error of sensitive weights. The two work together to help reduce the quantization error of low-bit weights and improve deployment stability with very few calibration samples and low computational complexity.

[0078] Example 2: Optional Gumbel-Softmax Adaptive Code Point Selection in NVFP4 Compatible Format In an extended implementation, the target deployment format is NVFP4 or a compatible low-precision floating-point format. This implementation does not replace the main process in Embodiment 1, but rather adapts the codebook construction in step 200 and the code point mapping in step 300 to a specific hardware format under constraints. For example, the NVFP4 compatible format may include FP4 E2M1 values, block-level E4M3 scaling factors, and tensor-level scaling factors. The FP4 positive amplitude grid can be a non-uniform grid supported by the target hardware. For each weight, the nearest FP4 candidate code point can be determined first based on the normalized weight value, and then a local candidate set can be constructed centered on this candidate code point. The candidate center of the local candidate set can be derived from rounding to the nearest, initialization based on quantization error compensation, format-aware adaptive rounding, or initialization based on multiple candidate code points. The candidate radius is exemplarily 1 or 2. Subsequently, soft allocation is learned within the local candidate set using Gumbel-Softmax. In Formula 12, This represents the soft assignment result on the local candidate set. Indicates the first Learnable assignable weights for each candidate codepoint Indicates the application to the first One candidate Gumbel noise, For temperature parameters, Indicates the first One candidate FP4 code point , Both represent summation over candidates within a local candidate set. Temperature parameters can be adjusted during training. Annealing is performed to gradually approximate the discrete selection in the soft assignment. After training, the soft assignment is hardened into discrete FP4 code points (i.e., discrete codebook indices) using argmax, and the block scale factor is kept consistent with the deployment format in the training or calibration forward using a pass-through estimator in E4M3 format. Optionally, an entropy regularization term can be introduced to make the assignment result of Gumbel-Softmax closer to the one-hot form. This extended implementation is beneficial for providing a more flexible code point selection space for each weight than fixed nearest rounding under the NVFP4 format legality constraint; it does not change the core processing logic of generalized Gaussian distribution fitting and row-by-row Fisher correction in Embodiment 1.

[0079] Example 3: Systems, Electronic Devices, and Computer-Readable Storage Media Corresponding to the above method embodiments, this embodiment also provides a low-bit weight quantization system for deploying neural network model inference, such as... Figure 2As shown, the system exemplarily includes a distribution fitting module, a codebook construction module, a block-by-block quantization module, a Fisher information calculation module, a line-by-line scaling correction module, and an output module.

[0080] The distribution fitting module is used to fit a generalized Gaussian distribution to the weight matrix of the target neural network layer to obtain the shape parameters. and scale parameters ; The codebook construction module is used to construct the codebook based on the shape parameters. and scale parameters Construct a zero-enclosed quantization codebook, where the zero value is fixed as one codebook entry of the quantization codebook, and the remaining codebook entries are symmetrically distributed in the positive and negative half-intervals; The block-by-block quantization module is used to divide the weight matrix into multiple blocks by rows, and map the weights within each block to the nearest codebook entry in the zero-containing quantized codebook based on the block scale factor, thus obtaining the block-by-block quantization weight matrix. The Fisher information calculation module is used to input calibration samples into the neural network model to perform forward propagation and calculate the diagonal Fisher information of the target neural network layer based on the calibration loss. The row-by-row scaling correction module is used to calculate the scalar scaling factor for each row, which minimizes the Fisher weighted reconstruction error between the original weights of the row and the block-by-block quantization weights scaled by the scalar scaling factor, using diagonal Fisher information as weights. The block-by-block quantization weights of the row are then scaled using this scalar scaling factor to obtain the corrected quantization weight matrix. The output module is used to output the corrected quantization weight matrix and the corresponding codebook index and scaling factor for low-bit inference deployment of the neural network model.

[0081] For details on the specific implementation of each module and their interrelationships, please refer to the corresponding descriptions in Embodiment 1 and Embodiment 2.

[0082] Furthermore, this embodiment also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, it implements the method described in Embodiment 1 or Embodiment 2. Optionally, the electronic device may also include components such as a communication interface and a bus.

[0083] Accordingly, this embodiment also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the method described in Embodiment 1 or Embodiment 2. The computer-readable storage medium can be, for example, a read-only memory, a random access memory, a disk, or an optical disk, or any other medium capable of storing program code.

[0084] Supplementary explanations regarding some implementation details To facilitate those skilled in the art in implementing the technical solutions of this application, the following further explains several implementation details in generalized Gaussian distribution fitting, zero-inclusion quantization codebook, block-by-block quantization, diagonal Fisher information calculation, line-by-line scale correction, and deployment format processing.

[0085] In one implementation, a predetermined number of weight elements are randomly selected from the weight matrix of each target neural network layer using a uniform random sampling method, and the moment estimation result is used as the shape parameter. and scale parameters The initial value is then iteratively updated using maximum likelihood estimation; when the values ​​obtained in two consecutive iterations are... and When all changes are less than the preset convergence threshold, or when the number of iterations reaches the preset upper limit, stop iterating and output the current value. and In another implementation, all elements of the weight matrix can be statistically analyzed directly, or neighboring layer parameters, global default parameters, or Gaussian distribution parameters can be used as degradation treatment when fitting fails.

[0086] In one implementation, the zero-inclusive quantization codebook includes at least one zero code point and multiple pairs of positive and negative non-zero code points; taking four-bit quantization as an example, the encoding index can represent a fixed zero code point, several pairs of positive and negative non-zero code points, and reserved code points required for alignment with the deployment format. During dequantization, the zero code point is fixedly mapped to a numerical value. Non-zero code points are set according to a positive-to-negative correspondence, and reserved code points can be mapped to zero values ​​or boundary code points according to the deployment format. This code point allocation method is used to describe the index organization method of the zero-containing quantization codebook, and does not require all low-bit formats to use the exact same index order.

[0087] In the Lloyd-Max iteration, if... The decision boundary between adjacent codebook entries can be determined based on these adjacent entries, and the codebook entries can be updated based on the centroid of the distribution within the corresponding interval. For the tail region, the standard deviation range obtained based on the fitted distribution can be used as the tail determination criterion; for example, when the weight value exceeds... When the range is multiples of the standard deviation, the corresponding region is considered the tail region. The value can be between 3 and 5. Minimum spacing threshold. It can be determined based on the maximum absolute codebook value of the zero-containment quantized codebook, the standard deviation of the fitted distribution, or the width of the tail interval; when the distance between adjacent tail code points obtained by Lloyd-Max iteration is less than When this happens, adjacent tail code points are adjusted to meet the minimum spacing requirement to reduce the risk of excessive clustering of tail code points.

[0088] In a column-shared diagonal Fisher information statistics method, the first... Fisher information on the diagonal of the column Can be sorted by the number of valid tokens and number of rows The average is expressed as: in, Indicates the first The calibration loss corresponding to each valid token The weight matrix represents the first... Line 1 Column weights. Sharing this column reduces storage and computational overhead. Alternatively, weights can be calculated for each element. This allows for a more granular representation of the sensitivity differences between different output channels within the same column.

[0089] When calculating diagonal Fisher information, if the gradient corresponding to a certain token is NaN, Inf, or its absolute value exceeds the gradient clipping threshold... If necessary, the token can be removed from the statistical set and the diagonal Fisher information can be recalculated. Gradient clipping threshold. This can be determined based on the statistical distribution of the absolute value of the gradient during the calibration phase, for example, by taking a preset quantile of the absolute value of the gradient or a multiple thereof. Lower limit threshold of the denominator. Based on each row of the current layer The statistical distribution is determined, for example, by taking it as the distribution of ... Positive numbers related to the median or lower quantiles.

[0090] In line-by-line scale correction, if Not greater than The block-by-block quantization weights of this row are all zero. Not greater than or by If the obtained scalar scaling factor is NaN or Inf, then let the scalar scaling factor of that row be... This means that scaling correction is not performed on that row. If , And if the calculation result is valid, then it can be done according to... Obtain the scalar scaling factor and truncate it to... Inside. For example, and Can be taken and For different layer types, the following can also be adopted. The cutoff range is adjusted to strike a balance between the strength of the correction and the risk of overscaling.

[0091] When processing the deployment format, the first... The scalar scaling factor of a row is combined with the block scaling factors of each block in that row to obtain the combined block scaling factor. When using the NVFP4-compatible format, the merged block scale factor can be projected to the nearest value that can be represented in the E4M3 format, and a format validity check is performed based on the relative deviation before and after projection; if there are multiple adjacent representable values, the value with the smaller relative deviation is selected. If the inference engine supports row scale fields, the row scale factor can also be saved separately and used in conjunction with the block scale factor before dequantization or matrix multiplication.

[0092] When using Gumbel-Softmax adaptive code point selection in the NVFP4 compatible format, this process can be considered as an optional method for local FP4 candidate code point selection. Specifically, a higher temperature can be initially set to maintain candidate code point exploration, and then the temperature can be gradually reduced to make the soft assignment approach a one-heat form. After calibration, discrete FP4 code points are obtained through the candidate code points corresponding to the maximum assignment weight, and the block scale factor is projected onto the E4M3 representable value. This optional method does not change the main process of generalized Gaussian distribution fitting, zero-inclusion quantization codebook construction, and line-by-line Fisher scaling correction.

[0093] To verify the above processing procedure, under the same model, same calibration samples, and same quantization bit width, comparisons can be made between the complete process, the removal of generalized Gaussian distribution fitting, the removal of zero code points, the removal of row-by-row Fisher scaling correction, the removal of denominator protection and scale truncation, and the removal of layer-by-layer activation cache updates. The reconstruction error, perplexity, or downstream task evaluation metrics can be compared in the comparison layer output. Alternatively, a simplified example using single-row weights can be used to demonstrate the process. , , The calculation process, and Degradation process when too small, all-zero blocks, or abnormal gradients occur.

[0094] Technical effects of the above embodiments The above embodiments have the following technical effects: By fitting the weight matrix with a generalized Gaussian distribution and constructing a zero-inclusion quantization codebook, the initial codebook distribution can better match the marginal distribution of the target neural network layer weights and reduce the quantization offset of near-zero weights; by block scaling and nearest-neighbor code point mapping, both local dynamic range and low-bit storage format can be taken into account; by solving the scalar scaling factor for each row of quantization weights using few-sample diagonal Fisher information, lightweight post-processing of block quantization results can be performed with lower computational complexity, and the weight error that is more sensitive to loss can be given more consideration in the correction; by denominator protection, abnormal token removal, degrading all-zero rows to uncorrected, and scale truncation, the risk of overcorrection caused by few-sample statistical noise can be reduced; by merging the row scaling factor into the block scaling factor, additional deployment fields can be reduced in some deployment formats; by updating the calibration activation layer by layer according to the forward propagation order, subsequent layers can continue to quantize and correct on inputs containing upstream quantization errors, thereby reducing the impact of multi-layer quantization error accumulation. Overall, this embodiment helps to improve the deployment stability and model inference availability after low-bit weight quantization while reducing weight storage usage and memory access bandwidth.

[0095] It should be noted that In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

Claims

1. A low-bit weight quantization method for deploying neural network model inference, characterized in that, include: Step S1: Fit a generalized Gaussian distribution to the weight matrix of the target neural network layer to obtain the shape parameters of the weight matrix. and scale parameters ; Step S2, based on the shape parameters and the scale parameters Construct a zero-inclusive quantization codebook, wherein the zero value is fixed as one codebook entry of the zero-inclusive quantization codebook, and the remaining codebook entries are symmetrically distributed in the positive and negative half-intervals; Step S3: Divide the weight matrix into multiple blocks by row. For each block, based on the block scale factor, map the weights within the block to the nearest codebook entry in the zero-inclusive quantization codebook to obtain a block-by-block quantization weight matrix. Step S4: Input the calibration sample into the neural network model to perform forward propagation, and calculate the diagonal Fisher information of the target neural network layer based on the calibration loss; Step S5: Using the diagonal Fisher information as weights, for each row of the weight matrix, solve for the scalar scaling factor that minimizes the Fisher weighted reconstruction error between the original weights of that row and the block quantization weights scaled by the scalar scaling factor, and scale the block quantization weights of that row using the scalar scaling factor to obtain the corrected quantization weight matrix. Specifically, the corrected quantization weight matrix, along with the corresponding codebook index and scaling factor, is output for low-bit inference deployment of the neural network model.

2. The method according to claim 1, characterized in that: In step S1, the shape parameters are obtained by performing maximum likelihood estimation or moment estimation on random subsamples of the weight matrix. and the scale parameters The number of elements in the random subsample is no more than The probability density function of the generalized Gaussian distribution is: in, For weight values, for The absolute value, The scale parameter is... For the shape parameters, It is the Gamma function; In step S2, based on the shape parameters and the scale parameters The non-zero codebook entries of the zero-containing quantized codebook are determined through Lloyd-Max iteration, and for the tail region exceeding the standard deviation range determined based on the generalized Gaussian distribution parameters, the minimum spacing threshold is applied. Tail homogenization is performed, among which This is the lower limit of the spacing between adjacent codebook entries at the tail of the zero-containing codebook; In step S3, the block size of each block Preferably, the block scale factor is 64, 128, or 256. The block scale factor is determined based on the maximum absolute value of the weights within the block. It is used to scale the weights within the block to the value range covered by the zero-inclusive quantized codebook, and then map them to the nearest codebook entry.

3. The method according to claim 1, characterized in that, In step S4: The number of calibration samples satisfy ; The calibration loss is at least one of the following: next token prediction cross-entropy loss, layer output reconstruction error, KL divergence, or teacher model output error; The diagonal Fisher information includes the diagonal Fisher information of each column of the target neural network layer. In the column-sharing statistical method, By number of valid tokens and the number of rows in the weight matrix On average: in, For the first Fisher information on the diagonal of the column, For the first The calibration loss corresponding to each valid token For the weight matrix of the first... Line 1 Column weights The number of valid tokens included in the statistics. The number of rows in the weight matrix; When the gradient corresponding to a certain token is NaN, Inf, or exceeds the gradient clipping threshold When this happens, the token is removed and the diagonal Fisher information is recalculated. .

4. The method according to claim 1, characterized in that, In step S5, for each row of the weight matrix Calculate row cross-correlation With effective signal energy : in, For the target neural network layer number 1 Fisher information on the diagonal of the column, For the weight matrix of the first Line 1 The original weights of the columns, For the block-by-block quantization weight matrix, the first... Line 1 Quantitative weights of columns This indicates all columns in that row. Summation; when Greater than the lower limit threshold of the denominator , Greater than 0 and When the calculation result is not NaN or Inf, let the scalar scaling factor... Otherwise, let Without correcting the line, the "otherwise" includes , The block-by-block quantization weights in this line are all zero, or The calculation result is either NaN or Inf; and the scalar scaling factor Cut off to ,in and Preferably 0.90 and 1.10, or for the attention projection layer, MLP projection layer, or output layer. ;in, The lower threshold value is the effective signal energy of the row. , These are the lower and upper bounds of the scalar scaling factor, respectively.

5. The method according to claim 1, characterized in that, Also includes: The scalar scaling factor of each row after correction is merged with the block scaling factor of each block in that row in step S3 to obtain the merged block scaling factor; the corrected quantization weight matrix is ​​packaged into INT4 format, NF4 format, custom 4-bit format or NVFP4 compatible format. When using the NVFP4 compatible format, the merged block scale factor is projected into a value that can be represented in the E4M3 format.

6. The method according to any one of claims 1 to 5, characterized in that, The method also includes step S6: according to the forward propagation order of the neural network model, steps S1 to S5 are executed sequentially for each neural network layer; after the quantization of the current neural network layer is completed, the forward propagation is re-executed using the corrected quantization weight matrix of the current neural network layer, and its output is used to update the calibration activation of the next neural network layer, so that the calibration activation used by the next neural network layer when executing steps S1 to S5 includes the quantization error of the current neural network layer.

7. The method according to claim 1, characterized in that, For NVFP4 format or compatible low-precision floating-point format, during codebook construction in step S2 and mapping in step S3, a local candidate set is maintained for each weight. The candidate center of the local candidate set is determined by rounding to the nearest, initialization based on quantization error compensation, format-aware adaptive rounding, or initialization based on multiple candidate code points. The candidate radius is 1 or 2. Gumbel-Softmax is used to learn soft allocation on the local candidate set according to the following formula. : in, For the first Learnable assigned weights for each candidate key entry It is Gumbel noise. For temperature parameters, For the first One candidate FP4 codebook entry , This represents the summation of candidates within the local candidate set; after training, the soft allocation is performed using argmax. Harden the codebook index to a discrete format and make the block scale factor consistent with the deployment format in the training forward pass-through estimator using the E4M3 format.

8. A low-bit weight quantization system for deploying neural network model inference, characterized in that, include: The distribution fitting module is used to fit a generalized Gaussian distribution to the weight matrix of the target neural network layer to obtain the shape parameters of the weight matrix. and scale parameters ; The codebook construction module is used to construct the codebook based on the shape parameters. and the scale parameters Construct a zero-inclusive quantization codebook, wherein the zero value is fixed as one codebook entry of the zero-inclusive quantization codebook, and the remaining codebook entries are symmetrically distributed in the positive and negative half-intervals; The block-by-block quantization module is used to divide the weight matrix into multiple blocks by rows, and for each block, based on the block scale factor, map the weights within the block to the nearest codebook entry in the zero-inclusive quantization codebook to obtain the block-by-block quantization weight matrix. The Fisher information calculation module is used to input calibration samples into the neural network model to perform forward propagation, and calculate the diagonal Fisher information of the target neural network layer based on the calibration loss; The row-by-row scaling correction module is used to calculate, for each row of the weight matrix, a scalar scaling factor that minimizes the Fisher weighted reconstruction error between the original weight of that row and the block-by-block quantization weight after scaling by the scalar scaling factor, using the diagonal Fisher information as weights, and then scaling the block-by-block quantization weight of that row with the scalar scaling factor to obtain the corrected quantization weight matrix. The output module is used to output the corrected quantization weight matrix and the corresponding codebook index and scaling factor for low-bit inference deployment of the neural network model.

9. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program thereon, which, when executed by a processor, implements the method of any one of claims 1 to 7.

Images