Model quantization method and apparatus, electronic device, medium, and program product

Abstract

The present disclosure provides a model quantization method and device, electronic equipment, medium and program product, which incorporates output length control into the quantization target through a preset hybrid loss function containing a distillation loss term and a stop loss term. The stop loss term is activated only when the full-precision model prediction end symbol probability exceeds a threshold value, avoiding interference with normal generation. The gradient of the stop loss term relative to the activation value is calculated to determine the importance weight of each token position, automatically identifying key positions sensitive to stop judgment. The original second-order gradient information matrix is weighted using the weight to construct a weighted matrix, so that the quantization process differentially protects key positions under a unified bit width, preferentially reducing the quantization error thereof. The final target quantization model not only maintains accuracy, but also suppresses output length expansion, reduces redundant calculation and user waiting time, and improves deployment experience and resource utilization.

Description

Technical Field

[0001] This disclosure relates to artificial intelligence technology, and in particular to a model quantization method, apparatus, electronic device, medium, and program product. Background Technology

[0002] With the rapid development of deep learning technology, the parameter scale of Large Language Models (LLMs) has grown exponentially—from 175 billion parameters in GPT-3 to trillions of parameters in today's models. The computational overhead and storage requirements brought about by model inference are becoming increasingly severe. Mainstream deployment platforms such as cloud servers, edge devices, and terminal products are facing the dual challenges of the "computation wall" and the "storage wall": massive parameters are frequently moved between computing units and memory, and memory access bandwidth has become a bottleneck for inference performance; at the same time, the multiplication and addition operations of a large number of parameters have led to a continuous increase in power consumption.

[0003] To alleviate the above problems, academia and industry have proposed model quantization technology. Quantization maps model parameters from high-precision floating-point numbers (such as FP32, FP16) to low-precision representations (such as INT8, INT4, or even binarization), thereby reducing storage space, reducing memory access bandwidth pressure, compressing the bit width of computing units, reducing power consumption, and enabling high-performance, energy-efficient inference deployment of large models.

[0004] Compared to full-precision models, the core difference of quantized models lies in the reduction of numerical representation precision. Model quantization involves two stages: training and inference. The training stage often employs quantization-aware training (QAT) or post-training quantization (PTQ) to maintain model accuracy by minimizing quantization error. The inference stage directly uses low-precision parameters for computation, significantly improving efficiency. Due to the massive number of parameters in large models, QAT requires a large amount of computational resources; therefore, current mainstream solutions all use PTQ.

[0005] In recent years, large-scale models have been evolving from traditional generative models to inference models, with typical examples including the OpenAI o1 series and DeepSeek-R1. Inference models perform more complex thought processes by extending the output length, which is often several times or even tens of times longer than that of traditional models. However, when existing mass production methods are applied to inference models, they lead to a significant increase in model output length (more than 15%), resulting in additional wasted computing resources and longer user wait times, severely restricting the large-scale deployment experience of inference models.

[0006] In-depth analysis reveals that existing quantization methods focus solely on "precision preservation" in their loss functions and optimization objectives, completely lacking any constraint mechanism on output length. The numerical errors introduced by quantization accumulate and propagate, interfering with the model's prediction of stop tokens and causing the model to tend to generate more redundant tokens before stopping. This problem is particularly severe in inference models—adding more than 15% redundancy to an already long output chain means more computational steps, higher bandwidth consumption, and longer user wait times, while also creating resource scheduling difficulties for service providers. However, current quantization technology research primarily focuses on precision preservation and efficiency improvement, leaving the control of output length largely unaddressed. Summary of the Invention

[0007] In view of the above-mentioned problems in related technologies, this disclosure provides a model quantization method, apparatus, electronic device, medium, and program product.

[0008] A first aspect of this disclosure provides a model quantization method, the method comprising:

[0009] Based on a preset hybrid loss function, a calibration set is used to perform forward propagation through the full-precision model to be quantized, obtaining the activation value and the log probability value of the terminator at each marker position; wherein, the preset hybrid loss function includes a distillation loss term and a stopping loss term, the distillation loss term is used to maintain the accuracy of the full-precision model to be quantized, and the stopping loss term is used to control the output length of the full-precision model to be quantized, and is activated when the probability of the full-precision model predicting the terminator exceeds a preset probability threshold;

[0010] Based on the activation value and the log-odds value of the terminator, determine the gradient information of the stopping loss term relative to the activation value;

[0011] Based on the gradient information, determine the importance weight of each marked position to the stopping loss term;

[0012] Using the importance weights, the original second-order gradient information matrix of the model weights of the full-precision model to be quantized, calculated based on the calibration set, is weighted to construct a weighted second-order gradient information matrix;

[0013] Based on the weighted second-order gradient information matrix, gradient-based post-training quantization is performed on the full-precision model to be quantized to obtain the target quantized model, which is then deployed to a hardware device. The target quantized model can be called by the hardware device to process corresponding tasks.

[0014] As an optional embodiment, the step of forward propagating through the full-precision model to be quantized using a calibration set based on a preset hybrid loss function to obtain the activation value and log odds value of the terminator at each marker position includes:

[0015] Obtain a calibration set, wherein each sample in the calibration set has a fixed-length word sequence;

[0016] The calibration set is input into the full-precision model to be quantized for forward propagation, wherein, during the forward propagation process, the full-precision model to be quantized outputs the activation value of the marker position and the prediction vector for the next word at each marker position.

[0017] Extract the logarithmic probability value corresponding to the terminator from the prediction vector;

[0018] Record the activation value of each of the marked positions and the log-odds value of the terminator; determine the stopping loss term of the preset hybrid loss function and the gradient information of the stopping loss term relative to the activation value based on the activation value of the marked position and the log-odds value of the terminator.

[0019] As an optional embodiment, determining the gradient information of the stopping loss term relative to the activation value based on the log-odds value of the activation value and the terminator includes:

[0020] For each marker position, the probability of the full-precision model predicting the end symbol is calculated using the log odds value of the end symbol;

[0021] The probability of the full-precision model predicting the end symbol to be quantized is compared with a preset probability threshold.

[0022] In response to the probability that the full-precision model to be quantized predicts the end symbol exceeding the preset probability threshold, the partial derivative of the stopping loss term with respect to the log probability value of the end symbol is calculated based on the difference between the log probability value of the end symbol and the target value, wherein the target value is the log probability value of the end symbol output by the full-precision model to be quantized at the same marker position.

[0023] Using the chain rule, the partial derivative is backpropagated to the activation value to obtain the gradient information of the stopping loss term relative to the activation value.

[0024] As an optional embodiment, determining the importance weight of each marker position to the stopping loss term based on the gradient information includes:

[0025] For any of the marked positions, calculate the second moment of the gradient information;

[0026] The second moment is determined as the importance weight of the marker position to the stopping loss term.

[0027] As an optional embodiment, calculating the second moment of the gradient information for any of the marked positions includes:

[0028] For any of the marked positions, the sum of squares of each element in the gradient information is calculated to serve as the second moment of the gradient information;

[0029] or,

[0030] For any of the marked positions, the expected value of the square of each element in the gradient information is calculated, and used as the second moment of the gradient information.

[0031] As an optional embodiment, the step of using the importance weights to weight the original second-order gradient information matrix of the model weights of the full-precision model to be quantized, calculated based on the calibration set, to construct a weighted second-order gradient information matrix includes:

[0032] The importance weight of each marked position is used as the diagonal element to form a diagonal matrix;

[0033] Multiplying the diagonal matrix by the original second-order gradient information matrix yields the weighted second-order gradient information matrix.

[0034] As an optional embodiment, the step of performing gradient-based post-training quantization on the full-precision model to be quantized based on the weighted second-order gradient information matrix to obtain the target quantized model includes:

[0035] The weighted second-order gradient information matrix is ​​used as the optimization target for error compensation, and the weights of the full-precision model to be quantized are grouped according to a preset grouping strategy.

[0036] The weights of each group of the full-precision model to be quantized are iteratively quantized according to preset iteration conditions to obtain the target quantized model.

[0037] As an optional embodiment, the step of iteratively quantizing each group of weights of the full-precision model to be quantized according to preset iteration conditions to obtain the target quantized model includes:

[0038] Determine the current group weights of the full-precision model to be quantized and perform quantization processing. The current group weights are the weights of the full-precision model to be quantized that are currently being quantized.

[0039] After the quantization of the current group weights is completed, the error compensation amount of the unquantized weights is calculated using the weighted second-order gradient information matrix, and the unquantized weights are updated according to the error compensation amount.

[0040] The above operations are performed iteratively until all weights of the full-precision model to be quantized are quantized, thus obtaining the target quantized model.

[0041] A second aspect of this disclosure provides a model quantization apparatus, the apparatus comprising:

[0042] The data acquisition module is used to perform forward propagation through the full-precision model to be quantized using a calibration set based on a preset hybrid loss function, to obtain the activation value and log probability value of the terminator at each marker position; wherein, the preset hybrid loss function includes a distillation loss term and a stopping loss term, the distillation loss term is used to maintain the accuracy of the full-precision model to be quantized, and the stopping loss term is used to control the output length of the full-precision model to be quantized, and is activated when the probability of the full-precision model to predict the terminator exceeds a preset probability threshold;

[0043] The gradient information determination module is used to determine the gradient information of the stopping loss term relative to the activation value based on the logarithmic probability value of the activation value and the terminator;

[0044] The importance weight determination module is used to determine the importance weight of each marked position to the stopping loss term based on the gradient information.

[0045] The weighting processing module is used to use the importance weights to weight the original second-order gradient information matrix of the model weights of the full-precision model to be quantized, calculated based on the calibration set, and construct a weighted second-order gradient information matrix.

[0046] The model quantization module is used to perform gradient-based post-training quantization on the full-precision model to be quantized based on the weighted second-order gradient information matrix to obtain a target quantized model, which is then deployed to a hardware device. The target quantized model can be called by the hardware device to process corresponding tasks.

[0047] As an optional embodiment, the data acquisition module includes:

[0048] A calibration set acquisition unit is used to acquire a calibration set, wherein each sample in the calibration set has a fixed-length word sequence;

[0049] A forward propagation unit is used to input the calibration set into the full-precision model to be quantized for forward propagation, wherein, during the forward propagation process, the full-precision model to be quantized outputs the activation value of the marker position and the prediction vector for the next word at each marker position.

[0050] The log odds value extraction unit is used to extract the log odds value corresponding to the end symbol from the prediction vector;

[0051] A numerical recording unit is used to record the activation value of each of the marked positions and the log-probability value of the end symbol; to determine the stopping loss term of the preset mixed loss function and the gradient information of the stopping loss term relative to the activation value based on the activation value of the marked position and the log-probability value of the end symbol.

[0052] As an optional embodiment, the gradient information determination module includes:

[0053] The first calculation unit is used to calculate the probability of the full-precision model predicting the end symbol for each marked position using the log odds value of the end symbol;

[0054] The comparison unit is used to compare the probability of the prediction terminator of the full-precision model to be quantized with a preset probability threshold.

[0055] The second calculation unit is used to respond to the fact that the probability of the predicted terminator by the full-precision model to be quantized exceeds the preset probability threshold, and to calculate the partial derivative of the stopping loss term with respect to the log probability value of the terminator based on the difference between the log probability value of the terminator and the target value, wherein the target value is the log probability value of the terminator output by the full-precision model to be quantized at the same marker position.

[0056] The gradient information acquisition unit is used to backpropagate the partial derivative to the activation value using the chain rule to obtain the gradient information of the stopping loss term relative to the activation value.

[0057] As an optional embodiment, the importance weight determination module includes:

[0058] The third calculation unit is used to calculate the second moment of the gradient information for any of the marked positions;

[0059] The importance weight determination unit is used to determine the second moment as the importance weight of the marker position to the stop loss term.

[0060] As an optional embodiment, the third computing unit includes:

[0061] The first calculation subunit is used to calculate the sum of squares of each element in the gradient information for any of the marked positions, so as to use the second moment of the gradient information;

[0062] or,

[0063] The second calculation subunit is used to calculate the expected value of the square of each element in the gradient information for any of the marked positions, so as to serve as the second moment of the gradient information.

[0064] As an optional embodiment, the weighting processing module includes:

[0065] Diagonal matrix construction unit, used to construct a diagonal matrix by taking the importance weight of each marked position as diagonal elements;

[0066] The matrix calculation unit is used to multiply the diagonal matrix with the original second-order gradient information matrix to obtain a weighted second-order gradient information matrix.

[0067] As an optional embodiment, the model quantization module includes:

[0068] The weight grouping unit is used to group the weights of the full-precision model to be quantized according to a preset grouping strategy, using the weighted second-order gradient information matrix as the optimization target for error compensation.

[0069] The quantization processing unit is used to iteratively quantize each group of weights of the full-precision model to be quantized according to preset iteration conditions to obtain the target quantized model.

[0070] As an optional embodiment, the quantization processing unit includes:

[0071] The weight quantization processing subunit is used to determine the current group weights of the full-precision model to be quantized and perform quantization processing. The current group weights are the weight regroups of the full-precision model to be quantized that are currently being quantized.

[0072] The error compensation calculation subunit is used to calculate the error compensation amount of the unquantized weights using the weighted second-order gradient information matrix after the quantization of the current group weights is completed, so as to update the unquantized weights according to the error compensation amount.

[0073] The iterative quantization subunit is used to iteratively perform the above operations until all weights of the full-precision model to be quantized are quantized, thus obtaining the target quantized model.

[0074] A third aspect of this disclosure provides an electronic device, including:

[0075] Memory, used to store computer program products;

[0076] A processor for executing a computer program product stored in memory, wherein when the computer program product is executed, it implements the method of any of the first aspects described above.

[0077] A fourth aspect of this disclosure provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the method of any one of the first aspects described above.

[0078] The fifth aspect of this disclosure provides a computer program product including computer program instructions that, when executed by a processor, implement any of the methods described in the first aspect above.

[0079] According to embodiments of this disclosure, output length control is explicitly incorporated into the quantization optimization objective through a preset hybrid loss function that includes a distillation loss term and a stopping loss term. The stopping loss term is activated only when the probability of the predicted terminator in the full-precision model exceeds a preset probability threshold. This avoids interference with the generation process at non-terminal positions, achieving precise constraint on the output length of the full-precision model. Furthermore, by calculating the gradient information of the stopping loss term relative to the activation value and determining the importance weight of each marker position, key positions sensitive to stopping judgment can be automatically identified. Then, this importance weight is used to weight the original second-order gradient information matrix, constructing a weighted second-order gradient information matrix. This allows the subsequent quantization process to differentially protect key positions, i.e., prioritizing the reduction of quantization error under a uniform quantization bit width. The resulting target quantization model not only maintains accuracy similar to the full-precision model through the distillation loss term but also suppresses output length expansion caused by quantization through the constraint of the stopping loss term and the protection of the weighted matrix. This ultimately reduces the growth of the output length, thereby reducing redundant computation and user waiting time, improving the deployment experience and resource utilization of the inference model.

[0080] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0081] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0082] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0083] Figure 1 One of the flowcharts for a model quantization method provided in one embodiment of this disclosure;

[0084] Figure 2 This is a second flowchart of a model quantization method according to an embodiment of the present disclosure;

[0085] Figure 3 This is the third flowchart of a model quantization method according to an embodiment of this disclosure;

[0086] Figure 4This is the fourth flowchart of a model quantization method according to an embodiment of the present disclosure;

[0087] Figure 5 This is the fifth flowchart of a model quantization method according to an embodiment of this disclosure;

[0088] Figure 6 This is a flowchart of a model quantization method according to an embodiment of the present disclosure;

[0089] Figure 7 This is a structural block diagram of a model quantization device according to an embodiment of the present disclosure;

[0090] Figure 8 This is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0091] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0092] The following description of at least one exemplary embodiment is merely illustrative and is not intended to limit the invention or its application or use in any way.

[0093] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0094] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0095] The embodiments of this invention can be applied to electronic devices such as computer systems / servers, which can operate with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with electronic devices such as computer systems / servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0096] Electronic devices such as computer systems / servers can be described in the general context of computer-executable instructions (such as program modules) executed by the computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are performed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0097] In order to accurately describe the technical content in this disclosure, and to accurately understand the embodiments of this disclosure, the terms used in the embodiments of this disclosure are explained or defined as follows:

[0098] (1) Full-precision model is the original model that has not been quantized with low precision before quantization. Its parameters are represented in a high-precision numerical format, including but not limited to single-precision floating point (FP32), half-precision floating point (FP16), BF16, double-precision floating point (FP64) or high-precision fixed-point number format.

[0099] (2) Reasoning Model: A large language model that performs more complex "thinking processes" by extending the output length. Typical examples include the OpenAI o1 series and DeepSeek-R1. Compared with traditional generative models, the output length of reasoning models may be several times or even tens of times longer.

[0100] (3) Model quantization is a technique that maps deep learning model parameters from high-precision numerical representations (such as FP32, FP16) to low-precision representations (such as INT8, INT4, or even binarization). Its purpose is to reduce model storage space, lower memory access bandwidth pressure, and simultaneously compress the bit width of computational units and reduce power consumption, thereby achieving high-performance, energy-efficient inference deployment of large models.

[0101] (4) Quantization-Aware Training (QAT) simulates quantization errors during training, enabling the model to learn and adapt to quantization methods for low-precision representations. By introducing pseudo-quantization operations during the training phase, the model parameters are allowed to adapt to the precision loss caused by quantization in advance, thus maintaining high precision during final quantization. Due to the large number of parameters in large models, QAT requires a large amount of computational resources.

[0102] (5) Post-Training Quantization (PTQ) is a method that quantizes the parameters directly after the model training is completed, without the need for retraining or fine-tuning. PTQ maintains model accuracy by minimizing quantization error.

[0103] (6) GPTQ (Gradient-based Post-Training Quantization) is a gradient-based post-training quantization method proposed by Frantar et al. in 2022. The core idea is to optimize the quantization parameters by minimizing the reconstruction error of the quantized weights. The specific implementation includes: performing sequential quantization by grouping the model weights by row, calculating the influence of the weights on the output error using the second-order gradient information of the Hessian matrix, and iteratively updating the quantization parameters to minimize the reconstruction error.

[0104] (7) The Hessian matrix is ​​a square matrix composed of the second-order partial derivatives of a multivariable function, which describes the local curvature of the function. In GPTQ, the Hessian matrix is ​​used to measure the degree of influence of weight changes on the output error. A larger diagonal element means that the weight is more sensitive to the output and requires more accurate compensation during quantization.

[0105] The technical solutions involved in the embodiments of this disclosure will be described in detail below.

[0106] Current mainstream quantization methods for large language models primarily revolve around maintaining model accuracy. The following are typical technical solutions:

[0107] (1) GPTQ (Gradient-based Post-Training Quantization), proposed by Frantar et al. in 2022 (Source: E. Frantar et al., "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers," arXiv:2210.17323, 2022). The core idea of ​​GPTQ is to optimize the quantization parameters by minimizing the weight reconstruction error after quantization. The specific implementation process is as follows:

[0108] ① Group the model weights by row, and quantize one group of weights at a time;

[0109] ② Calculate the influence of the weights on the output error using the second-order gradient information of the Hessian matrix;

[0110] ③ Perform more refined quantification on weights with greater influence, and more aggressive quantification on weights with less influence;

[0111] ④ Iteratively update the quantization parameters to minimize the reconstruction error.

[0112] (2) QuaRot (Quantization with Rotations) This method was proposed by Ashkboos et al. in 2024 (Source: N. Ashkboos et al., "QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs," arXiv:2404.00456, 2024). The core innovation of QuaRot lies in improving the quantization friendliness of the weights by introducing rotation operations. The specific implementation process is as follows:

[0113] ① Apply orthogonal rotation transformations to the activation values ​​and weights of the model respectively;

[0114] ②Smooth the weight distribution and reduce outliers by performing rotation operations;

[0115] ③ Perform conventional quantization operations in the rotated space;

[0116] ④ Automatically cancels out the effect of rotation on the final output during inference.

[0117] The current mainstream practice in industry is to combine multiple quantization methods with GPTQ to leverage the advantages of different approaches. For example, using GPTQ as the basic quantization framework and adding preprocessing steps such as QuaRot can further improve accuracy preservation. The advantage of this approach is that it can fully utilize the strengths of each method to achieve better quantization results.

[0118] Based on the above-mentioned existing technical solutions, the current quantitative methods have the following obvious drawbacks:

[0119] (1) Lack of constraint mechanism for output length

[0120] Existing quantization methods, such as GPTQ and QuaRot, focus solely on "accuracy preservation" in their loss functions and optimization objectives. Quantization is considered successful as long as the model outputs before and after quantization remain semantically consistent (i.e., perplexity (PPL) and accuracy are similar). However, numerical errors introduced during quantization accumulate and propagate, interfering with the model's prediction timing of the End of Sequence Token (EOS, a special term indicating the end of a sequence), causing the model to tend to generate more tokens before stopping. Since existing methods completely fail to incorporate output length into their optimization objectives, the lack of constraints on length changes during quantization makes output length expansion a "blind spot" in the process.

[0121] (2) The output length of the quantization inference model increases significantly, resulting in a waste of computing resources and deployment obstacles.

[0122] When existing quantitative methods are applied to inference models (such as the OpenAI o1 series, DeepSeek-R1, etc., models that perform complex thought processes by extending the output length), it leads to a significant increase in output length—experimental data shows it exceeds 15%. The causal chain of this problem is as follows:

[0123] Numerical errors introduced by quantization → The model's judgment of the "inference completed" state becomes ambiguous → The model tends to generate more redundant words → Output length increases.

[0124] For inference models, this problem is far more severe than that of traditional generative models. The core characteristic of inference models is improving reasoning quality by extending the thought process, resulting in output lengths that are already much longer than ordinary models. Adding more than 15% redundant output on top of this means that each inference requires more computational steps, consumes more storage bandwidth, and consumes more energy, directly translating into longer user wait times. For interactive applications (such as intelligent assistants and code generation assistants), longer wait times significantly reduce user experience; in scenarios with high real-time requirements (such as online customer service and instant translation), the expansion of output length may even lead to user churn. Furthermore, the uncontrollability of output length also presents scheduling and resource planning difficulties for service providers: because the actual output length of the quantization model cannot be predicted, the system needs to reserve a large amount of computing resources as a buffer, further reducing resource utilization. These factors severely restrict the large-scale commercial deployment of inference models, becoming a major obstacle to the practical application of quantization technology.

[0125] Figure 1 This is one of the flowcharts for a model quantization method according to an embodiment of this disclosure. Figure 1 As shown, the method may include the following steps:

[0126] Step 110: Based on the preset hybrid loss function, use the calibration set to perform forward propagation through the full-precision model to be quantized, and obtain the activation value and log odds value of the terminator at each marker position.

[0127] In this step, which is the data preparation stage for the entire quantization method, a forward propagation can be performed on the full-precision model to be quantized using a small batch of unlabeled calibration set samples based on a predefined hybrid loss function. This will obtain the activation value and log probability value of the terminator at each labeled position, providing the necessary data foundation for the subsequent calculation of the gradient of the stopping loss term and the importance weights.

[0128] The preset hybrid loss function (a predefined hybrid loss function) includes a distillation loss term and a stopping loss term. The distillation loss term is used to maintain the accuracy of the full-precision model to be quantized, while the stopping loss term is used to control the output length of the full-precision model to be quantized (i.e., to constrain the output length), and is activated when the probability of the full-precision model predicting the end symbol exceeds a preset probability threshold.

[0129] For example, a pre-defined hybrid loss function :

[0130]

[0131] in, It is the coefficient of the distillation loss term. It is a distillation loss item. It is the coefficient of the stopping loss term. It is a stop loss item.

[0132] For example, KL divergence can be used to maintain accuracy in distillation loss terms.

[0133]

[0134] in, It is the KL divergence. It is the output of the floating-point model. It is the output of the quantized model (the full-precision model to be quantized).

[0135] The stop loss term focuses on the difference in Logit values ​​as a terminator:

[0136] , formula (1)

[0137] in, As a probability threshold, For the indicator function, EOS is the stop token. It is the probability of the floating-point model outputting the EOS token (stop token) (after softmax). It is the logit value of the EOStoken output by the floating-point model (before softmax). It is the logit value of the EOS token output by the quantization model (before softmax). The stopping loss term is activated only when the floating-point model believes that it "should stop" (e.g., when the floating-point model outputs a logit greater than a threshold), avoiding interference with irrelevant generation steps and achieving precise length control.

[0138] The calibration set can contain 256 to 512 general texts (e.g., paragraphs from WikiText-2) or domain-specific corpora. Each sample is processed into a fixed-length sequence of terms (e.g., 2048 terms; truncated if too long, padded if too short) to ensure the model outputs the corresponding activation value and log-odds value for the terminator at each labeled position. The labeled position can be the output corresponding to each term in the sequence, the activation value can be the output vector of a certain intermediate layer of the model (e.g., the attention layer or feedforward layer), and the terminator is a special term in the vocabulary that represents the end of the sequence (usually denoted as EOS). Its log-odds value is the raw score (unnormalized by Softmax) given by the model at the output layer for that terminator; the larger the value, the higher the probability that the model considers the end at that position.

[0139] In the specific implementation process, step 110 can adopt various embodiments: In one embodiment, the calibration set can be input into the full-precision model to be quantized one by one, and the activation value of the last layer and the log probability value of the terminator can be recorded in a complete forward propagation, and stored as a matrix in positional order; In another embodiment, only the activation value of each layer can be saved to achieve more granular key position (terminator position) identification; In yet another embodiment, in order to reduce storage overhead, the terminator probability can be calculated in real time, and the data of the terminator position can be recorded only when the probability exceeds a preset threshold (such as 0.5), thereby focusing on the position where the model output may stop; In addition, for long sequence models, data can be sparsely collected every few terminators, or batch parallel processing can be used to accelerate data extraction.

[0140] For example, if the calibration set contains a question-and-answer sample "What is the capital of country A?", the model will output the corresponding activation value (such as a semantic vector) and the log-odds value of the terminator (such as -15.2, -12.8, etc.) when processing each lexical unit. These values ​​will be saved as is for subsequent gradient calculations. Through step 110, the statistical information necessary for subsequent length control can be obtained with low computational overhead (only one forward propagation is required). Moreover, this operation is independent of the model architecture and is compatible with various large autoregressive models such as GPT, LLaMA, and DeepSeek. At the same time, the size of the calibration set, sample length, and collection sparsity can be flexibly adjusted according to hardware resources and accuracy requirements, which is convenient for engineering implementation.

[0141] This embodiment can determine the activation value and the log-probability value of the terminator at each marker position, which can be used to calculate the gradient of the stopping loss term with respect to the activation value, thereby identifying key marker positions and guiding subsequent differential quantization, thus effectively suppressing the expansion of the output length while maintaining model accuracy.

[0142] Step 120: Determine the gradient information of the stopping loss term relative to the activation value based on the log-odds values ​​of the activation value and the terminator.

[0143] In this step, based on the activation values ​​and log-probability values ​​of the terminator at each marker position, the gradient information of the stopping loss term relative to the activation values ​​is calculated through backpropagation. This provides a quantitative basis for subsequently identifying key marker positions and assigning importance weights. The gradient information reflects the sensitivity of the stopping loss term to changes in the activation values. The larger the absolute value of the gradient, the more significant the impact of even small perturbations in the activation value at that marker position on the stopping loss value, thus indicating that the marker position is of high importance in controlling the output length.

[0144] In some embodiments, the gradient information of the stopping loss term relative to the activation value can be determined based on threshold judgment and mean squared error loss.

[0145] In other embodiments, various other alternative technical solutions may be employed, such as:

[0146] As an alternative approach, a threshold-independent continuous gradient weighting scheme can be used. Specifically, instead of setting a hard probability threshold, the log-probability value of the terminator or its probability can be directly used as a weighting factor, multiplied by the original gradient information. Specifically, for the stopping loss term being the square of the difference between the log-probability values ​​of the terminator, its partial derivative with respect to the log-probability value of the terminator is twice the difference value. This partial derivative is then multiplied by a position-related weight (such as the terminator probability itself or its logarithm) and backpropagated to the activation values. This avoids the dependence on and adjustment of a threshold, and can smoothly represent the contribution of each marker position to the stopping loss.

[0147] As an alternative approach, a gradient calculation scheme based on cross-entropy loss can be used. The stopping loss term can be defined as the cross-entropy loss between the full-precision model to be quantized and the target quantized model at the terminator position. Cross-entropy loss exhibits better gradient characteristics in classification problems; its gradient form is simpler and numerically stable. It avoids gradient explosion when the terminator probability is close to 0 or 1, making it suitable for situations with high terminator prediction confidence or extremely large model sizes.

[0148] Third, a gradient calculation scheme based on the absolute value of the difference between the log-odds values ​​is adopted. This scheme defines the stopping loss as the absolute value of the difference between the log-odds values ​​of the terminator, i.e., Lstop = |Logitfloat(EOS) - Logitquant(EOS)|, where its partial derivative with respect to the log-odds value of the terminator is the sign function. Compared with mean squared error, absolute value loss is insensitive to outliers, can produce more robust gradient estimation in the presence of noise or uneven quality of calibration set samples, and avoids the numerical amplification effect caused by squaring operations.

[0149] As an alternative approach, an efficient gradient calculation scheme based on the Hessian approximation can be employed. This scheme uses the second-order statistical information of the activation values ​​to approximately estimate the gradient of the stopping loss relative to the activation values. Specifically, the outer product matrix of the activation values ​​can be pre-calculated and inverted, then multiplied with the change in the stopping loss relative to the output to obtain the gradient estimate of the activation values. This avoids the complete backpropagation process, significantly reducing computational complexity in large-scale models and long sequence scenarios, making it suitable for real-time deployment environments with extremely high computational efficiency requirements.

[0150] As another alternative, a sparse gradient calculation scheme can be used. Based on certain criteria (such as activation norm, end-symbol probability, etc.), a subset of representative positions are selected for gradient calculation, while the gradients at the remaining positions are set to zero or estimated through interpolation. This can significantly reduce the computational load without significantly sacrificing accuracy, making it suitable for situations with extremely long sequences (e.g., exceeding 4096) or a large number of calibration set samples.

[0151] The above embodiments provide direct input data for the quantization calculation of importance weights, enabling accurate identification of the most critical marker positions for output length control. Furthermore, by leveraging the magnitude of gradient information, the importance differences between different marker positions can be automatically distinguished, thus achieving differentiated protection of critical marker positions under a unified quantization bit width. The introduction of gradient information establishes a clear mathematical connection between the length control objective and the quantization optimization objective, thereby organically integrating the output length constraint into mainstream quantization frameworks such as GPTQ, achieving dual optimization of accuracy preservation and length control.

[0152] Step 130: Determine the importance weight of each marked position to the stopping loss term based on the gradient information.

[0153] In this step, based on the gradient information of the stopping loss term relative to the activation value of each marker position, the importance weight of each marker position to the stopping loss term is further calculated. In this embodiment, the importance weight is a non-negative scalar that reflects the drastic change in the stopping loss term when the activation value of the marker position undergoes a small perturbation. That is, the larger the weight, the more critical the marker position is to the control of the output length, and it can be given higher priority protection in subsequent quantization processes.

[0154] In some embodiments, importance weights can be calculated using the concept of Fisher Information, i.e., the second moment of the gradient information. In other embodiments, the gradient squared expectation method can be used, averaging the squared gradients at the same position across multiple calibration samples to make the weights more stable; or the gradient absolute value expectation method can be used, taking the absolute value of the gradient instead of its square, which is less sensitive to large gradients and suitable for models with large dynamic ranges; or the trace or eigenvalue method of the gradient matrix can be used, characterizing importance through the trace (equivalent to the sum of squares) or the largest eigenvalue of the outer product matrix, the latter capturing the correlation of each dimension but with higher computational cost; or a probabilistic weighting method can be used, multiplying the sum of squared gradients by the probability of the terminator at that position, focusing the weights on positions where "the terminator" is likely to end; or an adaptive threshold truncation method can be used, setting upper and lower bounds on the calculated original weights, resetting excessively small weights to zero and truncating excessively large weights to obtain a sparse and controllable weight distribution. In this way, the transformation from gradient information to importance weights is achieved, providing a quantitative basis for the subsequent construction of the weighted Hessian matrix.

[0155] This embodiment transforms abstract gradient information into concrete, usable scalar weights, enabling the subsequent construction of the weighted Hessian matrix. It quantifies the sensitivity to stopping loss into a comparable value, clearly defining the importance differences between different marker positions. Furthermore, the weights are independent of model parameters and calibration data, relying solely on the gradient itself, and possess the ability to automatically adapt to different models and calibration sets. The magnitude of the weights enables automatic identification of key marker positions without manual annotation or additional prior knowledge. Finally, the importance weights are directly used to construct the weighted second-order gradient information matrix, guiding the GPTQ quantization process to prioritize protecting marker positions with high weights, achieving the dual effect of suppressing output length expansion while maintaining model accuracy.

[0156] Step 140: Using importance weights, the original second-order gradient information matrix of the model weights of the full-precision model to be quantized, calculated based on the calibration set, is weighted to construct a weighted second-order gradient information matrix.

[0157] In this step, the importance weights (scalar values ​​for each marker position) are fused with the pre-computed original second-order gradient information matrix (typically the Hessian matrix or its approximation) based on the calibration set to construct a weighted second-order gradient information matrix. This weighted matrix replaces the original second-order gradient information matrix for subsequent GPTQ quantization, giving higher optimization priority to marker positions with higher importance weights (i.e., positions critical to output length control) during quantization error compensation.

[0158] In the embodiments of this disclosure, the weighting processing methods include, but are not limited to, diagonal scaling, diagonal power weighting, threshold binarization weighting, relative sorting weighting, structured grouping weighting, and / or adaptive normalization weighting. For example:

[0159] (1) Diagonal power weighting: Instead of directly multiplying by the weight α, it is multiplied by α. P (p>0), the influence of importance weights can be amplified or reduced by adjusting the power exponent p. For example, when p=2, the weight difference between the marked positions is further widened, and the protection is more concentrated; when p=0.5, the weight distribution is made more even.

[0160] (2) Threshold binarization weighting: Set a threshold θ, set the diagonal elements corresponding to the marker positions with importance weight greater than θ to 1, and set the rest to 0. In this way, the resulting weighting matrix is ​​a selection matrix, which only retains the Hessian information of the most critical marker positions and completely ignores the other positions. It is suitable for scenarios that emphasize a few key positions to an extreme extent.

[0161] (3) Relative ranking weighting: Instead of using the original importance weight values ​​directly, new weights are assigned according to the order of the weights (such as linear or exponential decrease according to the ranking). This can avoid the problem of numerical instability caused by the original weight values ​​being too large or too small.

[0162] (4) Structured grouping and weighting: Multiple adjacent marker positions are treated as a group, the average or maximum value of the importance weights within the group is calculated, and then the Hessian matrix block corresponding to the group is uniformly weighted using this value. This method can reduce the granularity of calculation, improve efficiency, and at the same time retain local key information.

[0163] (5) Adaptive normalization weighting: First, normalize the importance weights (e.g., divide by the maximum value or sum them), and then multiply them with the original second-order gradient information matrix so that the overall magnitude of the weighted matrix is ​​comparable to the original matrix, thus avoiding numerical overflow or underflow due to weighting.

[0164] The weighted second-order gradient information matrix output by this embodiment will be input into the subsequent GPTQ quantization process, replacing the original second-order gradient information matrix for calculating error compensation. This achieves the technical effect of prioritizing the protection of critical marker positions under a uniform quantization bit width. It also achieves the embedding of the length control objective into the quantization optimization objective through mathematical fusion, enabling the GPTQ method, which originally only focused on reconstruction error, to perceive the importance differences in output length, thereby suppressing length expansion caused by quantization without adding extra overhead.

[0165] Step 150: Based on the weighted second-order gradient information matrix, perform gradient-based post-training quantization on the full-precision model to be quantized to obtain the target quantized model. The target quantized model is then deployed to the hardware device, and the target quantized model can be called by the hardware device to process corresponding tasks.

[0166] In this step, a weighted second-order gradient information matrix is ​​used to perform gradient-based post-training quantization (i.e., GPTQ and its variants) on the full-precision model to be quantized, ultimately obtaining the target quantized model, which can then be deployed to hardware devices for use by downstream tasks.

[0167] In this embodiment, a weighted second-order gradient information matrix can be used instead of the original Hessian matrix, thereby organically integrating the identified importance weights (marked positions sensitive to stopping loss) into the quantization error compensation process. Since the weighted second-order gradient information matrix amplifies the Hessian elements corresponding to key marked positions, these positions are naturally given higher optimization priority during compensation calculations. This results in smaller quantization errors at key positions and effective suppression of output length expansion while maintaining a uniform quantization bit width (e.g., W4A8). The target quantization model obtained after quantization maintains accuracy similar to the full-precision model (thanks to the constraint of the distillation loss term) while keeping the output length growth at a low level (thanks to the protection of the stopping loss term and the weighted matrix).

[0168] In this embodiment of the disclosure, the target quantization model can be deployed to hardware platforms such as cloud servers, edge devices or terminals, and called by applications or services to perform various natural language processing tasks such as text generation, code writing, and intelligent question answering, so as to obtain the corresponding task processing results.

[0169] In some implementations, the full-precision model to be quantized and the resulting target quantization model in this disclosure embodiment can be a large language model. The target quantization model can, after inference from an input natural language question (e.g., "Please explain what gradient descent is"), output the corresponding answer text and return it to the user or downstream application. Alternatively, the target quantization model can receive code completion snippets (e.g., "def fibonacci(n):") and output the completed code lines or function bodies for use by an integrated development environment or code editor. Furthermore, in intelligent customer service scenarios, the target quantization model can receive the user's dialogue history and current question, outputting reply text and operation suggestions to assist the customer service system in responding quickly. In addition, the target quantization model can also receive multimodal input (e.g., a fusion representation of text and image features) or structured data (e.g., query conditions in tables or JSON format), outputting data analysis conclusions, summaries, or decision suggestions. In the aforementioned tasks, the input object types of the target quantization model include, but are not limited to, plain text, code snippets, dialogue history, structured queries, and embedded vector sequences; the output object types include, but are not limited to, generated text, code completion, classification labels, question-answer pairs, answer fragments, and control commands. The target quantization model obtained through the method disclosed in this paper significantly shortens the output length while maintaining high accuracy, thereby reducing inference latency and computational resource consumption. It is particularly suitable for applications with high real-time requirements and high interaction frequency, improving hardware resource utilization and the end-user experience.

[0170] This step enables the transformation from the original high-precision model to a low-precision, high-efficiency, and length-controllable quantization model, providing key technical support for the large-scale commercial deployment of inference models.

[0171] Figure 2 This is a second flowchart illustrating a model quantization method according to an embodiment of this disclosure. Figure 2 As shown, a model quantization method may further include:

[0172] Step 111: Obtain the calibration set, where each sample in the calibration set has a fixed-length word sequence.

[0173] In this embodiment of the disclosure, a small-scale unlabeled text dataset, for example containing 256 to 512 samples, can be prepared in advance as a calibration set. This calibration set can be a collection of unlabeled text samples used to collect statistical features of the model before quantization. Each sample in the calibration set is processed into a fixed-length sequence of terms (e.g., 2048 terms). For example, if the original sample is too long, it is truncated; if the original sample is too short, it is padded. This ensures that all samples have the same number of labeled positions, facilitating aggregation and statistical analysis of the same position across different samples. This ensures that the model produces aligned outputs at each labeled position, facilitating subsequent position-based gradient information analysis.

[0174] Step 112: Input the calibration set into the full-precision model to be quantized and perform forward propagation. During the forward propagation, the full-precision model to be quantized outputs the activation value of the marker position and the prediction vector for the next word at each marker position.

[0175] In this step, each sample from the calibration set is sequentially input into the full-precision model to be quantized, and a standard forward propagation is performed. For each labeled position in the sample, the model outputs not only the intermediate activation value at that position (such as the output vector of a certain layer), but also a prediction vector for the next word (i.e., the logits vector, with a dimension equal to the vocabulary size). Forward propagation is the process of obtaining the output layer result from the input layer through layer-by-layer computation, without involving backpropagation or parameter updates. The activation value is a numerical vector output by a certain intermediate layer of the model (such as the attention layer, feedforward layer, or the last hidden layer), reflecting the model's internal representation of the input information at that position. The prediction vector is the original score vector (logits) generated by the output layer, whose dimension can be equal to the vocabulary size, and can be converted into a probability distribution using the softmax function.

[0176] Step 113: Extract the log odds value corresponding to the end symbol from the prediction vector.

[0177] For each predicted vector at a given position, its index in the vocabulary is found, and the component value corresponding to the End of Sequence Token (EOS) is extracted. This value is the logit value of the end of sequence token. The end of sequence token is a special word in the vocabulary that represents the end of the sequence. Once the model predicts the end of sequence token, it stops generating. The logit value is the model's raw score for a word token at the output layer; it can be positive or negative, and its absolute value is unbounded. It can be converted into a probability using softmax.

[0178] , formula (2)

[0179] This provides an object for the direct calculation of the stop loss term.

[0180] Step 114: Record the activation value and the log-odds value of the terminator at each marked position, so as to determine the stopping loss term and the gradient information of the stopping loss term relative to the activation value of the preset mixed loss function based on the activation value and the log-odds value of the terminator at the marked position.

[0181] The activation vector and the log-probability value (scalar) of the terminator corresponding to each marked position (position index t) can be stored in pairs, for example, in two matrices: activations[t, :] and logit_eos[t]. This recorded data will be used in subsequent steps to: ① calculate the activation stopping loss (determine if P_{float}(EOS) exceeds a threshold); ② calculate the partial derivative of the stopping loss with respect to the terminator logit; ③ obtain the gradient of the stopping loss with respect to the activation value through backpropagation using the chain rule.

[0182] For example, for a sequence of length 5, the recorded data might look like Table 1 below:

[0183] 1 0.23 -15.2 2 0.45 -12.8 3 0.67 -10.1 4 0.81 -8.3 5 0.92 -6.5

[0184] These data can be used in subsequent steps, such as first calculating the probability of the end symbol logit and determining whether it is greater than a threshold (e.g., 0.5); if it is greater (e.g., the probability of position 5 is softmax(-6.5) ≈ 0.001, which is still less than 0.5, then the stopping loss is not activated and the gradient is 0; however, at the end of the actual generation, there will be a position with a high probability of the end symbol, at which time the gradient is non-zero), then the gradient information of the stopping loss with respect to the activation value is calculated.

[0185] This embodiment completes the transformation from the original model to numerical records, providing accurate input for subsequent gradient information calculation and importance weight identification. By saving the activation value at each location, gradient information can be backpropagated to each labeled location in each layer, thus achieving fine-grained identification of key labeled locations. Simultaneously, the logit record terminator enables threshold determination and partial derivative calculation for stopping loss, ultimately embedding the length control objective into the quantization optimization process.

[0186] Through the above embodiments, this disclosure can extract two types of core data necessary for subsequent quantization control from a full-precision model with extremely low computational overhead (only one forward propagation): activation values ​​(for gradient propagation) and the logit terminator (for stopping loss calculation). This data can completely preserve the information of each marked position, enabling the automatic identification of key positions most sensitive to output length and guiding differentiated protection in the quantization process accordingly. Compared to existing quantization methods that do not collect this information at all, this establishes a data foundation for length-aware quantization, supports the optimized execution of the stopping loss term in the hybrid loss function, and ultimately achieves the technical effect of maintaining accuracy while significantly suppressing output length expansion.

[0187] Figure 3 This is the third flowchart of a model quantization method according to an embodiment of this disclosure. Figure 3 As shown,

[0188] Step 121: For each marker position, calculate the probability of the full-precision model predicting the end symbol using the log odds value of the end symbol.

[0189] For each marker position t, in this step, it can be converted into a probability using a normalized exponential function (softmax), for example, by the following formula:

[0190] , formula (3)

[0191] in, The terminology represents the vocabulary, with probability values ​​ranging from [0,1], reflecting the model's confidence in predicting the next lexical term as the end-of-word symbol at the current marker position. This transforms the logarithmic probability values, which have an indefinite absolute scale, into numerical values ​​with a clear probabilistic meaning, providing a unified reference benchmark (0 to 1) for threshold setting, facilitating cross-model and cross-task use.

[0192] Step 122: Compare the probability of the end symbol predicted by the full-precision model to be quantized with the preset probability threshold.

[0193] In this step, the P calculated in step 121 is... float (EOS t ) and preset probability threshold (For example, 0.5) is compared. The result of this comparison determines the indicator function. The value of is: if Pfloat(EOSt) > τ, then the indicator function If the value is 1, the stopping loss term is activated; otherwise, the indicator function... The stopping loss term is 0, and the subsequent gradient is also 0.

[0194] In this way, the stopping loss term is constrained only at the position where the model should stop, avoiding interference with the normal generation process (the probability of the early position terminator is extremely low), and achieving precise length control.

[0195] Step 123: In response to the fact that the probability of the predicted terminator by the full-precision model to be quantized exceeds a preset probability threshold, calculate the partial derivative of the stopping loss term with respect to the log probability of the terminator based on the difference between the log probability value of the terminator and the target value, where the target value is the log probability value of the terminator output by the full-precision model to be quantized at the same marker position.

[0196] When the condition in step 122 is met (i.e., P) float (EOS t If the value is greater than τ, the stopping loss term is activated, and then the stopping loss term L is calculated. stop Logit, the terminator logarithmic probability value of the full-precision model to be quantized. quant (EOS t The partial derivative of ). The specific calculation is as follows:

[0197] For a single labeled location in a single sample, the stopping loss term is (ignoring the indicator function, since the condition is already met):

[0198] , formula (4)

[0199] Step 124: Using the chain rule, the partial derivatives are backpropagated to the activation values ​​to obtain the gradient information of the stopping loss term relative to the activation values.

[0200] The log-odds value of the terminator is a function of the activation values ​​of the model (full-precision model). In this step, the partial derivatives are backpropagated to the activation values ​​using the chain rule:

[0201] , formula (5)

[0202] in, This depends on the structure of the output layer (e.g., a linear transformation). For a fully connected output layer, this derivative is the weight vector corresponding to the row of the terminal block in the output layer. Through backpropagation layer by layer, the gradient information (gradient vector) of the stopping loss term with respect to the activation values ​​of each layer can finally be obtained. .

[0203] In this way, by using the chain rule, the prediction error of the output layer for the terminator is backpropagated to the activation value of the inner layer, thereby enabling the assessment of the sensitivity of each activation unit at each marker position to the stopping loss.

[0204] Figure 4 This is flowchart four of an embodiment of a model quantization method disclosed herein. Figure 4 As shown, a model quantization method may further include the following steps:

[0205] Step 131: For any marked position, calculate the second moment of the gradient information.

[0206] For each labeled position t, the gradient vector (gradient information) of the stopping loss term relative to the activation value has been obtained in the preceding steps. (Its dimension can be equal to the number of elements in the activation value). In this step, the second moment of the gradient information is calculated. In statistics, the second moment refers to the expected value of its square. Treating each component of the gradient vector as a random variable, the sum of their squares (or the expected value of their squares) is the second moment. The second moment is a scalar, not a matrix, and is used to measure the squared strength of the gradient information.

[0207] In one embodiment of this disclosure, for any given marker location, the sum of squares of the elements in the gradient information is calculated as the second moment of the gradient information. This is achieved by squaring each element of the gradient vector at that marker location and then summing them, for example, by taking the square of the L2 norm.

[0208] , formula (6)

[0209] in, This represents the i-th component of the gradient vector at the t-th marker location. This embodiment is applicable to independently calculating the importance weight for each sample in the calibration set, is computationally simple, and does not require cross-sample storage.

[0210] In another embodiment of this disclosure, for any given marker location, the expected value of the square of each element in the gradient information is calculated as the second moment of the gradient information. When the calibration set contains multiple samples, for the same marker location (e.g., the t-th location among all samples), the square of the gradient component of each sample can be calculated first, and then the average (i.e., the expected value) can be calculated over all samples, for example:

[0211] , formula (7)

[0212] Where N is the number of samples in the calibration set. This represents the i-th component of the gradient vector at the t-th label location in the n-th sample. This embodiment can smooth out noise between different samples, making the importance weights more stable and generalizable.

[0213] This embodiment compresses high-dimensional gradient information into a single scalar, preserving gradient magnitude information (large gradient → large weight) while removing direction information, resulting in extremely efficient computation. Regarding the drastic changes in stopping loss caused by activation value variations, the two second-moment calculation methods provided in this disclosure cover different needs from simple and fast to stable and robust.

[0214] Step 132: Determine the second moment as the importance weight of the marker position to the stop loss term.

[0215] The scalar obtained in step 131 is further calculated This serves as the importance weight for the marker position t. This weight is non-negative and automatically satisfies the following: the larger the weight, the more sensitive the position is to stopping loss, and the more it needs protection during quantization. In this embodiment, a marker position with a high weight means that even a small change in the activation value will significantly affect the stopping signal determination.

[0216] In this way, abstract gradient information is transformed into numerical weights that can be directly used in the weighting matrix, enabling automatic identification and quantization of key marker positions. No manual annotation or prior knowledge is required; it is entirely data-driven and independent of specific model architectures. More importantly, these weights are directly embedded in the GPTQ optimization objective, allowing quantization methods that previously only focused on reconstruction error to perceive the differences in the importance of output length, thereby effectively suppressing length inflation while maintaining accuracy.

[0217] Figure 5 This is the fifth flowchart of a model quantization method according to an embodiment of this disclosure. Figure 5 As shown, a model quantization method may further include the following steps:

[0218] Step 141: Use the importance weight of each marked position as a diagonal element to form a diagonal matrix.

[0219] Step 142: Multiply the diagonal matrix with the original second-order gradient information matrix to obtain the weighted second-order gradient information matrix.

[0220] In this embodiment, the original second-order gradient information matrix of the model weights of the full-precision model to be quantized, calculated based on the calibration set, is weighted to construct a weighted second-order gradient information matrix. Specifically, the importance weights of each marker position are first arranged into a diagonal matrix, and then this diagonal matrix is ​​combined with the original second-order gradient information matrix (i.e., the Hessian matrix). Multiplying these two matrices yields the weighted second-order gradient information matrix. For example:

[0221] , formula (8)

[0222] in, Represented by importance weight vector A diagonal matrix with diagonal elements. The original Hessian matrix (original second-order gradient information matrix) is used for matrix multiplication, which is left matrix multiplication.

[0223] Assume the calibration set has T labeled locations (the total number of labeled locations in all samples, or the number of locations aligned to a fixed length), and each location corresponds to an importance weight. (t=1,2,…,T), arrange these weights in positional order to form a T×T diagonal matrix D:

[0224]

[0225] Thus, by using the diagonal matrix as a weighting operator, the goal of independently weighting each marker position is achieved. Since multiplying the diagonal matrix by other matrices is equivalent to multiplying the rows of the target matrix by the corresponding diagonal elements, this structure is suitable for the requirement of position-by-position differential protection, and is computationally simple and storage-efficient.

[0226] Furthermore, the diagonal matrix D constructed in step 141 is left-multiplied by the original second-order gradient information matrix H, that is:

[0227]

[0228] Here, H is the Hessian matrix in GPTQ, which can be obtained by accumulating the product of activation values ​​on the calibration set, for example... ,in Let H be the activation vector at the t-th label position. H is a square matrix whose dimension is equal to the number of weight parameters (or the dimension after grouping). Left multiplying by D results in each row of H being multiplied by the importance weight corresponding to the label position, for example:

[0229]

[0230] Where i represents the row index, corresponding to the marker position. The i-th row in H originally depicted the second-order gradient information of the i-th marker position, multiplied by... This results in rows with high importance weights having their values ​​amplified, while rows with low importance weights are compressed. The final result is... This is the weighted second-order gradient information matrix, which will be directly used for subsequent GPTQ quantization error compensation.

[0231] For example, let the original Hessian matrix (original second-order gradient information matrix) H and the diagonal matrix D be respectively:

[0232] ,

[0233] Combining the diagonal matrix D from the previous example, the weighted matrix... for:

[0234]

[0235] Thus, it can be seen that the element in the 3rd row originally has the largest weight (0.45), and its value is significantly amplified after multiplication (e.g., 0.045 in the lower left corner → the actual calculation is correct: 0.1 * 0.45 = 0.045, 1.35 = 3.0 * 0.45); while the element in the 2nd row has a very small weight (0.02), and its value is compressed after multiplication. Therefore, in the subsequent GPTQ quantization compensation, the weighted Hessian matrix will guide the algorithm to prioritize compensating the weights corresponding to the 3rd row, protecting the critical marker positions.

[0236] In this embodiment, by constructing a diagonal matrix and left-multiplying it by the original Hessian matrix, this method integrates the output length control objective into the mathematical framework of quantization optimization with extremely low computational cost (one matrix multiplication). Ultimately, the weighted matrix directly participates in the error compensation calculation of GPTQ, enabling the model to automatically provide higher protection for critical marker positions sensitive to stopping loss under a uniform quantization bit width, thereby controlling output length expansion while maintaining accuracy. Furthermore, this operation is independent of the model architecture and can be widely applied to post-training quantization of various large language models.

[0237] Figure 6 This is flowchart six of an embodiment of a model quantization method disclosed herein. Figure 6 As shown, a model quantization method sets a weighted second-order gradient information matrix as the optimization target for error compensation. The weights of the full-precision model to be quantized are grouped, and each group of weights is quantized sequentially using an iterative method. After each group is quantized, the error compensation amount for the unquantized weights is calculated using the weighted second-order gradient information matrix and updated. Finally, after all weights are quantized, the target quantized model is obtained. Compared with the related technology GPTQ, this embodiment uses a weighted second-order gradient information matrix. It replaces the original Hessian matrix H, thereby directly embedding importance weights (i.e. key marker positions sensitive to stopping loss) into the optimization process of quantization error compensation.

[0238] Specifically, this may include the following steps:

[0239] Step 151: The weighted second-order gradient information matrix is ​​used as the optimization target for error compensation, and the weights of the full-precision model to be quantized are grouped according to the preset grouping strategy.

[0240] The optimization objective of error compensation measures the impact of quantization error on the model output. GPTQ determines how to adjust unquantized weights to offset the error introduced by the quantized portion by minimizing this objective. The preset grouping strategy can be pre-defined to divide the weights into multiple subsets, quantizing one subset at a time. This can be achieved by fixing the number of columns in each group or dynamically adjusting it. One possible approach is to group by column (output feature dimension), with one or more columns per group.

[0241] For example, in related technologies, GPTQ error compensation is achieved by minimizing... The objective function is solved using the form H, where H is the Hessian matrix. In this scheme, the optimization objective of error compensation is: ,in, This represents the weight update amount.

[0242] Step 152: Iteratively quantize each group of weights of the full-precision model to be quantized according to the preset iteration conditions to obtain the target quantized model.

[0243] Specifically, the current group weights of the full-precision model to be quantized are determined and quantized. The current group weights are the weights of the full-precision model currently being quantized. After quantizing the current group weights, the error compensation for the unquantized weights is calculated using the weighted second-order gradient information matrix, and the unquantized weights are updated based on the error compensation. This process is iteratively repeated until all weights of the full-precision model to be quantized are quantized, resulting in the target quantized model.

[0244] The unquantized weights are those in all ownership reorganizations that follow the current group weights and have not yet been processed. The error compensation is an adjustment to the unquantized weights to minimize the error; it is calculated from the inverse of the weighted Hessian matrix and the quantization error of the current group.

[0245] In detail, following the grouping order (e.g., ascending column index), the next unquantized weight group is selected as the "current group." All weights in this group undergo uniform quantization (e.g., quantization to 4-bit integers), with a quantization formula such as:

[0246] , formula (9)

[0247] in, Here, z is the quantization step size, z is the zero point, and b is the bit width. After quantization, a low-precision integer representation is obtained, which is no longer involved in subsequent updates.

[0248] Quantizing the weights of the current group will introduce errors, for example... (Original floating-point weights minus quantization values). To reduce the impact of this error on the model output, the unquantized weights need to be compensated. GPTQ calculates the compensation by solving a linear system with the weighted Hessian matrix as the objective. For the case where only one column of weights is quantized (group size 1), the formula for calculating the error compensation is as follows:

[0249] , formula (10)

[0250] in: This is the quantization error of the current group weights (scalar, or vector if there are multiple columns in the group). The weighted second-order gradient information matrix (pre-calculated based on the calibration set and importance weights) is where i represents the index of the current group in the weight matrix (corresponding to...). (rows / columns in the middle) yes The i-th diagonal element, yes The i-th column.

[0251] Then the compensation amount Added to all weights that have not yet been quantized (i.e. This offsets the error caused by the quantized groups. If the grouping contains multiple columns, the formula can be extended to matrix form accordingly.

[0252] Finally, repeat the above process: select the next group as the current group, quantize, calculate the error compensation, and update the weights that have not yet been quantized. Once all reorganizations have been processed, the quantization process ends, and the weights of the resulting target quantized model are represented by low-precision integers, which can be directly used for efficient inference.

[0253] By using iterative quantization and error compensation based on the weighted Hessian matrix (weighted second-order gradient information matrix), differentiated protection under a unified quantization bit width is achieved: that is, without changing the bit width, but by optimizing the target weighting, the positions that are key to controlling the output length (with high importance weights) have smaller quantization errors.

[0254] According to embodiments of this disclosure, output length control is explicitly incorporated into the quantization optimization objective through a preset hybrid loss function that includes a distillation loss term and a stopping loss term. The stopping loss term is activated only when the probability of the predicted terminator in the full-precision model exceeds a preset probability threshold. This avoids interference with the generation process at non-terminal positions, achieving precise constraint on the output length of the full-precision model. Furthermore, by calculating the gradient information of the stopping loss term relative to the activation value and determining the importance weight of each marker position, key positions sensitive to stopping judgment can be automatically identified. Then, this importance weight is used to weight the original second-order gradient information matrix, constructing a weighted second-order gradient information matrix. This allows the subsequent quantization process to differentially protect key positions, i.e., prioritizing the reduction of quantization error under a uniform quantization bit width. The resulting target quantization model not only maintains accuracy similar to the full-precision model through the distillation loss term but also suppresses output length expansion caused by quantization through the constraint of the stopping loss term and the protection of the weighted matrix. This ultimately reduces the growth of the output length, thereby reducing redundant computation and user waiting time, improving the deployment experience and resource utilization of the inference model.

[0255] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0256] Figure 7 This is a structural block diagram of a model quantization device according to an embodiment of this disclosure. Figure 7 As shown in the figure, a model quantization apparatus according to an embodiment of the present disclosure includes:

[0257] The data acquisition module 710 is used to perform forward propagation through the full-precision model to be quantized using a calibration set based on a preset hybrid loss function, and to acquire the activation value and log probability value of the terminator at each marker position; wherein, the preset hybrid loss function includes a distillation loss term and a stopping loss term, the distillation loss term is used to maintain the accuracy of the full-precision model to be quantized, and the stopping loss term is used to control the output length of the full-precision model to be quantized, and is activated when the probability of the full-precision model to be quantized predicting the terminator exceeds a preset probability threshold;

[0258] The gradient information determination module 720 is used to determine the gradient information of the stopping loss term relative to the activation value based on the activation value and the log probability value of the terminator.

[0259] Importance weight determination module 730 is used to determine the importance weight of each marked position to the stopping loss term based on the gradient information;

[0260] The weighted processing module 740 is used to use the importance weights to weight the original second-order gradient information matrix of the model weights of the full-precision model to be quantized, calculated based on the calibration set, and construct a weighted second-order gradient information matrix.

[0261] The model quantization module 750 is used to perform gradient-based post-training quantization on the full-precision model to be quantized based on the weighted second-order gradient information matrix to obtain a target quantized model, so as to deploy the target quantized model to a hardware device, and the target quantized model can be called by the hardware device to process corresponding tasks.

[0262] As an optional embodiment, the data acquisition module 710 includes:

[0263] A calibration set acquisition unit is used to acquire a calibration set, wherein each sample in the calibration set has a fixed-length word sequence;

[0264] A forward propagation unit is used to input the calibration set into the full-precision model to be quantized for forward propagation, wherein, during the forward propagation process, the full-precision model to be quantized outputs the activation value of the marker position and the prediction vector for the next word at each marker position.

[0265] The log odds value extraction unit is used to extract the log odds value corresponding to the end symbol from the prediction vector;

[0266] A numerical recording unit is used to record the activation value of each of the marked positions and the log-probability value of the end symbol; to determine the stopping loss term of the preset mixed loss function and the gradient information of the stopping loss term relative to the activation value based on the activation value of the marked position and the log-probability value of the end symbol.

[0267] As an optional embodiment, the gradient information determination module 720 includes:

[0268] The first calculation unit is used to calculate the probability of the full-precision model predicting the end symbol for each marked position using the log odds value of the end symbol;

[0269] The comparison unit is used to compare the probability of the prediction terminator of the full-precision model to be quantized with a preset probability threshold.

[0270] The second calculation unit is used to respond to the fact that the probability of the predicted terminator by the full-precision model to be quantized exceeds the preset probability threshold, and to calculate the partial derivative of the stopping loss term with respect to the log probability value of the terminator based on the difference between the log probability value of the terminator and the target value, wherein the target value is the log probability value of the terminator output by the full-precision model to be quantized at the same marker position.

[0271] The gradient information acquisition unit is used to backpropagate the partial derivative to the activation value using the chain rule to obtain the gradient information of the stopping loss term relative to the activation value.

[0272] As an optional embodiment, the importance weight determination module 730 includes:

[0273] The third calculation unit is used to calculate the second moment of the gradient information for any of the marked positions;

[0274] The importance weight determination unit is used to determine the second moment as the importance weight of the marker position to the stop loss term.

[0275] As an optional embodiment, the third computing unit includes:

[0276] The first calculation subunit is used to calculate the sum of squares of each element in the gradient information for any of the marked positions, so as to use the second moment of the gradient information;

[0277] or,

[0278] The second calculation subunit is used to calculate the expected value of the square of each element in the gradient information for any of the marked positions, so as to serve as the second moment of the gradient information.

[0279] As an optional embodiment, the weighted processing module 740 includes:

[0280] Diagonal matrix construction unit, used to construct a diagonal matrix by taking the importance weight of each marked position as diagonal elements;

[0281] The matrix calculation unit is used to multiply the diagonal matrix with the original second-order gradient information matrix to obtain a weighted second-order gradient information matrix.

[0282] As an optional embodiment, the model quantization module 750 includes:

[0283] The weight grouping unit is used to group the weights of the full-precision model to be quantized according to a preset grouping strategy, using the weighted second-order gradient information matrix as the optimization target for error compensation.

[0284] The quantization processing unit is used to iteratively quantize each group of weights of the full-precision model to be quantized according to preset iteration conditions to obtain the target quantized model.

[0285] As an optional embodiment, the quantization processing unit includes:

[0286] The weight quantization processing subunit is used to determine the current group weights of the full-precision model to be quantized and perform quantization processing. The current group weights are the weight regroups of the full-precision model to be quantized that are currently being quantized.

[0287] The error compensation calculation subunit is used to calculate the error compensation amount of the unquantized weights using the weighted second-order gradient information matrix after the quantization of the current group weights is completed, so as to update the unquantized weights according to the error compensation amount.

[0288] The iterative quantization subunit is used to iteratively perform the above operations until all weights of the full-precision model to be quantized are quantized, thus obtaining the target quantized model.

[0289] According to embodiments of this disclosure, output length control is explicitly incorporated into the quantization optimization objective through a preset hybrid loss function that includes a distillation loss term and a stopping loss term. The stopping loss term is activated only when the probability of the predicted terminator in the full-precision model exceeds a preset probability threshold. This avoids interference with the generation process at non-terminal positions, achieving precise constraint on the output length of the full-precision model. Furthermore, by calculating the gradient information of the stopping loss term relative to the activation value and determining the importance weight of each marker position, key positions sensitive to stopping judgment can be automatically identified. Then, this importance weight is used to weight the original second-order gradient information matrix, constructing a weighted second-order gradient information matrix. This allows the subsequent quantization process to differentially protect key positions, i.e., prioritizing the reduction of quantization error under a uniform quantization bit width. The resulting target quantization model not only maintains accuracy similar to the full-precision model through the distillation loss term but also suppresses output length expansion caused by quantization through the constraint of the stopping loss term and the protection of the weighted matrix. This ultimately reduces the growth of the output length, thereby reducing redundant computation and user waiting time, improving the deployment experience and resource utilization of the inference model.

[0290] The model quantization device of this disclosure corresponds to the model quantization methods described above in terms of specific implementation and beneficial technical effects. The relevant contents can be referred to each other, and will not be repeated here.

[0291] Below, for reference Figure 8This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0292] Figure 8 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0293] like Figure 8 As shown, the electronic device includes one or more processors and memory.

[0294] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0295] The memory can store one or more computer program products, and the memory can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products can be stored on the computer-readable storage medium, and the processor can run the computer program products to implement the model quantization methods of the various embodiments of this disclosure described above and / or other desired functions.

[0296] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0297] In addition, the input device may also include, for example, a keyboard, a mouse, etc.

[0298] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0299] Of course, for the sake of simplicity, Figure 8 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0300] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the model quantization methods according to various embodiments of this disclosure as described in the foregoing portions of this specification.

[0301] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0302] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the model quantization methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0303] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0304] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0305] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0306] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0307] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0308] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0309] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0310] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A model quantization method, characterized in that, The method includes: Based on a preset hybrid loss function, a calibration set is used to perform forward propagation through the full-precision model to be quantized, obtaining the activation value and the log probability value of the terminator at each marker position; wherein, the preset hybrid loss function includes a distillation loss term and a stopping loss term, the distillation loss term is used to maintain the accuracy of the full-precision model to be quantized, and the stopping loss term is used to control the output length of the full-precision model to be quantized, and is activated when the probability of the full-precision model predicting the terminator exceeds a preset probability threshold; Based on the activation value and the log-odds value of the terminator, determine the gradient information of the stopping loss term relative to the activation value; Based on the gradient information, determine the importance weight of each marked position to the stopping loss term; Using the importance weights, the original second-order gradient information matrix of the model weights of the full-precision model to be quantized, calculated based on the calibration set, is weighted to construct a weighted second-order gradient information matrix; Based on the weighted second-order gradient information matrix, gradient-based post-training quantization is performed on the full-precision model to be quantized to obtain the target quantized model, which is then deployed to a hardware device. The target quantized model can be called by the hardware device to process corresponding tasks.

2. The method according to claim 1, characterized in that, The method, based on a preset hybrid loss function, uses a calibration set to perform forward propagation through the full-precision model to be quantized, obtaining the activation value and log-odds value of the terminator at each marker position, including: Obtain a calibration set, wherein each sample in the calibration set has a fixed-length word sequence; The calibration set is input into the full-precision model to be quantized for forward propagation, wherein, during the forward propagation process, the full-precision model to be quantized outputs the activation value of the marker position and the prediction vector for the next word at each marker position. Extract the logarithmic probability value corresponding to the terminator from the prediction vector; Record the activation value of each of the marked positions and the logarithmic probability value of the end symbol; The stopping loss term of the preset hybrid loss function and the gradient information of the stopping loss term relative to the activation value are determined based on the activation value of the marker position and the log probability value of the terminator.

3. The method according to claim 1, characterized in that, Determining the gradient information of the stopping loss term relative to the activation value based on the logarithmic probability value of the activation value and the terminator includes: For each marker position, the probability of the full-precision model predicting the end symbol is calculated using the log odds value of the end symbol; The probability of the full-precision model predicting the end symbol to be quantized is compared with a preset probability threshold. In response to the probability that the full-precision model to be quantized predicts the end symbol exceeding the preset probability threshold, the partial derivative of the stopping loss term with respect to the log probability value of the end symbol is calculated based on the difference between the log probability value of the end symbol and the target value, wherein the target value is the log probability value of the end symbol output by the full-precision model to be quantized at the same marker position. Using the chain rule, the partial derivative is backpropagated to the activation value to obtain the gradient information of the stopping loss term relative to the activation value.

4. The method according to any one of claims 1-3, characterized in that, The step of determining the importance weight of each marked position to the stopping loss term based on the gradient information includes: For any of the marked positions, calculate the second moment of the gradient information; The second moment is determined as the importance weight of the marker position to the stopping loss term.

5. The method according to claim 4, characterized in that, The step of calculating the second moment of the gradient information for any of the marked positions includes: For any of the marked positions, the sum of squares of each element in the gradient information is calculated to serve as the second moment of the gradient information; or, For any of the marked positions, the expected value of the square of each element in the gradient information is calculated, and used as the second moment of the gradient information.

6. The method according to any one of claims 1-3, characterized in that, The step of using the importance weights to weight the original second-order gradient information matrix of the model weights of the full-precision model to be quantized, calculated based on the calibration set, to construct a weighted second-order gradient information matrix includes: The importance weight of each marked position is used as the diagonal element to form a diagonal matrix; Multiplying the diagonal matrix by the original second-order gradient information matrix yields the weighted second-order gradient information matrix.

7. The method according to any one of claims 1-3, characterized in that, The step of performing gradient-based post-training quantization on the full-precision model to be quantized based on the weighted second-order gradient information matrix to obtain the target quantized model includes: The weighted second-order gradient information matrix is ​​used as the optimization target for error compensation, and the weights of the full-precision model to be quantized are grouped according to a preset grouping strategy. The weights of each group of the full-precision model to be quantized are iteratively quantized according to preset iteration conditions to obtain the target quantized model.

8. The method according to claim 7, characterized in that, The step of iteratively quantizing each group of weights of the full-precision model to be quantized according to preset iteration conditions to obtain the target quantized model includes: Determine the current group weights of the full-precision model to be quantized and perform quantization processing. The current group weights are the weights of the full-precision model to be quantized that are currently being quantized. After the quantization of the current group weights is completed, the error compensation amount of the unquantized weights is calculated using the weighted second-order gradient information matrix, and the unquantized weights are updated according to the error compensation amount. The above operations are performed iteratively until all weights of the full-precision model to be quantized are quantized, thus obtaining the target quantized model.

9. A model quantization device, characterized in that, The device includes: The data acquisition module is used to perform forward propagation through the full-precision model to be quantized using a calibration set based on a preset hybrid loss function, to obtain the activation value and log probability value of the terminator at each marker position; wherein, the preset hybrid loss function includes a distillation loss term and a stopping loss term, the distillation loss term is used to maintain the accuracy of the full-precision model to be quantized, and the stopping loss term is used to control the output length of the full-precision model to be quantized, and is activated when the probability of the full-precision model to predict the terminator exceeds a preset probability threshold; The gradient information determination module is used to determine the gradient information of the stopping loss term relative to the activation value based on the logarithmic probability value of the activation value and the terminator; The importance weight determination module is used to determine the importance weight of each marked position to the stopping loss term based on the gradient information. The weighting processing module is used to use the importance weights to weight the original second-order gradient information matrix of the model weights of the full-precision model to be quantized, calculated based on the calibration set, and construct a weighted second-order gradient information matrix. The model quantization module is used to perform gradient-based post-training quantization on the full-precision model to be quantized based on the weighted second-order gradient information matrix to obtain a target quantized model, which is then deployed to a hardware device. The target quantized model can be called by the hardware device to process corresponding tasks.

10. An electronic device, characterized in that, include: Memory, used to store computer program products; A processor for executing a computer program product stored in the memory, wherein when the computer program product is executed, it implements the method described in any one of claims 1-8.

11. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-8.

12. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-8.

Images