Model quantification method and device, storage medium and program product

Abstract

One or more embodiments of the invention provide a model quantification method and apparatus, a storage medium and a program product. The method comprises the steps of reconstructing a weight of at least one network layer in a to-be-quantized initial model based on an orthogonal rotation matrix; performing quantitative perception training on the reconstructed model to jointly optimize the weight of the at least one network layer and the corresponding rotation matrix; the optimized weight is quantified to generate a target model, and the weight precision of the target model is lower than that of the initial model.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of computer technology, and more particularly to a model quantization method, apparatus, storage medium, and program product. Background Technology

[0002] As large language models continue to grow in scale, their high computational and storage requirements severely limit their deployment in resource-constrained environments such as edge devices. Therefore, model quantization technology, by converting model parameters and activation values ​​from high precision to low-bit representation, can significantly reduce model size and computational overhead, making it a key means to promote the practical application of large models. However, during the quantization process, outliers in the weights or activation values ​​will greatly stretch the dynamic range of the quantized data, causing severe compression of the effective bits used to represent the vast majority of normal values, thus introducing significant quantization errors and causing a sharp decline in model performance. Therefore, how to effectively address the outlier problem while maintaining the advantage of low bit width and ensuring the accuracy of the quantized model has become a core requirement that the industry urgently needs to address in the pursuit of efficient deployment of large models.

[0003] In related technologies, various solutions have been proposed to mitigate the negative impact of outliers on quantization accuracy. For example, a smoothing factor can be introduced to shift the difficulty between activation values ​​and weights, balancing their quantization complexity; or a mixed-precision strategy can be adopted to preserve outliers in a high-precision format for processing. Alternatively, a rotation matrix can be used to optimize the data distribution, aiming to make the distribution more compact before quantization. However, these methods all fall under the category of Post-Training Quantization (PTQ), and their core limitation lies in the inability to fundamentally optimize the model weights themselves. Especially after fine-tuning the model, new outliers may reappear within the network layers, causing previously calibrated quantization parameters to become invalid, recurring performance degradation problems, and making it difficult to fundamentally guarantee the long-term stability of the quantized model during dynamic adjustment. Summary of the Invention

[0004] In view of the above, one or more embodiments of this specification provide the following technical solutions: According to a first aspect of one or more embodiments of this specification, a model quantization method is proposed, the method comprising: The weights of at least one network layer in the initial model to be quantized are reconstructed based on the orthogonal rotation matrix; Quantization-aware training is performed on the reconstructed model to jointly optimize the weights and corresponding rotation matrices of at least one network layer; The optimized weights are quantized to generate a target model, the weight accuracy of which is lower than that of the initial model.

[0005] According to a second aspect of one or more embodiments of this specification, a model quantization apparatus is provided, the apparatus comprising: The weight reconstruction unit is used to reconstruct the weights of at least one network layer in the initial model to be quantized based on the orthogonal rotation matrix. A quantization-aware training unit is used to perform quantization-aware training on the reconstructed model to jointly optimize the weights and corresponding rotation matrices of at least one network layer. The weight quantization unit is used to quantize the optimized weights to generate a target model, wherein the weight accuracy of the target model is lower than that of the initial model.

[0006] According to a third aspect of this specification, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0007] According to a fourth aspect of this specification, a computer program product includes a computer program / instructions that, when executed by a processor, implement the steps of the method described in the first aspect.

[0008] As demonstrated by the above embodiments, this specification effectively overcomes the fundamental shortcomings of existing post-training quantization techniques by deeply integrating orthogonal rotation matrices with quantization-aware training (QAT). This method abandons passively correcting outliers after model finalization, instead actively guiding weight reconstruction during training. By jointly optimizing the weights of the model's network layers and their corresponding rotation matrices, it enables the model to learn more easily quantized internal representations from the outset, thereby significantly improving the shape of the weight distribution and avoiding excessive stretching of the quantization dynamic range by outliers. In summary, this method not only ensures lower errors during quantization but also maintains quantization-friendly weights under the guidance of the rotation matrix's structural regularization term during subsequent model fine-tuning. This fundamentally ensures the long-term stability and robustness of the quantized model during dynamic adjustment, achieving a better balance between accuracy and efficiency. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating a model quantization method according to an embodiment disclosed in this specification; Figure 2 This is a schematic diagram illustrating the logical execution flow of a model quantization system according to an embodiment disclosed in this specification; Figure 3 This is a flowchart illustrating another model quantization method shown in the embodiments disclosed in this specification; Figure 4 This is a schematic structural diagram of an electronic device shown in the embodiments of this specification; Figure 5 This is a block diagram of a model quantization device illustrated in an embodiment of this specification. Detailed Implementation

[0010] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.

[0011] Figure 1 This is a schematic flowchart illustrating a model quantization method as shown in an exemplary embodiment of this specification. Figure 1 As shown, the method may include the following steps: Step S102: Reconstruct the weights of at least one network layer in the initial model to be quantized based on the orthogonal rotation matrix.

[0012] For at least one network layer in the initial model to be quantized, an orthogonal rotation matrix can be introduced. For example... Figure 2 As shown in step S202, the above weight reconstruction is the first step in the model quantization process. The orthogonal rotation matrix can be applied to the weight parameters of a specified network layer and reconstructed structurally through linear transformation, thereby optimizing the numerical distribution characteristics of the weights and laying the foundation for subsequent low-precision quantization. Its core lies in mapping the original weight space to a new representation space that is more conducive to quantization through matrix transformation.

[0013] The core of the above reconstruction mechanism lies in introducing an orthogonal rotation matrix as a structural regularization term into the subsequent quantization-aware training process. This is not passive data preprocessing, but rather actively guides the optimization direction of the weights, causing them to converge towards a region with low quantization error during training. Specifically, the rotation operation aims to make the reconstructed weight distribution more concentrated and compact, thus enabling the use of a limited number of low bits to more accurately represent the vast majority of weight values ​​during quantization, effectively reducing rounding errors. More importantly, this improvement in distribution characteristics fundamentally enhances the numerical stability of the model, ensuring that the quantized model exhibits smoother changes in internal activation and final output when faced with small perturbations in the input, avoiding drastic performance fluctuations caused by abnormal distributions or amplified quantization noise.

[0014] Step S104: Perform quantization-aware training on the reconstructed model to jointly optimize the weights and corresponding rotation matrices of at least one network layer.

[0015] After the initial reconstruction of the corresponding weights is completed, the quantization-aware training phase can begin. In this phase, the model does not directly undergo static quantization, but is retrained during the forward propagation process simulating quantization. In other words, this training process is jointly optimized; that is, not only are the weight parameters of the network layers themselves updated, but the parameters of the associated rotation matrices are also optimized simultaneously. This collaborative optimization mechanism ensures that the model can actively adapt to quantization noise during the learning process and guides the weights and rotation matrices to converge towards an optimal solution with smaller quantization error.

[0016] Theoretically, the ultimate goal of the above training process is to achieve joint optimization of the weights and rotation matrices, which can be expressed as minimizing the loss function of the quantized model on a specific task. Specifically, this joint optimization problem can be formally defined as: given that the rotation matrices Ri of each layer are orthogonal matrices, i.e. Under this constraint, we seek the optimal weight set {Wi} and rotation matrix set {Wi} such that, after quantization function... Overall model loss after mapping Minimize it, and its objective function is: in, This indicates that the forward propagation process of quantization is simulated during training; This represents the total number of layers in the initial model.

[0017] However, while the objective function described above fully describes the goal of the joint optimization, it is computationally intensive to directly solve for it due to the extremely high computational cost of simultaneously optimizing the weights of all layers and the rotation matrix under orthogonal constraints. Therefore, this specification provides a more efficient alternating optimization framework that decomposes the complex joint optimization problem into two iteratively solvable subproblems, thereby significantly reducing the computational burden while maintaining performance.

[0018] Specifically, for each of the at least one network layer mentioned above, before the optimization termination condition is met, the following two steps can be performed alternately: First, fix the current rotation matrix of the layer. At this point, the problem degenerates into standard quantization-aware training. Update the weight parameters of the layer using the backpropagation algorithm to better adapt it to quantization under the current rotation. Then, fix the weights that have just been updated for the layer, and instead, according to a preset rotation matrix search strategy, determine an optimal rotation matrix for the set of weights that makes its distribution more compact and its quantization error smaller, and update it. Through multiple alternating iterations of this "weight update-rotation search," the model can be guided to a quantization-friendly optimal state within a feasible computational cost. The following will further describe these two steps.

[0019] In the above alternating optimization framework, once the current rotation matrix of a certain network layer is fixed, the optimization phase for the weights of that layer begins, such as... Figure 2 As shown in step S206, this process can be viewed as a lower-level optimization problem. For example... Figure 2 As shown in steps S216-S218, the specific implementation process of this lower-layer optimization includes the following two steps: First, in the forward computation graph, pseudo-quantized nodes are inserted into the weight path and the output path of their activation values ​​of this layer to accurately simulate the effects of quantization and dequantization during training. Second, forward and backward propagation are performed based on the fine-tuning dataset to update the current weight parameters of this layer. In summary, this weight update process aims to directly minimize the error caused by the quantization of weights and activation values, and its lower-layer optimization objective function can be defined as: in, This represents the total quantitative error of the entire model under a specific weight configuration {Wi} and rotation matrix {Ri}; Wi: specifically refers to the weight tensor of the i-th layer; Ri is the orthogonal rotation matrix corresponding to the weights of the i-th layer; This indicates that a linear transformation L is applied to the weights Wi, which includes operations with the rotation matrix Ri, and its output is the transformed high-precision weight tensor. This indicates that the above high-precision weight tensor The result of performing the quantization operation Q is a low-bit representation; It is the Frobenius norm squared, i.e. mean square error, between the transformed and quantized weights of the i-th layer and the high-precision values ​​before the transformation. This term directly measures and penalizes the information loss caused by weight quantization. This represents the input activation tensor generated by the i-th layer when processing the j-th calibration data sample, where n is the number of samples sampled from the dataset; and The meanings are similar to those of the weight terms mentioned above, representing the effects on the activation tensor. High-precision results after a linear transformation involving rotation and the quantized results; This term is used to calculate the numerical error of the activation value of the i-th layer for the j-th sample after transformation and quantization. This ensures that the dynamic range distribution of the activation values ​​is quantization-friendly during forward propagation.

[0020] In summary, the above objective function passes through its first term, namely... This minimizes the static weight quantization error, while also passing its second term. That is... This minimizes dynamic, data-dependent activation quantization error. Together, these two factors guide the weights to optimize in a direction that simultaneously suppresses weight and activation quantization distortion, given a fixed rotation matrix.

[0021] Under the premise of a fixed rotation matrix, the above weight optimization process is formally equivalent to standard quantization-aware training, which is consistent with... Figure 2 This corresponds to step S218 in the previous section. However, its core advantage lies in the fact that the rotation matrix previously introduced in this specification has already performed a linear transformation on the distribution of weights or activations before the pseudo-quantization operation. This operation can effectively suppress outliers and compress the dynamic range. The improvement in distribution characteristics directly leads to a reduction in quantization error, which not only improves model accuracy but also reduces the interference of quantization error on gradient estimation during backpropagation. This makes the gradient approximation based on the Straight-Through Estimator (STE) more stable and accurate, thereby significantly improving the convergence and robustness of the entire quantization-aware training process.

[0022] After completing the lower-level optimization for the weights, such as Figure 2 As shown in step S204, the alternating optimization framework can then alternately transition to the upper-level optimization stage, i.e., updating the rotation matrix based on the currently fixed weights. This upper-level optimization problem can be formally described as: finding a set of optimal rotation matrices { under the constraint of satisfying the orthogonality of the rotation matrices}. This allows the model to minimize the lower-level quantization error function E, which is jointly defined by the weights and activations, thereby indirectly optimizing the task loss of the entire quantization model. Optimizing the rotation matrix under fixed weights is essentially a non-smooth manifold optimization problem involving orthogonal constraints. Furthermore, due to quantization operations, the objective function is non-differentiable, making traditional gradient-based methods difficult to apply directly. Therefore, this specification proposes a low-complexity rotation matrix search strategy. The core of this strategy is to shift the optimization space from a continuous manifold constraint to a finite, predefined set of rotation matrix candidates. This candidate set can include at least two of the following: a Walsh-Hadamard matrix (H), a standard orthogonal matrix with efficient and fast algorithms and sparsity; a Block-Hadamard matrix (B), which balances flexibility and computational cost by decomposing a large matrix into smaller Hadamard matrices; and an Identity Matrix (I), representing the baseline case without rotation. Based on this, the above-mentioned candidate set of rotation matrices can be represented in the form of, for example, {H, B, I}, {H, B}, {H, I}, {B, I}. Of course, other matrix types can be added to the above-mentioned candidate set of rotation matrices according to actual needs, and this specification does not impose any restrictions on this.

[0023] In other words, the process of determining and updating the current rotation matrix of this layer based on the above rotation matrix search strategy can be regarded as selecting the matrix that produces the minimum quantization error from the candidate rotation matrix set based on preset matrix selection conditions as the new rotation matrix for the current layer. Here, the so-called matrix selection conditions can be the objective function of the lower layer when using each candidate matrix. The value of is selected, and the smallest value is chosen. Alternatively, the current rotation matrix can be selected based on the sensitivity described below, which is not restricted in this specification.

[0024] In summary, this strategy of searching from a finite candidate set can transform complex manifold optimization into an efficient enumeration and evaluation process, significantly reducing computational complexity. Furthermore, different rotation matrices in the candidate set can possess different properties. For example, using randomly generated Walsh-Hadamard matrices not only benefits computationally from their fast matrix multiplication properties, reducing computational and storage overhead, but also helps improve the model's generalization ability. Thus, while providing near-optimal rotation solutions, it ensures the efficiency and practicality of the entire optimization process. Similarly, block-based Hadamard matrices are more conducive to balancing efficiency and effectiveness, or identity matrices guarantee zero computational resource overhead.

[0025] It is worth noting that although the above embodiments describe the alternating optimization framework's process as a loop of "updating weights first, then updating the rotation matrix" for clarity, from a complete optimization logic perspective, each effective iteration should begin with the determination of the rotation matrix, i.e., upper-level optimization, followed by weight updates, i.e., lower-level optimization. The reason this framework allows the first round of alternation to begin with lower-level optimization is that the initial step of "reconstructing weights based on the orthogonal rotation matrix" in step S102 provides an initialized rotation matrix for the network layer. This step is essentially equivalent to completing an implicit "upper-level optimization" before executing the first alternating loop, allowing subsequent loops to naturally start from the lower-level optimization step of "fixing the rotation matrix and updating weights." Conversely, for network layers that have not undergone this initial reconstruction step, their optimization process must strictly follow the logical order of "upper-level optimization first, then lower-level optimization," i.e., first initialize the rotation matrix, then update the weights. Furthermore, upper-level and lower-level optimizations will stop when the joint optimization objective corresponding to the above objective function is satisfied.

[0026] Returning to the matrix selection criteria mentioned in the rotation matrix search strategy above, such as... Figure 2 As shown in step S208, in the process of selecting the rotation matrix through upper-level optimization, this specification introduces a quantization sensitivity-driven hierarchical rotation mechanism to further improve the balance between quantization performance and computational efficiency. The core of this mechanism lies in the fact that different layers in a deep neural network have significantly different sensitivities to quantization errors. Therefore, compared to using a uniform rotation strategy, different types of rotation matrices can be adaptively allocated according to the sensitivity of each layer, thereby concentrating computational resources on the most critical layers that require the most protection.

[0027] In one embodiment, the sensitivity of the corresponding network layer to quantization error can be determined, and a rotation matrix matching the aforementioned sensitivity can be determined as the current rotation matrix based on a preset matrix sensitivity correspondence. This sensitivity can be quantified by comparing the difference between the first output distribution of the layer in floating-point state and the second output distribution in quantized state. The sensitivity can be measured using methods such as KL divergence (Kullback-Leibler Divergence), Jensen-Shannon Divergence (JS Divergence), Wasserstein distance (Earth-Mover Distance), and Hellinger distance, depending on actual needs. Below, using KL divergence as an example, a sensitivity measurement function Si is defined to measure the sensitivity of the i-th layer: in, and These represent the effects of the layer on the calibration samples under the floating-point model and the quantization model, respectively. The output distribution is given by , where n is the number of samples. A larger KL divergence value Si indicates a more significant change in the output distribution due to quantization, meaning it is more sensitive to quantization. The above output distribution can be approximated using the Softmax function or activation histogram normalization; this specification does not impose any restrictions on this method.

[0028] Based on the calculated sensitivity {Si} of each layer, the network layers can be divided into three sensitivity levels: high, medium, and low. A matrix-sensitivity correspondence is then established, and different rotation matrices are assigned to layers of different sensitivity levels. The first level, the high-sensitivity layer, is where Si is greater than a preset first threshold: it is extremely sensitive to quantization errors. For this purpose, a complete Walsh-Hadamard matrix can be used. And satisfying its orthogonality Under the premise of matrix multiplication: Achieving full-dimensional feature rotation effectively disperses feature orientation, suppresses outliers, and compresses the dynamic range, thereby minimizing quantization error.

[0029] The second level, the intermediate sensitivity layer, occurs when Si is between a preset first threshold and a second threshold, with the first threshold being greater than the second threshold. A block-based Hadamard matrix is ​​used, assuming the dimension d is divided into k blocks, each sub-block... (b=d / k): And perform local rotations independently for each sub-block: The above operations, while introducing appropriate distribution adjustments to reduce quantization errors, maintain low computational complexity, achieving a balance between effectiveness and efficiency.

[0030] The third level, the low-sensitivity layer, where Si is less than a preset second threshold, exhibits strong quantization robustness, with minimal changes in output distribution before and after quantization. In this case, the optimal strategy is to choose an identity matrix. That is, no rotation is performed, and the original weight expression is preserved: This follows the principle of minimum perturbation, avoids unnecessary computational overhead, and is beneficial to training stability and inference efficiency.

[0031] Those skilled in the art will understand that the rotation matrices matched for each of the above sensitivity levels can be directly selected from the candidate set of rotation matrices, or they can be selected from other independent sets. Furthermore, the classification of the above sensitivity levels can be further subdivided or merged based on the number of types of available rotation matrices.

[0032] In summary, the quantization sensitivity-driven mechanism described above, through dynamic evaluation and hierarchical configuration, ensures that the rotation enhancement operation is precisely applied to the network layers most susceptible to quantization damage, corresponding to... Figure 2 Steps S210-S212 involve determining the corresponding sensitivity level based on the determined sensitivity Si, and then updating the corresponding network layer with the rotation matrix corresponding to the sensitivity level. This not only achieves an optimal trade-off between quantization error suppression and computational overhead in the algorithm, but also significantly enhances the robustness and performance preservation of the final quantization model.

[0033] Of course, the layer rotation mechanism described above, based on initial sensitivity assessment, is typically considered a static strategy during model training. However, model training is a dynamic process: a network layer may be extremely sensitive to quantization errors in the early stages of training, but as the weight parameters are continuously optimized, its distribution characteristics may change, making it more robust to quantization. Using only a fixed rotation strategy throughout the entire training cycle will not perfectly adapt to the dynamic evolution of the model's internal state.

[0034] Based on this, this specification proposes a dynamic matrix update strategy to further enhance the model's adaptability during training, thereby continuously optimizing quantization performance and improving model robustness. For example... Figure 2 As shown in step S214, after the initial update of the rotation matrix based on sensitivity, it is determined whether to repeat the above process based on the matrix update conditions in the diagram. The core of this strategy is to transform the sensitivity assessment and rotation matrix selection from a one-time static operation into a periodic dynamic process. Specifically, in the quantization-sensory training phase, the above matrix update conditions include two types of judgment criteria: 1. Periodic update conditions, serving as the triggering conditions for updates, that is... Figure 2 The trigger condition for jumping back from S214 to S208 is such that the sensitivity of the layer can be recalculated every preset number of iterations, such as every k training steps. This indicator also considers how to update the rotation matrix.

[0035] 2. The freeze period trigger condition serves as the termination condition for the update, i.e. Figure 2The trigger condition for jumping back to the end of the step from S214 is that when the training progress reaches a preset threshold, the freeze mechanism is triggered, stopping all subsequent matrix updates. This threshold is a quantifiable and predefined objective standard, usually judged in one of the following ways: Method 1, based on the number of iterations / training steps: For example, when the total number of training iterations reaches 90% or 95% of the preset total number. This is the most direct and commonly used method; Method 2, based on validation set performance: For example, when the model's performance on the validation set, such as accuracy, no longer improves within N consecutive evaluation periods, or the improvement is lower than a certain threshold, the model is considered to be close to convergence, and a freeze can be triggered.

[0036] In summary, based on the previously calculated latest sensitivity Si and according to the preset matrix sensitivity correspondence, the current best-matching rotation matrix can be redefined and updated. The implementation of this dynamic strategy can be roughly divided into three stages: 1. Initialization Phase: At the start of training, all network layers are initialized based on their initial sensitivities. Determine its rotation matrix.

[0037] 2. Periodic Evaluation and Update Phase: During training, the sensitivity of each layer is periodically re-evaluated, and its rotation matrix is ​​dynamically adjusted. This allows a layer to change from its initial sensitivity level to another level as training progresses, thus switching its rotation matrix accordingly. For example, when changing from medium sensitivity to low sensitivity, the current rotation matrix of that layer can be switched from the block Hadamard matrix B to the identity matrix I, thereby achieving adaptive resource allocation.

[0038] 3. Late Freeze Phase: When the training is about to end according to the freeze period trigger condition in the matrix update condition above, the update of the rotation matrix can be stopped and frozen, thereby avoiding unstable disturbances introduced by matrix switching in the final stage of training and ensuring that the model can converge smoothly.

[0039] Through the dynamic update mechanism described above, the selection of the rotation matrix can continuously track and respond to changes in the model's internal state during training, achieving adaptive learning and fine-grained control of the rotation structure. This not only further enhances the suppression of quantization errors but also helps improve the generalization performance and stability of the quantization model.

[0040] Step S106: Quantize the optimized weights to generate a target model, wherein the weight accuracy of the target model is lower than that of the initial model.

[0041] After thorough quantization-aware training and joint optimization, the weights of the initial model have reached a stable and quantization-friendly state. Finally, a final quantization operation is performed on the optimized weights, such as converting the original high-precision FP32 values ​​to low-bit formats like INT8 or INT4. This step generates a target model with significantly lower weight precision than the initial model, effectively reducing model storage and computational overhead, and facilitating deployment in resource-constrained environments.

[0042] After completing quantization-aware training and quantizing the optimized weights to generate the target model, there are two efficient and practical ways to process the rotation matrix optimized during training: First, the optimized rotation matrix is ​​merged into the optimized corresponding weights. Specifically, before model deployment, the rotation operation can be pre-integrated into the weights using matrix multiplication. For example, for a layer of weights W and its corresponding rotation matrix R, the merged weight W′ can be calculated as W′=WR. Afterward, the quantization operation only needs to be applied to the merged weight W′, while the rotation matrix RR itself does not need to be retained.

[0043] Secondly, the optimized rotation matrix is ​​deployed independently as a fast orthogonal transformation operator for the target model. In this approach, the rotation matrix is ​​not merged with its applied weights, but rather serves as an independent, lightweight pre-processing or post-processing module. During actual inference, leveraging the structured characteristics of this rotation matrix, the rotation operation can be equivalently implemented using highly optimized fast transformation algorithms such as the Fast Walsh–Hadamard–Hartley Transform (FWHT), thereby avoiding large-scale matrix multiplications and significantly improving computational efficiency.

[0044] Those skilled in the art will understand that the reason why this step only quantizes the optimized weights, rather than quantizing the weights and rotation matrix together with low bits, is due to the essential properties and working mechanism of the rotation matrix: First, the rotation matrices involved in the various embodiments of this specification are all orthogonal matrices with strict mathematical structures. Their value lies in the mathematical properties brought about by their orthogonality. Therefore, the rotation matrix can be regarded as a fixed transformation operator, rather than parameters that need to learn subtle changes from data, like neural network weights. Second, low-bit quantization of the rotation matrix itself will inevitably introduce rounding errors, which will seriously destroy its strict orthogonality. Once the orthogonality is destroyed, the mathematical foundation on which the rotation matrix depends will no longer exist, and its core functions of scattering outliers and compressing dynamic range will also be lost. Finally, in the actual deployment of the model, the key step is to perform the "rotation" operation, rather than storing every element of the rotation matrix. For structured matrices, the computational complexity is extremely low. Therefore, we can also choose to retain the definition of the transformation operator in floating-point form as described in the second method above, and call an efficient transformation library during inference. This will not introduce significant storage or computational overhead, so there is no need to quantize it.

[0045] In summary, the aforementioned rotation matrix participates in the optimization process with its unique structured identity. Ultimately, it can be integrated into the inference process through merging or rapid transformation, thereby ensuring the final performance and efficiency of the quantized model while avoiding its own quantization.

[0046] Figure 3 This is a schematic flowchart illustrating a model quantization method as shown in an exemplary embodiment of this specification. Figure 3 As shown, the method may include the following steps: Step S302: Initialize the model.

[0047] In one embodiment, the model quantization system can reconstruct the weights of each network layer in the initial model to be quantized based on an orthogonal rotation matrix, so as to provide a technical basis for subsequent quantization-aware training.

[0048] Step S304: Perform one cycle of quantitative perception training.

[0049] In one embodiment, the system can begin executing a dynamic update strategy. First, the system loads a model that has undergone initial weight reconstruction, which is equivalent to completing an implicit upper-layer optimization, providing the network layers with the initial state of the rotation matrix. Then, based on this initial configuration, the system performs quantization-aware training for a preset number of iterations, such as one epoch or several steps. During this process, the system inserts pseudo-quantization nodes into the feedforward graph to simulate quantization effects and updates the network layer weight parameters through backpropagation and a pass-through estimator (STE), aiming to minimize task loss and quantization error; this process corresponds to lower-layer optimization.

[0050] Step S306: Determine whether the periodic matrix update condition is met.

[0051] In one embodiment, the system makes a judgment based on a preset periodic update condition, which is the triggering criterion for the dynamic update strategy. For example, the system checks whether the current number of training iterations since the last evaluation has reached a preset period, i.e., k training steps. If the condition is not met, the process returns to step S304 and continues to execute the weight update training for the next period; if the condition is met, it means entering the matrix evaluation and update window, and the process is triggered and proceeds to step S308.

[0052] Step S308: Re-evaluate the quantization sensitivity of each layer and update the rotation matrix accordingly.

[0053] In one embodiment, the system can pause the iterative update of weights and instead perform upper-layer optimization operations. Specifically, it first uses a calibration dataset to recalculate the latest quantization sensitivity for each network layer to be optimized. This calculation can be based on methods such as KL divergence, by comparing the difference between the first output distribution of the layer in floating-point state and the second output distribution in quantized state. Subsequently, the system calculates the latest... The sensitivity value is compared with a preset threshold range to redetermine the sensitivity level of each network layer. Assuming this includes high, medium, and low levels, and based on a preset matrix sensitivity correspondence, the new rotation matrix type to which the layer requiring modification should be switched is determined. Assuming these three levels correspond one-to-one with matrices H, B, or I, the actual matrix update operation is then performed. This achieves adaptive learning of the rotation structure.

[0054] Step S310: Determine whether the training freeze condition is met.

[0055] In one embodiment, the system can determine the termination criteria of the dynamic update strategy based on a preset freeze period trigger condition. For example, the system checks whether the current training progress has reached a preset proportion of the total number of iterations, such as 90% or 95%, or whether the validation set performance has not improved for N consecutive evaluation cycles. If the freeze condition is not met, the process returns to step S304 and begins the next "training-evaluation-update" cycle; if the freeze condition is met, it indicates that the model is close to convergence, and to avoid late-stage perturbations, the system will exit the update cycle, and the process will proceed to the final termination step S312.

[0056] Step S312: Freeze the rotation matrix and continue training until final convergence.

[0057] In one embodiment, once the training freeze condition is met, the system can exit the dynamic matrix update loop. In all subsequent training iterations, the rotation matrices of each layer will be fixed and no further updates will be made. Training will focus solely on fine-tuning the weights, continuing until full convergence. This avoids instability introduced by matrix switching at the end of training, ensuring smooth model convergence. Finally, the system will obtain a set of optimized weights and a defined rotation matrix, preparing for final weight quantization to generate an efficient target model.

[0058] Figure 4 This is a schematic structural diagram of an electronic device according to an exemplary embodiment. Please refer to... Figure 4 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile storage, and may also include other necessary hardware. The processor reads the corresponding computer program from the non-volatile memory into memory and then executes it, forming a model-based quantization device at the logical level. Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0059] Figure 5 This specification illustrates a block diagram of a model quantization device through embodiments. Please refer to... Figure 5 This device can be applied to, for example Figure 4 The device shown, for implementing the technical solution described in this specification, includes: The weight reconstruction unit 502 is used to reconstruct the weights of at least one network layer in the initial model to be quantized based on the orthogonal rotation matrix. The quantization-aware training unit 504 is used to perform quantization-aware training on the reconstructed model to jointly optimize the weights and corresponding rotation matrices of the at least one network layer. The weight quantization unit 506 is used to quantize the optimized weights to generate a target model, wherein the weight accuracy of the target model is lower than that of the initial model.

[0060] Optionally, the quantization perception training unit 504 is specifically used for: For each of the at least one network layer, the following optimization operations are performed alternately before the optimization termination condition is met: fix the current rotation matrix of the layer and perform quantization-aware training to update the current weights of the layer; and fix the current weights of the layer and determine and update the current rotation matrix of the layer based on a preset rotation matrix search strategy.

[0061] Optionally, the quantization perception training unit 504 is specifically used for: Insert pseudo-quantized nodes into the weight path and activation path of this layer; The current weights are updated based on forward and backward propagation.

[0062] The quantization perception training unit 504 is specifically used for: Based on preset matrix selection criteria, the rotation matrix corresponding to the minimum quantization error is selected from the preset rotation matrix candidate set.

[0063] Optionally, the candidate set of rotation matrices includes at least two of the following: Walsh-Hadamard matrix, block Hadamard matrix, and identity matrix.

[0064] Optionally, the quantization perception training unit 504 is specifically used for: Determine the sensitivity of this layer to quantization errors; Based on a preset matrix sensitivity correspondence, a rotation matrix matching the sensitivity level is determined as the current rotation matrix.

[0065] Optionally, the quantization perception training unit 504 is specifically used for: Calculate the first output distribution of this layer in floating-point state and the second output distribution in quantized state; The difference between the first output distribution and the second output distribution is used as the sensitivity.

[0066] Optionally, the device further includes: A matrix periodic matching unit is used to determine the sensitivity of the layer after a preset number of iterations, and to re-determine the rotation matrix that matches the sensitivity based on the matrix sensitivity correspondence as the current rotation matrix.

[0067] Optionally, the device further includes: A rotation matrix processing unit is used to merge the optimized rotation matrix into the corresponding optimized weights; or... The optimized rotation matrix is ​​deployed as a fast orthogonal transformation operator for the target model.

[0068] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0069] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0070] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0071] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0072] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0073] Computers suitable for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a GPS receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0074] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0075] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0076] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0077] Therefore, specific embodiments of the subject matter have been described. Furthermore, the processes depicted in the figures are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0078] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. A model quantization method, characterized in that, The method includes: The weights of at least one network layer in the initial model to be quantized are reconstructed based on the orthogonal rotation matrix; Quantization-aware training is performed on the reconstructed model to jointly optimize the weights and corresponding rotation matrices of at least one network layer; The optimized weights are quantized to generate a target model, the weight accuracy of which is lower than that of the initial model.

2. The method according to claim 1, characterized in that, The step of performing quantization-aware training on the reconstructed model includes: For each of the at least one network layer, the following optimization operations are performed alternately before the optimization termination condition is met: fix the current rotation matrix of the layer and perform quantization-aware training to update the current weights of the layer; and fix the current weights of the layer and determine and update the current rotation matrix of the layer based on a preset rotation matrix search strategy.

3. The method according to claim 2, characterized in that, The process of performing quantization-aware training to update the current weights of the layer includes: Insert pseudo-quantized nodes into the weight path and activation path of this layer; The current weights are updated based on forward and backward propagation.

4. The method according to claim 2, characterized in that, The method for determining and updating the current rotation matrix of the layer based on a preset rotation matrix search strategy includes: Based on preset matrix selection criteria, the rotation matrix corresponding to the minimum quantization error is selected from the preset rotation matrix candidate set.

5. The method according to claim 4, characterized in that, The candidate set of rotation matrices includes at least two of the following: Walsh-Hadamard matrix, block Hadamard matrix, and identity matrix.

6. The method according to claim 2, characterized in that, The current rotation matrix of this layer is determined and updated based on a preset rotation matrix search strategy, including: Determine the sensitivity of this layer to quantization errors; Based on a preset matrix sensitivity correspondence, a rotation matrix matching the sensitivity level is determined as the current rotation matrix.

7. The method according to claim 6, characterized in that, Determining the sensitivity of the layer to quantization errors includes: Calculate the first output distribution of this layer in floating-point state and the second output distribution in quantized state; The difference between the first output distribution and the second output distribution is used as the sensitivity.

8. The method according to claim 6, characterized in that, The method further includes: After a preset number of iterations, the sensitivity of the layer is determined, and a rotation matrix matching the sensitivity is re-determined based on the matrix sensitivity correspondence as the current rotation matrix.

9. The method according to claim 1, characterized in that, The method further includes: The optimized rotation matrix is ​​merged into the corresponding optimized weights; or... The optimized rotation matrix is ​​deployed as a fast orthogonal transformation operator for the target model.

10. A model quantization device, characterized in that, The device includes: The weight reconstruction unit is used to reconstruct the weights of at least one network layer in the initial model to be quantized based on the orthogonal rotation matrix. A quantization-aware training unit is used to perform quantization-aware training on the reconstructed model to jointly optimize the weights and corresponding rotation matrices of at least one network layer. The weight quantization unit is used to quantize the optimized weights to generate a target model, wherein the weight accuracy of the target model is lower than that of the initial model.

11. A computer-readable storage medium, characterized in that, It stores computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-9.

12. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-9.

Images