Model quantization method and apparatus, electronic device, and computer-readable storage medium

Abstract

The application discloses a model quantization method, comprising: determining each to-be-quantized module of a to-be-quantized model; using each candidate quantization configuration information to respectively quantize and dequantize the to-be-quantized module, to obtain each dequantized module corresponding to the to-be-quantized module, the candidate quantization configuration information being used for indicating a quantization manner of a model parameter; determining a module output error between an output of the dequantized module and an output of the corresponding to-be-quantized module; and selecting, as a target, a target quantization configuration information corresponding to each to-be-quantized module from each candidate quantization configuration information, to quantize the corresponding to-be-quantized module through the target quantization configuration information. The scheme provided by the application can improve the flexibility and accuracy of model quantization, and effectively reduce the influence of quantization on model precision.

Description

Technical Field

[0001] This application relates to the field of intelligent model technology, specifically to a model quantization method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] With the rapid development of computer technology, machine learning models are widely deployed in various intelligent applications. However, high-precision models typically have a large number of parameters, making it difficult to run efficiently on resource-constrained devices. Weight quantization is a mainstream method for reducing model size and accelerating inference. To minimize the accuracy loss caused by quantization within a limited storage budget, hybrid precision quantization can be used, which involves setting different quantization parameters for different modules in the model.

[0003] When performing mixed-precision quantization, related technologies typically assign corresponding quantization parameters to different types of modules, such as using higher precision for convolutional layers and lower precision for fully connected layers.

[0004] However, even among modules of the same type, different modules often have different sensitivities to quantization, resulting in insufficient flexibility and accuracy in quantization allocation. Summary of the Invention

[0005] This application provides a model quantization method, apparatus, electronic device, and computer-readable storage medium, which can improve the flexibility and accuracy of model quantization and effectively reduce the impact of quantization on model accuracy. The specific solution is as follows: Firstly, this application provides a model quantization method, including: Identify the individual modules of the model to be quantized; The module to be quantized is quantized and dequantized using each candidate quantization configuration information to obtain each dequantized module corresponding to the module to be quantized. The candidate quantization configuration information is used to indicate the quantization method of the model parameters. Determine the module output error between the output of the inverse quantization module and the output of the corresponding module to be quantized; With the goal of reducing the module output error corresponding to each of the modules to be quantized, target quantization configuration information corresponding to the module to be quantized is selected from each of the candidate quantization configuration information, so as to quantize the corresponding module to be quantized through the target quantization configuration information.

[0006] Secondly, this application provides a model quantization apparatus, comprising: The module determination unit is used to determine each module to be quantized in the model to be quantized. A quantization unit is used to quantize and dequantize the module to be quantized using each candidate quantization configuration information to obtain each dequantized module corresponding to the module to be quantized. The candidate quantization configuration information is used to indicate the quantization method of the model parameters. An error determination unit is used to determine the module output error between the output of the inverse quantization module and the output of the corresponding module to be quantized. The quantization selection unit is used to select the target quantization configuration information corresponding to the module to be quantized from the candidate quantization configuration information with the goal of reducing the module output error corresponding to each of the modules to be quantized, so as to quantize the corresponding module to be quantized through the target quantization configuration information.

[0007] Thirdly, this application also provides an electronic device, including: a processor, a memory, and computer program instructions stored in the memory and executable on the processor; the processor, when executing the computer program instructions, implements the method as described in the first aspect.

[0008] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in the first aspect.

[0009] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0010] Compared with the prior art, this application has the following advantages: The model quantization method provided in this application determines each module of the model to be quantized, and then uses each candidate quantization configuration information to quantize and dequantize the module to be quantized, obtaining each dequantized module corresponding to the module to be quantized. The candidate quantization configuration information is used to indicate the quantization method of the model parameters. That is, for each module to be quantized, the module to be quantized and dequantized is performed using the quantization method indicated by each candidate quantization configuration information, obtaining each dequantized module corresponding to the module to be quantized. Then, the module output error between the output of each dequantized module corresponding to the module to be quantized and the output of the corresponding module to be quantized is determined. This module output error can objectively reflect the degree of influence of different quantization configurations on the accuracy of the module to be quantized. Based on this, with the goal of reducing the module output error corresponding to each module to be quantized, suitable target quantization configuration information is selected from the candidate quantization configuration information for each module to be quantized.

[0011] This quantization configuration selection mechanism based on module-level output error feedback can accurately measure the actual impact of different quantization configurations on each module to be quantized, thus selecting the target configuration with minimal impact on model accuracy from among the candidate quantization configurations. Compared to the traditional method of assigning fixed precision solely based on module type, this application, by determining the module output error under different candidate configurations, can more precisely identify the differences in quantization sensitivity among different modules of the same type, improving the accuracy of mixed-precision quantization. It selects a quantization configuration that matches the characteristics of each module, enhancing the flexibility and accuracy of model quantization and effectively reducing the impact of quantization on model accuracy. Attached Figure Description

[0012] Figure 1 This is a schematic diagram illustrating the application scenario of the solution provided in this application.

[0013] Figure 2 This is a flowchart illustrating an example of the model quantization method provided in the embodiments of this application.

[0014] Figure 3 This is a flowchart illustrating another example of the model quantization method provided in the embodiments of this application.

[0015] Figure 4 This is a structural block diagram of the electronic device provided in this application. Detailed Implementation

[0016] To enable those skilled in the art to better understand the technical solutions of this application, the application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. However, this application can be implemented in many other ways different from those described below. Therefore, based on the embodiments provided in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0017] It should be noted that the terms "first," "second," "third," etc., in the claims, specification, and drawings of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. Such data are interchangeable where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown or described in this application. Furthermore, the terms "comprising," "having," and their variations are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0018] It should be understood that in the embodiments of this application, "at least one" means one or more, and "more than one" means two or more. "And / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the related objects before and after it are in an "or" relationship. "Contains A, B and / or C" means containing any one, two, or three of A, B, and C.

[0019] It should be understood that in the embodiments of this application, "B corresponding to A", "B corresponding to A", "A corresponds to B" or "B corresponds to A" means that B is associated with A, and B can be determined based on A. Determining B based on A does not mean that B is determined solely based on A; B can also be determined based on A and / or other information.

[0020] To facilitate understanding of the various embodiments of this application, the application background of the embodiments will be explained.

[0021] With the rapid development of computer technology, machine learning models are widely deployed in various intelligent applications. However, high-precision models typically have a large number of parameters, making it difficult to run efficiently on resource-constrained terminal devices. Weight quantization is a mainstream method for reducing model size and accelerating inference. To minimize the accuracy loss caused by quantization within a limited storage budget, mixed-precision quantization can be used, which involves setting different quantization parameters for different modules in the model. Related techniques typically assign corresponding quantization parameters to different types of modules when performing mixed-precision quantization, such as using higher precision for convolutional layers and lower precision for fully connected layers. However, even modules of the same type often have different sensitivities to quantization, resulting in insufficient flexibility and accuracy in quantization allocation.

[0022] To address the above issues, embodiments of this application provide a model quantization method, apparatus, electronic device, and computer-readable storage medium, which can improve the flexibility and accuracy of model quantization and effectively reduce the impact of quantization on model accuracy.

[0023] The model quantization method provided in this application can be applied to scenarios where intelligent models are deployed in various resource-constrained edge computing devices, mobile terminals, embedded systems, and intelligent devices sensitive to inference latency and power consumption. Examples include deploying large language models, image recognition models, or speech processing models in terminal devices. In these scenarios, the model quantization method provided in this application can be used to quantize the intelligent models deployed on the terminal devices. To facilitate understanding of the method embodiments of this application, their application scenarios are described. Please refer to... Figure 1 , Figure 1This is a schematic diagram illustrating an application scenario of the solution provided in the embodiments of this application. This application scenario is merely an illustrative example and is not intended to limit the specific application scenario. Figure 1 As shown, in this application scenario, a server 102 and a terminal device 101 are provided. In this embodiment, the terminal device 101 and the server 102 establish a connection through network communication to perform data transmission.

[0024] Server 102 possesses high computing power. Server 102 can be a server, featuring high-speed central processing unit (CPU) computing power, long-term reliable operation, powerful input / output (I / O) external data throughput, and better scalability. Server 102 can be a single server or a server cluster. Server 102 is used to quantize the model to be quantized using the model quantization method provided in this application, generating a low-bit quantized model adapted to resource-constrained devices. Subsequently, server 102 can package the quantized model along with corresponding quantization parameters (such as scaling factors, zero points, etc.) and transmit it to terminal device 101 via the network. Server 102 can also provide other specific services to terminal device 101, such as user information access, website access, application access, etc., which are not specifically limited in this application.

[0025] Terminal device 101 can be an electronic device with display and data processing functions, such as a mobile phone, tablet computer, smartwatch, desktop computer, smart TV, VR device, in-vehicle device, wearable device, or laptop computer. Terminal device 101 receives the quantized model and corresponding quantization parameters sent by server 102, and saves the quantized model in a quantized format. During inference, terminal device 101 performs on-demand dequantization or low-bit computation in the quantization operator or inference kernel based on the quantization parameters to obtain the numerical representation required for inference.

[0026] Terminal device 101 and server 102 can communicate using various communication systems, such as wired or wireless communication systems.

[0027] In other application scenarios, it may only include terminal device 101 or only server 102, without relying on communication between the two. For example, in some localized deployment scenarios, terminal device 101 has sufficient computing power to perform the model quantization process without interacting with server 102. Alternatively, in an offline environment, server 102 pre-compiles model quantization for later use. The above different deployment modes can be flexibly selected according to actual application needs, and this application does not impose any restrictions on them.

[0028] Example 1 The first embodiment of this application provides a model quantization method. This method can be applied to electronic devices, such as servers, desktop computers, laptops, mobile phones, tablets, smartwatches, smart TVs, VR devices, in-vehicle devices, wearable devices, and terminal devices, all of which have data processing capabilities. Since the model quantization method involves numerous computational operations, it can run on devices with strong computing power, such as servers. Of course, it can also run on other types of electronic devices; this application does not impose specific limitations. The following is a detailed description... Figure 2 The model quantization method provided in this embodiment will be explained. For example... Figure 2 As shown, the model quantization method provided in the first embodiment of this application includes the following steps S110 to S140.

[0029] Step S110: Determine each module to be quantized in the model to be quantized.

[0030] The model to be quantized is a pre-trained model, which can be a neural network model, a convolutional neural network (CNN), a recurrent neural network (RNN), a Transformer architecture model, etc. The model to be quantized can be a large language model (LLM), such as a large pre-trained language model based on the Transformer architecture, or a deep neural network model used for computer vision, speech recognition, or other tasks.

[0031] The module to be quantized refers to the module in the model that needs to be quantized. This application does not limit the granularity of the module to be quantized in its embodiments. For example, the module to be quantized can be various network layers in the model, such as linear layers, convolutional layers, attention layers, etc.; or, the module to be quantized can also be various functional sub-networks in the model, such as the attention module or feedforward network module in the Transformer model, where a functional sub-network can contain multiple network layers; or, the module to be quantized can also be a single tensor or weighted array in the model; or, the module to be quantized can also be other computational units in the model with specific functions. In practical applications, the granularity of the module to be quantized can be flexibly selected according to the model's architectural characteristics and quantization requirements to balance quantization accuracy and computational efficiency. Those skilled in the art can flexibly set the granularity of the module to be quantized according to actual accuracy requirements and deployment constraints.

[0032] In this embodiment, when determining the modules to be quantized in the model, the model can be divided into modules based on a preset module partitioning rule. For example, the preset module partitioning rule can be based on the type of network layer, treating all convolutional layers, fully connected layers, or attention layers as independent modules to be quantized. Alternatively, it can be based on functional substructure, treating the multi-head attention mechanism and feed-forward network in the Transformer architecture as separate modules to be quantized. Furthermore, the preset module partitioning rule can also be based on the topology of the computation graph, grouping layers with specific dependencies into a single module. This approach comprehensively covers all parts of the model that require quantization, laying the foundation for subsequent quantization configuration. This application does not limit the specific module partitioning method.

[0033] Step S120: Use each candidate quantization configuration information to quantize and dequantize the module to be quantized, respectively, to obtain each dequantized module corresponding to the module to be quantized. The candidate quantization configuration information is used to indicate the quantization method of the model parameters.

[0034] In this embodiment, a candidate quantization configuration information set can be pre-set. The candidate quantization configuration information in the candidate quantization configuration information set may include at least one of the following: quantization bit count (e.g., 2-bit, 4-bit, 8-bit, etc.), quantization block size (e.g., group-32, group-64, group-128, etc.), quantization symmetry (symmetric quantization or asymmetric quantization), quantization granularity (layer-by-layer quantization, channel-by-channel quantization, or group-by-group quantization), and backend quantizer type, but is not limited thereto. The candidate quantization configuration information can also be understood as candidate quantization actions, and the candidate quantization configuration information set can be understood as a candidate quantization action space.

[0035] Among these parameters, the number of quantization bits determines the storage space occupied by the quantized parameters and their representation precision; the quantization block size determines how many groups the parameters are divided into during the quantization process, with each group independently calculating quantization parameters (such as scaling factors and zeros). Smaller block sizes usually capture the distribution characteristics of local parameters better, thus improving quantization precision, but increasing storage overhead, while larger block sizes have the opposite effect; quantization symmetry determines whether the quantization range is symmetrical about the zeros, with asymmetric quantization typically adapting more flexibly to weight distributions with larger biases; quantization granularity refers to the range of application of the quantization parameters, with channel-by-channel or group-by-group quantization providing finer-grained precision control compared to layer-by-layer quantization; and the backend quantizer type indicates the specific quantization algorithm implementation, such as linear quantization, logarithmic quantization, or nonlinear quantization. By combining these configuration parameters from different dimensions, a diverse set of candidate quantization configuration information can be constructed to cover various quantization strategies, from high compression ratio and low precision to low compression ratio and high precision, providing a rich selection space for subsequent selection of target configurations for each module to be quantized.

[0036] In practice, for each module to be quantized, the candidate quantization configurations can be iterated through. For a selected candidate quantization configuration, the model parameters (such as weight parameters) in the module to be quantized are first quantized according to the quantization method indicated by the configuration (such as the number of quantization bits, granularity, etc.), converting floating-point parameters into low-bit integer parameters. Subsequently, the quantized integer parameters are dequantized using the corresponding dequantization parameters, restoring them to floating-point representation, thus obtaining the dequantized module under the current candidate configuration. This "quantization-dequantization" process can simulate the numerical changes of the model during the inference phase, allowing for a preliminary assessment of the impact of the quantization configuration on the module's output values ​​without actually deploying a low-bit model.

[0037] Each inverse quantization module corresponds to a module to be quantized; that is, the modules representing the floating-point parameters obtained after quantization using various candidate quantization configurations and subsequent inverse quantization. Each inverse quantization module corresponds to a candidate quantization configuration, reflecting the reconstruction result of the model parameters under that specific quantization constraint. By obtaining these inverse quantization modules, the degree to which different quantization configurations retain the original parameter information can be directly compared in the floating-point domain, providing a data foundation for subsequent evaluation of quantization error and selection of target quantization strategies.

[0038] In one implementation, prior to step S120, the step of determining the candidate quantization configuration information set in step S120a may be included.

[0039] Step S120a: Based on the target deployment scenario of the model to be quantized, determine the candidate quantization configuration information set corresponding to the model to be quantized. The target deployment scenario includes at least one of the following attributes of the deployed device: hardware type, inference engine type, supported quantization operator implementation, storage budget, performance budget, allowed number of bits, supported block format, and metadata format.

[0040] The aforementioned target deployment scenario refers to the set of hardware constraints and software support capabilities of the environment in which the model to be quantized is deployed and run. Specifically, the target deployment scenario defines the boundary conditions that the quantization process must follow to ensure that the generated quantized model can be correctly loaded and efficiently inferred on the specified device. For example, if the target deployment scenario is a specific mobile terminal model, its hardware type may limit it to supporting only INT8 or INT4 quantization, the inference engine may only support a specific block size (such as group-128), and the storage budget may strictly limit the total size of the model file. Therefore, when determining the candidate quantization configuration information set, it is necessary to screen out quantization configurations that meet the attribute requirements of the target deployment scenario and exclude those configurations that cannot be implemented in this specific scenario, thereby ensuring the effectiveness and practicality of subsequent quantization steps.

[0041] For example, if the target deployment scenario is a mobile GPU that supports a specific quantization operator, the candidate quantization configuration information set can be limited to the number of bits (e.g., 4-bit, 8-bit) and block format supported by the hardware. If the target deployment scenario is a storage-constrained device, the candidate quantization configuration information set can focus on low bit counts and larger block sizes to reduce metadata overhead. By combining the hardware and software constraints of the target deployment scenario to filter candidate quantization configuration information, invalid evaluations of unsupported or inefficient quantization methods can be avoided, thereby improving the efficiency of quantization search.

[0042] For example, in the specific implementation process of determining the candidate quantization configuration information set according to the target deployment scenario, for low-bit half-quadratic quantization (HQQ) / round-to-nearest (RTN) quantization scenarios, one or more of the following action spaces (i.e., candidate quantization configuration information sets) can be configured to cover different deployment requirements: Fixed-granularity bit-only action space: This includes configurations such as 3-bit group-128 and 4-bit group-128. This type of action space is suitable for scenarios where the quantization granularity is pre-fixed. It only requires searching along the bit dimension, effectively reducing the search space and improving the efficiency of quantization configuration selection. Specifically, 3-bit group-128 refers to using 3 bits of quantization precision and grouping model parameters into groups of 128 for independent quantization calculation. This configuration maintains low storage overhead while preserving more weight distribution details through smaller grouping granularity, making it suitable for scenarios with certain accuracy requirements and hardware support for 3-bit quantization. 4-bit group-128 achieves a good balance between storage compression and inference accuracy.

[0043] NPU per-channel restricted action space: This includes configurations such as 4-bit channel-wise and 8-bit channel-wise. This type of action space can be used when the NPU (Neural Processing Unit) in the target deployment scenario only exposes a per-channel quantization kernel. Subsequent quantization strategies determine which modules are upgraded to 8-bit to achieve higher precision preservation. Specifically, 4-bit channel-wise grouping of model parameters by channel dimension, with all parameters within each channel sharing the same set of quantization parameters (such as scaling factors and zeros), and each parameter having 4 quantization bits, provides finer precision control than layer-by-layer quantization while maintaining lower storage overhead. 8-bit channel-wise sharing of the same set of quantization parameters within each channel, with 8 quantization bits, is suitable for scenarios requiring high precision but still needing a certain compression ratio. By restricting the quantization granularity to per-channel, it ensures that the generated quantized model can directly utilize the hardware-accelerated kernel provided by the NPU, avoiding increased inference latency caused by software-simulated fine-grained quantization (such as group-wise).

[0044] Extended action space: Includes configurations such as 3-bit group-64, 3-bit group-128, 4-bit channel-wise, 4-bit group-64, and 4-bit group-128. The extended action space provides a richer combination of quantization configurations, enabling a more precise trade-off between accuracy and compression rate, suitable for deployment scenarios with high quantization quality requirements.

[0045] More broadly, the action space of the candidate quantization configuration information set can be extended to multiple bit count levels such as W2 / W3 / W4 / W5 / W6 / W8, different block sizes, and different local quantizer backends (such as RTN, HQQ, AWQ, GPTQ, etc.). If the quantization backend in the target deployment scenario has a significant performance preference for a specific block size, the block size can be fixed in advance, and then the search can be performed on the bit count dimension of each module to reduce search complexity and ensure compatibility with the target backend. Here, W2 refers to using 2-bit quantization precision, while W3 to W8 correspond to 3-bit to 8-bit quantization precision, respectively. As the bit count increases, the model's representational power gradually increases, and the quantization error decreases accordingly, but the storage overhead and computational complexity also increase.

[0046] The following examples illustrate how to set up candidate quantization configuration information sets, using several typical edge deployment scenarios as examples: NPU Scenario: If the target NPU only exposes per-channel quantization kernels, the candidate quantization configuration information set can be set to {4-bit channel-wise, 8-bit channel-wise}. Subsequently, based on the characteristics of each module to be quantized, it can be adaptively determined which modules use 8-bit quantization to protect accuracy and which modules use 4-bit quantization to compress size.

[0047] In a processor (CPU) scenario based on an Advanced RISC Machine (ARM) design: If the target ARM CPU is more favorable for 4-bit / 8-bit packing operations, the candidate quantization configuration information set can be set to {4-bit, 8-bit} × {group-32, group-64, group-128}, which is the combination of traversing the number of bits and the block size; or, if it is determined in advance that the inference performance is better under the group-128 configuration, the block size is fixed to group-128, and only the allocation of the number of bits is searched.

[0048] Storage-priority scenarios: If the target deployment scenario has strict limitations on model storage space (e.g., extremely tight storage budget), the candidate quantization configuration information set can be biased towards low bit count configurations (such as W2, W3, W4) and larger block sizes (such as group-128 or larger) to minimize model size and meet storage constraints.

[0049] Performance-priority scenarios: If the target deployment scenario has high requirements for inference latency or throughput, the candidate quantization configuration information set should exclude formats that lack efficient quantization kernel support in the target backend, and only retain the combination of bit count and block size with better throughput performance under that backend, so as to ensure that the quantized model can achieve the expected inference performance.

[0050] Quality-first scenarios: If the target deployment scenario has high requirements for maintaining model accuracy (e.g., for task accuracy-sensitive applications), then higher-precision configuration options, such as 8-bit quantization or finer-grained block configuration, can be added to the candidate quantization configuration information set. At the same time, the total quantization cost of each module can be constrained by the effective-bit budget, so as to minimize the accuracy loss under the limited total budget.

[0051] Through the above methods, those skilled in the art can flexibly configure the candidate quantization configuration information set according to the actual target deployment scenario and multi-dimensional requirements such as accuracy, performance, and storage, so that the subsequent quantization configuration selection can not only meet the engineering feasibility of hardware deployment, but also achieve the corresponding accuracy-efficiency trade-off effect.

[0052] This implementation method dynamically constructs a candidate quantization configuration information set based on the hardware constraints and performance requirements of the target deployment scenario, which can effectively narrow the space range of subsequent quantization strategy search and avoid invalid traversal of incompatible or inefficient configurations.

[0053] Step S130: Determine the module output error between the output of the inverse quantization module and the output of the corresponding module to be quantized.

[0054] In this embodiment, after obtaining the dequantization modules corresponding to the module to be quantized, the difference between the output of each dequantization module and the output of the corresponding module to be quantized is further determined, i.e., the module output error. This module output error is used to measure the degree of deviation between the output result of the module to be quantized after quantization and dequantization processing and the original output result under a certain candidate quantization configuration information. It is an important basis for subsequent evaluation of the impact of different quantization configurations on model accuracy.

[0055] Specifically, for any module to be quantized, the input data of the module to be quantized (e.g., the output of the previous module, the original input sample or its corresponding calibration activation) can be input to the module to be quantized and each of the corresponding inverse quantization modules to obtain the output results of the module to be quantized and the output results of each inverse quantization module. Then, the error between the output result of each inverse quantization module and the output result of the module to be quantized is calculated as the module output error of the module to be quantized under the candidate quantization configuration information.

[0056] Optionally, the input data (i.e., calibration activation) used to calculate the output error of the module can be obtained from preset calibration data. The calibration dataset can be a portion of the training or validation set, or it can be samples randomly generated based on the distribution of the input data of the model to be quantized. The calibration data can be unlabeled data or samples synthesized using a generative model that conform to the distribution of real data. Those skilled in the art can flexibly choose the size and source of the calibration dataset according to the actual application scenario and computing resources to balance the accuracy of error assessment and computational efficiency.

[0057] When calculating the output error of a module, various error metrics can be employed. For example, mean squared error, average error, cosine similarity or its derivative cosine distance, and the signal-to-noise ratio (SNR) of the output tensor can be used to measure the error between two outputs. In practical applications, one or more error metrics can be selected and combined based on the type of the module to be quantized, the data distribution characteristics of the output tensor, and the quantization accuracy assessment requirements. For instance, for modules with a large range of output values, mean squared error or SNR can be used; for modules whose output is a high-dimensional feature vector (such as the output of an attention layer), cosine similarity can be used; and for modules sensitive to outliers, average error can be prioritized. By flexibly selecting or combining these error metrics, the impact of different candidate quantization configurations on the output quality of the module to be quantized can be more comprehensively evaluated, thus providing a more accurate reference for subsequently selecting the target quantization configuration.

[0058] In one specific implementation, step S130 can be achieved by the following steps S131 to S133.

[0059] Step S131: Input the calibration data into the model to be quantized to perform complete forward inference, and obtain the calibration activation and pre-quantization module output corresponding to each module to be quantized.

[0060] During the propagation of calibration data layer by layer, the output of each module to be quantized before quantization (i.e., the original module) can be saved and recorded as the pre-quantization module output. Simultaneously, for each module to be quantized, its input data (i.e., the output value of the previous module) can also be saved and recorded as the calibration activation. By performing a complete forward inference on the model to be quantized, the pre-quantization module output and its corresponding calibration activation of each module can be obtained.

[0061] Step S132: Use the inverse quantization module corresponding to the module to be quantized to infer the corresponding calibration activation and obtain the output of the quantized module.

[0062] Specifically, for any module to be quantized, the calibration activation corresponding to the module can be input into the dequantization module corresponding to that module for inference calculation, and the output result of the dequantization module can be obtained, which is recorded as the output of the quantized module. The dequantization module is a floating-point parameter module corresponding to a certain candidate quantization configuration information, obtained by quantizing and dequantizing the parameters of the module to be quantized. The "inference" in step S132 does not refer to rerunning the forward process of the entire model to be quantized, but only to running the calculation process of the dequantization module corresponding to the module to be quantized itself, that is, using the calibration activation as the input of the dequantization module, and obtaining its output result after parameter calculation by the dequantization module. In this way, for each module to be quantized, the output of its original module (the module output before quantization) and the output of the dequantization module under each candidate quantization configuration (the module output after quantization) can be obtained under the same input conditions.

[0063] Step S133: Determine the module output error between the output of the quantized module and the output of the module before quantization, and use it as the module output error between the output of the inverse quantization module and the output of the corresponding module to be quantized.

[0064] Since the output of the module after quantization and the output of the module before quantization are calculated by the dequantization module and the original module to be quantized respectively under the same input (i.e., the same calibration activation), this error can truly reflect the actual impact of the quantization operation on the output accuracy of the module under this candidate quantization configuration. In other words, this module output error is the module output error of the dequantization module relative to the output of the module to be quantized under the same input conditions.

[0065] In the above implementation, the pre-quantization module output and calibration activation of each module to be quantized can be saved through a single forward inference of calibration data. Subsequent error evaluations of each module and each candidate quantization configuration can be performed independently and in parallel, without repeating the entire forward inference process of the model, thus significantly reducing computational overhead. Furthermore, since the calculation of the post-quantization module output only involves the forward operation of a single module to be quantized and does not depend on the calculation results of subsequent modules, the error evaluations between modules are independent, facilitating subsequent selection of differentiated quantization configurations at the module granularity and providing favorable conditions for parallelization acceleration.

[0066] The calibration activation used for error assessment in the above embodiments relies on inputting calibration data into the model to be quantized for forward inference; however, this application is not limited to this. In other embodiments, the calibration activation can also be obtained in other ways, such as directly reading from a pre-stored activation value dataset, or randomly generating and propagating it according to the input data distribution of the model to be quantized. Any method that can obtain the pre-quantization module output and the dequantization module output of the module to be quantized under the same input conditions, and calculate the module output error accordingly, falls within the protection scope of this application.

[0067] In one implementation, step S130 can be implemented according to the following steps S134 to S135.

[0068] Step S134: Determine the direct module output error between the output of the inverse quantization module and the output of the module to be quantized under the same input.

[0069] The above-mentioned direct module output error can be calculated using any applicable error metric such as mean square error, average error, cosine similarity, or signal-to-noise ratio, which reflects the actual deviation between the output of the dequantization module and the original module output.

[0070] Step S135: Based on the output error between the module to be quantized and the direct module output error, obtain the relative module output error, which is used as the module output error of the inverse quantization module relative to the output of the module to be quantized.

[0071] Specifically, the aforementioned relative module output error can be the ratio between the direct module output error and the output of the module to be quantized, used to measure the severity of the direct error relative to the original output itself. By converting the direct error into a relative error form, the incomparability of errors caused by differences in the magnitude of output values ​​between different modules to be quantized can be eliminated, making the module output errors between different modules and between different candidate quantization configurations comparable. This provides a unified and fair evaluation benchmark for subsequent selection of optimal cross-module quantization configurations.

[0072] As an example, the direct module output error can be divided by the norm (such as the L2 norm) of the output of the module to be quantized to obtain the normalized relative module output error. Alternatively, the direct module output error can be divided by the mean or maximum value of the output of the module to be quantized to obtain the amplitude-based relative error. Or, when using the mean squared error as the direct error, the mean squared error can be calculated first, and then divided by the variance of the output of the module to be quantized to obtain the relative error in the form of an explanation of variance proportion. The above methods for calculating relative errors can be flexibly selected according to the actual application scenario and evaluation requirements; this application does not limit this choice.

[0073] For example, the relative module output error can be determined by the following formula (1).

[0074] (1) in, This represents the relative module output error of the module to be quantized, m, under candidate quantization configuration a; This represents the direct module output error between the output of the dequantization module and the original output of the module to be quantized under candidate quantization configuration a; This represents the raw output of the module to be quantized; The Frobenius Norm is used to measure the overall magnitude of a tensor. This is a very small positive constant used to prevent the denominator from being zero and to ensure the numerical stability of the calculation. The relative module output error calculated by formula (1) can eliminate the evaluation bias caused by the difference in the magnitude of the output activation values ​​of different modules.

[0075] When the module to be quantized is a linear module (e.g., a fully connected layer or a convolutional layer, whose core operation can be represented as matrix multiplication or tensor convolution), the module output error can be calculated directly based on the module input activation and weight difference without actually running the forward inference of the dequantization module.

[0076] Specifically, for the m-th linear module to be quantized, its forward computation can be expressed as: ,in These are the original weight parameters for module m. Let y be the calibration activation input to module m, and y be the full-precision output tensor of module m. For candidate quantization configuration information (i.e., candidate quantization action) a, module m performs quantization and dequantization according to the candidate quantization configuration information a, and the resulting dequantization weight is: Under this candidate quantization configuration a, the operator output perturbation (i.e., direct module output error) caused by the quantization action can be expressed as: ,in, This refers to the direct module output error of the m-th module under candidate quantization configuration a, used to represent the weight quantization error. The output error is propagated to the module output through linear transformation.

[0077] This implementation method effectively eliminates the error measurement bias caused by different output scales of each module to be quantized by converting the direct module output error into the relative module output error. This makes the error evaluation results between different modules within the same model and between different candidate quantization configurations more comparable, which is beneficial for subsequent unified quantization configuration selection and global optimization decision-making among multiple modules.

[0078] In an optional implementation, if calibration data is difficult to obtain or inconvenient to use, step S130 can also be implemented as follows: obtain the original model parameters of the module to be quantized and the corresponding dequantization parameters of the dequantization module, and determine the estimated value of the module output error based on the degree of difference between the original model parameters and the dequantization parameters. Specifically, since the weight reconstruction error generated by the model parameters during quantization and dequantization will be transmitted to the output of the module, the mean square error, average error, or other statistical distance measure between the original model parameters and the dequantization parameters can be directly used as an approximate evaluation value of the module output error. This method does not require inputting calibration data into the model for forward inference, nor does it require obtaining the input activation data of the module; it can complete the approximate evaluation of the module output error based solely on the parameter information of the module to be quantized itself. This is particularly advantageous for scenarios where calibration data is difficult to obtain, the distribution of input data is unknown, or data privacy is involved and inconvenient to use. At the same time, since forward inference calculation is not required, this method has a faster evaluation speed and lower computational resource consumption, making it suitable for scenarios that require rapid screening of a large number of candidate quantization configurations.

[0079] Step S140: With the goal of reducing the module output error corresponding to each of the modules to be quantized, select the target quantization configuration information corresponding to the module to be quantized from each of the candidate quantization configuration information, so as to quantize the corresponding module to be quantized through the target quantization configuration information.

[0080] Specifically, the objective can be to reduce the total error of the output errors of each module to be quantized. Target quantization configuration information corresponding to the modules to be quantized can be selected from the candidate quantization configuration information. For example, an objective function containing all modules to be quantized can be constructed. This objective function is the weighted sum or total of the output errors of each module under the selected candidate quantization configuration. Through traversal or search algorithms, a combination is found in the set of candidate quantization configuration information corresponding to each module to minimize the value of this objective function. In specific implementations, optimization strategies such as dynamic programming, integer linear programming, or heuristic search can be used to solve for this combination. This quantization configuration selection method, which aims to minimize the total error, can balance the quantization accuracy loss of each module from a global perspective, avoiding the overall model performance degradation caused by local optimization.

[0081] Alternatively, the target quantization configuration information can be selected independently for each module to be quantized. That is, for each module to be quantized... The module output error is compared under each candidate quantization configuration information, and the candidate quantization configuration information with the smallest error value (or less than a preset error threshold) is selected as the target quantization configuration information for that module. This independent selection implementation method can find the corresponding quantization configuration for each module, thereby achieving accuracy optimization for each module.

[0082] In one implementation, step S140 may select the target quantization configuration information corresponding to the module to be quantized from each of the candidate quantization configuration information according to the following steps S141~S143.

[0083] Step S141: Determine the importance level of the module to be quantized, whereby the importance level represents the degree of impact of the perturbation of the module to be quantized on the overall output of the model to be quantized.

[0084] In the embodiments of this application, the sensitivity of different modules to the overall output accuracy of the model may vary significantly. For example, in the Transformer architecture, some linear layers corresponding to attention heads may be more sensitive to quantization, and their parameter perturbations will significantly affect the final output quality of the model; while other modules (such as some feedforward network layers) may have higher robustness and are not sensitive to quantization perturbations.

[0085] The importance level is used to characterize the impact of a perturbation in the output of a module to be quantized on the overall output result after the perturbation propagates to the final output of the model. The higher the importance level, the more easily the output change of the module will cause a decrease in the overall accuracy of the model, and more attention should be paid to the quantization error of the module when selecting quantization configurations; conversely, the lower the importance level, the higher the tolerance of the module to quantization, and the quantization configuration with a higher compression ratio can be given priority.

[0086] In practice, determining the importance of a module to be quantized can be done in various ways. For example, forward inference can be performed on the calibration data of the model to be quantized, adding artificial perturbations to the output of each module (e.g., adding Gaussian noise, randomly discarding some output neurons), and then observing the magnitude of change in the final output of the model (e.g., loss function value or specific task metric) relative to the unperturbed output. The importance of the module can be determined based on this magnitude of change. The greater the magnitude of change, the more important the module, and the higher its corresponding importance level.

[0087] In one specific embodiment, the importance of each module can be determined based on the gradient information of the model parameters. Specifically, the importance of the module to be quantized can be determined according to the following steps S141a~S141c.

[0088] Step S141a: Input the calibration data into the model to be quantized for forward inference to obtain the inference result.

[0089] The details regarding calibration data can be found above and will not be repeated here. After inputting the calibration data into the model to be quantized for forward inference, the data sequentially passes through each module to be quantized in the model (such as embedding layers, attention layers, feedforward network layers, etc.), outputting the model's inference results. The specific form of the inference results depends on the task type of the model to be quantized. For classification tasks, the inference results can be the predicted probability distributions of each category; for regression tasks, the inference results can be continuous predicted values; for language modeling tasks (such as large language models), the inference results can be the probability distribution of the next word, etc. This inference result will serve as the basis for subsequent loss calculations.

[0090] Step S141b: Determine the loss corresponding to the inference result.

[0091] Specifically, after obtaining the inference result of the model to be quantized on the calibration data, the loss value can be calculated based on the difference between the inference result and the true result in the calibration data (such as the true label or the true following text). When the calibration data is unlabeled, a loss function can be constructed using self-supervised learning, for example, by using the difference between the predicted result of the next sentence and the true next sentence in the calibration data to calculate the loss.

[0092] When the calibration data contains multiple samples, the loss value for each sample can be calculated separately, and then the loss values ​​of multiple samples can be summed or averaged to obtain the overall loss value, which can be used for subsequent backpropagation calculations. In one embodiment, to reduce computational overhead, only a small number of calibration samples (e.g., 32 or 64 samples) can be used to calculate the loss, and this application does not limit this to that.

[0093] Step S141c: Backpropagate the module to be quantized based on the loss to obtain the gradient corresponding to the module to be quantized, and determine the importance of the module to be quantized based on the gradient.

[0094] Backpropagation uses a chain rule to calculate the partial derivatives of the loss function with respect to the parameters of each layer of the model, from the output layer to the input layer. These partial derivatives are the gradients. The specific implementation of backpropagation will not be elaborated here.

[0095] Through the backpropagation process described above, the gradient value corresponding to each module in the model to be quantized can be obtained. The importance of a certain module can be determined based on its gradient information. The higher the gradient, the greater the impact of a small change in the module's parameters on the loss function, that is, the more significant the module's contribution to the overall accuracy of the model, and therefore the higher its importance.

[0096] Specifically, the determination of importance can include at least one of the following methods: Determine the norm (e.g., L1 norm, L2 norm, or infinite norm) of the gradient of each model parameter in the module to be quantized. Use this norm value as the importance of the module. The larger the L2 norm of the gradient, the more sensitive the loss function is to the changes in the parameters of the module, that is, the more important the module is to the overall output of the model. Determine the sum or average of the gradients of each model parameter in the module to be quantized, and use this value as the importance of the module. This method can also reflect the overall influence of the module parameters on the loss function; The importance of each model parameter in the module to be quantized is determined by the product of the gradient and the corresponding parameter value (i.e., gradient-weighted parameters). This method not only considers the magnitude of the gradient but also the magnitude of the parameter itself, which can more comprehensively reflect the impact of parameter changes on the loss.

[0097] In this embodiment, the gradient information obtained through backpropagation accurately reflects the sensitivity of the loss function to changes in the parameters of each module. This sensitivity is directly related to the impact of each module on the overall output of the model. Therefore, the importance determined based on the gradient has high reliability. Furthermore, the gradient information calculation can reuse the calibration data and forward inference results obtained in step S130. Only one additional backpropagation is needed on top of the original forward inference, eliminating the need for additional data collection or multiple model inferences, resulting in relatively low computational overhead.

[0098] After obtaining the gradient corresponding to the module to be quantized, the sum of the gradient norms of all parameters in the module can be used to determine the importance of the module. Alternatively, the sum of squares, expectation, or maximum value of the gradients of each parameter can be used as the importance of the module. This application does not specifically limit this.

[0099] For example, the importance of the module to be quantified can be determined by the following formula (2).

[0100] (2) in This indicates the importance of the m-th module to be quantized. This represents the loss function value calculated based on the calibration data x. This represents the set of model parameters for the m-th module to be quantized. This represents the gradient of the loss function with respect to the parameters of the m-th module to be quantized. This represents the Frobenius norm (i.e., the square root of the sum of the squares of all elements of a matrix). Indicates the calibration dataset x Calculate the expected value. This formula, by calculating the expected value of the Frobenius norm of the gradient, comprehensively reflects the average influence of changes in the parameters of the module to be quantized on the overall loss under this calibration data distribution.

[0101] This embodiment determines the importance of each module to be quantized by calculating the gradient through backpropagation of the loss during the inference process. This accurately reflects the true sensitivity of each module to the overall model performance, thus providing a basis for subsequent differentiated quantization configuration. Compared to importance assessment methods based on heuristic rules or experience, the gradient-based importance determination method can adaptively capture the actual contribution differences of each module under different model architectures and task scenarios, avoiding the subjectivity and limitations of manually setting weights.

[0102] In other specific embodiments, the importance can also be determined based on prior knowledge such as the ratio of the number of parameters of a module to its output dimension, or the position of the module in the model computation graph (e.g., modules closer to the output layer usually have a greater impact on the final output than modules closer to the input layer). This application does not limit this.

[0103] Step S142: Weight the output error of the corresponding module according to the importance of the module to be quantized to obtain the weighted module output error.

[0104] Specifically, the importance level can be used to weight the output errors of the corresponding modules to obtain the weighted output error of the module to be quantized under each candidate quantization configuration information. Through the above weighting operation, the output error of the module with a high importance level is amplified in the subsequent target configuration selection, thereby obtaining higher priority protection in the optimization process; the output error of the module with a low importance level is relatively compressed, and it can withstand a slightly higher quantization error in exchange for a higher compression ratio.

[0105] Those skilled in the art can flexibly choose the weighting method according to actual needs. For example, they can convert the importance level into a weighting coefficient to weight the module output error through product, exponential operation, or other nonlinear mapping methods. For example, the importance level can be normalized to the [0, 1] interval and used directly as the weighting coefficient, or the importance level can be used directly as the weighting coefficient. As long as the weighting method reflects the general trend of "the higher the importance level, the greater the weighted module output error", this application does not limit it in this regard.

[0106] Step S143: With the goal of reducing the output error of the weighted module corresponding to each of the modules to be quantized, select the target quantization configuration information corresponding to the module to be quantized from the candidate quantization configuration information.

[0107] The execution process of step S143 can be referred to the description of step S140 above, and will not be detailed here.

[0108] For any given module to be quantized, there is a corresponding weighted module output error value under each candidate quantization configuration. To reduce the impact of quantization on the output accuracy of this module, a configuration that minimizes the weighted module output error should be selected from the candidate quantization configurations.

[0109] In one implementation, the target quantization configuration information for each module to be quantized can be selected from the candidate quantization configuration information, with the goal of reducing the total error of the weighted module output error for each module to be quantized, or with the goal of minimizing the total error of the weighted module output error for each module to be quantized. This allows for a global balance of quantization accuracy loss among different modules in the model. In another implementation, the target quantization configuration information can be selected independently for each module to be quantized. This involves comparing the weighted module output errors of the module under each candidate configuration and selecting the candidate configuration with the smallest error value (or less than a preset error threshold) as the target quantization configuration information for that module. This approach reduces computational complexity and improves the efficiency of quantization configuration selection, making it particularly suitable for scenarios with a large number of modules or a complex candidate configuration space.

[0110] This implementation weights the output errors of each module by introducing the importance level of each module to be quantized. This allows the quantization configuration selection process to fully consider the differentiated impact of different modules on the overall model accuracy. Compared to directly optimizing the errors of all modules indiscriminately, this method prioritizes the allocation of limited high-quality quantization resources (such as high bit count configuration, small block size configuration, etc.) to critical modules that are more sensitive to quantization, while allocating configurations with higher compression ratios to non-critical modules with stronger robustness. This results in better overall model accuracy preservation under the same global resource constraints, achieving refined and differentiated allocation of quantization resources. Furthermore, the calculation of importance can reuse the calibration data and forward inference results obtained in step S130, without additional data acquisition or model training costs, thus exhibiting high practicality and computational efficiency.

[0111] In one implementation, step S140 can also be implemented according to the following steps S144~S145.

[0112] Step S144: Determine the storage cost of the module to be quantized using the candidate quantization configuration information.

[0113] In this embodiment, different candidate quantization configuration information typically corresponds to different storage costs. Storage cost refers to the amount of storage space occupied by the module in a storage device (such as memory, flash memory, video memory, etc.) after quantizing the model parameters according to a certain candidate quantization configuration information. The size of the storage cost depends on factors such as the number of quantization bits, the size of the quantization block, and the number of parameters in the module itself. For example, the lower the number of quantization bits, the fewer storage bits each parameter occupies, resulting in lower storage costs; the larger the block size, the fewer the number of quantization parameters (such as scaling factors and zeros), and the smaller the metadata overhead, resulting in lower storage costs.

[0114] In one implementation, for any module to be quantized, when using a candidate quantization configuration, the storage cost of the module to be quantized can be determined based on the storage overhead of the quantized model parameters and the storage overhead of the metadata introduced during the quantization process. In other words, the storage cost of the module to be quantized can be determined based on the number of weight bits and the number of metadata bits corresponding to the candidate quantization configuration information.

[0115] The storage overhead of the quantized model parameters is determined based on the number of parameters of the module to be quantized and the number of quantization bits indicated by the candidate quantization configuration information. The larger the number of parameters and the higher the number of quantization bits, the greater the storage overhead. The metadata storage overhead includes, but is not limited to, auxiliary information such as quantization scaling factor and quantization zero point. This overhead depends on the parameter distribution of the module to be quantized, the quantization block size indicated by the candidate quantization configuration information, and the storage bit width of the metadata itself.

[0116] The number of weight bits corresponding to the candidate quantization configuration information refers to the number of bits occupied by each model parameter when quantizing the model parameters in the module to be quantized. The number of metadata bits refers to the total number of bits required to store quantization auxiliary information (such as scaling factors, zero points, etc.). In specific calculations, the number of metadata bits depends on the quantization block strategy and the metadata storage format. For example, if per-channel quantization is used, each channel requires one scaling factor and one zero point, and the total amount of metadata is proportional to the number of channels; if per-tensor quantization is used, the entire module only requires one set of scaling factors and zero points, resulting in minimal metadata overhead; if per-block quantization is used, the total amount of metadata is proportional to the number of blocks. Furthermore, the storage bit width of the metadata itself (such as using 16-bit or 32-bit floating-point numbers to store scaling factors) also affects the number of metadata bits.

[0117] Based on the above definitions, the storage cost of the module to be quantized can be calculated using the following formula: Storage Cost = Number of Parameters × Number of Weight Bits + Number of Metadata Items × Number of Metadata Bits. Where, the number of parameters is the total number of model parameters in the module to be quantized, the number of weight bits is the number of bits specified in the candidate quantization configuration information to represent each model parameter, the number of metadata items is the total amount of auxiliary information such as scaling factors and zero points generated by the module during quantization, and the number of metadata bits is the number of bits required to store each metadata item. This formula allows for the precise calculation of the storage space occupied by the module to be quantized under a specific candidate quantization configuration, thus providing a quantitative basis for subsequent storage cost constraints.

[0118] Step S145: With the quantization storage cost meeting the preset cost condition as a constraint, and with the goal of reducing the module output error corresponding to each of the modules to be quantized, select the target quantization configuration information corresponding to the module to be quantized from each of the candidate quantization configuration information. The quantization storage cost is the sum of the storage costs corresponding to each of the modules to be quantized, or the average effective storage cost obtained by weighting the parameters of each module to be quantized.

[0119] Specifically, for a set of selected quantization configuration schemes (i.e., selecting one candidate quantization configuration information for each module to be quantized), the quantization storage cost can be the sum of the storage costs of each module to be quantized in the set of schemes. Using the sum of storage costs as a global storage constraint index can intuitively reflect the overall storage space occupied after model quantization, and facilitate direct comparison with hardware storage capacity limitations.

[0120] Alternatively, to eliminate the impact of differences in the number of parameters of different modules on the total cost calculation, the average effective storage cost can be obtained by weighting the number of parameters of each module to be quantized, that is, calculating the average storage overhead per unit parameter. This can more fairly evaluate the storage efficiency of modules of different sizes under the same bit width, eliminate the cost measurement bias caused by excessive differences in the number of parameters of different modules, and make the quantized storage costs under different number of modules or different division granularities comparable. At the same time, the calculation method of average effective storage cost can smooth the cost differences between modules and avoid the dominance of a few modules with extremely large number of parameters in the overall cost assessment.

[0121] The aforementioned preset cost conditions can be that the quantization storage cost does not exceed a preset storage cost threshold, or that the compression ratio of the quantization storage cost relative to the original model storage cost does not exceed a preset compression ratio, or that the quantization storage cost is within a certain preset reasonable range, that is, it cannot be too high (exceeding the storage budget) nor too low (which may mean that excessive compression leads to a serious decrease in accuracy), thereby achieving a balance between storage cost and model accuracy.

[0122] Step S145 can use the weighted sum (or direct summation) of the module output errors of each module as the optimization objective. Among the various configuration combinations that satisfy the quantization storage cost constraint, the set of configurations that minimizes the weighted sum or direct summation is selected as the target quantization configuration information for each module to be quantized. This optimization process can be implemented by traversing all combinations (when the number of combinations is small), dynamic programming (when the number of candidate configurations and modules is moderate), integer linear programming, or heuristic algorithms.

[0123] This implementation introduces a quantized storage cost constraint, measured by the storage cost of each module. This tightly integrates the quantized configuration selection with the actual storage budget requirements of the target deployment scenario, ensuring that the selected quantized configuration scheme meets the hardware storage resource limitations in actual deployment and improving the engineering feasibility of the quantization scheme. Furthermore, using the average storage cost of each module as a constraint indicator ensures that the quantized storage cost is unaffected by changes in the granularity of the module partitioning (i.e., regardless of whether the model is divided into multiple fine-grained modules or a few coarse-grained modules, the average storage cost stably reflects the overall compression level), thereby improving the applicability and stability of the method under different module partitioning strategies.

[0124] In one specific embodiment, in step S145, the target quantization configuration information of each module to be quantized can be obtained by constructing a decomposable cost table and solving it using dynamic programming or integer linear programming.

[0125] For example, for the m-th module to be quantized and the a-th candidate quantization configuration information, a cost function can be defined. This cost function is used to comprehensively measure the total cost incurred by applying the candidate quantization configuration information to this module. The total cost... It can be determined according to the following formula (3).

[0126] (3) in, For the first m The importance of each module to be quantified. This represents the module output error of the m-th module to be quantized under the a-th candidate quantization configuration information.

[0127] By combining the module's importance with its output error through the above multiplication operation, the overall cost of the module under this configuration is obtained. Modules with higher importance have their output errors amplified to a greater extent, thus tending to be assigned a better (i.e., smaller error) quantization configuration in subsequent optimizations; conversely, modules with lower importance have their output errors compressed and can tolerate relatively large quantization errors in exchange for a higher compression ratio.

[0128] Based on the aforementioned cost function, a decomposable cost table can be constructed. This decomposable cost table uses a first dimension (e.g., rows) to represent each module to be quantized, and a second dimension (e.g., columns) to represent each candidate quantization configuration. Each element in the table represents the comprehensive cost of the corresponding module under the corresponding configuration. Since the costs of each module are independent and cumulative, the cost table is decomposable (i.e., the total cost is the sum of the costs of each module). This property allows subsequent combinatorial optimization problems to be solved using optimization methods such as dynamic programming or integer linear programming.

[0129] When constructing the decomposable cost table, for the m-th module to be quantized, there is a corresponding storage cost under each candidate quantization configuration information.

[0130] Specifically, the average (or sum) of the storage costs of each module under its selected configuration can be used as the quantized storage cost. With the quantized storage cost meeting the preset cost conditions as a constraint, and with the goal of minimizing the sum of the comprehensive costs of all modules, the corresponding target quantized configuration information can be selected for each module to be quantized from the candidate quantized configuration information.

[0131] For example, the corresponding target quantization configuration information can be selected for each module to be quantized by solving the constraint optimization problem shown in the following formula (4).

[0132] (4) in, This represents the number of parameters for the m-th module to be quantized. This indicates that the m-th module to be quantized is using the... The unit parameter storage cost when configuring candidate quantization information (i.e., the sum of the weight bits and the amortized metadata bits). This represents the preset storage cost threshold. This represents the set of candidate quantization configurations for the m-th module to be quantized. This represents the candidate quantization configuration selected for the m-th module. This represents the overall cost of the m-th module to be quantized under the selected configuration.

[0133] By solving the above optimization problem, a set of target candidate quantization configuration combinations can be obtained, which minimizes the overall quantization error of the model while satisfying the overall storage cost constraint. Those skilled in the art can flexibly choose dynamic programming or integer linear programming to solve the problem based on the number of modules to be quantized, the number of candidate configurations, and computing resources in the actual application. When the number of modules and candidate configurations is small, an exhaustive search approach can be used; when the problem size is moderate, dynamic programming is an efficient and easy-to-implement choice; when there are multiple complex constraints (such as simultaneous storage budget, latency budget, and effective bit count budget), integer linear programming provides stronger modeling and expressive capabilities.

[0134] The model quantization method provided in this application determines each module of the model to be quantized, and then uses each candidate quantization configuration information to quantize and dequantize the module to be quantized, obtaining each dequantized module corresponding to the module to be quantized. The candidate quantization configuration information is used to indicate the quantization method of the model parameters. That is, for each module to be quantized, the module to be quantized and dequantized is performed using the quantization method indicated by each candidate quantization configuration information, obtaining each dequantized module corresponding to the module to be quantized. Then, the module output error between the output of each dequantized module corresponding to the module to be quantized and the output of the corresponding module to be quantized is determined. This module output error can objectively reflect the degree of influence of different quantization configurations on the accuracy of the module to be quantized. Based on this, with the goal of reducing the module output error corresponding to each module to be quantized, suitable target quantization configuration information is selected from the candidate quantization configuration information for each module to be quantized.

[0135] This quantization configuration selection mechanism based on module-level output error feedback can accurately measure the actual impact of different quantization configurations on each module to be quantized, thus selecting the target configuration with minimal impact on model accuracy from among the candidate quantization configurations. Compared to the traditional method of assigning fixed precision solely based on module type, this application, by determining the module output error under different candidate configurations, can more precisely identify the differences in quantization sensitivity among different modules of the same type, improving the accuracy of mixed-precision quantization. It selects a quantization configuration that matches the characteristics of each module, enhancing the flexibility and accuracy of model quantization and effectively reducing the impact of quantization on model accuracy.

[0136] The model quantization method provided in this application is illustrated by a specific example below. Figure 3 As shown, the model quantization method provided in this application includes the following steps S1 to S9.

[0137] Step S1: Obtain the pre-trained LLM model weights W, the unlabeled text set, and the target end-side backend constraints and storage / performance budget.

[0138] The pre-trained LLM model weights W represent the model parameters of the model to be quantized. The unlabeled text set is the calibration data, which is used for subsequent calibration activation acquisition, module sensitivity calculation, and action condition operator error calculation. The target-side backend constraints include at least one of the following: the hardware type of the deployed device (e.g., CPU, GPU, NPU, DSP) and the inference engine type (e.g., MNN, llama.cpp / GGUF), i.e., the aforementioned target deployment scenario. These constraints are used to determine the generation rules for the candidate quantization configuration set and the calculation of metadata storage overhead.

[0139] Step S2: Generate candidate action space based on the target-side backend constraints and storage / performance budget. (That is, to determine the candidate quantitative configuration information set).

[0140] The target end-side backend constraints include at least one of the following: hardware type of the deployed device (e.g., CPU, GPU, NPU, DSP) and inference engine type (e.g., MNN, llama.cpp / GGUF, etc.). The deployment constraints also include at least one of the following: supported quantization operator implementation (kernel support), supported bit width, supported block size, whether channel-wise quantization is supported, scaling factor and zero-point metadata format, bandwidth budget, and prefill / decode performance targets. The target end-side backend constraints refer to the target deployment scenario described above.

[0141] The candidate quantization configuration information includes at least one of the following: number of quantization bits, quantization block size, and quantization granularity. It may also further include the type of backend quantizer (such as RTN, HQQ, AWQ, GPTQ, etc.).

[0142] For example, depending on the target deployment scenario, the candidate action space may include any one or more of the following: In a fixed-granularity bit-only scenario, the candidate quantization configuration information set can be set to {3-bit group-128, 4-bit group-128}, that is, selection is made only between different numbers of bits, and the quantization block size is fixed at group-128.

[0143] In scenarios where per-channel NPUs are limited, if the target NPU only supports per-channel quantization kernels, the candidate quantization configuration information set can be set to {4-bit channel-wise, 8-bit channel-wise}, that is, to select between per-channel channel-wise quantization with different bit numbers.

[0144] In scenarios where the target ARM CPU is more compatible with 4-bit / 8-bit data packet packing, the candidate quantization configuration information set can be set to {4-bit, 8-bit}×{group-32, group-64, group-128}, which is a combination of the number of traversed bits and the block size.

[0145] Step S3: Obtain the calibration activation corresponding to each module to be quantized.

[0146] The calibration data is input into the model to be quantized for forward inference. During the inference process, the input activation values ​​of each module to be quantized (such as linear layers, attention layers, etc.) are saved and recorded as calibration activation values. The calibration activation is used to subsequently calculate the module output error under each candidate quantization configuration information. The calibration data can specifically be an unlabeled text set.

[0147] Step S4: Calculate the importance level S(m) of each module to be quantified (i.e. the importance level in step S141).

[0148] Step S5: Calculate the effective bit cost B(m,a) under each candidate quantization configuration information (the storage cost in step S144).

[0149] The specific execution process of steps S4 and S5 is described above and will not be detailed here.

[0150] Step S6: Determine the action condition operator error AOE for each module to be quantized under each candidate quantization configuration information: E(m,a) (i.e., the module output error in step S130).

[0151] Specifically, for the m-th module to be quantized and the a-th candidate quantization configuration information, the model parameters of the module to be quantized are quantized and dequantized according to the candidate quantization configuration information, and dequantization weights are constructed. Subsequently, calibration activation was performed based on the data collected in step S3. Calculate the operator output perturbation caused by the quantization action under this candidate quantization configuration. (i.e., the direct module output error in step S134), and calculate the action condition operator error E(m,a) (i.e., the relative module output error in step S135).

[0152] The specific implementation method of this step can be found in the relevant descriptions of module output error determination in steps S130, S134 and S135 above, which will not be repeated here.

[0153] Step S7: Construct a module-action decomposable cost table (i.e., the weighted module output error in step S142).

[0154] Based on the importance level S(m) and action condition operator error E(m,a) determined in the preceding steps, the comprehensive cost of each module to be quantized under each candidate quantization configuration information is calculated (i.e., the weighted module output error in step S142). This comprehensive cost weights the module output error by importance level, and answers two questions simultaneously: Which module is more important (S(m) term)? Which quantization action would cause less damage to this module (E(m,a) term)? Modules with higher importance levels have their output errors amplified to a greater extent, thus tending to be assigned quantization configurations with smaller errors in subsequent optimizations.

[0155] A decomposable cost table is constructed based on the comprehensive cost C(m,a) of each module. The decomposable cost table represents each module to be quantized along a first dimension and each candidate quantization configuration along a second dimension. Each element in the table represents the comprehensive cost of the corresponding module under the corresponding configuration. Since the costs of each module are independent and cumulative, this cost table is decomposable (i.e., the total cost is the sum of the costs of each module). For the specific implementation of this step, please refer to the descriptions of importance weighting and the construction of the decomposable cost table in steps S141 to S142 above; they will not be repeated here.

[0156] Step S8: Under the condition of satisfying the storage budget constraint, solve the target quantization allocation scheme (i.e. the optimization solution in step S145).

[0157] The objective is to minimize the sum of the comprehensive costs of all modules to be quantized (i.e., the total error corresponding to the weighted module output error), and the constraint is that the weighted average of the effective bit cost of each module to be quantized does not exceed the preset storage budget B̄. The target quantization configuration information is then obtained by solving this problem. For details on the specific implementation of this step, please refer to the relevant descriptions of the optimization solution in steps S143 to S145 above; they will not be repeated here.

[0158] Step S9: Quantize each module to be quantized according to the solution results and write them into the deployment format.

[0159] After obtaining the target quantization configuration information, select the corresponding bit width, block size, or channel-wise quantization method for each module to be quantized, and use the corresponding quantization backend (such as RTN, HQQ, AWQ, GPTQ, or MNN quantization backend) to implement dequantization weights. The model construction and quantization parameter optimization involve quantizing each module to obtain quantized model parameters. Subsequently, the quantized model and its corresponding quantization parameters (including scaling factors, zero points, and other metadata) are written into the target deployment format (such as MNN, llama.cpp / GGUF, etc.). The solution and allocation process for this quantization configuration is offline, without increasing additional computational overhead during inference. The quantized model can be directly deployed to the target backend for efficient inference.

[0160] For details regarding the specific content in this example, please refer to the explanation above; further details will not be provided here.

[0161] Example 2 The second embodiment of this application also provides a model quantization apparatus corresponding to the model quantization method embodiment provided in the first embodiment. Since the apparatus embodiment is basically similar to the method embodiment, it is described simply. For details of the relevant technical features and their effects, please refer to the corresponding descriptions of the model quantization method embodiments provided above. The model quantization apparatus provided in this embodiment includes: The module determination unit is used to determine each module to be quantized in the model to be quantized. A quantization unit is used to quantize and dequantize the module to be quantized using each candidate quantization configuration information to obtain each dequantized module corresponding to the module to be quantized. The candidate quantization configuration information is used to indicate the quantization method of the model parameters. An error determination unit is used to determine the module output error between the output of the inverse quantization module and the output of the corresponding module to be quantized. The quantization selection unit is used to select the target quantization configuration information corresponding to the module to be quantized from the candidate quantization configuration information with the goal of reducing the module output error corresponding to each of the modules to be quantized, so as to quantize the corresponding module to be quantized through the target quantization configuration information.

[0162] Example 3 The third embodiment of this application also provides an electronic device, and the following description of the embodiment of the electronic device is merely illustrative. The embodiment of this electronic device is as follows: Please refer to Figure 4 Understanding the above electronic devices, Figure 4 This is a schematic diagram of an electronic device. The electronic device provided in this embodiment includes: a processor 1001, a memory 1002, a communication bus 1003, and a communication interface 1004; The memory 1002 is used to store computer instructions for data processing, which, when read and executed by the processor 1001, execute the method described in the first embodiment.

[0163] Example 4 The fourth embodiment of this application also provides a computer-readable storage medium for implementing the method of any one of the first or second embodiments. The embodiments of the computer-readable storage medium provided in this application are described in a relatively simple manner; relevant parts can be found in the corresponding descriptions of the above method embodiments. The embodiments described below are merely illustrative.

[0164] The computer-readable storage medium provided in this embodiment stores computer instructions, which, when executed by a processor, implement the steps described in the first embodiment.

[0165] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0166] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0167] 1. Computer-readable media includes both permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined in this application, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0168] 2. Those skilled in the art will understand that embodiments of this application can provide methods, systems, or computer program products. Therefore, embodiments of this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0169] 3. This application embodiment may involve the use of user data. In practical applications, user-specific personal data may be used within the scope permitted by applicable laws and regulations of the country in which the application is located (e.g., with the user's explicit consent and effective notification to the user, etc.). Furthermore, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0170] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

Claims

1. A model quantization method, characterized in that, include: Identify the individual modules of the model to be quantized; The module to be quantized is quantized and dequantized using each candidate quantization configuration information to obtain each dequantized module corresponding to the module to be quantized. The candidate quantization configuration information is used to indicate the quantization method of the model parameters. Determine the module output error between the output of the inverse quantization module and the output of the corresponding module to be quantized; With the goal of reducing the module output error corresponding to each of the modules to be quantized, target quantization configuration information corresponding to the module to be quantized is selected from each of the candidate quantization configuration information, so as to quantize the corresponding module to be quantized through the target quantization configuration information.

2. The method according to claim 1, characterized in that, Before using the candidate quantization configuration information to quantize and dequantize the module to be quantized, the method further includes: Based on the target deployment scenario of the model to be quantized, a candidate quantization configuration information set corresponding to the model to be quantized is determined. The target deployment scenario includes at least one of the following attributes of the deployed device: hardware type, inference engine type, supported quantization operator implementation, storage budget, performance budget, allowed number of bits, supported block format, and metadata format.

3. The method according to claim 1, characterized in that, Determining the module output error between the output of the inverse quantization module and the output of the corresponding module to be quantized includes: The calibration data is input into the model to be quantized for forward inference to obtain the calibration activation and pre-quantization module output corresponding to each module to be quantized. The inverse quantization module corresponding to the module to be quantized is used to infer the corresponding calibration activation to obtain the output of the quantized module. The module output error between the output of the quantized module and the output of the module before quantization is determined as the module output error between the output of the dequantization module and the output of the corresponding module to be quantized.

4. The method according to claim 1, characterized in that, Determining the module output error between the output of the inverse quantization module and the output of the corresponding module to be quantized includes: Determine the direct module output error between the output of the dequantization module and the output of the module to be quantized under the same input; Based on the error between the output of the module to be quantized and the output of the direct module, the relative module output error is obtained, which is used as the module output error of the output of the inverse quantization module relative to the output of the module to be quantized.

5. The method according to claim 1, characterized in that, The step of selecting the target quantization configuration information corresponding to the module to be quantized from the candidate quantization configuration information with the goal of reducing the module output error corresponding to each of the modules to be quantized includes: Determine the importance level of the module to be quantized, whereby the importance level represents the degree of impact of the perturbation of the module to be quantized on the overall output of the quantization model; The weighted module output error is obtained by weighting the corresponding module output error according to the importance of the module to be quantized. With the goal of reducing the output error of the weighted module corresponding to each of the quantized modules, the target quantization configuration information corresponding to the quantized module is selected from the candidate quantization configuration information.

6. The method according to claim 5, characterized in that, Determining the importance level of the module to be quantified includes: The calibration data is input into the model to be quantized for forward inference to obtain the inference result; Determine the loss corresponding to the inference result; Backpropagation is performed on the module to be quantized based on the loss to obtain the gradient corresponding to the module to be quantized, and the importance of the module to be quantized is determined based on the gradient.

7. The method according to claim 5, characterized in that, The step of selecting the target quantization configuration information corresponding to the module to be quantized from the candidate quantization configuration information with the goal of reducing the output error of the weighted module corresponding to each of the modules to be quantized includes: With the goal of reducing the total error corresponding to the weighted module output error of each of the quantized modules, the target quantization configuration information corresponding to the quantized module is selected from the candidate quantization configuration information.

8. The method according to claim 1, characterized in that, The step of selecting the target quantization configuration information corresponding to the module to be quantized from the candidate quantization configuration information with the goal of reducing the module output error corresponding to each of the modules to be quantized includes: The storage cost of the module to be quantized is determined using the candidate quantization configuration information. The quantization storage cost is constrained by meeting a preset cost condition, and the goal is to reduce the module output error corresponding to each of the modules to be quantized. The target quantization configuration information corresponding to the module to be quantized is selected from each of the candidate quantization configuration information. The quantization storage cost is the sum of the storage costs corresponding to each of the modules to be quantized, or the average effective storage cost obtained by weighting the parameters of each module to be quantized.

9. The method according to claim 7, characterized in that, The step of determining the storage cost of the module to be quantized using the candidate quantization configuration information includes: The storage cost of the module to be quantized is determined based on the number of weight bits and metadata bits corresponding to the candidate quantization configuration information.

10. The method according to any one of claims 1 to 9, characterized in that, The candidate quantization configuration information includes at least one of the following: the number of quantization bits for quantizing model parameters, the quantization block size, the quantization granularity, and the type of backend quantizer.

11. A model quantization device, characterized in that, include: The module determination unit is used to determine each module to be quantized in the model to be quantized. A quantization unit is used to quantize and dequantize the module to be quantized using each candidate quantization configuration information to obtain each dequantized module corresponding to the module to be quantized. The candidate quantization configuration information is used to indicate the quantization method of the model parameters. An error determination unit is used to determine the module output error between the output of the inverse quantization module and the output of the corresponding module to be quantized. The quantization selection unit is used to select the target quantization configuration information corresponding to the module to be quantized from the candidate quantization configuration information with the goal of reducing the module output error corresponding to each of the modules to be quantized, so as to quantize the corresponding module to be quantized through the target quantization configuration information.

12. An electronic device, characterized in that, include: Processor, memory, and computer program instructions stored in said memory and executable on the processor; When the processor executes the computer program instructions, it implements the method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-10.

Images