Hardware-aware model quantization method, system, medium, product and terminal

Abstract

The application provides a hardware perception-based model quantization method, system, medium, product and terminal, a hybrid execution mechanism is constructed between a high-level software framework and a bottom layer hardware behavior model, only core calculation operators in a quantization calculation graph of an initial quantization model which need to be executed on target hardware are handed over to a hardware behavior model for bit precision simulation, real calculation characteristics of the target hardware are simulated in a quantization process, and other auxiliary, non-computation-intensive operators are reserved in the high-level software framework for execution, so that the simulation verification efficiency is improved, the simulation precision and the execution efficiency are considered, and the goal that the quantized model runs more efficiently and has lower precision loss on the target hardware is achieved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a hardware-aware model quantization method, system, medium, product, and terminal. Background Technology

[0002] In recent years, large language models (LLMs) based on the Transformer architecture have achieved great success in the field of artificial intelligence. However, such models typically have an extremely high number of parameters (reaching hundreds of billions or even trillions), resulting in the consumption of enormous computing resources and memory bandwidth during deployment and inference. This characteristic severely restricts the deployment and application of large language models on resource-constrained devices (such as mobile devices and edge devices).

[0003] To address this issue, model quantization techniques have emerged. Quantization converts high-precision floating-point parameters (e.g., FP32, FP16) and activation values ​​in the model into low-precision fixed-point integer representations (e.g., INT8, INT4), effectively compressing model size, reducing memory usage, and accelerating the inference process using low-precision computing units provided by hardware. Currently, mainstream quantization methods mainly include Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT). Weight-only quantization, such as W4A16 (4-bit weights, 16-bit activation values), effectively alleviates the memory bandwidth bottleneck by compressing the weights to 4 bits while maintaining the precision of the activation values.

[0004] Although existing quantization techniques have achieved model compression and acceleration to some extent, current quantization methods generally suffer from a core problem of lacking hardware-awareness, namely, the mismatch between general-purpose quantization algorithms and the computational characteristics of specific hardware. This manifests in the following aspects:

[0005] First, existing quantization methods typically ignore the differences in hardware microarchitecture: standard quantization algorithms are usually based on idealized mathematical models that assume the hardware can perfectly execute the defined quantization and dequantization operations. However, different AI chips (such as ASICs, FPGAs, and GPUs) differ significantly in microarchitectural details such as multiply-accumulate unit structures, data path designs, rounding mode implementations, and overflow handling mechanisms. This difference leads to a systematic deviation between the quantization behavior simulated by software and the actual computational behavior performed by the hardware, making it difficult to accurately predict the performance of the quantization model on real hardware.

[0006] Secondly, due to the aforementioned biases, the accuracy loss of quantized models becomes difficult to predict: a model that performs well in quantization and validation within standard frameworks (such as PyTorch and TensorFlow) may experience unexpected accuracy degradation after deployment to specific target hardware (such as the WD1 chip). Troubleshooting and optimizing for such accuracy losses typically requires repeated actual hardware deployment and testing, a process that is not only time-consuming and labor-intensive but also significantly impacts the overall efficiency of model deployment.

[0007] Finally, traditional simulation verification methods are inefficient: although theoretically, accuracy deviations can be discovered through low-level hardware simulation, using full-process low-level simulation (such as RTL-level simulation) to verify the behavior of the quantized model on the target hardware is extremely slow for large language models, making it impractical for use during algorithm iteration cycles. This makes rapid and accurate hardware behavior verification during the quantization stage a difficult contradiction to resolve. Summary of the Invention

[0008] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a hardware-aware model quantization method, system, medium, product and terminal to solve the technical problem of additional accuracy loss caused by the mismatch between general quantization algorithms and specific hardware.

[0009] To achieve the above and other related objectives, a first aspect of this application provides a hardware-aware model quantization method, comprising: performing preliminary low-precision quantization on a pre-trained neural network model according to a preset high-level software framework to obtain an initial quantized model, and constructing a quantization computation graph of the initial quantized model and determining the initial quantization parameters of each operator in the quantization computation graph; constructing a hardware behavior model based on the hardware resource status of the initial quantized model to be deployed on the target hardware, and constructing a hybrid execution engine accordingly; and performing multiple rounds of hybrid simulation computation on the quantization computation graph of the initial quantized model based on the hybrid execution engine and the initial quantization parameters of each operator in the quantization computation graph to obtain the final target quantized model.

[0010] In some embodiments of the first aspect of this application, the type of operator includes core computation operators; wherein, the method of performing multiple rounds of hybrid simulation computation on the quantization computation graph of the initial quantization model based on the hybrid execution engine and the initial quantization parameters of each operator in the quantization computation graph to obtain the final target quantization model includes: using the hybrid execution engine to traverse the quantization computation graph, and distributing and scheduling different types of operators in the quantization computation graph according to preset rules, thereby executing the computation of different types of operators to generate low-precision simulation computation results corresponding to each core computation operator; performing optimization operations on the quantization computation graph based on the low-precision simulation computation results corresponding to each core computation operator; and performing multiple rounds of hybrid simulation computation on the quantization computation graph after the optimization operation to obtain the final target quantization model.

[0011] In some embodiments of the first aspect of this application, the type of the operator includes a general auxiliary operator; wherein, the method of using the hybrid execution engine to traverse the quantization computation graph and distribute and schedule different types of operators in the quantization computation graph according to preset rules to execute the computation of different types of operators includes: if the type of the operator is a general auxiliary operator, then calling the preset high-level software framework to perform the computation; if the type of the operator is a core computation operator, then performing operator fusion operation and operator remapping operation to call the corresponding hardware functional module in the hardware behavior model to perform the computation.

[0012] In some embodiments of the first aspect of this application, the method of performing operator fusion operation and operator remapping operation includes: if the type of the operator is a core computing operator, then performing an operator interception operation to generate a logical combination operator sequence; performing an operator fusion operation on the logical combination operator sequence to generate a composite operator; and performing an operator remapping operation on the composite operator to construct a mapping relationship between the composite operator and the hardware functional modules in the hardware behavior model.

[0013] In some embodiments of the first aspect of this application, the method of calling the corresponding hardware functional module in the hardware behavior model to perform calculations includes: according to the mapping relationship between the composite operator and the hardware functional module in the hardware behavior model, calling the corresponding hardware functional module in the hardware behavior model to perform calculations to generate low-precision simulation calculation results corresponding to the core calculation operator; returning the low-precision simulation calculation results corresponding to the core calculation operator to the execution environment of the preset high-level software framework, so as to execute the calculations of subsequent operators.

[0014] In some embodiments of the first aspect of this application, the method of performing the optimization operation of the quantization calculation graph based on the low-precision simulation calculation results corresponding to each core calculation operator includes: performing error analysis on the low-precision simulation calculation results corresponding to each core calculation operator based on the preset benchmark results corresponding to each core calculation operator, so as to lock the target operator to be optimized from each core calculation operator; adjusting the initial quantization parameters of the target operator to be optimized, so as to perform the optimization operation of the quantization calculation graph.

[0015] To achieve the above and other related objectives, a second aspect of this application provides a hardware-aware model quantization system, comprising: an initial quantization module, configured to perform preliminary low-precision quantization on a pre-trained neural network model according to a preset high-level software framework to obtain an initial quantized model, and to construct a quantization computation graph of the initial quantization model and determine the initial quantization parameters of each operator in the quantization computation graph; a hybrid execution engine construction module, configured to construct a hardware behavior model based on the hardware resource status of the initial quantization model to be deployed to the target hardware, and thereby construct a hybrid execution engine; and an optimization module, configured to perform multiple rounds of hybrid simulation computation on the quantization computation graph of the initial quantization model based on the hybrid execution engine and the initial quantization parameters of each operator in the quantization computation graph, to obtain the final target quantization model.

[0016] To achieve the above and other related objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the hardware-aware model quantization method described above.

[0017] To achieve the above and other related objectives, a fourth aspect of this application provides a computer program product comprising computer program code that, when executed on a computer, causes the computer to implement the hardware-aware model quantization method as described above.

[0018] To achieve the above and other related objectives, a fifth aspect of this application provides an electronic terminal, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the hardware-aware model quantization method described above.

[0019] As described above, the hardware-aware model quantization method, system, medium, product, and terminal of this application construct a hybrid execution mechanism between a high-level software framework and a low-level hardware behavior model. Only the core computational operators that need to be executed on the target hardware in the quantization computation graph of the initial quantization model are handled by the hardware behavior model for bit-precision simulation. This simulates the actual computational characteristics of the target hardware during the quantization process, while other auxiliary, non-computationally intensive operators (such as data shape transformations) are retained and executed within the high-level software framework. This improves simulation verification efficiency, balances simulation accuracy and execution efficiency, and achieves the goal of more efficient operation of the quantized model on the target hardware with lower accuracy loss. Specifically, this is reflected in the following aspects:

[0020] (1) Reduce quantization accuracy loss: By introducing a hardware behavior model simulation with bit precision, this invention incorporates the actual computational behavior of the hardware into the closed loop of quantization optimization, effectively avoiding additional accuracy loss caused by differences in software and hardware computation, so that the final generated target quantization model has higher accuracy on the target hardware.

[0021] (2) Shorten the model deployment and simulation verification cycle: By introducing a hybrid execution engine, only the core calculation is simulated with high precision but time-consuming hardware, while a large number of auxiliary operations are performed in the high-level software framework, which improves the overall simulation speed, shortens the time for full model bit precision simulation, and accelerates the model development and deployment iteration process.

[0022] (3) Improve the efficiency of the model on the target hardware: By introducing hardware awareness, the quantization process can adapt to the computing characteristics of the hardware in a targeted manner, effectively reducing the additional accuracy loss caused by the mismatch between traditional quantization and the hardware.

[0023] (4) Good versatility and scalability: For different hardware platforms, as long as the corresponding hardware behavior model and calling interface are provided, the present invention can be used to perform targeted hardware perception quantization. Attached Figure Description

[0024] Figure 1 The diagram shown is a flowchart of a hardware-aware model quantization method according to an embodiment of this application.

[0025] Figure 2 The diagram shown is a flowchart illustrating the process of generating a target quantization model in one embodiment of this application.

[0026] Figure 3 The diagram shown is a schematic diagram of the hybrid execution and operator remapping mechanism in one embodiment of this application.

[0027] Figure 4 The diagram shown is a schematic block diagram of a hardware-aware model quantization system according to an embodiment of this application.

[0028] Figure 5 The diagram shown is a structural schematic of an electronic terminal according to an embodiment of this application. Detailed Implementation

[0029] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0030] Before providing a further detailed description of the present invention, the nouns and terms used in the embodiments of the present invention are explained, and the nouns and terms used in the embodiments of the present invention are subject to the following interpretations:

[0031] <1> Transformer: A self-attention model whose core is the self-attention mechanism, which can process sequential data (such as sentences) in parallel, thereby capturing long-range dependencies within the data more efficiently.

[0032] <2> ASIC: Application-Specific Integrated Circuit, is an integrated circuit that is specifically designed and manufactured for a particular application or purpose.

[0033] <3> FPGA: Field Programmable Gate Array, is a semi-custom integrated circuit that allows users to configure its internal logic circuit structure through programming after manufacturing, enabling it to achieve specific hardware functions.

[0034] <4> GPU: Graphics Processing Unit, originally designed as a dedicated processor for parallel processing of computer graphics (such as rendering game graphics). Due to its massively parallel architecture (with thousands of cores), it is well-suited for handling large-scale matrix operations in deep learning and is therefore widely used for model training and inference.

[0035] <5> PyTorch: A deep learning framework known for its dynamic computation graph, intuitive programming interface, and excellent debugging capabilities.

[0036] <6> TensorFlow: A deep learning framework characterized by a static computation graph, suitable for large-scale deployment and production environments.

[0037] <7> RTL-level simulation: Register-transfer-level simulation, is the process of simulating the precise operating behavior of the entire chip or module at each clock cycle in a software environment using its "register-transfer-level" design code before chip manufacturing.

[0038] <8> Qwen2.5-3B: The 2.5th generation of the Tongyi Qianwen series of models, a lightweight large language model with approximately 3 billion parameters.

[0039] <9> WD1 chip: It is an artificial intelligence chip whose hardware structure and instruction set are specifically optimized for core artificial intelligence calculations such as matrix multiplication and convolution.

[0040] <10> Cycle-Accurate: A cycle-accurate model is a software simulation model that not only simulates the mathematical logic results (i.e., bit precision) of hardware computing units, but also rigorously simulates the generation process of these results in the time dimension.

[0041] <11> CNN: Convolutional Neural Network, is a type of feedforward neural network specifically designed for processing data with a grid-like topology (such as images, videos, and audio).

[0042] <12> CPU: Central Processing Unit.

[0043] To facilitate understanding of the embodiments of this application, firstly, in conjunction with Figure 1 Detailed explanation. Figure 1 This document illustrates a flowchart of a hardware-aware model quantization method according to an embodiment of the present invention. The hardware-aware model quantization method in this embodiment mainly includes the following steps:

[0044] S101: Based on the preset high-level software framework, perform preliminary low-precision quantization on the pre-trained neural network model to obtain an initial quantization model, and construct the quantization calculation graph of the initial quantization model and determine the initial quantization parameters of each operator in the quantization calculation graph.

[0045] In this embodiment, PyTorch is selected as the preset high-level software framework, and a large language model pre-trained on the training dataset is loaded. Subsequently, PyTorch's built-in quantization tools are used to perform preliminary low-precision quantization operations to obtain an initial quantization model. The quantization computation graph is then constructed within the preset high-level software framework, and the initial quantization parameters of each operator in the quantization computation graph are determined. These initial quantization parameters include scaling factors and zero-point values.

[0046] In this embodiment, the types of operators include core computation operators, general auxiliary operators, and quantization auxiliary operators. Core computation operators include, for example, matrix multiplication operators (MatMul) and convolution operators (Conv). General auxiliary operators include, for example, shape reshape operators (Reshape) and transpose operators (Transpose). Quantization auxiliary operators include, for example, dequantize operators (Dequantize) and quantize operators (Quantize).

[0047] S102: Based on the hardware resource status of the target hardware to be deployed according to the initial quantization model, construct a hardware behavior model, and construct a hybrid execution engine accordingly.

[0048] In this embodiment, after obtaining the initial quantization model, a corresponding hardware behavior model is constructed based on the hardware resource conditions of the target hardware (such as the WD1 chip), including limited on-chip memory capacity, the size of the multiply-accumulate unit array, and non-standard overflow / saturation handling logic. The hardware behavior model can simulate the numerical behavior of the target hardware when executing core computational operators such as quantized convolution / fully connected layers, in order to verify the true performance of the quantization model on the target hardware.

[0049] In this embodiment, the hardware behavior model (CModel) is a software model that can simulate the target hardware computing logic with bit-level precision. It is used to predict the execution behavior and numerical results of the quantization algorithm on the real chip before the model is deployed.

[0050] In this embodiment, a hybrid execution engine is constructed based on this hardware behavior model. This hybrid execution engine uses a preset high-level software framework and hardware behavior model to perform the calculation of the quantization computation graph. Only the core computation operators that need to be executed on the target hardware in the model quantization computation graph are handed over to the hardware behavior model for bit-precision simulation, while other auxiliary and non-computationally intensive operators (such as data shape transformation) are kept in the high-level software framework for execution. This hybrid execution strategy improves the speed of model verification while ensuring the accuracy of critical path computation behavior, and takes into account both simulation accuracy and execution efficiency.

[0051] S103: Based on the hybrid execution engine and the initial quantization parameters of each operator in the quantization calculation graph, perform multiple rounds of hybrid simulation calculation on the quantization calculation graph of the initial quantization model to obtain the final target quantization model.

[0052] In this embodiment, as Figure 2The diagram illustrates the process of generating a target quantization model in an embodiment of the present invention. The operators include core computational operators; wherein, based on the hybrid execution engine and the initial quantization parameters of each operator in the quantization computation graph, multiple iterations of hybrid simulation computation are performed on the quantization computation graph of the initial quantization model to obtain the final target quantization model, including:

[0053] S1031: The hybrid execution engine is used to traverse the quantization calculation graph and distribute and schedule different types of operators in the quantization calculation graph according to preset rules, so as to execute the calculation of different types of operators and generate low-precision simulation calculation results corresponding to each core calculation operator.

[0054] In this embodiment, the operator type includes general auxiliary operators; wherein, the method of using the hybrid execution engine to traverse the quantization computation graph and distributing and scheduling different types of operators in the quantization computation graph according to preset rules to execute the computation of different types of operators includes:

[0055] (1) If the operator is a general auxiliary operator, the preset high-level software framework is invoked to perform the calculation.

[0056] (2) If the type of the operator is a core computing operator, then the operator fusion operation and the operator remapping operation are performed to call the corresponding hardware function module in the hardware behavior model to perform the calculation.

[0057] In this embodiment, the methods for performing operator fusion and operator remapping operations include:

[0058] (1) If the type of the operator is a core computation operator, then perform an operator interception operation to generate a logical combination operator sequence.

[0059] (2) Perform operator fusion operation on the logical combination operator sequence to generate composite operators.

[0060] (3) Perform operator remapping operation on the composite operator to construct the mapping relationship between the composite operator and the hardware functional modules in the hardware behavior model.

[0061] In this embodiment, a pre-prepared calibration dataset is input into the initial quantization model. The hybrid execution engine, starting from the initial node of the quantization computation graph, sequentially visits each operator according to preset rules. Upon encountering a general auxiliary operator, the native interface of the high-level software framework is invoked to perform calculations and obtain the results. The next operator then performs subsequent calculations based on these results. These general auxiliary operators typically do not involve core numerical calculations; they only perform tensor format conversion and dimension adjustment. The calculation results are directly updated to the intermediate representation layer, maintaining the normal flow of the quantization computation graph. Continuing to access the next operator, if a core computation operator is encountered, an interception mechanism is activated. After performing operator interception, operator fusion, and operator remapping operations, the corresponding hardware functional module in the hardware behavior model is invoked to perform calculations, generating low-precision simulation results corresponding to the core computation operator. These results are then returned to the execution environment of the high-level software framework as input for subsequent operators, continuing the execution of the remaining part of the quantization computation graph. This ensures a high degree of consistency between the software simulation logic and the hardware execution behavior.

[0062] In this embodiment, if a core computational operator is encountered, an interception mechanism is activated to identify combinations of operators that are separate at the software level but constitute an atomic operation at the hardware level. Upon identification, not only the core computational operator but also its associated quantization auxiliary operators are intercepted to generate a logical combination operator sequence. After the operator interception operation is completed, an operator fusion operation is performed on the logical combination operator sequence to integrate multiple discrete operator nodes into a unified composite operator. The operator fusion operation eliminates the overhead of storing and transmitting intermediate results between operators while maintaining complete computational semantics, transforming a computational process that originally required multiple calls into a single computational unit. Next, an operator remapping operation is performed on the generated composite operator. Based on the architectural characteristics of the hardware behavior model, by analyzing the computational characteristics and numerical accuracy requirements of the composite operator, functional modules capable of simulating its execution behavior are found in the hardware behavior model. A correspondence is established between the composite operator and the underlying hardware functional modules, forming a mapping from software operators to hardware simulation units.

[0063] In this embodiment, exemplarily, when the hybrid execution engine accesses the shape reshape operator, it immediately invokes the native interface of the high-level software framework to quickly perform shape transformation calculations on the CPU. When the hybrid execution engine accesses the matrix multiplication operator (MatMul), it identifies the dequantize and quantize operators associated with it. Therefore, it intercepts this set of associated operators as a whole, obtaining a logical combination operator sequence of dequantize → matrix multiplication operator → quantize (Dequantize → Matmul → Quantize). This logical combination operator sequence is then fused into a quantized matrix multiplication composite operator (quantizedmatmul). Based on the computational characteristics of this composite operator, the quantized matrix multiplication composite operator (quantizedmatmul) is remapped to the corresponding hardware functional module in the hardware functional module.

[0064] In this embodiment, the method of calling the corresponding hardware function module in the hardware behavior model to perform calculations includes:

[0065] (1) Based on the mapping relationship between the composite operator and the hardware functional module in the hardware behavior model, the corresponding hardware functional module in the hardware behavior model is called to perform calculations to generate the low-precision simulation calculation results corresponding to the core calculation operator.

[0066] (2) Return the low-precision simulation calculation results corresponding to the core calculation operator to the execution environment of the preset high-level software framework, so as to execute the calculation of subsequent operators.

[0067] In this embodiment, the hardware behavior model is built based on the microarchitectural characteristics of the target hardware. It can simulate the numerical behavior of the actual hardware when processing quantization calculations. By simulating the hardware-level computing pipeline, it obtains output results that conform to the actual operating characteristics of the target hardware. It can truly reflect the actual operation of the quantization model on the hardware, avoid additional accuracy loss caused by differences in software and hardware calculations, and make the final generated quantization model have higher accuracy on the target hardware.

[0068] It is worth noting that since the general auxiliary operator only changes the view of the data and does not perform any multiplication or addition operations, its computational overhead is extremely low. Therefore, allowing the general auxiliary operator to continue performing calculations within the high-level software framework avoids unnecessary hardware simulation overhead, leverages software-level operational efficiency, and improves calibration speed. Furthermore, by intercepting the core computational operators and their associated quantization auxiliary operators and entrusting them to the hardware behavior model for processing, it is ensured that key computational steps accurately reflect the actual numerical characteristics of the target hardware, effectively capturing numerical deviations caused by differences in hardware microarchitecture. This hybrid execution framework maintains the execution speed of the calibration process while ensuring the accuracy of quantization parameter calibration, providing a reliable accuracy guarantee for the subsequent deployment of the quantization model on the target hardware and resolving the inconsistency between pure software simulation and actual hardware performance.

[0069] S1032: Based on the low-precision simulation results corresponding to each of the core computation operators, perform optimization operations on the quantization computation graph. This includes:

[0070] (1) Based on the preset benchmark results corresponding to each core computational operator, perform error analysis on the low-precision simulation calculation results corresponding to each core computational operator in order to lock the target operator to be optimized from each core computational operator.

[0071] (2) Adjust the initial quantization parameters of the target operator to be optimized in order to perform the optimization operation of the quantization calculation graph.

[0072] S1033: Based on the quantization calculation graph after optimization, perform multiple rounds of hybrid simulation calculations to obtain the final target quantization model.

[0073] In this embodiment, a hybrid execution engine starts from the initial node of the quantization computation graph and sequentially visits each operator according to preset rules. When a general auxiliary operator is encountered, the native interface of the high-level software framework is invoked to perform the calculation and obtain the result. The next operator then performs the calculation of subsequent operators in the quantization computation graph based on the result. The process continues to traverse the next operator. If a core computation operator is encountered, an interception mechanism is activated to perform operator interception, operator fusion, and operator remapping operations. This calls the corresponding hardware functional module in the hardware behavior model to perform the calculation and generate a low-precision simulation result corresponding to the core computation operator. The low-precision simulation result is then returned to the execution environment of the high-level software framework as input for subsequent operators, and the remaining part of the quantization computation graph continues to be executed. The process involves obtaining the preset benchmark result corresponding to the core computational operator in the pre-trained neural network model. This benchmark result is a high-precision computation result. The low-precision simulation result corresponding to the core computational operator is compared with the benchmark result to analyze the error. If the quantization error between the two meets a preset threshold, the initial quantization parameters of the core computational operator are not adjusted, and the calculation of subsequent operators continues using the low-precision simulation result. If the quantization error does not meet the preset threshold, the low-precision simulation result is returned to the execution environment of the high-level software framework as input for the next operator. The core computational operator is then locked to the target operator to be optimized. Based on the quantization error, the initial quantization parameters of the target operator to be optimized are adjusted for the next round of hybrid simulation calculation of the quantization graph. This process is iterated until the accuracy loss of the quantization model under hybrid execution simulation meets the preset requirements. Finally, the target quantization model optimized by hardware awareness is output.

[0074] It is worth noting that the hardware-aware model quantization method of this invention constructs a hybrid execution mechanism between a high-level software framework and a low-level hardware behavior model. Only the core computational operators in the initial quantization model's quantization computation graph that need to be executed on the target hardware are handled by the hardware behavior model for bit-precision simulation. This simulates the actual computational characteristics of the target hardware during the quantization process, while other auxiliary, non-computationally intensive operators (such as data shape transformations) are retained and executed within the high-level software framework. This improves simulation verification efficiency, balances simulation accuracy and execution efficiency, and achieves the goal of more efficient operation of the quantized model on the target hardware with lower accuracy loss. Specifically, this is reflected in the following aspects:

[0075] (1) Reduce quantization accuracy loss: By introducing a hardware behavior model simulation with bit precision, this invention incorporates the actual computational behavior of the hardware into the closed loop of quantization optimization, effectively avoiding additional accuracy loss caused by differences in software and hardware computation, so that the final generated target quantization model has higher accuracy on the target hardware.

[0076] (2) Shorten the model deployment and simulation verification cycle: By introducing a hybrid execution engine, only the core calculation is simulated with high precision but time-consuming hardware, while a large number of auxiliary operations are performed in the high-level software framework, which improves the overall simulation speed, shortens the time for full model bit precision simulation, and accelerates the model development and deployment iteration process.

[0077] (3) Improve the efficiency of the model on the target hardware: By introducing hardware awareness, the quantization process can adapt to the computing characteristics of the hardware in a targeted manner, effectively reducing the additional accuracy loss caused by the mismatch between traditional quantization and the hardware.

[0078] (4) Good versatility and scalability: For different hardware platforms, as long as the corresponding hardware behavior model and calling interface are provided, the present invention can be used to perform targeted hardware perception quantization.

[0079] Furthermore, the following provides specific application examples of hardware-aware model quantization methods.

[0080] like Figure 3 The diagram illustrates the hybrid execution and operator remapping mechanism in an embodiment of the present invention. This embodiment aims to perform W4A16 quantization on a Linear layer (fully connected layer) of the large model Qwen2.5-3B within the PyTorch framework and deploy it to the WD1 chip.

[0081] Step 1: Preliminary low-precision quantization operation

[0082] The Qwen2.5-3B model was initially quantized using W4A16, resulting in a quantization computation graph containing different types of operators. Here, W4 indicates that the precision of the fully connected layer weight matrix (W) is reduced from the original 16-bit floating-point (FP16) or 32-bit floating-point (FP32) to 4-bit integers (INT4). A16 indicates that the activation values ​​are maintained at the precision of 16-bit floating-point (FP16).

[0083] The hybrid execution engine of this invention is launched. When the model executes a transpose operation before the Linear layer (fully connected layer), the transpose operation is recognized as a general auxiliary operator, which is efficiently executed on a general-purpose processor by calling the native interface of the PyTorch framework, and generates an FP16 precision input tensor as its output.

[0084] Step 2: Operator interception, remapping, and hardware behavior model invocation

[0085] (1) Remapping of high-level logic to hardware primitives: The hybrid execution engine recognizes the next Linear layer (fully connected layer), whose logic in the software can be decomposed into Dequantize -> MatMul -> Quantize (dequantization operator -> matrix multiplication operator -> quantization operator). At this point, operator interception is triggered. The engine does not execute these three separate operations in PyTorch, but instead performs operator fusion and operator remapping operations on this continuous operation, and remaps it as a single function called wd1_quantized_matmul in the hardware behavior model.

[0086] (2) Atomized hardware behavior simulation: The design of this function completely imitates the corresponding hardware function module on the WD1 chip. This module receives 4-bit quantization weights and 16-bit activation values ​​as inputs, and completes the complete process of decompression, multiplication and addition, accumulation, rounding and requantization internally, and directly outputs the low-precision output tensor after hardware calculation.

[0087] Step 3: Data Feedback and Error Analysis

[0088] (1) After the hardware behavior model completes the bit-precision calculation, it returns a low-precision output tensor. The data of this low-precision output tensor is fed back into the PyTorch memory space for use by subsequent native operators in the quantization computation graph.

[0089] (2) After the entire model (or a sub-module) has been executed by hybrid process, the low-precision output tensor is compared with the preset benchmark result of the original large model to calculate the error.

[0090] (3) Hardware behavior error tracing: Through comparison, it was found that the output error of the Lth layer in the model was significantly higher than that of other layers. Further analysis of the internal state and calculation trajectory of the hardware behavior model revealed that when the WD1 chip processes multiplication and accumulation within a specific numerical range, its internal truncation logic introduces a small negative bias.

[0091] Step 4: Closed-loop feedback and iterative optimization

[0092] Parameter optimization for hardware characteristics. To address the negative bias issue identified above, the parameter optimization module initiates an optimization strategy. It fine-tunes the quantization zeros corresponding to the weights of layer L, creating a slight positive bias to counteract the hardware-introduced negative bias.

[0093] After adjusting the parameters, the above hybrid simulation calculation process was re-executed. After several iterations, the simulation output error of the Lth layer was significantly reduced, and the accuracy loss of the entire model converged to within the preset target threshold.

[0094] Finally, the optimized target quantization model is output. This target quantization model can be directly loaded onto the WD1 chip to achieve high-precision and high-efficiency operation results that are highly consistent with the hardware behavior model simulation.

[0095] It should be noted that the hardware-aware model quantization method of the present invention enables the quantization process to make more targeted use of hardware characteristics. Without sacrificing (or even improving) accuracy, a more aggressive quantization strategy (such as a lower number of bits) can be adopted to achieve higher computational efficiency and lower power consumption.

[0096] Secondly, the hardware behavior model (CModel) in this invention is not limited to a specific implementation. Any simulation model that can simulate the computational behavior of the target hardware with bit-level precision, such as the Cycle-Accurate model or the RTL-level simulation model, is applicable to this invention. Furthermore, the quantization scheme of this invention is not limited to W4A16; it is also applicable to other quantization schemes, such as W8A8 and W4A4. Its core idea lies in using bit-precision simulation to guide and optimize any type of quantization process. Finally, the method of this invention is applicable not only to large language models but also to other types of deep neural network models, such as CNN models in the field of computer vision, as long as the target hardware provides a corresponding bit-precision behavior model.

[0097] It should be noted that, in the embodiments of this application, the words "exemplary" or "for example" indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0098] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0099] Figure 4 This is a schematic block diagram of a hardware-aware model quantization system provided in an embodiment of this application. Figure 4As shown, the hardware-aware model quantization system 400 includes:

[0100] The initial quantization module 401 is used to perform preliminary low-precision quantization operations on the pre-trained neural network model according to a preset high-level software framework to obtain an initial quantization model, and to construct a quantization calculation graph of the initial quantization model and determine the initial quantization parameters of each operator in the quantization calculation graph.

[0101] The hybrid execution engine building module 402 is used to build a hardware behavior model based on the hardware resource status of the target hardware to be deployed by the initial quantization model, and thereby build a hybrid execution engine.

[0102] The optimization module 403 is used to perform multiple rounds of hybrid simulation calculations on the quantization calculation graph of the initial quantization model based on the hybrid execution engine and the initial quantization parameters of each operator in the quantization calculation graph, so as to obtain the final target quantization model.

[0103] It should be understood that the specific process of each module performing the above-mentioned steps has been described in detail in the above method embodiments, and will not be repeated here for the sake of brevity.

[0104] It should also be understood that the module division in the embodiments of this application is illustrative and only represents a logical functional division; in actual implementation, there may be other division methods. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0105] Figure 5 This is a schematic block diagram of an electronic terminal provided in an embodiment of this application. An electronic terminal includes a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the hardware-aware model quantization method described above. Figure 5 As shown, the electronic terminal 500 includes at least one processor 501, a memory 502, at least one network interface 503, and a user interface 505. The various components in the device are coupled together via a bus system 504. It is understood that the bus system 504 is used to implement communication between these components. In addition to a data bus, the bus system 504 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 5 The general will label all buses as bus systems.

[0106] The user interface 505 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0107] It is understood that memory 502 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0108] In this embodiment of the invention, the memory 502 is used to store various types of data to support the operation of the electronic terminal 500. Examples of this data include: any executable program for operation on the electronic terminal 500, such as the operating system 5021 and application programs 5022; the operating system 5021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 5022 may contain various applications, such as a media player, browser, etc., for implementing various application services. The hardware-aware model quantization method provided in this embodiment of the invention can be included in the application program 5022.

[0109] The methods disclosed in the above embodiments of the present invention can be applied to processor 501, or implemented by processor 501. Processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 501 or by instructions in the form of software. The processor 501 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 501 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 501 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0110] In an exemplary embodiment, the electronic terminal 500 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to execute the aforementioned method.

[0111] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the hardware-aware model quantization method as described above.

[0112] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, the computer implements the hardware-aware model quantization method as described above.

[0113] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0114] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0115] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0116] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0117] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0118] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0119] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs, DVDs), or semiconductor media (e.g., solid-state disks, SSDs, etc.).

[0120] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0121] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0122] In summary, this application provides a hardware-aware model quantization method, system, medium, product, and terminal. It constructs a hybrid execution mechanism between a high-level software framework and a low-level hardware behavior model. Only the core computational operators that need to be executed on the target hardware in the quantization computation graph of the initial quantization model are simulated with bit precision by the hardware behavior model. This simulates the actual computational characteristics of the target hardware during the quantization process, while other auxiliary, non-computationally intensive operators (such as data shape transformations) are retained and executed in the high-level software framework. This improves simulation verification efficiency, balances simulation accuracy and execution efficiency, and achieves the goal of more efficient operation of the quantized model on the target hardware with less accuracy loss. Therefore, this application effectively overcomes the various shortcomings of existing technologies and has high industrial applicability.

[0123] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A hardware-aware model quantization method, characterized in that, The method comprises the following steps: According to the preset high-level software framework, a preliminary low-precision quantization operation is performed on the pre-trained neural network model to obtain an initial quantization model, and a quantization calculation graph of the initial quantization model is constructed and initial quantization parameters of each operator in the quantization calculation graph are determined; According to the hardware resource status of the target hardware to which the initial quantization model needs to be deployed, a hardware behavior model is constructed, and a hybrid execution engine is constructed based on the hardware behavior model; Based on the hybrid execution engine and the initial quantization parameters of each operator in the quantization calculation graph, multiple rounds of iteration of hybrid simulation calculation are performed on the quantization calculation graph of the initial quantization model to obtain a final target quantization model; The type of the operator includes a core calculation operator; wherein, based on the hybrid execution engine and the initial quantization parameters of each operator in the quantization calculation graph, multiple rounds of iteration of hybrid simulation calculation are performed on the quantization calculation graph of the initial quantization model to obtain a final target quantization model, and the way includes: The hybrid execution engine traverses the quantization calculation graph, and different types of operators in the quantization calculation graph are distributed and scheduled according to a preset rule, so as to perform calculation of different types of operators and generate low-precision simulation calculation results corresponding to each core calculation operator; the type of the operator includes a general auxiliary operator; wherein, the hybrid execution engine traverses the quantization calculation graph, and different types of operators in the quantization calculation graph are distributed and scheduled according to a preset rule, so as to perform calculation of different types of operators; if the type of the operator is a general auxiliary operator, a preset high-level software framework is called to perform calculation; if the type of the operator is a core calculation operator, operator fusion operation and operator remapping operation are performed to call the corresponding hardware function module in the hardware behavior model to perform calculation; the operator fusion operation and the operator remapping operation include: if the type of the operator is a core calculation operator, operator interception operation is performed to generate a logical combination operator sequence; operator fusion operation is performed on the logical combination operator sequence to generate a composite operator; operator remapping operation is performed on the composite operator to construct a mapping relationship between the composite operator and the hardware function module in the hardware behavior model; According to the low-precision simulation calculation results corresponding to each core calculation operator, optimization operation of the quantization calculation graph is performed; Based on the quantization calculation graph after the optimization operation, multiple rounds of iteration of hybrid simulation calculation are performed to obtain a final target quantization model.

2. The hardware-aware model quantization method of claim 1, wherein, The way of calling the corresponding hardware function module in the hardware behavior model to perform calculation includes: According to the mapping relationship between the composite operator and the hardware function module in the hardware behavior model, the corresponding hardware function module in the hardware behavior model is called to perform calculation to generate low-precision simulation calculation results corresponding to the core calculation operator; The low-precision simulation calculation results corresponding to the core calculation operator are returned to the execution environment of the preset high-level software framework to perform calculation of subsequent operators.

3. The hardware-aware model quantization method of claim 1, wherein, The manner of performing the optimization operation of the quantization computation graph according to the low-precision simulation computation result corresponding to each core computation operator includes: According to the preset reference result corresponding to each core computation operator, error analysis is performed on the low-precision simulation computation result corresponding to each core computation operator, so as to lock a target operator to be optimized from each core computation operator; The initial quantization parameter of the target operator to be optimized is adjusted to perform the optimization operation of the quantization computation graph. 4.A hardware-aware model quantization system, comprising: Comprise: An initial quantization module is configured to perform preliminary low-precision quantization operation on a pre-trained neural network model according to a preset high-level software framework to obtain an initial quantization model, construct a quantization computation graph of the initial quantization model, and determine initial quantization parameters of each operator in the quantization computation graph; A hybrid execution engine construction module is configured to construct a hardware behavior model according to hardware resource conditions of a target hardware to which the initial quantization model needs to be deployed, and construct a hybrid execution engine based on the hardware behavior model; An optimization module is configured to perform multiple rounds of iteration of hybrid simulation computation on the quantization computation graph of the initial quantization model based on the hybrid execution engine and the initial quantization parameters of each operator in the quantization computation graph to obtain a final target quantization model; The type of the operator includes a core computation operator; wherein the manner of performing multiple rounds of iteration of hybrid simulation computation on the quantization computation graph of the initial quantization model based on the hybrid execution engine and the initial quantization parameters of each operator in the quantization computation graph to obtain a final target quantization model includes: The hybrid execution engine is used to traverse the quantization computation graph, and different types of operators in the quantization computation graph are distributed and scheduled according to a preset rule, so as to perform computation of different types of operators and generate low-precision simulation computation results corresponding to each core computation operator; the type of the operator includes a general auxiliary operator; wherein the manner of using the hybrid execution engine to traverse the quantization computation graph and distributing and scheduling different types of operators in the quantization computation graph according to a preset rule to perform computation of different types of operators includes: if the type of the operator is a general auxiliary operator, a preset high-level software framework is called to perform computation; if the type of the operator is a core computation operator, operator fusion operation and operator remapping operation are performed to call a corresponding hardware function module in the hardware behavior model to perform computation; the manner of performing operator fusion operation and operator remapping operation includes: if the type of the operator is a core computation operator, operator interception operation is performed to generate a logical combination operator sequence; operator fusion operation is performed on the logical combination operator sequence to generate a composite operator; operator remapping operation is performed on the composite operator to construct a mapping relationship between the composite operator and a hardware function module in the hardware behavior model; According to the low-precision simulation computation result corresponding to each core computation operator, the optimization operation of the quantization computation graph is performed; Based on the quantization computation graph after the optimization operation, multiple rounds of iteration of hybrid simulation computation are performed to obtain a final target quantization model.

5. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, which is executed by a processor, implements the hardware-aware model quantization method according to any one of claims 1 to 3.

6. A computer program product, characterised in that, The computer program product comprises computer program code, which, when executed on a computer, causes the computer to implement the hardware-aware model quantization method according to any one of claims 1 to 3.

7. An electronic terminal comprising a memory, a processor and a computer program stored on the memory, characterized in that, The processor executes the computer program to implement the hardware-aware model quantization method according to any one of claims 1 to 3.

Images