Quantization and inference method and device of mixed precision model, storage medium and equipment

Abstract

This application discloses a quantization and inference method, apparatus, storage medium, and device for mixed-precision models, belonging to the field of deep learning technology. In the offline stage, a mixed-precision configuration table is generated based on the floating-point model, including configuration information for the optimal static quantization bit width and hardware platform type selected for each layer. Weights are quantized according to the static quantization bit width to obtain the mixed-precision model. During the inference stage of deploying the mixed-precision model, the computational tasks of each layer are scheduled to a target hardware platform matching the hardware platform type within the target terminal. The distribution characteristics of activation values ​​output from historical layers are statistically analyzed in real time, and the quantization parameters for the current layer are dynamically calculated based on the statistical results. The quantization parameters are used to quantize the activation values ​​input to the current layer, and the quantized weights are used for inference on the target hardware platform. This application achieves globally optimal cross-platform inference latency, dynamically adjusts the quantization range, and maintains stable output precision.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a method, apparatus, storage medium and device for quantization and inference of mixed precision models. Background Technology

[0002] As the demand for deploying artificial intelligence models on general-purpose devices such as smartphones, tablets, and personal computers (PCs) grows, we must address the contradiction between the large size and high computational cost of these deep learning models and the limited resources of these devices. These resource constraints range from dedicated embedded neural network processing units (NPUs) / digital signal processing units (DSPs) in mobile phones to general-purpose graphics processing units (GPUs) / central processing units (CPUs) in PCs. Model quantization is key to resolving this contradiction. It converts floating-point numbers in the model into low-precision fixed-point numbers, significantly reducing model size and accelerating inference. The core of quantization is determining a scaling factor S (i.e., the quantization range) and a zero point Z to achieve the mapping from floating-point to fixed-point numbers.

[0003] The quantization methods in related technologies include static fixed-point quantization and static mixed-precision quantization.

[0004] Static fixed-point quantization: After the model is trained, this quantization method uses a small amount of data to statistically analyze the floating-point distribution range of the weights and activation values ​​of each layer of the model. Then, based on this fixed range, a unique scaling factor S and zero point Z are calculated and remain unchanged throughout the inference process. It usually adopts a uniform INT8 precision.

[0005] Disadvantages: The weights and activation values ​​of different layers in the model vary significantly. Using a static and uniform quantization range leads to several problems: for layers with a concentrated distribution of activation values, the quantization range is too large, resulting in significant loss of quantization accuracy; for layers with extreme activation values, the quantization range is stretched, causing the loss of most numerical details and resulting in extremely large quantization errors. Therefore, the static fixed-point quantization method cannot effectively address the differences in data distribution between layers and the incompleteness of the initial calibration set, leading to uncontrollable quantization errors and making it difficult to meet the accuracy requirements of general-purpose terminal deployments.

[0006] Static mixed precision quantization: This quantization method manually or pre-sets some important layers in the model to be set to higher precision (such as INT16 or FP16), and most layers to lower precision (such as INT8 or INT4).

[0007] Disadvantages: This allocation is static and lacks a general hardware basis. It does not consider the differences in the actual acceleration capabilities of different general-purpose terminal hardware architectures (such as mobile phone NPUs, PC GPUs, or high-performance CPUs) for different low precisions (such as INT4 and INT8). For example, a PC GPU may be highly efficient at accelerating INT8 but have poor support for INT4, while a mobile phone NPU may be the opposite. Static allocation cannot achieve optimal cross-platform hardware and software collaboration. More importantly, it does not consider the dynamic data inputs faced by the model during actual runtime (such as data with different resolutions and complexities). The static quantization range cannot adapt to the drift of runtime data distribution, resulting in large and unstable precision fluctuations in actual inference. Therefore, static mixed precision quantization is difficult to achieve high-performance and high-reliability deployment across platforms. Summary of the Invention

[0008] This application provides a quantization and inference method, apparatus, storage medium, and device for mixed-precision models, addressing the problem that static fixed-point quantization struggles to meet the accuracy loss requirements of general-purpose terminal deployments, and that static mixed-precision quantization results in large and unstable accuracy fluctuations during inference, hindering high-performance and high-reliability deployment across platforms. The technical solution is as follows: According to a first aspect of this application, a method for quantization and inference of a mixed-precision model is provided, the method comprising: During the offline phase, a mixed precision configuration table is generated based on the pre-trained floating-point model. The mixed precision configuration table includes configuration information for an optimal static quantization bit width and hardware platform type selected for each layer in the floating-point model. The mixed precision configuration table is generated based on the sensitivity scores of each layer in the floating-point model to different quantization precisions, as well as the inference speed of multiple hardware platforms in different target terminals for different bit width operators. The weights in the floating-point model are quantized according to the static quantization bit width in the configuration information to obtain a mixed-precision model. During the inference phase of deploying the mixed-precision model to the target terminal, the computational tasks of each layer are scheduled to the target hardware platform within the target terminal that matches the hardware platform type. The distribution characteristics of activation values ​​output from historical layers are statistically analyzed in real time, and the quantization parameters of the current layer are dynamically calculated based on the statistical results. The activation values ​​of the current layer input are quantized using the dynamically calculated quantization parameters, and the quantized activation values ​​are inferred using the quantized weights on the corresponding target hardware platform.

[0009] In one possible implementation, generating the mixed-precision configuration table based on the pre-trained floating-point model includes: A hierarchical quantization sensitivity analysis was performed on the floating-point model to obtain the sensitivity scores of each layer to different quantization precisions. On multiple target terminals, the actual inference speed of different hardware platforms for different quantization bit width operators was tested, and a multi-platform hardware acceleration cost model was established. Based on the sensitivity score and the hardware acceleration cost model, an optimal static quantization bit width and corresponding hardware platform type are determined for each layer of the floating-point model by optimizing the search algorithm. A mixed-precision configuration table is generated based on the static quantization bit width of each layer and the corresponding hardware platform type.

[0010] In one possible implementation, the hierarchical quantization sensitivity analysis of the floating-point model to obtain sensitivity scores for each layer to different quantization precisions includes: During the inference process of the floating-point model using the dataset, the accuracy loss of weights and activation values ​​under different quantization precisions is analyzed layer by layer. The sensitivity score of each layer to quantization accuracy is calculated based on the aforementioned accuracy loss.

[0011] In one possible implementation, the step of testing the actual inference speed of different hardware platforms for different quantization bit-width operators on multiple target terminals and establishing a multi-platform hardware acceleration cost model includes: For each target terminal, acquire multiple hardware platforms within the target terminal; For each hardware platform, speed tests were performed on operators with different quantization bit widths using the hardware platform to obtain the inference speed; Create a multi-platform hardware acceleration cost model based on hardware platform type, quantization bit width, and inference speed.

[0012] In one possible implementation, the step of determining an optimal static quantization bit width and corresponding hardware platform type for each layer of the floating-point model based on the sensitivity score and the hardware acceleration cost model through an optimized search algorithm includes: The optimization objective is to minimize the average inference latency across all target terminals, while ensuring that the overall quantization accuracy loss of the configured model does not exceed a preset threshold. The optimal combination of static quantization bit width and hardware platform type is searched for at each layer using a hardware-precision joint search algorithm.

[0013] In one possible implementation, the real-time statistical analysis of the distribution characteristics of activation values ​​output by the historical layer, and the dynamic calculation of the quantization parameters of the current layer based on the statistical results, include: The distribution characteristics of the activation values ​​output by the previous layer are statistically analyzed to obtain the statistical results; The statistical results are then merged with the historical statistical results stored in the sliding window. The quantization parameters of the current layer are dynamically calculated based on the fused statistical results.

[0014] In one possible implementation, the step of dynamically calculating the quantization parameters of the current layer based on the fused statistical results includes: Determine the quantization boundary value of the current layer based on the fused statistical results; Calculate the quantization scaling factor based on the quantization boundary value and the target quantization bit width; Calculate the quantization zero point based on the quantization boundary value and the quantization scaling factor; The calculated quantization scaling factor and the quantization zero point are determined as the final quantization parameters.

[0015] According to a second aspect of this application, a quantization and inference apparatus for a mixed-precision model is provided, the apparatus comprising: The configuration table generation module is used to generate a mixed precision configuration table based on the pre-trained floating-point model during the offline stage. The mixed precision configuration table includes configuration information for an optimal static quantization bit width and hardware platform type selected for each layer in the floating-point model. The mixed precision configuration table is generated based on the sensitivity scores of each layer in the floating-point model to different quantization precisions, as well as the inference speed of multiple hardware platforms in different target terminals for different bit width operators. The model quantization module is used to quantize the weights in the floating-point model according to the static quantization bit width in the configuration information to obtain a mixed-precision model. The task scheduling module is used to schedule the computational tasks of each layer to a target hardware platform that matches the hardware platform type within the target terminal during the inference phase of deploying the mixed-precision model to the target terminal. The parameter calculation module is used to statistically analyze the distribution characteristics of activation values ​​output from historical layers in real time, and dynamically calculate the quantization parameters of the current layer based on the statistical results. The task inference module is used to quantize the activation value of the current layer input using the dynamically calculated quantization parameters, and to perform inference calculation on the quantized activation value using the quantized weights on the corresponding target hardware platform.

[0016] According to a third aspect of this application, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the quantization and inference method of the mixed-precision model as described above.

[0017] According to a fourth aspect of this application, a computer device is provided, the computer device including the quantization and inference apparatus for the above-described mixed precision model.

[0018] The beneficial effects of the technical solution provided in this application include at least the following: In the offline phase, by selecting an optimal static quantization bit width and hardware platform type for each layer in the floating-point model, it ensures that while maintaining accuracy, it achieves hardware-software co-optimization based on the actual inference efficiency differences of various hardware platforms for different bit-width operators. This maximizes the utilization of the acceleration characteristics of different hardware platforms and achieves globally optimal cross-platform inference latency. In the inference phase, the quantization parameters are adjusted in real time based on the distribution characteristics of activation values ​​of each historical layer during inference. When the input features fluctuate significantly (such as from a quiet environment to a noisy environment in speech recognition), the quantization range is dynamically adjusted to suppress the accumulation of quantization errors caused by the shift in activation value distribution. This maintains stable model output accuracy in dynamic scenarios and avoids a sudden drop in accuracy due to mismatch in static quantization range. The mixed-precision configuration table and floating-point model generated offline are platform-independent. Combined with a general-purpose inference engine that can dynamically adjust the quantization range, this allows a single deployment package to be applicable to multiple hardware architectures, significantly reducing the cost of optimizing and maintaining multiple model versions for different terminals. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of a quantization and inference method for a mixed-precision model provided in one embodiment of this application; Figure 2 This is a flowchart of a quantization and inference method for a mixed-precision model provided in one embodiment of this application; Figure 3 This is a schematic diagram of the inference flow of a mixed-precision model provided in one embodiment of this application; Figure 4 This is a structural block diagram of a quantization and inference device for a mixed-precision model provided in one embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0022] like Figure 1The diagram illustrates a flowchart of a quantization and inference method for a mixed-precision model according to an embodiment of this application. This quantization and inference method for a mixed-precision model can be applied to computer devices. The quantization and inference method for a mixed-precision model may include: Step 101: In the offline stage, a mixed precision configuration table is generated based on the pre-trained floating-point model. The mixed precision configuration table includes configuration information for an optimal static quantization bit width and hardware platform type selected for each layer in the floating-point model. The mixed precision configuration table is generated based on the sensitivity scores of each layer in the floating-point model to different quantization precisions, as well as the inference speed of multiple hardware platforms in different target terminals for different bit width operators.

[0023] A floating-point model is a model whose weights are floating-point numbers.

[0024] When generating a mixed-precision configuration table, it is necessary to analyze the sensitivity score of each layer in the floating-point model to different quantization precisions, that is, to evaluate the tolerance of each layer in the floating-point model to quantization errors and avoid over-quantization of sensitive layers that could lead to precision collapse. On the other hand, it is necessary to quantize the inference speed of different types of hardware platforms for specific precision operators, so as to find the optimal configuration for each layer in the combination space of candidate precisions (INT4 / INT8 / FP16, etc.) and candidate hardware platforms (NPU / GPU / CPU, etc.).

[0025] Taking a floating-point model as the visual Transformer model as an example, suppose that during sensitivity analysis, it is found that the QKV projection layer of the attention module is sensitive to quantization (high sensitivity score), and it is recommended to configure it with higher precision (such as INT8); while some residual connection layers are not sensitive to quantization (low sensitivity score), and it is recommended to configure them with lower precision (such as INT4). Suppose that during hardware performance testing, it is shown that the mobile NPU significantly accelerates INT4 convolution, while the PC GPU is better at INT8 matrix operations. Finally, the joint optimization result is: the mixed precision configuration table specifies that the QKV projection layer uses INT8 and is scheduled to the GPU, and the residual connection layer uses INT4 and is scheduled to the NPU. While ensuring accuracy, the overall latency is reduced by 40%. The corresponding configuration information includes: QKV projection layer - INT8 - GPU, residual connection layer - INT4 - NPU.

[0026] Step 102: Quantize the weights in the floating-point model according to the static quantization bit width in the configuration information to obtain a mixed precision model.

[0027] Based on the static quantization bit width (such as INT8, INT4 or FP16) specified for each layer in the configuration information, static quantization processing is performed on the weights of the corresponding layers in the floating-point model, converting the high-precision floating-point weights into low-precision fixed-point or precision-preserving representations to generate a mixed-precision model.

[0028] Specifically, for each layer in the floating-point model, the static quantization bit width configured for that layer is read from the mixed-precision configuration table, the original floating-point weights of that layer are extracted, and the floating-point weights are quantized according to the static quantization bit width. Different layers may use different static quantization bit widths (e.g., some layers use INT8, some layers use INT4, and a few key layers retain FP16), thus forming a mixed-precision model.

[0029] After quantization, the original floating-point model is converted into a mixed-precision model composed of modules with different precision weights. This mixed-precision model, along with the mixed-precision configuration table and the inference engine integrating a dynamic range adjustment operator, can be packaged into a platform-independent deployment package. During mixed-precision inference, the inference engine can adjust the dynamic range of each quantization parameter in real time using the built-in dynamic range adjustment operator, thus achieving adaptive quantization to the input data distribution. Quantization parameters mainly include scaling factors and zeros. The scaling factor is the linear mapping ratio between the floating-point numerical field and the fixed-point integer field, determining the range of floating-point values ​​represented by each integer unit. Zeros are the corresponding positions of floating-point zeros in the quantized integer field, used to maintain numerical alignment before and after quantization.

[0030] During inference, the scaling factor is updated in real time based on the actual distribution of activation values ​​within the current batch or sliding window. This avoids quantization overflow due to insufficient range and resolution loss due to excessive range when the input data range changes. When the scaling factor is updated due to changes in the input distribution, the zero point is recalculated to maintain the accuracy of the quantization mapping and avoid systematic errors introduced by zero-point offset.

[0031] Step 103: During the stage of deploying the mixed-precision model to the target terminal for inference, the computational tasks of each layer are scheduled to the target hardware platform that matches the hardware platform type within the target terminal.

[0032] After obtaining the deployment package, you can deploy it to the target terminal, which can be a mobile terminal (such as a smartphone, tablet, etc.) or a desktop terminal (such as a PC).

[0033] After installing the deployment package, the available hardware platforms of the target terminal can be identified, including but not limited to NPU, GPU, CPU, and dedicated AI acceleration chips (such as DSP, TPU, etc.). Then, the hardware platform type specified for each layer is read from the mixed-precision configuration table, and the hardware platform type is matched with the actual available hardware platforms of the target terminal. Based on the matching results, the computing tasks of each layer are dynamically allocated to the matched target hardware platform.

[0034] For example, convolutional layers configured for INT4 are preferentially scheduled to NPUs that support the INT4 instruction set, while matrix layers configured for INT8 are scheduled to GPU Tensor Cores. If multiple layers recommend the same hardware platform type, dynamic task sorting and pipeline arrangement can be performed based on the current load, memory usage, and data locality of each hardware. If the specified hardware platform is unavailable or does not support the required precision (e.g., GPU does not support INT4), the layer is automatically downgraded to alternative hardware (e.g., CPU), and the performance impact is recorded.

[0035] Step 104: Statistically analyze the distribution characteristics of activation values ​​output by historical layers in real time, and dynamically calculate the quantization parameters of the current layer based on the statistical results.

[0036] During inference, the activation values ​​output by the historical layers (the first N layers, N≥1) are statistically analyzed in real time to obtain their dynamic distribution characteristics. Based on the statistical results, the quantization parameters required for the current layer are adaptively calculated. These quantization parameters include scaling factors and zero points.

[0037] Step 105: Quantize the activation value of the current layer input using dynamically calculated quantization parameters, and perform inference calculation on the quantized activation value using the quantized weights on the corresponding target hardware platform.

[0038] Specifically, the activation values ​​of the current layer input can be quantized using dynamically calculated quantization parameters, and inference calculations can be performed on the corresponding target hardware platform using the offline quantized weights and the dynamically quantized activation values.

[0039] In summary, the quantization and inference method for the mixed-precision model provided in this application, in the offline stage, selects an optimal static quantization bit width and hardware platform type for each layer in the floating-point model. This ensures that, while meeting accuracy requirements, it achieves hardware-software co-optimization based on the actual inference efficiency differences of various hardware platforms for operators with different bit widths. This maximizes the utilization of the acceleration characteristics of different hardware platforms and achieves globally optimal cross-platform inference latency. In the inference stage, the quantization parameters are adjusted in real time according to the distribution characteristics of activation values ​​of each historical layer during inference. When the input features fluctuate significantly (such as from a quiet environment to a noisy environment in speech recognition), the quantization range is dynamically adjusted to suppress the accumulation of quantization errors caused by the shift in activation value distribution. This maintains stable model output accuracy in dynamic scenarios and avoids a sudden drop in accuracy due to mismatch in static quantization range. The offline-generated mixed-precision configuration table and floating-point model are platform-independent. Combined with a general-purpose inference engine that can dynamically adjust the quantization range, this allows a single deployment package to be applicable to multiple hardware architectures, significantly reducing the cost of optimizing and maintaining multiple model versions for different terminals.

[0040] like Figure 2The diagram illustrates a flowchart of a quantization and inference method for a mixed-precision model according to an embodiment of this application. This method can be applied to computer devices. The quantization and inference method for the mixed-precision model may include: Step 201: In the offline stage, perform hierarchical quantization sensitivity analysis on the floating-point model to obtain the sensitivity scores of each layer to different quantization precisions.

[0041] Specifically, performing hierarchical quantization sensitivity analysis on the floating-point model to obtain the sensitivity scores of each layer to different quantization precipitates can include: analyzing the precision loss of weights and activation values ​​at different quantization precipitates layer by layer during the inference process of the floating-point model using a dataset; and calculating the sensitivity score of each layer to quantization precision based on the precision loss.

[0042] For each layer in the floating-point model, the original floating-point weights of that layer are obtained, and quantization simulations are performed according to different quantization precipitates to obtain multiple quantized weights. The activation values ​​corresponding to the input data are quantized to the corresponding quantization precipitate. For each quantization precipitate, the layer is used to perform inference calculations on the quantized weights and quantized activation values ​​to obtain the quantized simulation output. This quantized simulation output is compared with the original floating-point output to calculate the inter-layer output error for each quantization precipitate. Finally, based on each inter-layer output error, the sensitivity score of that layer to different quantization precipitates is calculated. A high sensitivity score indicates that the layer may be a key layer in the network (such as an attention mechanism, small convolutional kernel, classification head, etc.), and it is recommended to retain higher precision. A low sensitivity score indicates that the layer may be a large convolutional kernel, batch normalization layer, etc., and it is recommended to quantize to a lower precision.

[0043] Step 202: On multiple target terminals, test the actual inference speed of different hardware platforms for different quantization bit width operators, and establish a multi-platform hardware acceleration cost model.

[0044] Specifically, on multiple target terminals, the actual inference speed of different hardware platforms for different quantization bit width operators is tested, and a multi-platform hardware acceleration cost model is established. This can include: for each target terminal, obtaining multiple hardware platforms within the target terminal; for each hardware platform, using the hardware platform to perform speed tests on different quantization bit width operators to obtain the inference speed; and creating a multi-platform hardware acceleration cost model based on the hardware platform type, quantization bit width, and inference speed.

[0045] For each target terminal, the available hardware platforms can be identified, including but not limited to NPU, GPU, CPU, and dedicated AI acceleration chips (such as DSP, TPU, etc.). For each hardware platform, each operator is used to perform multiple inferences on the same input using different quantization bit widths, and parameters such as average latency and power consumption are recorded. A mapping relationship between hardware platform type, quantization bit width, and latency is established, and a multi-platform hardware acceleration cost model is constructed to predict the inference time of a certain layer when using a certain quantization bit width on a certain hardware platform.

[0046] Step 203: Based on the sensitivity score and hardware acceleration cost model, an optimal static quantization bit width and corresponding hardware platform type are determined for each layer of the floating-point model by optimizing the search algorithm.

[0047] Specifically, based on sensitivity scoring and hardware acceleration cost models, an optimal static quantization bit width and corresponding hardware platform type are determined for each layer of the floating-point model through an optimized search algorithm. This may include: configuring constraints that the overall quantization accuracy loss of the model does not exceed a preset threshold, and minimizing the average inference latency on all target terminals; and using a hardware-precision joint search algorithm to search for the optimal combination of static quantization bit width and hardware platform type for each layer.

[0048] The constraint is that the overall accuracy loss of the model does not exceed a set threshold (e.g., 3%), and the optimization objective is to minimize the average inference latency across all target terminals. Based on the above, heuristic search, evolutionary algorithms, or reinforcement learning methods can be used to select an optimal static baseline quantization bit width (e.g., INT4 or INT8) for each layer. Specifically, in each iteration, a candidate mixed-precision configuration is first generated (i.e., each layer is assigned a quantization bit width). Then, a sensitivity score is used to predict the accuracy loss under this configuration. Next, a multi-platform hardware acceleration cost model is used to predict the inference latency of this configuration on each hardware platform. Then, it is evaluated whether the accuracy constraint is satisfied, the objective function value (latency) is calculated, and finally, a globally optimal mixed-precision configuration table is output, which contains the optimal quantization bit width and hardware platform type combination for each layer.

[0049] Step 204: Generate a mixed precision configuration table based on the static quantization bit width of each layer and the corresponding hardware platform type.

[0050] The mixed precision configuration table includes configuration information for selecting an optimal static quantization bit width and hardware platform type for each layer in the floating-point model. The mixed precision configuration table is generated based on the sensitivity scores of each layer in the floating-point model to different quantization precisions, as well as the inference speed of multiple hardware platforms in different target terminals for operators with different bit widths.

[0051] Step 205: Quantize the weights in the floating-point model according to the static quantization bit width in the configuration information to obtain a mixed-precision model.

[0052] Based on the static quantization bit width (such as INT8, INT4 or FP16) specified for each layer in the configuration information, static quantization processing is performed on the weights of the corresponding layers in the floating-point model, converting the high-precision floating-point weights into low-precision fixed-point or precision-preserving representations to generate a mixed-precision model.

[0053] Step 206: During the stage of deploying the mixed-precision model to the target terminal for inference, the computational tasks of each layer are scheduled to the target hardware platform that matches the hardware platform type within the target terminal.

[0054] After the deployment package is installed, the available hardware platforms of the target terminal can be identified. Then, the mixed precision configuration table is read to specify the hardware platform type for each layer. The hardware platform type is matched with the actual available hardware platforms of the target terminal. Finally, based on the matching results, the computing tasks of each layer are dynamically allocated to the matched target hardware platform.

[0055] Step 207: Statistically analyze the distribution characteristics of activation values ​​output from historical layers in real time, and dynamically calculate the quantization parameters of the current layer based on the statistical results.

[0056] During inference, the activation values ​​output by the historical layers (the first N layers, N≥1) are statistically analyzed in real time to obtain their dynamic distribution characteristics. Based on the statistical results, the quantization parameters required for the current layer are adaptively calculated. These quantization parameters include scaling factors and zero points.

[0057] The process of real-time statistical analysis of the distribution characteristics of activation values ​​output from historical layers and dynamic calculation of quantization parameters for the current layer based on the statistical results may include: statistical analysis of the distribution characteristics of activation values ​​output from the previous layer to obtain statistical results; fusion of the statistical results with historical statistical results stored in a sliding window; and dynamic calculation of quantization parameters for the current layer based on the fused statistical results.

[0058] Specifically, the activation values ​​output by the previous layer can be statistically analyzed to obtain the results, including but not limited to: the minimum and maximum values, mean, standard deviation, and variance of the activation values ​​in the current batch. A window of size N is maintained within the sliding window, cyclically storing the statistical results collected in the most recent N inference iterations. The statistical results from the previous layer can be weighted and fused with the statistical results within the sliding window to obtain a fused statistical result. Finally, the quantization parameters of the current layer are dynamically calculated based on the fused statistical result. Different weights can be assigned to the data within the window based on their temporal proximity, with more recent data receiving higher weight.

[0059] In this embodiment, dynamically calculating the quantization parameters of the current layer based on the fused statistical results may include: determining the quantization boundary value of the current layer based on the fused statistical results; calculating the quantization scaling factor based on the quantization boundary value and the target quantization bit width; calculating the quantization zero point based on the quantization boundary value and the quantization scaling factor; and determining the calculated quantization scaling factor and quantization zero point as the final quantization parameters.

[0060] When calculating the quantization parameters, the effective range boundary for quantization is extracted from the fused distribution estimate. To avoid overflow caused by instantaneous fluctuations, the effective range boundary can be appropriately extended to obtain the quantization boundary value R. min and R max For symmetric quantization, the quantization scaling factor S = max(R) min R max ) / Q max , where Q max This represents the maximum positive value within the target integer range, such as Q corresponding to INT8. max The value is 127; for asymmetric quantization, the quantization scaling factor S = (R max -R min ) / (Q max -Q min ), Q max and Q min This represents the maximum and minimum values ​​of the target integer range, such as Q for INT8. max Q is 255. min The quantization zero is 0, Z = round[Q] min -(R) min / S)], and ensure that Z is within the range of integer representation, such as Z∈[-128, 127] for INT8.

[0061] Step 208: Quantize the activation value of the current layer input using dynamically calculated quantization parameters, and perform inference calculation on the quantized activation value using the quantized weights on the corresponding target hardware platform.

[0062] Specifically, the activation values ​​of the current layer input can be quantized using dynamically calculated quantization parameters, and inference calculations can be performed on the corresponding target hardware platform using the offline quantized weights and the dynamically quantized activation values.

[0063] In summary, the quantization and inference method for the mixed-precision model provided in this application, in the offline stage, selects an optimal static quantization bit width and hardware platform type for each layer in the floating-point model. This ensures that, while meeting accuracy requirements, it achieves hardware-software co-optimization based on the actual inference efficiency differences of various hardware platforms for operators with different bit widths. This maximizes the utilization of the acceleration characteristics of different hardware platforms and achieves globally optimal cross-platform inference latency. In the inference stage, the quantization parameters are adjusted in real time according to the distribution characteristics of activation values ​​of each historical layer during inference. When the input features fluctuate significantly (such as from a quiet environment to a noisy environment in speech recognition), the quantization range is dynamically adjusted to suppress the accumulation of quantization errors caused by the shift in activation value distribution. This maintains stable model output accuracy in dynamic scenarios and avoids a sudden drop in accuracy due to mismatch in static quantization range. The offline-generated mixed-precision configuration table and floating-point model are platform-independent. Combined with a general-purpose inference engine that can dynamically adjust the quantization range, this allows a single deployment package to be applicable to multiple hardware architectures, significantly reducing the cost of optimizing and maintaining multiple model versions for different terminals.

[0064] Figure 3 A schematic diagram of the reasoning process is shown, which includes the following stages: 1. Initialization (1) Obtain user-input voice or text data; (2) Perform preprocessing on the input data, such as standardization; (3) Identify the available hardware resources of the current terminal, including NPU / GPU / CPU; (4) Load the corresponding mixed precision configuration table according to the hardware recognition results and obtain the static quantization parameters of each layer.

[0065] 2. Core layer computation (Layer n, INT8) (1) Determine that Layer n uses INT8 precision according to the mixed precision configuration table, and schedule the task to the optimal target hardware platform according to the hardware identification result, such as scheduling to the INT8 Tensor Core of the GPU; (2) Use quantization parameters (S and Z) to quantize the input activation values ​​using INT8; (3) Perform INT8 precision Transformer calculations on the target hardware platform; (4) Generate the output result L_N_Out of Layer n and use it as the input of the next layer.

[0066] 3. Dynamic range adjustment (1) Obtain the activation value L_N_Out output by Layer n; (2) Monitor the distribution characteristics of activation values ​​(minimum, maximum, distribution pattern), and also pay attention to sudden user voice interruptions and environmental noise interference; (3) New quantization parameters (S and Z) are calculated based on real-time statistical calculations.

[0067] 4. Embedded layer computation (Layer n+1, INT4) (1) Obtain new quantization parameters (S and Z); (2) Use the new quantization parameters to perform INT4 quantization on the input activation values; (3) Schedule the INT4 computing task to the corresponding target hardware platform, such as the NPU INT4 core on a mobile phone; (4) Perform INT4 precision embedding layer calculations on the target hardware platform; (5) Generate the output result L_N+1_Out of Layer n+1 and use it as the input of the next layer.

[0068] like Figure 4 The diagram illustrates a structural block diagram of a quantization and inference apparatus for a mixed-precision model according to an embodiment of this application. This apparatus can be applied in a computer device and includes: The configuration table generation module 410 is used to generate a mixed precision configuration table based on the pre-trained floating-point model during the offline stage. The mixed precision configuration table includes configuration information for an optimal static quantization bit width and hardware platform type selected for each layer in the floating-point model. The mixed precision configuration table is generated based on the sensitivity scores of each layer in the floating-point model to different quantization precisions, and the inference speed of multiple hardware platforms in different target terminals for different bit width operators. The model quantization module 420 is used to quantize the weights in the floating-point model according to the static quantization bit width in the configuration information to obtain a mixed precision model; The task scheduling module 430 is used to schedule the computational tasks of each layer to a target hardware platform that matches the hardware platform type within the target terminal during the inference phase of deploying the mixed precision model to the target terminal. The parameter calculation module 440 is used to statistically analyze the distribution characteristics of activation values ​​output by historical layers in real time, and dynamically calculate the quantization parameters of the current layer based on the statistical results. The task inference module 450 is used to quantize the activation value of the current layer input using the dynamically calculated quantization parameters, and to perform inference calculation on the quantized activation value using the quantized weights on the corresponding target hardware platform.

[0069] In an optional embodiment, the configuration table generation module 410 is further configured to: A hierarchical quantization sensitivity analysis was performed on the floating-point model to obtain the sensitivity scores of each layer to different quantization precisions. On multiple target terminals, the actual inference speed of different hardware platforms for different quantization bit width operators was tested, and a multi-platform hardware acceleration cost model was established. Based on the sensitivity score and the hardware acceleration cost model, an optimal static quantization bit width and corresponding hardware platform type are determined for each layer of the floating-point model by an optimized search algorithm.

[0070] In an optional embodiment, the configuration table generation module 410 is further configured to: During the inference process of the floating-point model using the dataset, the accuracy loss of weights and activation values ​​under different quantization precisions is analyzed layer by layer. The sensitivity score of each layer to quantization accuracy is calculated based on the aforementioned accuracy loss.

[0071] In an optional embodiment, the configuration table generation module 410 is further configured to: For each target terminal, acquire multiple hardware platforms within the target terminal; For each hardware platform, speed tests were performed on operators with different quantization bit widths using the hardware platform to obtain the inference speed; Create a multi-platform hardware acceleration cost model based on hardware platform type, quantization bit width, and inference speed.

[0072] In an optional embodiment, the configuration table generation module 410 is further configured to: The optimization objective is to minimize the average inference latency across all target terminals, while ensuring that the overall quantization accuracy loss of the configured model does not exceed a preset threshold. The optimal combination of static quantization bit width and hardware platform type is searched for at each layer using a hardware-precision joint search algorithm.

[0073] In an optional embodiment, the parameter calculation module 440 is further configured to: The distribution characteristics of the activation values ​​output by the previous layer are statistically analyzed to obtain the statistical results; The statistical results are then merged with the historical statistical results stored in the sliding window. The quantization parameters of the current layer are dynamically calculated based on the fused statistical results.

[0074] In an optional embodiment, the parameter calculation module 440 is further configured to: Determine the quantization boundary value of the current layer based on the fused statistical results; Calculate the quantization scaling factor based on the quantization boundary value and the target quantization bit width; Calculate the quantization zero point based on the quantization boundary value and the quantization scaling factor; The calculated quantization scaling factor and the quantization zero point are determined as the final quantization parameters.

[0075] In summary, the quantization and inference apparatus for the mixed-precision model provided in this application, during the offline phase, selects an optimal static quantization bit width and hardware platform type for each layer in the floating-point model. This ensures that, while maintaining accuracy, it achieves hardware-software co-optimization based on the actual inference efficiency differences of various hardware platforms for operators with different bit widths. This maximizes the utilization of the acceleration characteristics of different hardware platforms and achieves globally optimal cross-platform inference latency. During the inference phase, the quantization parameters are adjusted in real time based on the distribution characteristics of activation values ​​of each historical layer during inference. When the input features fluctuate significantly (such as from a quiet environment to a noisy environment in speech recognition), the quantization range is dynamically adjusted to suppress the accumulation of quantization errors caused by the shift in activation value distribution. This maintains stable model output accuracy in dynamic scenarios and avoids a sudden drop in accuracy due to mismatch in static quantization range. The offline-generated mixed-precision configuration table and floating-point model are platform-independent. Combined with a general-purpose inference engine that can dynamically adjust the quantization range, this allows a single deployment package to be applied to multiple hardware architectures, significantly reducing the cost of optimizing and maintaining multiple model versions for different terminals.

[0076] One embodiment of this application provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the quantization and inference method for a mixed-precision model as described above.

[0077] One embodiment of this application provides a computer device, which includes the quantization and inference apparatus for the above-described arbitrary mixed-precision model.

[0078] It should be noted that the quantization and inference device for mixed-precision models provided in the above embodiments is only illustrated by the division of the functional modules described above. In practical applications, the functions described above can be assigned to different functional modules as needed, that is, the internal structure of the quantization and inference device for mixed-precision models can be divided into different functional modules to complete all or part of the functions described above. In addition, the quantization and inference device for mixed-precision models provided in the above embodiments and the quantization and inference method embodiments for mixed-precision models belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0079] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0080] The above description is not intended to limit the embodiments of this application. Any adjustments, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of the embodiments of this application.

Claims

1. A quantization and inference method for a mixed-precision model, characterized in that, The method includes: During the offline phase, a mixed precision configuration table is generated based on the pre-trained floating-point model. The mixed precision configuration table includes configuration information for an optimal static quantization bit width and hardware platform type selected for each layer in the floating-point model. The mixed precision configuration table is generated based on the sensitivity scores of each layer in the floating-point model to different quantization precisions, as well as the inference speed of multiple hardware platforms in different target terminals for different bit width operators. The weights in the floating-point model are quantized according to the static quantization bit width in the configuration information to obtain a mixed-precision model. During the inference phase of deploying the mixed-precision model to the target terminal, the computational tasks of each layer are scheduled to the target hardware platform within the target terminal that matches the hardware platform type. The distribution characteristics of activation values ​​output from historical layers are statistically analyzed in real time, and the quantization parameters of the current layer are dynamically calculated based on the statistical results. The activation values ​​of the current layer input are quantized using the dynamically calculated quantization parameters, and the quantized activation values ​​are inferred using the quantized weights on the corresponding target hardware platform.

2. The quantization and inference method for the mixed-precision model according to claim 1, characterized in that, The step of generating a mixed-precision configuration table based on a pre-trained floating-point model includes: A hierarchical quantization sensitivity analysis was performed on the floating-point model to obtain the sensitivity scores of each layer to different quantization precisions. On multiple target terminals, the actual inference speed of different hardware platforms for different quantization bit width operators was tested, and a multi-platform hardware acceleration cost model was established. Based on the sensitivity score and the hardware acceleration cost model, an optimal static quantization bit width and corresponding hardware platform type are determined for each layer of the floating-point model by optimizing the search algorithm. A mixed-precision configuration table is generated based on the static quantization bit width of each layer and the corresponding hardware platform type.

3. The quantization and inference method for the mixed-precision model according to claim 2, characterized in that, The hierarchical quantization sensitivity analysis of the floating-point model yields sensitivity scores for each layer to different quantization precisions, including: During the inference process of the floating-point model using the dataset, the accuracy loss of weights and activation values ​​under different quantization precisions is analyzed layer by layer. The sensitivity score of each layer to quantization accuracy is calculated based on the aforementioned accuracy loss.

4. The quantization and inference method for the mixed-precision model according to claim 2, characterized in that, The process involves testing the actual inference speed of different hardware platforms for different quantization bit-width operators on multiple target terminals, and establishing a multi-platform hardware acceleration cost model, including: For each target terminal, acquire multiple hardware platforms within the target terminal; For each hardware platform, speed tests were performed on operators with different quantization bit widths using the hardware platform to obtain the inference speed; Create a multi-platform hardware acceleration cost model based on hardware platform type, quantization bit width, and inference speed.

5. The quantization and inference method for the mixed-precision model according to claim 2, characterized in that, Based on the sensitivity score and the hardware acceleration cost model, an optimized search algorithm is used to determine an optimal static quantization bit width and corresponding hardware platform type for each layer of the floating-point model, including: The optimization objective is to minimize the average inference latency across all target terminals, while ensuring that the overall quantization accuracy loss of the configured model does not exceed a preset threshold. The optimal combination of static quantization bit width and hardware platform type is searched for at each layer using a hardware-precision joint search algorithm.

6. The quantization and inference method for a mixed-precision model according to any one of claims 1 to 5, characterized in that, The real-time statistical analysis of the distribution characteristics of activation values ​​output by the historical layer, and the dynamic calculation of quantization parameters for the current layer based on the statistical results, include: The distribution characteristics of the activation values ​​output by the previous layer are statistically analyzed to obtain the statistical results; The statistical results are then merged with the historical statistical results stored in the sliding window. The quantization parameters of the current layer are dynamically calculated based on the fused statistical results.

7. The quantization and inference method for the mixed-precision model according to claim 6, characterized in that, The dynamic calculation of the quantization parameters of the current layer based on the fused statistical results includes: Determine the quantization boundary value of the current layer based on the fused statistical results; Calculate the quantization scaling factor based on the quantization boundary value and the target quantization bit width; Calculate the quantization zero point based on the quantization boundary value and the quantization scaling factor; The calculated quantization scaling factor and the quantization zero point are determined as the final quantization parameters.

8. A quantization and inference device for a mixed-precision model, characterized in that, The device includes: The configuration table generation module is used to generate a mixed precision configuration table based on the pre-trained floating-point model during the offline stage. The mixed precision configuration table includes configuration information for an optimal static quantization bit width and hardware platform type selected for each layer in the floating-point model. The mixed precision configuration table is generated based on the sensitivity scores of each layer in the floating-point model to different quantization precisions, as well as the inference speed of multiple hardware platforms in different target terminals for different bit width operators. The model quantization module is used to quantize the weights in the floating-point model according to the static quantization bit width in the configuration information to obtain a mixed-precision model. The task scheduling module is used to schedule the computational tasks of each layer to a target hardware platform that matches the hardware platform type within the target terminal during the inference phase of deploying the mixed-precision model to the target terminal. The parameter calculation module is used to statistically analyze the distribution characteristics of activation values ​​output from historical layers in real time, and dynamically calculate the quantization parameters of the current layer based on the statistical results. The task inference module is used to quantize the activation value of the current layer input using the dynamically calculated quantization parameters, and to perform inference calculation on the quantized activation value using the quantized weights on the corresponding target hardware platform.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to implement the quantization and inference method for a mixed-precision model as described in any one of claims 1 to 7.

10. A computer device, characterized in that, The computer device includes: the quantization and inference apparatus for the mixed-precision model as described in claim 8.

Images