A deep learning model quantization method, electronic device, and computer program product

Abstract

The application relates to the technical field of machine learning, and discloses a deep learning model quantization method, an electronic device and a computer program product. The method comprises the following steps: performing post-training quantization processing on a deep learning model to be quantized by using a constructed original calibration set; if the deep learning model after the post-training quantization processing does not meet the end-side accuracy requirement, using quantization parameters of each layer activation value obtained by the post-training quantization processing as initialization values of corresponding quantizers in quantization-aware training, and performing quantization-aware training processing on the deep learning model. The method uses valuable information in the post-training quantization process to optimize the effect of the quantization-aware training, and can further improve the inference performance of the quantized deep learning model.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and in particular to a method for quantizing deep learning models, electronic devices, and computer program products. Background Technology

[0002] Quantization of deep learning models refers to the technique of compressing the model, improving inference speed and reducing energy consumption by reducing the numerical precision of model parameters. Its core idea is to convert the weights and activation values, which were originally represented by high-precision floating-point numbers (such as 32-bit floating-point numbers FP32), into low-precision integers (such as 8-bit integers INT8) for storage and calculation while maintaining model performance as much as possible.

[0003] Currently, two common model quantization methods are Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT). PTQ eliminates the need to retrain the model, determining quantization parameters by statistically analyzing the distribution of weights and activation values ​​across layers using a calibration dataset. It offers advantages such as simplicity and rapid deployment. QAT, on the other hand, simulates the quantization process during model training, allowing model parameters to adapt to quantization errors. This typically results in higher model accuracy but requires additional training time and computational resources.

[0004] However, in practical industrial deployments, regardless of whether post-training quantization or quantization-aware training is used, the improvement in inference performance of quantized deep learning models remains relatively limited. Summary of the Invention

[0005] In view of this, embodiments of this application provide a deep learning model quantization method, electronic device, and computer program product, which optimize the effect of quantization perception training by utilizing valuable information in the post-training quantization process, thereby further improving the inference performance of the quantized deep learning model.

[0006] The first aspect of this application provides a method for quantizing deep learning models, including:

[0007] The deep learning model to be quantized is trained and then quantized using the constructed original calibration set. If the deep learning model after training and quantization does not meet the edge accuracy requirements, the quantization parameters of the activation values ​​of each layer obtained after training and quantization will be used as the initialization values ​​of the corresponding quantizers in quantization-aware training, and quantization-aware training will be performed on the deep learning model.

[0008] The technical solution of this application uses a constructed original calibration set to perform post-training quantization processing on the deep learning model to be quantized. If the deep learning model after post-training quantization does not yet meet the edge accuracy requirements, the inheritance mechanism of post-training quantization parameters is activated. The quantization parameters of the activation values ​​of each layer obtained after post-training quantization are used as the initialization values ​​of the corresponding quantizers in quantization-aware training, and quantization-aware training processing continues on the deep learning model. Since the quantization parameters obtained after post-training quantization are closer to the optimal parameter values ​​than randomly initialized values, the above process, by using the quantization parameters of the activation values ​​obtained after post-training quantization as the initialization values ​​of the corresponding quantizers in quantization-aware training, can provide a high-quality initial value for quantization parameters for the quantization-aware training process, thereby accelerating convergence and improving the final accuracy during model training. In summary, the above process is equivalent to using valuable information from the post-training quantization process to optimize the effect of quantization-aware training, which can further improve the inference performance of the quantized deep learning model.

[0009] In one implementation of this application, before performing post-quantization processing on the deep learning model to be quantized using the constructed original calibration set, the method further includes: Check whether the deep learning model meets the performance requirements of the edge device; If the deep learning model does not meet the performance requirements of the edge device, then the deep learning model is trained with sparse perception, and the structure of the deep learning model is modified according to a predefined rule base until the deep learning model meets the performance requirements of the edge device.

[0010] In one implementation of this application, detecting whether a deep learning model meets the edge performance requirements includes: Deploy and measure the benchmark performance metrics of the deep learning model on the target edge device; Use an implicit quantization engine to probe the upper limit of the performance metrics of a deep learning model after quantization; By comparing the baseline performance metrics with the upper limit of the performance metrics, it can be determined whether the deep learning model meets the performance requirements of the edge device.

[0011] In one implementation of this application, the deep learning model undergoes quantization-aware training processing, including: Based on data perturbation analysis, identify the sensitive and non-sensitive layers of the deep learning model; The sensitive and non-sensitive layers undergo mixed-precision quantization training.

[0012] In one implementation of this application, based on data perturbation analysis, the sensitive and non-sensitive layers of the deep learning model are identified, including: Noise used to simulate quantization error is injected into the original calibration set to obtain a noise calibration set; The deep learning model is used to perform forward inference on the original calibration set and the noisy calibration set respectively, so as to obtain the original activation value and the noisy activation value of each network layer of the deep learning model respectively. Based on the original activation value and noisy activation value of each network layer, the sensitivity score of each network layer is determined. Sensitive and insensitive layers are determined from each network layer based on their respective sensitivity scores.

[0013] In one implementation of this application, the sensitivity score of each network layer is determined based on its original activation value and noisy activation value, including: For each network layer, the distribution change rate of the output activation value of the network layer is determined by comparing the difference between the original activation value and the noisy activation value. The sensitivity score of the network layer is then determined based on the distribution change rate. The distribution change rate is positively correlated with the difference in activation value, and the sensitivity score is positively correlated with the distribution change rate.

[0014] In one implementation of this application, after performing quantization-aware training on the deep learning model, the method further includes: If the deep learning model after quantization-aware training does not meet the edge accuracy requirements, then the non-sensitive layers with the highest sensitivity scores are set as sensitive layers, and the deep learning model is retrained with mixed-precision quantization-aware training.

[0015] In one implementation of this application, if the deep learning model after training and quantization does not meet the edge accuracy requirements, the method further includes the following before performing quantization-aware training on the deep learning model: A lightweight generative adversarial network is used to perform distribution-aligned data augmentation on the original calibration set with a preset test set distribution as the target, resulting in an augmented calibration set. The deep learning model is then retrained and quantized using the augmented calibration set.

[0016] A second aspect of this application provides a deep learning model quantization apparatus, comprising: The post-training quantization module is used to perform post-training quantization on the deep learning model to be quantized using the constructed original calibration set. The quantization-aware training module is used to perform quantization-aware training on the deep learning model if the quantized deep learning model does not meet the edge accuracy requirements. In this case, the quantization parameters of the activation values ​​of each layer obtained after quantization are used as the initialization values ​​of the corresponding quantizer in the quantization-aware training.

[0017] A third aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the deep learning model quantization method provided in the first aspect of this application.

[0018] A fourth aspect of this application provides a computer program product that, when run on an electronic device, causes the electronic device to execute the deep learning model quantization method provided in the first aspect of this application.

[0019] A fifth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the deep learning model quantization method provided in the first aspect of this application.

[0020] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0021] Figure 1 This is a flowchart of a deep learning model quantization method provided in an embodiment of this application; Figure 2 This is a schematic diagram of an operation process for precision-oriented post-training quantization and intelligent evaluation provided in an embodiment of this application; Figure 3 This is a schematic diagram of the operation process of intelligent initialization mixed precision quantization perception training provided in an embodiment of this application; Figure 4 This is a schematic diagram of the operation process of the deep learning model quantization method provided in this application embodiment in a practical application scenario; Figure 5 This is a schematic diagram of the structure of a deep learning model quantization device provided in an embodiment of this application; Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail. Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0023] In the deployment of deep learning models in industry, model quantization is a key technical means to improve the inference speed and reduce power consumption of deep learning models on the edge. Currently, commonly used model quantization methods mainly include post-training quantization and quantization-aware training. However, the accuracy loss of post-training quantization is unpredictable, and the quantization effect of quantization-aware training often fails to meet expectations. Ultimately, regardless of which model quantization method is used, it is impossible to achieve a satisfactory improvement in model inference performance.

[0024] To address the aforementioned technical problems, embodiments of this application provide a deep learning model quantization method, an electronic device, and a computer program product. By utilizing valuable information from the post-training quantization process to optimize the effectiveness of quantization-aware training, the inference performance of the quantized deep learning model can be further improved. For more specific technical implementation details of the embodiments of this application, please refer to the various method embodiments described below.

[0025] It should be understood that the implementing entity of the various method embodiments proposed in this application can be various types of electronic devices, such as mobile phones, tablets, desktop computers, wearable devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), large-screen TVs, etc. The embodiments of this application do not impose any restrictions on the specific type of electronic device.

[0026] Please see Figure 1 This application illustrates a deep learning model quantization method provided in an embodiment, comprising: 101. Use the constructed original calibration set to perform post-quantization processing on the deep learning model to be quantized after training; A deep learning model to be quantized refers to any deep learning model that has not yet been quantized and uses high-precision floating-point numbers, such as an FP32 model. The technical solution of this application adopts a collaborative working mode of initial calibration after training and fine-tuning through quantization-aware training. That is, the deep learning model is first trained and then quantized. If the accuracy of the quantized deep learning model is found to be insufficient, quantization-aware training is continued. This solves the problem of the fragmented process between the two quantization methods and effectively improves the efficiency and effect of model quantization. During the post-training quantization process of the deep learning model, the dynamic range of activation values ​​of each layer can be statistically analyzed using the constructed original calibration set, thereby determining the optimal quantization parameters for each layer's activation values. The quantization parameters mainly include the scaling factor S and the zero-point offset Z.

[0027] Furthermore, in existing quantization decision-making processes, technical personnel often need to rely on experience to repeatedly try and fail, attempting various solutions such as post-training quantization, lightweight model modification, and quantization-aware training. This process is time-consuming, labor-intensive, and the results are difficult to guarantee. To address this issue, this application proposes an automated analysis technique that can solve quantization performance bottlenecks. This technique can accurately locate model structural defects that cause performance problems. Specific technical implementation details are provided below.

[0028] In one implementation of this application, before performing post-quantization processing on the deep learning model to be quantized using the constructed original calibration set, the method further includes: (1) Detect whether the deep learning model meets the performance requirements of the edge device; (2) If the deep learning model does not meet the performance requirements of the edge, the deep learning model is trained with sparse perception, and the structure of the deep learning model is modified according to the predefined rule base until the deep learning model meets the performance requirements of the edge.

[0029] Before quantizing a deep learning model after training, it's essential to check if it meets the performance requirements of the target device. This primarily involves assessing whether quantizing the model on the existing hierarchical structure will result in the expected performance improvement when deployed on the target device. If the model doesn't meet these requirements, subsequent quantization will not yield the desired performance gains. Therefore, a lightweighting and reconstruction phase is necessary. This phase involves hardware capability probing and structured sparsity application, performing sparsity-aware training on the deep learning model. Engines supporting sparse inference, such as TensorRT, can be used to quickly deploy and evaluate the inference speed and power consumption of the sparse but unquantized model on the target device. This approach aims to quickly and cost-effectively validate the potential of existing model structures to benefit from hardware sparse acceleration. For example, we can first check whether the hardware (e.g., GPU) of the target edge device supports 2:4 structured sparse acceleration. If it does, we automatically apply a 2:4 structured sparse mask to all weights of the model. That is, we retain the two with the largest absolute values ​​out of every four consecutive weight elements, and set the rest to zero. This generates a sparse model copy. We then test the inference performance gain of this model copy and record the 2:4 structured sparse mask. This sparse mask will be used to implement sparsity-aware training to ensure that quantization and sparsity are optimized together, maximizing model accuracy. Another aspect of this step involves modifying the structure of the deep learning model according to a predefined rule base. Specifically, we can perform computationally intensive modifications and quantization-friendly optimizations, such as replacing complex activation functions with simplified versions, or changing a serial structure to a parallel structure to facilitate operator fusion, etc. This completes the modification of the model structure. The modified model is then re-evaluated to check if it meets the edge performance requirements. If it still does not, it re-enters the model lightweighting and reconstruction stage. This cycle repeats until the modified deep learning model meets the edge performance requirements. If the deep learning model meets the performance requirements of the edge device, it means that after subsequent model quantization, the deep learning model can achieve the expected performance improvement, and at this time, the subsequent model quantization operation can be started.

[0030] In one implementation of this application, detecting whether a deep learning model meets the edge performance requirements includes: (1) Deploy and measure the benchmark performance metrics of the deep learning model on the target edge device; (2) Use an implicit quantization engine to probe the upper limit of the performance metrics of deep learning models after quantization; (3) By comparing the benchmark performance index with the upper limit of the performance index, determine whether the deep learning model meets the edge performance requirements.

[0031] After obtaining the deep learning model to be quantized, its baseline accuracy and performance metrics can be deployed and measured on the target edge device. Baseline accuracy is primarily used to evaluate whether the quantized model meets edge accuracy requirements during subsequent quantization, while baseline performance metrics can be used to evaluate whether the unquantized model meets edge performance requirements. Specifically, after deploying the deep learning model to the target edge device, quantization engines with FP32 and FP16 accuracy can be built to measure baseline performance metrics such as inference speed and power consumption, serving as the performance basis for subsequent structural optimization. Additionally, implicit quantization engines such as TensorRT can be used to probe the optimal performance achievable under ideal conditions after deep learning model quantization, serving as the upper limit of performance metrics. For example, the TensorRT implicit quantization engine can be used to automatically quantize an FP32 model into an INT8 model and evaluate the upper limit of the INT8 model's performance metrics, providing key reference metrics for subsequent decisions. Next, a performance-oriented quantization decision-making process is initiated. The upper limit of the implicit quantization performance metrics is compared with the benchmark performance metrics. Based on the comparison results, it is determined whether the quantized deep learning model, when deployed and run on the target edge device, can achieve the expected performance improvement. If so, the deep learning model is deemed to meet the edge performance requirements; otherwise, it is deemed not to meet the edge performance requirements. For example, edge model performance typically includes metrics such as inference latency and GPU load rate. If the inference latency of the implicitly quantized INT8 model is reduced by 10% to 40% or more compared to the FP16 model, and the GPU load rate of the implicitly quantized INT8 model is reduced by 5% to 40% or more compared to the FP16 model, then the performance gain from quantization is considered significant, and the edge performance requirements are met. Conversely, if the reduction in inference latency and GPU load rate of the implicitly quantized INT8 model compared to the FP16 model is small, it indicates that the performance gain from quantization is limited, and the edge performance requirements are not met.

[0032] After determining that the deep learning model meets the edge performance requirements, the model quantization process begins. The constructed original calibration set is used to perform post-training quantization on the deep learning model, thereby obtaining the optimal quantization parameters for each layer's activation values. After completing the post-training quantization, it is checked whether the current deep learning model meets the edge accuracy requirements. Here, the baseline accuracy described earlier can be obtained. The current accuracy of the quantized deep learning model is compared with this baseline accuracy. If the decrease in current accuracy compared to the baseline accuracy is within an acceptable range, the edge accuracy requirements are considered met; otherwise, they are considered not to meet the edge accuracy requirements.

[0033] If the post-quantized deep learning model meets the edge accuracy requirements, the inference engine obtained after post-quantization can be deployed to the target edge device for actual performance and accuracy evaluation, thus completing the model quantization task. If the post-quantized deep learning model does not meet the edge accuracy requirements, two different processing methods can be adopted. The first is a calibration set optimization mechanism based on generative adversarial networks, and the second is quantization-aware training based on the quantization parameters obtained after post-quantization. These two processing methods will be introduced in turn below.

[0034] In one implementation of this application, if the deep learning model after training and quantization does not meet the edge accuracy requirements, the method further includes the following before performing quantization-aware training on the deep learning model: A lightweight generative adversarial network is used to perform distribution-aligned data augmentation on the original calibration set with a preset test set distribution as the target, resulting in an augmented calibration set. The deep learning model is then retrained and quantized using the augmented calibration set.

[0035] For the calibration set optimization mechanism based on generative adversarial networks (GANs), when the accuracy of the model obtained after post-training quantization is deemed insufficient, a calibration set enhancement process is automatically initiated. This process uses a lightweight GAN to perform distribution-aligned data augmentation on the original calibration set, targeting a pre-defined test set distribution, to generate more representative synthetic calibration data, thus obtaining an enhanced calibration set. Subsequently, the deep learning model is retrained and quantized using the enhanced calibration set, and the re-quantized deep learning model is checked to see if it meets the edge accuracy requirements. If it meets the edge accuracy requirements, the inference engine is output and the quantization task ends. If it still does not meet the edge accuracy requirements, the model undergoes quantization-aware training. Specifically, for the distribution gap analysis between the test set and the calibration set, an intermediate layer inherent in the deep learning model can be used as a feature extractor. Methods such as KL divergence are used to calculate the mean, variance, and distribution shape of the two datasets in the feature space. A lightweight, conditionalized feature-enhancing generative adversarial network (GAN) is pre-trained. The generator's input includes random noise *z* and a distribution conditional vector *c* from the test set. The distribution conditional vector *c* contains various statistical information extracted from the test set features, such as the global feature mean and variance. The GAN's loss function aims to characterize the differences in statistical moments (e.g., mean, standard deviation) between the generated data features and the test set features. Subsequently, the trained GAN is used for targeted data synthesis to obtain an enhanced calibration set. For example, the GAN can generate corresponding image data, which may not be visually realistic but is highly aligned with the test set in the model's feature space. Because the enhanced calibration set encompasses feature distributions closer to the test set, the dynamic range of activation values ​​and the quantization scaling factor obtained during retraining and post-quantization calibration using the enhanced calibration set will be more accurate, thus effectively improving the post-training quantization performance.

[0036] As an example, Figure 2 This is a schematic diagram illustrating the operation flow of accuracy-oriented post-training quantization and intelligent evaluation provided in an embodiment of this application. Figure 2 First, the deep learning model is trained and quantized using the original calibration set. Then, the current deep learning model is evaluated to see if it meets the edge accuracy requirements. If it does not meet the edge accuracy requirements, a calibration set optimization mechanism based on generative adversarial networks is initiated. The original calibration set is augmented using the test set and the trained generative adversarial network to obtain an augmented calibration set. After that, the deep learning model is trained and quantized using the augmented calibration set. Then, the current deep learning model is evaluated again to see if it meets the edge accuracy requirements. If it does, the quantization task ends; otherwise, the subsequent quantization-aware training process begins.

[0037] 102. If the deep learning model after training and quantization does not meet the edge accuracy requirements, the quantization parameters of the activation values ​​of each layer obtained after training and quantization are used as the initialization values ​​of the corresponding quantizers in quantization-aware training, and quantization-aware training is performed on the deep learning model.

[0038] When performing quantization-aware training based on quantization parameters obtained after post-training quantization, the training does not start from scratch. Instead, it introduces a mechanism for inheriting post-training quantization parameters. The quantization parameters of the activation values ​​obtained from post-training quantization are used as the initial values ​​for the corresponding quantizers in each layer during quantization-aware training. Since the quantization parameters obtained after post-training quantization are closer to the optimal parameter values ​​than randomly initialized values, they provide a high-quality initial value for the quantization-aware training process, thereby accelerating convergence and improving final accuracy during model training. In practice, a post-training quantization calibration step can be set during quantization-aware training. After the calibration data undergoes one forward propagation, the activation value quantization parameters can be obtained using statistical methods and embedded in the quantization model weights, which will then be used as the initial values ​​for the quantization parameters during subsequent quantization model training.

[0039] After completing the quantization-aware training, it is determined whether the current deep learning model meets the edge-side accuracy requirements. If it does, the inference engine obtained from the quantization-aware training can be deployed to the target edge device for actual performance and accuracy evaluation, thus concluding the model quantization task. If it does not meet the requirements, quantization failure information can be recorded, triggering further model optimization actions, such as improving the model structure again, performing sensitivity layer analysis, and adjusting training parameters. This forms a continuously iterative, automatically correcting closed-loop optimization system until a deep learning model that simultaneously meets the performance and accuracy requirements is obtained. Furthermore, all optimization processes and results can be recorded in an experience knowledge base for continuous optimization of thresholds and strategy selection at each decision node. For example, the operation logs and conclusions of each process node in the model optimization process can be compiled and archived to facilitate analysis and decision-making by relevant technical personnel regarding the degree of model quantization.

[0040] Considering that deep learning models typically have a large number of network layers, performing low-precision INT8 quantization on all layers may lead to excessively large accumulated errors in some sensitive layers; while performing high-precision FP16 quantization on all layers may not maximize performance gains. To balance this issue, this application proposes a hybrid-precision quantization-aware training method. It first identifies the sensitive and insensitive layers in the deep learning model, and then performs quantization processing on the sensitive and insensitive layers with different precisions respectively. This effectively balances accumulated error and quantization performance gains. The specific technical implementation details are described below.

[0041] In one implementation of this application, the deep learning model undergoes quantization-aware training processing, including: (1) Based on data perturbation analysis, identify the sensitive and non-sensitive layers of the deep learning model; (2) Perform mixed-precision quantitative perception training on sensitive and non-sensitive layers.

[0042] In identifying sensitive and insensitive layers, this application employs an automated sensitive layer identification method based on data perturbation analysis. This method injects noise into the calibration set and analyzes the changes in the activation value distribution of each network layer's output. This allows for precise quantization of the sensitivity of each network layer to quantization errors, thereby distinguishing between sensitive and insensitive layers. After determining the sensitive and insensitive layers of the deep learning model, a mixed-precision quantization configuration is automatically generated, and mixed-precision quantization-aware training is performed on both the sensitive and insensitive layers. For example, each sensitive layer can be quantized to FP16 precision, while each insensitive layer can be quantized to INT8 precision. This avoids excessive cumulative error in sensitive layers while ensuring sufficient quantization performance gains. Furthermore, if the model accuracy after quantization-aware training is still insufficient, the learning rate strategy can be automatically adjusted or the BN layer parameters can be frozen, and quantization-aware training can be performed again.

[0043] In one implementation of this application, based on data perturbation analysis, the sensitive and non-sensitive layers of the deep learning model are identified, including: (1) Inject noise to the original calibration set to simulate quantization error to obtain a noise calibration set; (2) Use a deep learning model to perform forward inference on the original calibration set and the noisy calibration set respectively, so as to obtain the original activation value and the noisy activation value of each network layer of the deep learning model respectively; (3) Determine the sensitivity score of each network layer based on its original activation value and noise activation value; (4) Based on the sensitivity scores of each network layer, determine the sensitive and non-sensitive layers from each network layer.

[0044] First, noise is injected into the original calibration set to simulate quantization errors, resulting in a noisy calibration set. For example, small amounts of noise can be injected into each calibration image in the original calibration set. This noise is not random and can be used to simulate rounding errors caused by INT8 precision quantization. The calibration images after noise injection constitute the noisy calibration set. Then, an unquantized deep learning model is used to perform forward inference on both the original and noisy calibration sets, simultaneously collecting two sets of activation values ​​through the outputs of all network layers. The activation values ​​obtained based on the original calibration set are called original activation values, and the activation values ​​obtained based on the noisy calibration set are called noisy activation values. Each network layer can obtain its own original and noisy activation values; for example, for network layer i, the original activation value A_orig_i and the noisy activation value A_noised_i can be obtained. Next, based on the original and noisy activation values ​​of each network layer, the sensitivity score of each network layer is determined. Generally, the greater the difference between the original activation values ​​and the noisy activation values ​​of a network layer, the higher its sensitivity to input perturbations. Even small input noise can cause significant distortion in the output distribution of that network layer, meaning it is more sensitive to quantization errors. In this case, a higher sensitivity score can be assigned to the network layer; otherwise, a lower sensitivity score is assigned. Finally, based on the sensitivity scores of each network layer, sensitive and insensitive layers are determined. Specifically, network layers with sensitivity scores exceeding a set threshold X can be identified as sensitive layers, while those with sensitivity scores below X can be identified as insensitive layers. Alternatively, all network layers can be sorted from highest to lowest sensitivity score, and the top K% of layers with the highest sensitivity scores can be selected as sensitive layers. Or, network layers with sensitivity scores exceeding M times the standard deviation of the average sensitivity score of all network layers can be selected as sensitive layers. The thresholds X, K, and M can be dynamically adjusted based on historical quantization data of similar models in an empirical knowledge base.

[0045] In one implementation of this application, the sensitivity score of each network layer is determined based on its original activation value and noisy activation value, including: For each network layer, the distribution change rate of the output activation value of the network layer is determined by comparing the difference between the original activation value and the noisy activation value. The sensitivity score of the network layer is then determined based on the distribution change rate. The distribution change rate is positively correlated with the difference in activation value, and the sensitivity score is positively correlated with the distribution change rate.

[0046] In determining the sensitivity score of each network layer based on its original activation values ​​and noisy activation values, for each network layer, the rate of change of the output activation value distribution of that network layer can be determined by comparing the difference between the original activation value and the noisy activation value. The sensitivity score of the network layer is then determined based on this rate of change. A larger difference in activation values ​​corresponds to a larger rate of change in the distribution, and vice versa; that is, the rate of change in the distribution is positively correlated with the difference in activation values, and the sensitivity score is positively correlated with the rate of change in the distribution. For example, for network layer i, by comparing the difference between the original activation value A_orig_i and the noisy activation value A_noised_i, the rate of change in the output activation value distribution of network layer i can be quantified, and the sensitivity of network layer i to input noise disturbances can be quantified accordingly. Here, KL divergence can be introduced as a metric. A larger KL divergence value indicates that even small input noise can cause a large distortion in the output distribution of network layer i, thus determining a higher sensitivity score for network layer i.

[0047] In one implementation of this application, after performing quantization-aware training on the deep learning model, the method further includes: If the deep learning model after quantization-aware training does not meet the edge accuracy requirements, then the non-sensitive layers with the highest sensitivity scores are set as sensitive layers, and the deep learning model is retrained with mixed-precision quantization-aware training.

[0048] If the deep learning model trained with quantization awareness still does not meet the accuracy requirements on the edge, an effective approach is to appropriately add sensitive layers. This involves setting the non-sensitive layers with the highest sensitivity scores as sensitive layers and then retraining the deep learning model with mixed-precision quantization awareness. For example, assuming the accuracy of the deep learning model after quantization awareness training is not far from the accuracy requirement, some non-sensitive layers with high sensitivity scores can be modified into sensitive layers, and then the deep learning model can be trained with mixed-precision quantization awareness again. This will increase the number of network layers quantized to FP16 accuracy and decrease the number of network layers quantized to INT8 accuracy, thus further improving the accuracy of the deep learning model. However, if the accuracy of the deep learning model after quantization awareness training is far from the accuracy requirement, a large number of sensitive layers need to be added to meet the accuracy requirement. This will lead to a significant reduction in quantization performance gains. To avoid this problem, the model lightweighting and reconstruction steps described earlier can be performed for a more in-depth model structure reconstruction.

[0049] As an example, Figure 3 This is a schematic diagram illustrating the operation flow of intelligent initialization hybrid precision quantization perception training provided in an embodiment of this application. Figure 3 In this process, the quantization parameters of the activation values ​​of each layer obtained after training and quantization are used as the initialization values ​​of the corresponding quantizers in quantization-aware training, and sensitivity layer analysis is performed on the deep learning model. Then, sensitive layers and non-sensitive layers are distinguished, and mixed-precision quantization-aware training is performed on the deep learning model. After the quantization-aware training is completed, it is determined whether the deep learning model meets the edge precision requirements. If it does, the quantization task ends; otherwise, a sensitive layer is added appropriately, and mixed-precision quantization-aware training is performed on the deep learning model again.

[0050] The technical solution of this application uses a constructed original calibration set to perform post-training quantization processing on the deep learning model to be quantized. If the deep learning model after post-training quantization does not yet meet the edge accuracy requirements, the inheritance mechanism of post-training quantization parameters is activated. The quantization parameters of the activation values ​​of each layer obtained after post-training quantization are used as the initialization values ​​of the corresponding quantizers in quantization-aware training, and quantization-aware training processing continues on the deep learning model. Since the quantization parameters obtained after post-training quantization are closer to the optimal parameter values ​​than randomly initialized values, the above process, by using the quantization parameters of the activation values ​​obtained after post-training quantization as the initialization values ​​of the corresponding quantizers in quantization-aware training, can provide a high-quality initial value for quantization parameters for the quantization-aware training process, thereby accelerating convergence and improving the final accuracy during model training. In summary, the above process is equivalent to using valuable information from the post-training quantization process to optimize the effect of quantization-aware training, which can further improve the inference performance of the quantized deep learning model.

[0051] To facilitate understanding of the deep learning model quantization method proposed in the embodiments of this application, a practical application scenario is given below. Please refer to... Figure 4 This diagram illustrates the operation flow of the deep learning model quantization method provided in this application embodiment under a practical application scenario. Figure 4In this process, a deep learning model to be quantized is obtained, deployed on the target edge device, and its baseline accuracy and performance metrics are measured. Then, the TensorRT implicit quantization engine is used to probe the upper limit of the achievable performance metrics of the quantized model, and the baseline performance metrics and upper limit are used to determine whether the edge performance requirements are met. If the edge performance requirements are not met, the model structure can be lightweighted, modified to be computationally intensive and quantization-friendly, and the process returns to the step of probing the upper limit of the achievable performance metrics using the TensorRT implicit quantization engine. If the edge performance requirements are met, the deep learning model is trained and then quantized using a calibration set, and the processed deep learning model is judged to meet the edge accuracy requirements. If the edge accuracy requirements are met, the inference engine obtained after training and quantization is deployed. The model quantization task ends when the target edge device is used for actual performance and accuracy evaluation. If the edge accuracy requirement is not met, the quantization parameters of the activation values ​​of each layer obtained after training quantization are used as the initialization values ​​of the corresponding quantizers in quantization-aware training, and sensitivity layer analysis is performed on the deep learning model. Then, sensitive and non-sensitive layers are distinguished, and mixed-precision quantization-aware training is performed on the deep learning model. It is then determined whether the trained deep learning model meets the edge accuracy requirement. If the edge accuracy requirement is met, the inference engine obtained from quantization-aware training is deployed to the target edge device for actual performance and accuracy evaluation, and the model quantization task ends. If the edge accuracy requirement is not met, the learning rate strategy is readjusted and / or the BN layer parameters are frozen, and the mixed-precision quantization-aware training step for the deep learning model is returned to be executed. It can be seen that... Figure 4 A standardized processing flow integrating both performance and accuracy has been constructed, replacing the traditional manual trial-and-error approach with a cyclical model of measurement, judgment, and decision-making. Furthermore, this standardized process incorporates optimization strategies for different failure causes; for example, it automatically triggers model lightweighting and structural modifications when performance is substandard, and automatically triggers training hyperparameter adjustments when accuracy is substandard, thus achieving intelligent fault recovery.

[0052] In summary, the technical solution of this application, by introducing data perturbation sensitivity analysis, generating adversarial calibration set optimization, and inheriting quantization parameters after training, maximizes deployment performance while ensuring model accuracy, and significantly improves the efficiency and success rate of model quantization deployment.

[0053] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0054] The above mainly describes a method for quantizing deep learning models. The following section will describe a device for quantizing deep learning models.

[0055] Please see Figure 5 One embodiment of a deep learning model quantization device in this application includes: The post-training quantization module 501 is used to perform post-training quantization on the deep learning model to be quantized using the constructed original calibration set. The quantization-aware training module 502 is used to perform quantization-aware training on the deep learning model if the deep learning model after training and quantization does not meet the edge accuracy requirements. In this case, the quantization parameters of the activation values ​​of each layer obtained after training and quantization are used as the initialization values ​​of the corresponding quantizer in the quantization-aware training.

[0056] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a deep learning model quantization method as described in any of the above embodiments.

[0057] This application also provides a computer program product that, when run on an electronic device, causes the electronic device to execute a deep learning model quantization method as shown in any of the above embodiments.

[0058] Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device 6 of this embodiment includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, it implements the steps in the embodiments of the various deep learning model quantization methods described above, for example... Figure 1 Steps 101 to 102 are shown. Alternatively, when the processor 60 executes the computer program 62, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 5 The functions of modules 501 to 502 are shown.

[0059] The computer program 62 can be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 62 in the electronic device 6.

[0060] The processor 60 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0061] The memory 61 can be an internal storage unit of the electronic device 6, such as a hard disk or memory. The memory 61 can also be an external storage device of the electronic device 6, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, the memory 61 can include both internal and external storage units of the electronic device 6. The memory 61 is used to store the computer program and other programs and data required by the electronic device 6. The memory 61 can also be used to temporarily store data that has been output or will be output.

[0062] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0063] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0064] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0065] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0066] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0067] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0068] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0069] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0070] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for quantizing deep learning models, characterized in that, include: The deep learning model to be quantized is trained and then quantized using the constructed original calibration set. If the deep learning model after training and quantization does not meet the edge accuracy requirements, the quantization parameters of the activation values ​​of each layer obtained after training and quantization are used as the initialization values ​​of the corresponding quantizers in quantization-aware training, and quantization-aware training is performed on the deep learning model.

2. The method as described in claim 1, characterized in that, Before performing post-training quantization on the deep learning model to be quantized using the constructed original calibration set, the method further includes: Detect whether the deep learning model meets the edge performance requirements; If the deep learning model does not meet the edge performance requirements, then the deep learning model is subjected to sparse perception training, and the structure of the deep learning model is modified according to a predefined rule base until the deep learning model meets the edge performance requirements.

3. The method as described in claim 2, characterized in that, The step of detecting whether the deep learning model meets the edge performance requirements includes: Deploy and measure the benchmark performance metrics of the deep learning model on the target edge device; The implicit quantization engine is used to probe the upper limit of the performance metrics of the deep learning model after quantization; By comparing the baseline performance metric with the upper limit of the performance metric, it is determined whether the deep learning model meets the edge performance requirements.

4. The method as described in claim 1, characterized in that, The quantization-aware training process for the deep learning model includes: Based on data perturbation analysis, the sensitive and non-sensitive layers of the deep learning model are identified; The sensitive layer and the non-sensitive layer are subjected to mixed-precision quantization perception training processing.

5. The method as described in claim 4, characterized in that, The method of identifying sensitive and non-sensitive layers in the deep learning model based on data perturbation analysis includes: Noise used to simulate quantization error is injected into the original calibration set to obtain a noise calibration set; The deep learning model is used to perform forward inference on the original calibration set and the noisy calibration set respectively, so as to obtain the original activation value and the noisy activation value of each network layer of the deep learning model respectively; Based on the original activation value and the noise activation value of each network layer, the sensitivity score of each network layer is determined respectively; Based on the sensitivity scores of each network layer, the sensitive layer and the non-sensitive layer are determined from the network layers.

6. The method as described in claim 5, characterized in that, The step of determining the sensitivity score of each network layer based on its original activation value and noisy activation value includes: For each network layer, the distribution change rate of the output activation value of the network layer is determined by comparing the activation value difference between the original activation value and the noisy activation value of the network layer, and the sensitivity score of the network layer is determined based on the distribution change rate; wherein, the distribution change rate is positively correlated with the activation value difference, and the sensitivity score is positively correlated with the distribution change rate.

7. The method as described in claim 5, characterized in that, After performing quantization-aware training on the deep learning model, the method further includes: If the deep learning model after quantization-aware training does not meet the edge accuracy requirement, then the non-sensitive layers with the highest sensitivity scores are set as sensitive layers, and the deep learning model is re-trained with mixed-precision quantization-aware training.

8. The method according to any one of claims 1 to 7, characterized in that, If the deep learning model after training and quantization does not meet the edge accuracy requirements, then before performing quantization-aware training on the deep learning model, the method further includes: A lightweight generative adversarial network is used to perform distribution-aligned data augmentation on the original calibration set with a preset test set distribution as the target, resulting in an augmented calibration set. The deep learning model is then retrained and quantized using the augmented calibration set.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the deep learning model quantization method as described in any one of claims 1 to 8.

10. A computer program product, characterized in that, When the computer program product is run on an electronic device, the electronic device performs the deep learning model quantization method as described in any one of claims 1 to 8.

Images