Mixing precision quantification strategy determination method and device

Abstract

The invention discloses a hybrid precision quantization strategy determination method, which comprises the steps of determining a quantization bit width selection range of each operator in a neural network model according to a plurality of different quantization precisions supported by target hardware; according to each operator of the neural network model and the corresponding quantization bit width selection range, generating a candidate mixed precision quantization combination, the candidate mixed precision quantization combination being used for quantizing each operator; selecting K candidate mixing precision quantization combinations from all the candidate mixing precision quantization combinations by using a genetic algorithm; the trained XGBOOST cost model is utilized to conduct reasoning evaluation on the K candidate mixing precision quantification combinations, and the optimal mixing precision quantification combination is determined based on the reasoning evaluation result of each candidate mixing precision quantification combination; and automatically deploying the neural network model to target hardware according to the optimal mixing precision quantification combination. According to the method, the optimal mixing precision quantification combination conforming to the target hardware can be automatically generated.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and in particular to a method and apparatus for determining mixed-precision quantization strategies. Background Technology

[0002] With the development of neural network algorithms, the improvement in network performance is accompanied by a synchronous increase in the demand for computing resources (e.g., processing, memory, bandwidth). This poses a great challenge to deploying neural network models on resource-constrained edge devices, thus limiting the use of deep learning in edge scenarios.

[0003] Model quantization compression schemes can reduce the memory footprint of over-parameterized models and decrease the resource requirements of neural networks without significantly compromising accuracy. Model quantization algorithms are broadly categorized into two types based on whether the model requires retraining: training-time quantization and post-training quantization. Training-time quantization is not widely used in real-world scenarios due to privacy concerns regarding retraining datasets and the extremely time-consuming and resource-intensive nature of retraining. Post-training quantization, on the other hand, quantizes the model's parameters and activation values ​​using only a small dataset for calibration, eliminating the need for retraining and enabling the deployment of neural networks on resource-constrained edge devices.

[0004] Device hardware typically supports multiple precision levels. While using lower precision in neural network inference can achieve faster inference speeds, it also leads to greater precision loss and larger result errors. Therefore, a balance between inference precision and speed is necessary. Related technologies propose a hybrid approach based on model segmentation, combining multiple quantization methods for a unified model to address the low inference efficiency of quantized models in practical neural network deployments. However, in actual deployed neural networks, due to the mutual influence of computations within the same layer or between different layers, this method does not consider the potential impact between quantized model layers. Furthermore, the different underlying characteristics of different hardware platforms also affect the precision of model quantization, making it difficult to determine the optimal hybrid precision optimization strategy empirically. Summary of the Invention

[0005] The purpose of this application is to provide a method for determining mixed-precision quantization strategies, which not only considers the underlying hardware characteristics of the hardware platform, but also the influence of inter-layer computation in the neural network model, and can automatically generate the optimal mixed-precision quantization combination that conforms to the target hardware.

[0006] To achieve the above objectives, this application provides a method for determining a mixed-precision quantization strategy, the method comprising:

[0007] Based on the multiple different quantization precipitates that the target hardware can support, determine the range of quantization bit width selection for each operator in the neural network model;

[0008] Based on the various operators of the neural network model and the corresponding quantization bit width selection range, candidate mixed-precision quantization combinations are generated, which are used to quantize the various operators of the neural network model.

[0009] A genetic algorithm is used to select K candidate mixed-precision quantization combinations from all candidate mixed-precision quantization combinations.

[0010] The trained XGBOOST cost model is used to perform inference evaluation on K candidate mixed precision quantization combinations. Based on the inference evaluation results of each candidate mixed precision quantization combination, the optimal mixed precision quantization combination is determined.

[0011] The neural network model is automatically deployed to the target hardware based on the optimal combination of mixed precision quantization.

[0012] Furthermore, the steps for selecting K candidate mixed-precision quantization combinations from all candidate mixed-precision quantization combinations using a genetic algorithm include:

[0013] Construct an initial population, where each individual in the initial population corresponds to a candidate mixture precision quantization combination;

[0014] Using TOP1 accuracy as the fitness function, the fitness value of each individual in the initial population is calculated;

[0015] Selection operation: Select multiple individuals with high fitness values ​​from the initial population as offspring;

[0016] Crossover and mutation operations: Perform crossover and mutation operations on the selected offspring population to obtain a new generation of offspring population;

[0017] Iterative training is performed on the new generation of offspring. When the iteration termination condition is met, the individuals in the current population are sorted according to their fitness values, and the top K individuals are selected as K candidate mixed precision quantization combinations.

[0018] Furthermore, the XGBOOST cost model is used to perform inference evaluation on the K candidate mixed-precision quantization combinations. Based on the inference evaluation results of each candidate mixed-precision quantization combination, the steps to determine the optimal mixed-precision quantization combination include:

[0019] Set the inference threshold for the neural network model. The inference threshold includes the inference accuracy threshold and the inference speed threshold.

[0020] K candidate mixed-precision quantization combinations are input into the XGBOOST cost model, which outputs the predicted mixed-precision quantization combination and the corresponding predicted inference result. If the predicted inference result meets the inference threshold, the predicted mixed-precision quantization combination is determined to be the optimal mixed-precision quantization combination. Otherwise, the XGBOOST cost model is optimized and trained. The predicted inference result includes the predicted inference accuracy and the predicted inference speed.

[0021] Furthermore, the steps for optimizing the XGBOOST cost model training include:

[0022] Based on the predicted mixed precision quantization combination, a deep learning compiler is used to automatically deploy the neural network model. The deployed neural network model is then simulated and inferred on the target hardware to obtain the actual inference accuracy and actual inference speed.

[0023] The training set is updated using the model structure of the neural network model, the quantized combination of predicted mixed precision, the actual inference accuracy, and the actual inference speed as new training samples.

[0024] Obtain the accuracy error between the predicted inference accuracy and the actual inference accuracy, and obtain the speed error between the predicted inference speed and the actual inference speed. Determine the loss function based on the accuracy error and the speed error, and update the parameters of the XGBOOST cost model based on the loss function.

[0025] The XGBOOST cost model is iteratively trained using the updated training set until the inference result output by the trained XGBOOST cost model meets the inference threshold, and / or all candidate mixed precision quantization combinations are searched.

[0026] Furthermore, the predicted inference results output by the XGBOOST cost model are used as the fitness function of the genetic algorithm.

[0027] Furthermore, the training steps for the XGBOOST cost model include:

[0028] Based on the model structure and mixed precision quantization combination of the neural network model, a test quantization model is deployed using a deep learning compiler. The test quantization model is then tested on hardware to obtain inference accuracy and inference speed. The model structure, mixed precision quantization combination, inference accuracy, and inference speed of the neural network model are used as training samples for the training set. The training set is then input into the XGBOOST cost model for training, and the XGBOOST cost model is output.

[0029] To achieve the above objectives, this application provides a hybrid precision quantization strategy determination apparatus, the apparatus comprising:

[0030] The acquisition module is used to determine the range of quantization bit width selection for each operator in the neural network model based on the multiple different quantization precipitates that the target hardware can support.

[0031] The determination module is used to generate all candidate mixed-precision quantization combinations based on each operator of the neural network model and the corresponding quantization bit width selection range. The candidate mixed-precision quantization combinations are used to quantize each operator of the neural network model.

[0032] The genetic algorithm module is used to select K candidate mixed-precision quantization combinations from all candidate mixed-precision quantization combinations using a genetic algorithm.

[0033] The model prediction module is used to perform inference evaluation on K candidate mixed precision quantization combinations using the trained XGBOOST cost model, and to determine the optimal mixed precision quantization combination based on the inference evaluation results of each candidate mixed precision quantization combination.

[0034] The model deployment module is used to automatically deploy neural network models to target hardware based on the optimal combination of mixed precision quantization.

[0035] Furthermore, the model prediction module is specifically used for:

[0036] Set the inference threshold for the neural network model. The inference threshold includes the inference accuracy threshold and the inference speed threshold.

[0037] K candidate mixed-precision quantization combinations are input into the XGBOOST cost model, which outputs the predicted mixed-precision quantization combination and the corresponding predicted inference result. If the predicted inference result meets the inference threshold, the predicted mixed-precision quantization combination is determined to be the optimal mixed-precision quantization combination. Otherwise, the XGBOOST cost model is optimized and trained. The predicted inference result includes the predicted inference accuracy and the predicted inference speed.

[0038] Furthermore, the model prediction module is specifically used for:

[0039] Based on the predicted mixed precision quantization combination, a deep learning compiler is used to automatically deploy the neural network model. The deployed neural network model is then simulated and inferred on the target hardware to obtain the actual inference accuracy and actual inference speed.

[0040] The training set is updated using the model structure of the neural network model, the quantized combination of predicted mixed precision, the actual inference accuracy, and the actual inference speed as new training samples.

[0041] Obtain the accuracy error between the predicted inference accuracy and the actual inference accuracy, and obtain the speed error between the predicted inference speed and the actual inference speed. Determine the loss function based on the accuracy error and the speed error, and update the parameters of the XGBOOST cost model based on the loss function.

[0042] The XGBOOST cost model is iteratively trained using the updated training set until the inference result output by the trained XGBOOST cost model meets the inference threshold, and / or all candidate mixed precision quantization combinations are searched.

[0043] For the purposes described above, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described above.

[0044] This application utilizes a genetic algorithm and the XGBOOST cost model to determine the optimal mixed-precision quantization strategy. It considers not only the underlying hardware characteristics of the hardware platform but also the impact of inter-layer computation in the neural network model, automatically generating the optimal mixed-precision quantization combination that meets the target hardware. The neural network model deployed based on the optimal mixed-precision quantization combination minimizes quantization precision loss while maintaining higher Top-1 precision, all while compressing the model size as much as possible. Attached Figure Description

[0045] Figure 1 This is a first flowchart of the method for determining mixed precision quantization strategy according to the embodiments of this application;

[0046] Figure 2 This is a second flowchart of the method for determining mixed precision quantization strategies according to the embodiments of this application;

[0047] Figure 3 This is a third flowchart of the method for determining mixed precision quantization strategies according to the embodiments of this application;

[0048] Figure 4 This is the fourth flowchart of the method for determining mixed precision quantization strategy according to the embodiments of this application;

[0049] Figure 5 This is a schematic diagram of the data structure of the XGBOOST cost model training set provided in the embodiments of this application;

[0050] Figure 6 This is a schematic diagram of the XGBOOST cost model structure provided in the embodiments of this application;

[0051] Figure 7 This is a schematic diagram of the optimal mixed precision quantization combination of the ResNet18 network generated according to the embodiments of this application;

[0052] Figure 8 This is a schematic diagram comparing the acceleration effects of the mixed precision algorithm provided in the embodiments of this application;

[0053] Figure 9 This is a schematic diagram comparing experimental results of mixed precision quantization combinations of ResNet18 and ResNet50 networks provided in the embodiments of this application;

[0054] Figure 10 This is a system block diagram of the hybrid precision quantization strategy determination device provided in the embodiments of this application;

[0055] Figure 11 A schematic diagram of the structure of a computer device provided in the embodiments of this application. Detailed Implementation

[0056] The present application will be described in detail below with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present application. Any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the protection scope of the present application.

[0057] Operations related to neural networks include training, inference, and quantization. Training refers to adjusting the network structure using input samples through a specific algorithm to ensure the network's output matches the expected value. Inference refers to deploying a pre-trained neural network model into real-world business scenarios, such as image classification and object detection. Since inference directly faces the customer, its performance is crucial. Therefore, improving computational speed by compressing the model size becomes a time-sensitive requirement, and quantization is a typical model compression method. Model quantization refers to converting floating-point calculations into low-precision point calculations during model computation, effectively reducing computational intensity, parameter size, and memory consumption. In actual deployed neural networks, the interrelationships between calculations within the same layer or between different layers make it difficult to determine the optimal mixed-precision optimization strategy empirically. This application proposes a mixed-precision quantization strategy determination method that automatically determines the optimal mixed-precision quantization combination for the neural network model using a genetic algorithm and the XGBoost model to complete the neural network deployment. By comprehensively considering the influence between different layers of the neural network model, the optimal mixed-precision quantization combination is effectively determined, resulting in better mixed-precision quantization results. It should be noted that this application determines the combination of quantization bit widths of different layers of a neural network in a mixed-precision quantization scenario, without involving specific quantization algorithms and quantization parameters. Without deviating from the basic principles of this application, those skilled in the art can use any suitable post-training algorithm to quantize weights and activation values.

[0058] Please refer to Figure 1 This application provides a method for determining a mixed-precision quantization strategy, which includes steps S101-S105.

[0059] Step S101: Based on the multiple different quantization accuracies that the target hardware can support, determine the range of quantization bit width selection for each operator in the neural network model.

[0060] In neural networks, operators typically refer to the basic mathematical operations used in the layers of the neural network. For example, convolution operators indicate a convolution operation, and pooling operators indicate a pooling operation. Based on these operators, neural network models can be built and trained, and these models can be deployed on various devices for forward inference.

[0061] The hardware of the devices may differ. For example, different devices may be configured with hardware (chips) from different manufacturers, such as x86 CPUs (Central Processing Units), GPUs (Graphics Processing Units), ARM (Advanced RISC Machines) processors, ASIC chips designed specifically for deep learning computing, or AI (Artificial Intelligence) processors. Different hardware may support different quantization precision types, such as 4-bit, 8-bit, 16-bit, and 32-bit. In the embodiments of this application, a single device may be equipped with one or more types of hardware. The neural network model to be deployed on the device needs to perform forward inference on one of these pieces of hardware, which can be referred to as the target hardware.

[0062] Based on the various quantization precisions supported by the target hardware, the quantization bit width selection range for each operator in the neural network model is determined. For example, the Rockchip RK3588 hardware supports bit widths of 4, 8, and 16 bits. When deploying the lightweight convolutional neural network ResNet18 to the Rockchip RK3588 platform (target hardware), the quantizable bit width selection is set to 4, 8, and 16 bits according to the quantization requirements and hardware characteristics. That is, the quantization bit width selection range for each operator in the lightweight convolutional neural network is 4, 8, and 16 bits. In actual deployment, the supported quantization precisions can be determined according to the hardware product specifications; specific hardware support information can be obtained by consulting the relevant hardware manual. By setting the quantization bit width selection range for each operator in the neural network model based on the quantization precision types supported by the hardware, the overall search space can be pruned, accelerating the search for the optimal quantization combination scheme.

[0063] Step S102: Based on the various operators of the neural network model and the corresponding quantization bit width selection range, generate all candidate mixed-precision quantization combinations. The candidate mixed-precision quantization combinations are used to quantize the various operators of the neural network model.

[0064] Each operator in a neural network model has a corresponding range of quantization bit widths. Based on the quantization bit width chosen by each operator, combining the selected quantization bit widths of all operators yields a candidate mixed-precision quantization combination. This process is repeated to obtain all candidate mixed-precision quantization combinations. Taking the above example, the Rockchip RK3588 hardware supports bit widths of 4, 8, and 16 bits. Assuming the neural network model has four layers (four operators), and the first operator selects 4 bits, the second 8 bits, the third 16 bits, and the fourth 4 bits, then a candidate mixed-precision quantization combination would be a combination of 4 bits, 8 bits, 16 bits, and 4 bits. This process can be repeated to obtain 4... 3 There are 3 candidate mixed-precision quantization combinations. The aforementioned ResNet18 convolutional neural network can form 3 18 A total of 10 candidate mixed-precision quantization combinations are evaluated. This comprehensive approach, based on all mixed-precision quantization combinations, ensures a complete assessment of all quantization combinations and yields more accurate results.

[0065] In one embodiment of this application, the quantization bit width of the first layer and the quantization bit width of the last layer of the neural network are determined based on prior knowledge. Then, based on the remaining operators of the neural network model and their corresponding quantization bit width selection ranges, all candidate mixed-precision quantization combinations are generated. Taking the above embodiment as an example, originally there were 3... 18 Based on this implementation method, 3 candidate mixed-precision quantization combinations were generated. 16 This reduces the number of candidate mixed-precision quantization combinations.

[0066] Step S103: Use a genetic algorithm to select K candidate mixed precision quantization combinations from all candidate mixed precision quantization combinations.

[0067] Genetic algorithms are a type of random search algorithm based on biological mechanisms. They are based on the assumption that a new population will be better than the previous one. This embodiment utilizes a genetic algorithm to search and optimize all candidate mixed-precision quantization combinations, identifying the K best combinations to provide better candidate combinations for determining the optimal mixed-precision quantization combination in subsequent processes, thus shortening the model prediction and evaluation time.

[0068] Step S104: Use the trained XGBOOST cost model to perform inference evaluation on K candidate mixed precision quantization combinations, and determine the optimal mixed precision quantization combination based on the inference evaluation results of each candidate mixed precision quantization combination.

[0069] The XGBoost cost model is a powerful gradient boosting tree algorithm used to solve classification and regression problems, i.e., decision problems. The XGBoost cost model forms a powerful model by combining multiple weak learners (usually decision trees). Each decision tree is trained based on the residuals of the previous tree, thus progressively improving the model's predictive ability. This model provides an intuitive method for evaluating feature importance, helping to understand the model's dependence on different features and revealing which features are most critical for making correct decisions, thus helping to explain the model's decision-making process. This embodiment treats the mixed-precision quantization combination strategy of the neural network model as a decision problem, with the quantization bit width selection of each layer as a sub-problem. The trained XGBoost cost model is used to evaluate the inference performance of K candidate mixed-precision quantization combinations. Based on the evaluation results of each candidate combination, the optimal mixed-precision quantization combination is determined. The XGBoost cost model can automatically generate mixed-precision quantization combination strategies, and this implementation is transferable to different neural network models and different hardware platforms.

[0070] Step S105: Based on the optimal combination of mixed precision quantization, automatically deploy the neural network model to the target hardware.

[0071] Based on the determined optimal combination of mixed-precision quantization, the quantization bit width of each operator in the neural network model is further determined. Deploying this neural network model on the target hardware then allows for better performance. For example, automatic model deployment can be achieved through a deep learning compiler, such as the open-source platform TVM proposed by Tianqi Chen or Huawei Ascend's CANN.

[0072] This embodiment determines all candidate mixed-precision quantization combinations based on the characteristics of the hardware platform. It then uses a genetic algorithm and the XGBoost cost model to determine the optimal mixed-precision quantization strategy. This approach considers not only the underlying hardware characteristics of the platform but also the impact of inter-layer computations in the neural network model, automatically generating the optimal mixed-precision quantization scheme that fits the target hardware and achieving better model quantization results. Furthermore, the neural network model deployed based on the optimal mixed-precision quantization combination minimizes quantization accuracy loss while maintaining higher Top-1 accuracy, all while compressing the model size as much as possible.

[0073] As an optional implementation, such as Figure 2As shown, the steps for using a genetic algorithm to search and optimize, and to determine the K best candidate mixed-precision quantization combinations from all candidate combinations, include:

[0074] Step S201: Construct an initial population, where each individual in the initial population corresponds to a candidate mixture precision quantization combination;

[0075] Step S202: Calculate the fitness value of each individual in the initial population according to the fitness function, using TOP1 accuracy as the evaluation function.

[0076] For any individual in the initial population, the neural network model is deployed using the candidate mixture precision quantization combination corresponding to that individual, and the inference precision is used as the fitness value of that individual.

[0077] Step S203: Selection operation: Calculate the fitness values ​​of all individuals in the initial population, and select multiple individuals with high fitness values ​​in the initial population as offspring.

[0078] Step S204, Crossover and Mutation: Perform crossover and mutation operations on the selected offspring population to obtain a new generation of offspring population;

[0079] Step S205: Perform iterative training on the new generation of offspring. When the iteration termination condition is met, sort each individual in the current population according to its fitness value, and take the top K individuals as K candidate mixed precision quantization combinations.

[0080] The iteration termination condition is the maximum number of generations that can be set, and / or the completion of searching all candidate mixed-precision quantization combinations. This implementation uses a genetic algorithm to obtain K optimal candidate mixed-precision quantization combinations, reducing the computational load for subsequent cost models.

[0081] Optionally, the predicted inference result output by the XGBOOST cost model can be used as the fitness function of the genetic algorithm to reduce the workload of the genetic algorithm's optimization search. Based on the steps of the genetic algorithm described above, a search and optimization process is performed to obtain K candidate mixed-precision quantization combinations, which will not be elaborated further here.

[0082] As an optional implementation, such as Figure 3 As shown, the inference performance of K candidate mixed-precision quantization combinations is evaluated using the trained XGBOOST cost model. The steps for determining the optimal mixed-precision quantization combination based on the evaluation results of each candidate combination include:

[0083] Step S301: Set the inference threshold of the neural network. The inference threshold includes the inference accuracy threshold and the inference speed threshold.

[0084] Setting inference accuracy and speed thresholds for a neural network can be done based on the specific deployment scenario and experience. For example, these thresholds can be obtained by testing the neural network model at its original accuracy (FP32) and the lowest computational accuracy supported by the target hardware. Generally, inference at the original accuracy yields the highest accuracy but also the longest computation time; the inference accuracy threshold can be determined by reducing this threshold by a certain margin. Conversely, using the lowest computational accuracy yields the fastest computation time but the lowest accuracy; the computation time can be reduced by a certain margin to determine the inference speed threshold. Setting these thresholds helps prevent extreme search results in subsequent processes.

[0085] Step S302: Input the K candidate mixed precision quantization combinations into the XGBOOST cost model, output the predicted mixed precision quantization combination and the corresponding predicted inference result. If the predicted inference result meets the inference threshold, determine the predicted mixed precision quantization combination as the optimal mixed precision quantization combination. Otherwise, perform model optimization training of the XGBOOST cost model. The predicted inference result includes the predicted inference accuracy and the predicted inference speed.

[0086] K candidate mixed-precision quantization combinations are input into the XGBOOST cost model. The XGBOOST cost model predicts the quantization inference result for each candidate mixed-precision quantization combination, thus obtaining K predicted inference precision and predicted inference speed. The cost model scores the K predicted inference precision and predicted inference speed, and outputs the predicted mixed-precision quantization combination, along with the corresponding predicted inference precision and predicted inference speed, based on the scoring results. If the predicted inference precision meets the inference precision threshold and the predicted inference speed meets the inference speed threshold, the predicted mixed-precision quantization combination is determined to be the optimal mixed-precision quantization combination; otherwise, the XGBOOST cost model needs further training. This implementation evaluates and predicts candidate mixed-precision quantization combinations using the XGBOOST cost model, enabling the prediction of the deployment effect of different quantization strategies, achieving a faster and more efficient search and comparison process, and thus finding the optimal mixed quantization strategy.

[0087] As an optional implementation, such as Figure 4 As shown, the steps for model optimization training of the XGBOOST cost model include:

[0088] Step S401: Based on the predicted mixed precision quantization combination, automatically deploy the neural network model using a deep learning compiler, and perform simulation inference on the target hardware to obtain the actual inference accuracy and actual inference speed.

[0089] If the predicted inference accuracy does not reach the inference accuracy threshold, and / or the predicted inference speed does not reach the inference speed threshold, simulation inference processing is required. Based on the simulation inference, the actual inference accuracy and actual inference speed are obtained. For example, the deep learning compiler can be a TVM or CANN deep learning compiler.

[0090] Step S402: Update the training set using the model structure of the neural network model, the predicted mixed precision quantization combination, the actual inference accuracy, and the actual inference speed as new training samples.

[0091] The simulation prediction results are stored in a database as sample data for the training set of the XGBOOST cost model, and the model is further trained to improve the performance of the cost model.

[0092] Step S403: Obtain the accuracy error between the predicted inference accuracy and the actual inference accuracy, and obtain the speed error between the predicted inference speed and the actual inference speed. Determine the loss function based on the accuracy error and the speed error, and update the parameters of the XGBOOST cost model based on the loss function.

[0093] Step S404: Iteratively train the XGBOOST cost model using the updated training set until the inference result output by the trained XGBOOST cost model meets the inference threshold, and / or search all candidate mixed precision quantization combinations.

[0094] The accuracy and speed errors are used as the loss values ​​for the XGBOOST cost model, and the network parameters are adjusted based on these loss values. The optimal mixed-precision quantization combination and corresponding inference results are recorded at each iteration, and the training set is updated. The new training set is used to train the model, allowing its performance to gradually reach its optimal level during iteration. When the iteration terminates, the final optimal mixed-precision quantization combination is output, resulting in a better-performing XGBOOST cost model.

[0095] This application's implementation combines the XGBOOST cost model with simulation inference. First, the XGBOOST cost model predicts mixed-precision quantization combinations. Then, the actual inference results are obtained through simulation testing. The actual inference results are used as training data to further train the XGBOOST cost model. The error between the model's prediction results and the simulation test results is used as the loss value of the cost model to adjust its network parameters, further improving its performance. The search process guided by the XGBOOST cost model, combined with error correction from real model inference data and model retraining, achieves a faster and more efficient search and comparison process for combination strategies, enabling faster convergence and outputting the mixed-precision quantization combination with the best overall performance.

[0096] To ensure the accuracy of the cost model, historical data is used to train the XGBOOST cost model. Based on the model structure and mixed-precision quantization combination of the neural network model, a test quantization model is deployed using a deep learning compiler. This test quantization model is then tested on hardware to obtain inference accuracy and speed. The model structure, mixed-precision quantization combination, inference accuracy, and inference speed of the neural network model are used as training samples in the training set. This process is repeated to obtain the training set, as shown in Figure 5, which represents the data structure of the training set as: D = {(ei, si, ci)}, where ei, si, and ci represent the model structure, mixed-precision quantization combination, and inference result, respectively. The inference result includes inference accuracy and inference speed. The training set is then input into the XGBOOST cost model to be trained, and the model is trained. After training, the XGBOOST cost model is obtained.

[0097] like Figure 6 The diagram shows the network structure of the XGBOOST cost model. The XGBOOST cost model is an ensemble learning algorithm based on gradient descent iteration. This model uses decision trees as base learners for ensemble learning. The algorithm continuously adds trees by feature splitting and growing. Each time a tree is added, it learns a new function. The residual of the previous prediction is used to learn the new function. Each tree corresponds to a leaf node, and each leaf node corresponds to a score. The sum of the scores of each tree is the predicted value of the sample.

[0098] The specific calculation process of the XGBOOST cost model is as follows:

[0099] The model's final prediction results It can be represented as a ensemble tree model of k stacked elements.

[0100]

[0101] Where K represents the number of trees, Let F represent the predicted value of the i-th sample, F represent all binary trees, and f represent a specific binary tree. k (x i ) represents the score obtained by a sample at a leaf node in a tree.

[0102] To train the model, we use the model's built-in regularization objective function and optimization method. The regularization objective function is:

[0103]

[0104] The formula mainly consists of two parts: the loss function. and the regularization function Ω(f k loss function It is a differentiable convex function, and the predicted value is calculated using the mean square error. and the actual test value y i The differences between them. Regularization function Ω(f) k The expansion of the k-th tree is given by the following formula:

[0105]

[0106] Where T is the number of leaves, between 0 and 1, and multiplied by T to reduce the complexity of each leaf, and is a parameter used for scaling penalty to avoid overfitting.

[0107] When the objective function is obtained in the t-th tree, the regularization objective function can be expressed as:

[0108]

[0109] Among them, each f t This represents the independent tree generated by instance i in step t.

[0110] To reduce the computational cost of the objective function, a second-order Taylor approximation is used for transformation, resulting in the following formula:

[0111]

[0112] in,

[0113]

[0114] By removing the constant term, the objective function can be simplified to:

[0115]

[0116] Using the simplified objective function obtained, the model performance after splitting a leaf node in the tree can be iteratively evaluated. If the performance of the tree model improves after the split, the change is accepted; otherwise, the splitting stops. This method determines the optimal split point for each tree that minimizes the objective function, while the regularization term can compensate for overfitting during training.

[0117] The XGBoost cost model allows you to configure parameters to customize the model for the training dataset. For example, hyperparameters and a rank loss function can be set. Hyperparameters can be Eta and gamma. To provide the training dataset to the XGBoost cost model, feature preprocessing is required. For example, classification encoding or one-hot encoding can be used for preprocessing. To apply the rank loss function during model training, the training dataset needs to be sorted. For example, sorting can be done by grouping according to a type of CNN model or by grouping the entire dataset.

[0118] In one specific embodiment of this application, the target classification model used consists of two pre-trained models on the ImageNet dataset: a ResNet18 network and a ResNet50 network, and the hardware platform used is an Nvidia GPU3090. The method compared to this application is the existing quantization method ZeroQ, which is a relatively accepted fixed-bit post-training quantization method in the industry, capable of post-training quantization with a small or even no calibration set. Figure 7 This is a schematic diagram of the optimal mixed-precision quantization combination of the ResNet18 network obtained according to the embodiments of this application. As can be seen from the figure, the optimal quantization bit width selected for each layer of the network shows that the first few layers of the ResNet18 network are highly sensitive to computation, so the allocated quantization bit width is relatively high. Conversely, the quantization bit width allocated to the later layers is lower. Figure 8 This diagram illustrates the speedup effect of the hybrid precision algorithm based on the present invention. Compared to the existing deployment framework TVM, the ResNet18 network quantized model achieves a speedup of 2.45x compared to the TVM INT8 deployment model. Figure 9 The diagram shows a comparison of experimental results for the hybrid precision quantization combination of ResNet18 and ResNet50 networks. As can be seen from the diagram, the hybrid bit quantization combination of ResNet18 and ResNet50 networks in this application, compared with the existing technology, maintains higher Top-1 precision while compressing the model size as much as possible.

[0119] like Figure 10As shown, this application provides a hybrid precision quantization strategy determination device, the device comprising:

[0120] The acquisition module 1001 is used to determine the quantization bit width selection range corresponding to each operator in the neural network model based on the multiple different quantization precipitates that the target hardware can support.

[0121] The determining module 1002 is used to generate all candidate mixed-precision quantization combinations based on each operator of the neural network model and the corresponding quantization bit width selection range. The candidate mixed-precision quantization combinations are used to quantize each operator of the neural network model.

[0122] Genetic algorithm module 1003 is used to select K candidate mixed-precision quantization combinations from all candidate mixed-precision quantization combinations using a genetic algorithm;

[0123] The model prediction module 1004 is used to perform inference evaluation on K candidate mixed precision quantization combinations using the trained XGBOOST cost model, and to determine the optimal mixed precision quantization combination based on the inference evaluation results of each candidate mixed precision quantization combination.

[0124] The model deployment module 1005 is used to automatically deploy the neural network model to the target hardware based on the optimal combination of mixed precision quantization.

[0125] The acquisition module 1001 sets the quantization bit width selection range for each operator in the neural network model based on multiple different quantization precipitates supported by the hardware. The determination module 1002, based on the selected quantization bit width for each operator, combines the selected quantization bit widths to obtain a candidate mixed-precision quantization combination, and so on, to obtain all candidate mixed-precision quantization combinations. The genetic algorithm module 1003 uses a genetic algorithm to search and optimize, selecting K candidate mixed-precision quantization combinations from all candidate combinations. The model prediction module 1004 uses a trained XGBOOST cost model to evaluate the inference performance of the K candidate mixed-precision quantization combinations, evaluating the inference precision and inference speed of the output K predictions to determine the optimal mixed-precision quantization combination. The model deployment module 1005 automatically deploys the model using a deep learning compiler. This embodiment combines the characteristics of the hardware platform and uses a genetic algorithm and XGBOOST cost model to determine the optimal mixed precision quantization strategy. It not only considers the underlying hardware characteristics of the hardware platform, but also the impact of inter-layer computation in the neural network model. It automatically generates the optimal mixed precision quantization scheme that conforms to the target hardware and obtains better model quantization results.

[0126] As an optional implementation, the model prediction module 1004 is specifically used for:

[0127] Set the inference threshold for the neural network model. The inference threshold includes the inference accuracy threshold and the inference speed threshold.

[0128] K candidate mixed-precision quantization combinations are input into the XGBOOST cost model, which outputs the predicted mixed-precision quantization combination and the corresponding predicted inference result. If the predicted inference result meets the inference threshold, the predicted mixed-precision quantization combination is determined to be the optimal mixed-precision quantization combination. Otherwise, the XGBOOST cost model is optimized and trained. The predicted inference result includes the predicted inference accuracy and the predicted inference speed.

[0129] As an optional implementation, the model prediction module 1004 is specifically used for:

[0130] Based on the predicted mixed precision quantization combination, a deep learning compiler is used to automatically deploy the neural network model. The deployed neural network model is then simulated and inferred on the target hardware to obtain the actual inference accuracy and actual inference speed.

[0131] The training set is updated using the model structure of the neural network model, the quantized combination of predicted mixed precision, the actual inference accuracy, and the actual inference speed as new training samples.

[0132] Obtain the accuracy error between the predicted inference accuracy and the actual inference accuracy, and obtain the speed error between the predicted inference speed and the actual inference speed. Determine the loss function based on the accuracy error and the speed error, and update the parameters of the XGBOOST cost model based on the loss function.

[0133] The XGBOOST cost model is iteratively trained using the updated training set until the inference result output by the trained XGBOOST cost model meets the inference threshold, and / or all candidate mixed precision quantization combinations are searched.

[0134] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-described mixed-precision quantization strategy determination methods.

[0135] Figure 11 This is a schematic diagram of the hardware structure of a computer device provided in an embodiment of this application. Figure 11 The computer device shown includes: a processor 1101, a communication interface 1102, a memory 1103, and a communication bus 1104. The processor 1101, communication interface 1102, and memory 1103 communicate with each other via the communication bus 1104. Figure 11The connection method between the processor 1101, communication interface 1102, and memory 1103 shown is merely exemplary. In the implementation process, the processor 1101, communication interface 1102, and memory 1103 may also use other connection methods besides the communication bus 1104 to communicate with each other.

[0136] Memory 1103 can be used to store computer programs, which may include instructions and data to implement the steps of any of the mixed-precision quantization strategy determination methods described above. In this embodiment, memory 1103 can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, optical memory, and registers. Memory 1103 may include a hard disk and / or RAM.

[0137] Processor 1101 can be a general-purpose processor, which can be a processor that performs specific steps and / or operations by reading and executing a computer program (e.g., a computer program) stored in memory (e.g., memory 1103). The general-purpose processor may use data stored in memory (e.g., memory 1103) during the execution of said steps and / or operations. The general-purpose processor can be, for example, but not limited to, a central processing unit (CPU). Furthermore, processor 1101 can also be a special-purpose processor, which can be a processor specifically designed to perform specific steps and / or operations. Special-purpose processors can be, for example, but not limited to, ASICs and FPGAs. Additionally, processor 1101 can also be a combination of multiple processors, such as a multi-core processor.

[0138] Communication interface 1102 may include input / output (I / O) interfaces, physical interfaces, and logical interfaces for interconnecting devices within the network device, as well as interfaces for interconnecting the network device with other devices (e.g., network devices). The communication network may be Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. Communication interface 1102 may be a module, circuit, transceiver, or any device capable of enabling communication.

[0139] In implementation, each step of the above method can be completed by the integrated logic circuitry of the hardware in the processor 1101 or by instructions in software form. The method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules in the processor. The software modules can be located in random access memory (RAM), read-only memory (ROM), programmable read-only memory (ROM), electrically erasable programmable memory (EPR), registers, or other mature storage media in the art. This storage medium is located in memory 1103. The processor 1101 reads the information in memory 1103 and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are not provided here.

[0140] Although preferred embodiments of the present application have been disclosed for illustrative purposes, those skilled in the art will recognize that various modifications, additions, and substitutions are possible without departing from the scope and spirit of the present application as disclosed in the appended claims.

Claims

1. A method for determining a mixed-precision quantization strategy, characterized in that, The method includes: Based on the multiple different quantization precipitates that the target hardware can support, determine the range of quantization bit width selection for each operator in the neural network model; Based on the various operators of the neural network model and the corresponding quantization bit width selection range, all candidate mixed-precision quantization combinations are generated, and the candidate mixed-precision quantization combinations are used to quantize the various operators of the neural network model. A genetic algorithm is used to select K candidate mixed-precision quantization combinations from all candidate mixed-precision quantization combinations. The trained XGBOOST cost model is used to perform inference evaluation on the K candidate mixed precision quantization combinations. Based on the inference evaluation results of each candidate mixed precision quantization combination, the optimal mixed precision quantization combination is determined. Based on the optimal combination of mixed precision quantization, the neural network model is automatically deployed to the target hardware.

2. The method for determining a hybrid precision quantization strategy as described in claim 1, characterized in that, The step of selecting K candidate mixed-precision quantization combinations from all candidate mixed-precision quantization combinations using a genetic algorithm includes: Construct an initial population, where each individual in the initial population corresponds to a candidate mixture precision quantization combination; Using TOP1 accuracy as the fitness function, the fitness value of each individual in the initial population is calculated; Selection operation: Select multiple individuals with high fitness values ​​from the initial population as offspring; Crossover and mutation operations: Perform crossover and mutation operations on the selected offspring population to obtain a new generation of offspring population; Iterative training is performed on the new generation of offspring. When the iteration termination condition is met, the individuals in the current population are sorted according to their fitness values, and the top K individuals are selected as K candidate mixed precision quantization combinations.

3. The method for determining a hybrid precision quantization strategy as described in claim 1, characterized in that, The step of using the XGBOOST cost model to perform inference evaluation on the K candidate mixed-precision quantization combinations, and determining the optimal mixed-precision quantization combination based on the inference evaluation results of each candidate mixed-precision quantization combination, includes: Set the inference threshold of the neural network model, the inference threshold including an inference accuracy threshold and an inference speed threshold; K candidate mixed precision quantization combinations are input into the XGBOOST cost model, and the predicted mixed precision quantization combination and the corresponding predicted inference result are output. If the predicted inference result meets the inference threshold, the predicted mixed precision quantization combination is determined to be the optimal mixed precision quantization combination; otherwise, the XGBOOST cost model is optimized and trained. The predicted inference result includes the predicted inference accuracy and the predicted inference speed.

4. The method for determining a hybrid precision quantization strategy as described in claim 3, characterized in that, The steps for optimizing and training the XGBOOST cost model include: Based on the predicted mixed precision quantization combination, a deep learning compiler is used to automatically deploy the neural network model. The deployed neural network model is then simulated and inferred on the target hardware to obtain the actual inference accuracy and actual inference speed. The training set is updated using the model structure of the neural network model, the quantized combination of predicted mixed precision, the actual inference accuracy, and the actual inference speed as new training samples. The accuracy error between the predicted inference accuracy and the actual inference accuracy, and the speed error between the predicted inference speed and the actual inference speed are obtained. A loss function is determined based on the accuracy error and the speed error, and the parameters of the XGBOOST cost model are updated based on the loss function. The XGBOOST cost model is iteratively trained using the updated training set until the inference result output by the trained XGBOOST cost model meets the inference threshold, and / or all candidate mixed precision quantization combinations are searched.

5. The method for determining a hybrid precision quantization strategy as described in claim 3, characterized in that, The method further includes: The inference result predicted by the XGBOOST cost model is used as the fitness function of the genetic algorithm.

6. The method for determining a hybrid precision quantization strategy as described in claim 1, characterized in that, The training steps for the XGBOOST cost model include: Based on the model structure and mixed precision quantization combination of the neural network model, a test quantization model is deployed using a deep learning compiler. The test quantization model is then tested on hardware to obtain inference accuracy and inference speed. The model structure, mixed precision quantization combination, inference accuracy, and inference speed of the neural network model are used as training samples for the training set. The training set is then input into the XGBOOST cost model for training, and the XGBOOST cost model is output.

7. A hybrid precision quantization strategy determination device, characterized in that, The device includes: The acquisition module is used to determine the range of quantization bit width selection for each operator in the neural network model based on the multiple different quantization precipitates that the target hardware can support. The determination module is used to generate all candidate mixed-precision quantization combinations based on the various operators of the neural network model and their corresponding quantization bit width selection ranges. These candidate mixed-precision quantization combinations are used to quantize the various operators of the neural network model. The genetic algorithm module is used to select K candidate mixed-precision quantization combinations from all candidate mixed-precision quantization combinations using a genetic algorithm. The model prediction module is used to perform inference evaluation on the K candidate mixed precision quantization combinations using the trained XGBOOST cost model, and to determine the optimal mixed precision quantization combination based on the inference evaluation results of each candidate mixed precision quantization combination. The model deployment module is used to automatically deploy the neural network model to the target hardware based on the optimal combination of mixed precision quantization.

8. The hybrid precision quantization strategy determination device as described in claim 7, characterized in that, The model prediction module is specifically used for: Set the inference threshold of the neural network model, the inference threshold including an inference accuracy threshold and an inference speed threshold; K candidate mixed precision quantization combinations are input into the XGBOOST cost model, and the predicted mixed precision quantization combination and the corresponding predicted inference result are output. If the predicted inference result meets the inference threshold, the predicted mixed precision quantization combination is determined to be the optimal mixed precision quantization combination; otherwise, the XGBOOST cost model is optimized and trained. The predicted inference result includes the predicted inference accuracy and the predicted inference speed.

9. The hybrid precision quantization strategy determination device as described in claim 8, characterized in that, The model prediction module is also specifically used for: Based on the predicted mixed precision quantization combination, a deep learning compiler is used to automatically deploy the neural network model. The deployed neural network model is then simulated and inferred on the target hardware to obtain the actual inference accuracy and actual inference speed. The training set is updated using the model structure of the neural network model, the quantized combination of predicted mixed precision, the actual inference accuracy, and the actual inference speed as new training samples. The accuracy error between the predicted inference accuracy and the actual inference accuracy, and the speed error between the predicted inference speed and the actual inference speed are obtained. A loss function is determined based on the accuracy error and the speed error, and the parameters of the XGBOOST cost model are updated based on the loss function. The XGBOOST cost model is iteratively trained using the updated training set until the inference result output by the trained XGBOOST cost model meets the inference threshold, and / or all candidate mixed precision quantization combinations are searched.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1-6.

Images