Neural network model quantification method and device, medium and equipment

Abstract

The embodiment of the invention discloses a quantification method and device of a neural network model, a medium and equipment, and the method comprises the steps: determining a to-be-quantized first neural network model; pseudo quantization processing is carried out on the first neural network model to obtain a second neural network model, and the second neural network model comprises pseudo quantization nodes and operators; determining a target calibration mode corresponding to each pseudo quantization node; determining a target calibration model corresponding to the second neural network model based on a target calibration mode corresponding to each pseudo quantization node; determining a target quantization parameter corresponding to each operator in the target calibration model; and based on the target calibration model and the target quantization parameters corresponding to the operators, determining a quantization model corresponding to the first neural network model. The embodiment of the invention can reduce the calculation precision loss of the model and improve the precision of the model.

Description

Technical Field

[0001] The present disclosure relates to model quantization technology, and in particular to a quantization method, apparatus, medium, and device for a neural network model. Background Art

[0002] With the continuous advancement of deep learning technology, the structure of neural network models (models) has become increasingly complex and highly parameterized. This poses a challenge to device-side deployment given limited hardware computing resources. To address this issue and reduce model computational complexity, models are typically quantized before deployment. This technique has become a widely used technique in the field of model compression. Model quantization converts floating-point operations in the model into integer operations, thereby reducing the storage and computational overhead of neural network models. Model quantization methods include post-training quantization (PTQ) and quantization-aware training (QAT). Compared to quantization-aware training, post-training quantization avoids re-tuning model parameters and relies only on a small amount of unlabeled data (referred to as calibration data) for calibration. This reduces training costs and has led to its widespread adoption. In related technologies, models quantized after training typically employ a fixed calibration method and a uniform quantization parameter, which can result in significant loss of computational accuracy after model quantization. Summary of the Invention

[0003] Embodiments of the present disclosure provide a method, apparatus, medium, and device for quantizing a neural network model to reduce the loss of computational accuracy of the model during model quantization and improve the computational accuracy of the model.

[0004] A first aspect of an embodiment of the present disclosure provides a quantization method for a neural network model, comprising: determining a first neural network model to be quantized; performing pseudo-quantization processing on the first neural network model to obtain a second neural network model, wherein the second neural network model includes pseudo-quantization nodes and operators; determining a target calibration method corresponding to each of the pseudo-quantization nodes; determining a target calibration model corresponding to the second neural network model based on the target calibration method corresponding to each of the pseudo-quantization nodes; determining a target quantization parameter corresponding to each operator in the target calibration model; and determining a quantization model corresponding to the first neural network model based on the target calibration model and the target quantization parameters corresponding to each of the operators.

[0005] According to a second aspect of the embodiments of the present disclosure, a quantization device for a neural network model is provided, comprising: a first processing module for determining a first neural network model to be quantized; a second processing module for performing pseudo-quantization processing on the first neural network model to obtain a second neural network model, wherein the second neural network model comprises pseudo-quantization nodes and operators; a third processing module for determining a target calibration method corresponding to each of the pseudo-quantization nodes; a fourth processing module for determining a target calibration model corresponding to the second neural network model based on the target calibration method corresponding to each of the pseudo-quantization nodes; a fifth processing module for determining a target quantization parameter corresponding to each operator in the target calibration model; and a sixth processing module for determining a quantization model corresponding to the first neural network model based on the target calibration model and the target quantization parameters corresponding to each of the operators.

[0006] According to a third aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the quantization method of the neural network model described in any of the above embodiments of the present disclosure.

[0007] According to a fourth aspect of the embodiments of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; and the processor for reading the executable instructions from the memory and executing the instructions to implement the quantization method of the neural network model described in any of the above embodiments of the present disclosure.

[0008] A fifth aspect of the embodiments of the present disclosure provides a computer program product. When the instructions in the computer program product are executed by a processor, the quantization method of the neural network model provided by any of the above embodiments of the present disclosure is executed.

[0009] Based on the quantization method, apparatus, medium, and device of the neural network model provided by the above-mentioned embodiments of the present disclosure, during the quantization process of the first neural network model, corresponding target calibration methods are determined for the pseudo-quantization nodes of different layers in the pseudo-quantized second neural network model, so that the pseudo-quantization nodes of different layers can adopt different target calibration methods, that is, the pseudo-quantization nodes of different layers have different quantization threshold selection strategies, ensuring the accuracy and effectiveness of the quantization thresholds of each layer to meet the computational accuracy requirements of the model, and then determining the target calibration model based on the target calibration methods corresponding to each pseudo-quantization node, and determining the corresponding target quantization parameters for each operator in the target calibration model, so that different operators can correspond to different target quantization parameters, thereby obtaining a quantization model with a hybrid calibration method and hybrid quantization parameters. It can be seen that because the quantization model in the present disclosure is based on the target calibration methods determined for different pseudo-quantization nodes in the neural network model and the target quantization parameters determined for operators of different layers, the quantization parameters of operators of different layers can meet the computational accuracy requirements of the model, effectively reducing the computational accuracy loss of the model on the basis of meeting the terminal deployment of the quantization model, thereby improving the computational accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is an exemplary application scenario of the quantization method of the neural network model provided by the present disclosure;

[0011] Figure 2 is a flowchart of a quantization method for a neural network model provided by an exemplary embodiment of the present disclosure;

[0012] Figure 3 is a flowchart of a quantization method for a neural network model provided by another exemplary embodiment of the present disclosure;

[0013] Figure 4 is a flowchart of a target calibration method according to an exemplary embodiment of the present disclosure;

[0014] Figure 5 is a flowchart of a target calibration method according to another exemplary embodiment of the present disclosure;

[0015] Figure 6 is a flowchart of a quantization method for a neural network model provided by yet another exemplary embodiment of the present disclosure;

[0016] Figure 7 is a flowchart of a quantization method for a neural network model provided by another exemplary embodiment of the present disclosure;

[0017] Figure 8 is a schematic diagram of a flow chart of determining the quantization sensitivity of an operator under a preset quantization parameter provided by an exemplary embodiment of the present disclosure;

[0018] Figure 9 is a schematic diagram of a flow chart of determining the quantization sensitivity of an operator under a preset quantization parameter provided by another exemplary embodiment of the present disclosure;

[0019] Figure 10 is a schematic diagram of a flow chart of determining the quantization sensitivity of an operator under a preset quantization parameter provided by yet another exemplary embodiment of the present disclosure;

[0020] Figure 11 is a flowchart of a method for searching for a calibration threshold layer by layer provided by an exemplary embodiment of the present disclosure;

[0021] Figure 12 This is a flowchart of a method for searching quantization accuracy layer by layer provided by an exemplary embodiment of the present disclosure;

[0022] Figure 13 is a flowchart of a quantization method for a neural network model provided by an exemplary embodiment of the present disclosure;

[0023] Figure 14 1 is a schematic structural diagram of a quantization device for a neural network model provided by an exemplary embodiment of the present disclosure;

[0024] Figure 15 is a structural diagram of a quantization device for a neural network model provided by another exemplary embodiment of the present disclosure;

[0025] Figure 16 This is a structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] To explain the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. It should be understood that the present disclosure is not limited to the example embodiments.

[0027] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0028] Overview of the Disclosure

[0029] Among the technologies related to model quantization, model quantization methods can include post-training quantization (PTQ) and quantization-aware training (QAT). For post-training quantization, a fixed calibration method and unified quantization parameters are usually used for quantization. The post-training quantization process involves a large number of quantization parameters, including quantization accuracy, quantization granularity, quantization method, etc. Different calibration methods and different quantization parameters may have different quantization accuracy for different layers in the post-training quantized model. Fixed calibration methods and unified quantization parameters are difficult to meet the computational accuracy requirements of quantization at different layers, which can easily cause a large loss of computational accuracy after model quantization.

[0030] Exemplary Overview

[0031] Figure 1 This is an exemplary application scenario of the quantization method of the neural network model provided by the present disclosure. Figure 1 As shown, the neural network model 11 can be any neural network model, for example, the neural network model 11 can be any model in the target detection model, semantic segmentation model, speech recognition model, classification model, etc. The network structure of the neural network model 11 can be, for example, a convolutional neural network, a recurrent neural network, a Transformer neural network, etc., without specific limitation. The neural network model 11 can include multiple operators, such as Figure 1 Operators 111, 112, ..., and 11n, where n is a positive integer, are each operand that can be used to perform a specific function, such as multiplication, addition, multiplication-addition, or convolution. When quantizing the trained neural network model 11, the neural network model quantization apparatus 12 of the disclosed embodiment can be used to execute the neural network model quantization method of the disclosed embodiment, quantizing the neural network model 11 to obtain a quantized model corresponding to the neural network model 11. Specifically, the neural network model 11 is determined as a first neural network model to be quantized; the first neural network model is pseudo-quantized to obtain a second neural network model, which includes pseudo-quantization nodes and operators; a target calibration method corresponding to each pseudo-quantization node is determined; based on the target calibration method corresponding to each pseudo-quantization node, a target calibration model corresponding to the second neural network model is determined; a target quantization parameter corresponding to each operator in the target calibration model is determined; and based on the target quantization parameter corresponding to the target calibration model and each operator, a quantization model corresponding to the first neural network model is determined. The quantized model is then deployed on a terminal device 13. The terminal device 13 can be an in-vehicle terminal, a mobile phone, a tablet, or the like.

[0032] The embodiment of the present disclosure determines the corresponding target calibration method for the pseudo-quantization nodes of different layers in the second neural network model after pseudo-quantization, so that the pseudo-quantization nodes of different layers can adopt different target calibration methods, that is, the pseudo-quantization nodes of different layers have different quantization threshold selection strategies, ensuring the accuracy and effectiveness of the quantization threshold of each layer to meet the calculation accuracy requirements of the model, and then determining the target calibration model based on the target calibration method corresponding to each pseudo-quantization node, and determining the corresponding target quantization parameter for each operator in the target calibration model, so that different operators can correspond to different target quantization parameters, thereby obtaining a quantization model with a mixed calibration method and mixed quantization parameters. It can be seen that because the quantization model in the present disclosure is based on the target calibration method determined by different pseudo-quantization nodes in the neural network model, and the target quantization parameters determined for operators of different layers, the quantization parameters of operators of different layers can meet the calculation accuracy requirements of the model, and on the basis of meeting the terminal deployment of the quantization model, effectively reduce the calculation accuracy loss of the model, thereby improving the calculation accuracy of the model.

[0033] Exemplary Methods

[0034] Figure 2 This is a flow chart of a quantization method of a neural network model provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as servers, terminal devices, etc. Figure 2 As shown, the method of the embodiment of the present disclosure may include the following steps:

[0035] Step 210: Determine a first neural network model to be quantized.

[0036] The first neural network model (or original model) can be any neural network model, see the above-mentioned neural network model 11. The first neural network model is a pre-trained neural network model. The first neural network model is a floating-point model.

[0037] Step 220: Perform pseudo-quantization processing on the first neural network model to obtain a second neural network model, where the second neural network model includes pseudo-quantization nodes and operators.

[0038] Among them, the first neural network model includes operators for completing various calculations, such as convolution operators, addition operators, multiplication operators, etc. Pseudo-quantization processing refers to the operation of inserting a pseudo-quantization node (or calibration node) before the operator that supports quantization in the first neural network model. The pseudo-quantization node can first quantize the data input to the pseudo-quantization node, and then dequantize it to simulate the quantization behavior of the hardware and perceive the quantization error. Operators that need to be inserted with pseudo-quantization nodes may include but are not limited to convolution, activation functions, multiplication and addition operators, etc., and are not specifically limited. They can be set according to actual needs. The pseudo-quantization node can sense the dynamic range of activation values ​​and weights, determine the calibration threshold based on the calibration method, and then simulate the behavior of hardware introducing errors through quantization and dequantization based on the calibration threshold. The second neural network model includes the operators in the first neural network model and the pseudo-quantization nodes inserted before the operators that support quantization in the first neural network model.

[0039] Step 230: Determine the target calibration method corresponding to each pseudo quantization node.

[0040] Calibration methods can include KL (Kullback-Leibler) calibration, MAX (maximum absolute value) calibration, and Percentile calibration. The target calibration method corresponding to any pseudo-quantization node can be one of the aforementioned calibration methods. The target calibration methods corresponding to any two pseudo-quantization nodes can be the same or different.

[0041] KL calibration, also known as KL divergence calibration, calculates the KL divergence (i.e., the amount of information loss) of the activation value distribution before and after quantization to find the range of values ​​that minimizes the quantization error (i.e., the quantization interval, or quantization range). By dynamically adjusting the quantization interval, the quantized distribution is made as close as possible to the original floating-point data distribution. The final adjusted quantization interval boundary values ​​(i.e., upper and lower limits) are the calibration thresholds.

[0042] MAX calibration directly takes the maximum absolute value MAX of the activation value in the calibration data as the upper limit of the quantization range. For symmetric quantization, the lower limit of the quantization range can be symmetrically set to -MAX. For asymmetric quantization, the minimum value can be separately counted as the lower limit.

[0043] Percentile calibration is to determine the quantization range based on the percentile value (such as 99% or other values) of the activation value in the calibration data, and then determine the calibration threshold to exclude the influence of extreme outliers.

[0044] In some optional embodiments, the target calibration method corresponding to each pseudo-quantization node can be determined according to a pre-configured calibration method determination rule. The pre-configured calibration method determination rule may include a rule for the user to set the target calibration method, a rule for searching the calibration method layer by layer to determine the target calibration method, a rule for using a preset default calibration method as the target calibration method, etc., which can be specifically set according to actual needs. The rule for the user to set the target calibration method means that when the model is quantized, if the user specifies the corresponding target calibration method for each pseudo-quantization node according to the needs, the target calibration method corresponding to each pseudo-quantization node is determined based on the target calibration method set by the user, that is, the user can dynamically configure the target calibration method corresponding to each pseudo-quantization node in real time through the software according to the model quantization needs. The rule for determining the target calibration method by searching the calibration threshold layer by layer means that when the user does not specify the target calibration method or the user chooses to search the calibration method layer by layer, the optimal target calibration method is determined by searching multiple calibration methods layer by layer based on the quantization errors corresponding to each calibration method. The preset default calibration method is a pre-set calibration method. For example, the preset default calibration method is MAX calibration. In actual application, the user can specify the target calibration method as Percentile calibration.

[0045] Step 240 : Determine a target calibration model corresponding to the second neural network model based on the target calibration methods corresponding to the pseudo quantization nodes.

[0046] Among them, the target calibration model refers to the model after each pseudo-quantization node in the second neural network model is calibrated using the corresponding target calibration method. After determining the target calibration method corresponding to each pseudo-quantization node, the target calibration model corresponding to the second neural network model can be determined based on the target calibration method corresponding to each pseudo-quantization node, that is, the calibration method corresponding to each pseudo-quantization node is determined to be the target calibration method corresponding to each pseudo-quantization node. For example, the target calibration method used by pseudo-quantization node a is KL calibration, the target calibration method used by pseudo-quantization node b is MAX calibration, and the target calibration method used by pseudo-quantization node c is KL calibration.

[0047] Step 250: Determine the target quantization parameters corresponding to each operator in the target calibration model.

[0048] The target quantization parameters corresponding to any operator may include at least one of quantization precision, quantization granularity, and quantization mode. Quantization precision refers to the bit width of the values ​​used in the quantization process. For example, quantization precisions include INT2, INT4, INT8, INT16, FLOAT16, and so on. Quantization granularity refers to the adjustable range of the quantization parameter, which determines the flexibility and accuracy retention of the model quantization. For example, quantization granularity can include per-layer (per-Tensor) quantization and per-channel (per-Channel) quantization. Per-layer quantization refers to the use of uniform scaling factors, zero points, and other parameters (also known as quantization sub-parameters) across all layers. Per-channel quantization refers to setting quantization sub-parameters separately for each channel, which helps improve model calculation accuracy. A layer is the basic structural unit of a neural network, performing a series of calculations on input data and passing the calculation results to the next layer, such as the input layer, convolutional layer, pooling layer, or fully connected layer. A channel is the depthwise dimension of a feature map or tensor, such as the three color channels of an RGB image or the multi-channel feature maps of the input / output tensors of a convolutional layer. Quantization methods can include symmetric quantization and asymmetric quantization. Symmetric quantization means that the quantization range is symmetric around the zero point and is suitable for scenarios with symmetrical activation value distribution, such as the Tanh activation function (or activation operator). Asymmetric quantization refers to quantization that introduces a zero-point offset, that is, the quantization range is not symmetric around the zero point and is suitable for scenarios with asymmetric activation value distribution, such as the Sigmoid activation function.

[0049] In some optional embodiments, the target quantization parameter corresponding to each operator can be determined based on preconfigured quantization parameter determination rules. Preconfigured quantization parameter determination rules may include rules for user-set quantization parameters, rules for layer-by-layer quantization parameter search to determine the target quantization parameter corresponding to each operator, and rules for using a preset default quantization parameter as the target quantization parameter. User-set quantization parameter rules refer to the user specifying the corresponding quantization parameter for each operator based on quantization requirements, and then determining the target quantization parameter corresponding to each operator based on the user-set quantization parameter. That is, when performing model quantization, the user can dynamically configure the target quantization parameter corresponding to each operator based on requirements. Layer-by-layer quantization parameter search refers to the process of determining the optimal target quantization parameter for each operator by layer-by-layer searching for multiple quantization parameters, if the user does not specify a quantization parameter, based on the quantization error corresponding to each quantization parameter. A preset default quantization parameter is a pre-set quantization parameter. For example, the preset default quantization precision is INT4. In actual applications, the user can specify the target quantization precision of the operator as INT8.

[0050] Step 260: Determine a quantization model corresponding to the first neural network model based on the target calibration model and the target quantization parameters corresponding to each operator.

[0051] Among them, after determining the target calibration model and the target quantization parameters corresponding to each operator, the target quantization parameters corresponding to each operator can be used as the quantization parameters corresponding to each operator in the target calibration model to obtain a quantization model. For example, the pseudo quantization node a adopts KL calibration, the pseudo quantization node b adopts MAX calibration, the quantization accuracy adopted by operator A is INT8-INT4, the quantization granularity is layer-by-layer quantization, and the quantization method is symmetric quantization. The quantization accuracy adopted by operator B is INT8-INT8, the quantization granularity is channel-by-channel quantization, and the quantization method is asymmetric quantization. INTi-INTj means that the quantization accuracy of the activation value is INTi, and the quantization accuracy of the weight value is INTj. Both i and j are positive integers. i and j can be equal or unequal. That is to say, the quantization accuracy of the activation value and the quantization accuracy of the weight value of any operator can be the same or different to meet the different accuracy requirements of the activation value and the weight value.

[0052] The quantization method of the neural network model provided in this embodiment, during the quantization process of the first neural network model, determines corresponding target calibration methods for the pseudo-quantization nodes of different layers in the pseudo-quantized second neural network model, so that the pseudo-quantization nodes of different layers can adopt different target calibration methods, that is, the pseudo-quantization nodes of different layers have different quantization threshold selection strategies, ensuring the accuracy and effectiveness of the quantization thresholds of each layer to meet the computational accuracy requirements of the model, and then determines a target calibration model based on the target calibration methods corresponding to each pseudo-quantization node, and determines corresponding target quantization parameters for each operator in the target calibration model, so that different operators can correspond to different target quantization parameters, thereby obtaining a quantization model with a hybrid calibration method and hybrid quantization parameters. It can be seen that because the quantization model in the present disclosure is based on the target calibration methods determined for different pseudo-quantization nodes in the neural network model and the target quantization parameters determined for operators of different layers, the quantization parameters of operators of different layers can meet the computational accuracy requirements of the model, effectively reducing the computational accuracy loss of the model while meeting the terminal deployment of the quantization model, thereby improving the computational accuracy of the model.

[0053] Figure 3 It is a flowchart of a quantization method of a neural network model provided by another exemplary embodiment of the present disclosure.

[0054] In some optional embodiments, in the above Figure 2 Based on the embodiment shown, Figure 3 As shown, step 230 of determining the target calibration mode corresponding to each pseudo quantization node may include:

[0055] Step 2310: Obtain calibration data for model quantization.

[0056] The calibration data may be data extracted from the training set and / or validation set corresponding to the first neural network model. The training set is a data set used in the training process of the first neural network model. The training set may include multiple training samples and label data corresponding to each training sample. The validation set is a data set used to verify whether the trained model meets the training end conditions. The validation set may include multiple training samples and label data corresponding to each training sample. Some data may be extracted from the training set and / or validation set as calibration data in a random manner or other manner.

[0057] Step 2320: Process the calibration data based on the first neural network model to obtain first output data and input data corresponding to each operator in the first neural network model.

[0058] Among them, the first neural network model is the original model that has not been quantized. For example, the first neural network model is a floating-point model or a high-precision fixed-point model, which needs to be quantized into a fixed-point model with lower precision. The calibration data is first processed based on the first neural network model to obtain the first output data of the calibration data under the original model and the input data corresponding to each operator in the original model. The first output data is the output result obtained by the first neural network model to infer the calibration data. The input data corresponding to each operator in the first neural network model refers to the data input to each operator during the inference process of the calibration data. For example, the input data of the input operator (i.e., the input layer) is the calibration data, and the input data of any intermediate operator is the output data of the previous operator of the operator.

[0059] Step 2330 : Based on the input data corresponding to each preset calibration method and each operator in at least one preset calibration method, determine the calibration threshold corresponding to each pseudo quantization node in each preset calibration method.

[0060] Among them, at least one preset calibration method may include at least one of the calibration methods such as KL calibration, MAX calibration, and Percentile calibration. The calibration threshold corresponding to any pseudo-quantization node under any preset calibration method refers to the boundary value of the input data of the operator corresponding to the pseudo-quantization node, which may include an upper limit value (or upper limit value) and a lower limit value (or lower limit value). It can be understood that since the pseudo-quantization node is a node inserted before the operator, for example, the pseudo-quantization node b is inserted before the operator B, the output data of the operator C before the operator B is used as the input data of the pseudo-quantization node b, and the output data of the pseudo-quantization node b is used as the input data of the operator B, then the operator B in the first neural network model is the operator after the pseudo-quantization node b in the second neural network model.

[0061] In some optional embodiments, based on the second neural network model, the quantization processing step can be skipped at the pseudo-quantization node to process the calibration data, and the input data corresponding to each operator can be obtained through the pseudo-quantization node. Skipping quantization processing refers to skipping quantization and dequantization of the data input to the pseudo-quantization node and directly outputting floating-point data, that is, capturing the floating-point input data of the operator for analysis and determination of the input data range of the operator.

[0062] In some optional embodiments, for any pseudo-quantization node, the input data range corresponding to the pseudo-quantization node can be statistically analyzed according to each preset calibration method to obtain the calibration threshold corresponding to the pseudo-quantization node under each preset calibration method. For example, taking MAX calibration as an example, the calibration data includes N samples, and the input data corresponding to the pseudo-quantization node b includes the input data corresponding to the N samples. By calculating the maximum absolute value MAX of the input data corresponding to the N samples, the calibration threshold corresponding to the pseudo-quantization node b under the MAX calibration method is obtained.

[0063] Step 2340 : Based on the calibration thresholds and first output data corresponding to each pseudo quantization node under each preset calibration mode, a layer-by-layer search for the calibration threshold is performed to determine the target calibration mode corresponding to each pseudo quantization node.

[0064] Among them, since each pseudo-quantization node has a corresponding calibration threshold under each preset calibration method, the calibration threshold is searched layer by layer, that is, the calibration method is searched layer by layer. Each pseudo-quantization node corresponds to a layer, that is, for each pseudo-quantization node in turn, the pseudo-quantization node is determined to be quantized separately, and the other parts all use the output data of the model under the original operator in the first neural network model. According to the output data and the first output data, the quantization error corresponding to the pseudo-quantization node can be determined, and then based on the quantization error of the pseudo-quantization node under each preset calibration method, the target calibration method that minimizes the quantization error is determined.

[0065] In the embodiment of the present disclosure, the target calibration method corresponding to each pseudo-quantization node is determined by searching the calibration threshold layer by layer, so that each pseudo-quantization node can adopt the optimal target calibration method, thereby effectively reducing the computational accuracy loss of the quantization model relative to the original model.

[0066] Figure 4 It is a flowchart of a target calibration method provided by an exemplary embodiment of the present disclosure.

[0067] In some optional embodiments, such as Figure 4As shown, step 2340, based on the calibration thresholds and first output data corresponding to each pseudo quantization node under each preset calibration mode, determines the target calibration mode corresponding to each pseudo quantization node by searching the calibration threshold layer by layer, which may include:

[0068] Step 310 : determining calibration models corresponding to respective preset calibration modes based on calibration thresholds corresponding to respective pseudo quantization nodes in respective preset calibration modes.

[0069] Among them, the calibration model determined by the calibration threshold obtained by KL calibration can be called the KL calibration model, the calibration model determined by the calibration threshold obtained by MAX calibration can be called the MAX calibration model, and the calibration model determined by the calibration threshold obtained by Percentile calibration can be called the Percentile calibration model. The calibration model corresponding to any preset calibration method means that each pseudo-quantization node adopts the preset calibration method, that is, in the KL calibration model, each pseudo-quantization node adopts the calibration threshold obtained by KL calibration as the boundary value of the quantization range.

[0070] In some optional embodiments, for each pseudo-quantization node, the scaling factor, zero point, and other quantization sub-parameters corresponding to the pseudo-quantization node can be calculated based on the calibration threshold corresponding to each preset calibration method. Optionally, in the process of determining the target calibration method corresponding to each pseudo-quantization node, pre-configured quantization parameters can be used to participate in the quantization process to search for the optimal target calibration method under the pre-configured quantization parameters. The pre-configured quantization parameters may, for example, include pre-configured quantization accuracy, pre-configured quantization granularity, pre-configured quantization method, etc. For example, the quantization accuracy adopts int8, the quantization granularity adopts layer-by-layer quantization, and the quantization method adopts symmetric quantization.

[0071] Step 320: Determine second output data corresponding to each calibration model based on the calibration data.

[0072] The second output data corresponding to each calibration model refers to the output data of the model obtained by inferring the calibration data through each calibration model. That is, the calibration data is input to the calibration model and the calibration model outputs the data after inference. For example, the corresponding second output data is obtained by inferring the KL calibration model and the MAX calibration model.

[0073] Step 330 : Determine a first calibration model from the calibration models based on the second output data and the first output data corresponding to the calibration models.

[0074] Among them, the first output data is the output data of the calibration data after inference by the unquantized original model, and the second output data is the output data of the calibration data after inference by the calibration model. The pseudo-quantization nodes in the calibration model are quantized and dequantized. Therefore, the error between the second output data corresponding to the calibration model and the first output data can represent the quantization error of the calibration model relative to the original model (i.e., the first neural network model). Based on the principle of minimizing the quantization error, the first calibration model (or reference calibration model) can be determined from each calibration model. The quantization error can be determined by one or more of cosine similarity (CS), mean squared error (MSE), mean relative error (MRE), signal-to-noise ratio (SNR), etc.

[0075] Step 340 : Determine the first quantization sensitivity corresponding to each pseudo quantization node based on the calibration data and the first calibration model.

[0076] Among them, the first quantization sensitivity corresponding to any pseudo-quantization node refers to the degree of influence of the quantization of the pseudo-quantization node on the calculation accuracy of the model when the pseudo-quantization node is quantized separately in the first calibration model. The quantization sensitivity corresponding to different pseudo-quantization nodes characterizes the degree of deviation of the output data of different layers relative to the first output data after quantization. Taking the pseudo-quantization node b as an example, quantizing the pseudo-quantization node b separately means that only the pseudo-quantization node b performs quantization and inverse quantization to simulate the quantization processing of the hardware, and other pseudo-quantization nodes except the pseudo-quantization node b are not quantized (that is, the quantization processing of other pseudo-quantization nodes is turned off), which is equivalent to other pseudo-quantization nodes directly transmitting the output data of their previous operator to their next operator, maintaining data consistent with the first neural network model, so as to effectively determine the loss of model calculation accuracy caused by the quantization of the pseudo-quantization node b.

[0077] In some optional embodiments, the first quantization sensitivity may be determined by separately quantizing a quantization error between the model output data of the pseudo-quantization node and the first output data. The quantization error may be determined by one or more of cosine similarity, mean square error, mean relative error, signal-to-noise ratio, and the like.

[0078] Step 350 : Determine a target calibration method corresponding to each pseudo quantization node based on the first quantization sensitivity corresponding to each pseudo quantization node and the second calibration model in each calibration model.

[0079] Among them, the second calibration model is the calibration model other than the first calibration model among the calibration models. The first quantization sensitivity corresponding to each pseudo-quantization node characterizes the influence degree of each pseudo-quantization node on the model calculation accuracy. The higher the sensitivity, the greater the quantization error introduced.

[0080] In some optional embodiments, the number of the second calibration models can be one or more. For example, if the first calibration model is determined to be the KL calibration model, the second calibration models can include the MAX calibration model, the Percentile calibration model, etc.

[0081] In some optional embodiments, for each pseudo-quantization node, based on the calibration data and each second calibration model, the quantization sensitivity of separately quantizing this pseudo-quantization node in the second calibration model can be determined. Then, in combination with the first quantization sensitivity corresponding to each pseudo-quantization node and the quantization sensitivity corresponding to each pseudo-quantization node in the second calibration model, the target calibration method corresponding to each pseudo-quantization node can be determined. For example, for each pseudo-quantization node, the calibration method corresponding to the calibration model with the smallest quantization sensitivity is determined as the target calibration method corresponding to this pseudo-quantization node. Exemplarily, the KL calibration model is the first calibration model, the second calibration models include the MAX calibration model and the Percentile calibration model, the first quantization sensitivity of the pseudo-quantization node b corresponding to the KL calibration model is e1, the quantization sensitivities corresponding to the MAX calibration model and the Percentile calibration model are e2 and e3 respectively, and e1 < e3 < e2, then the target calibration method corresponding to the pseudo-quantization node b is KL calibration.

[0082] In some optional embodiments, in order to further reduce the calculation amount, for the second calibration model, only the quantization sensitivities corresponding to some pseudo-quantization nodes with higher first quantization sensitivities can be calculated. For the pseudo-quantization nodes with lower first quantization sensitivities, it indicates that the quantization error introduced by the first calibration model is smaller, and the calibration method corresponding to the first calibration model can be used as the target calibration method for these pseudo-quantization nodes with lower first quantization sensitivities.

[0083] In an embodiment of the present disclosure, the second output data corresponding to each calibration model is determined through calibration data, and then the second output data of the calibration data under the calibration model reasoning and the first output data of the calibration data under the original model reasoning are combined to determine the optimal first calibration model from each calibration model, and then the first quantization sensitivity of each pseudo-quantization node corresponding to the first calibration model is determined. Since the first quantization sensitivity characterizes the sensitivity of each pseudo-quantization node to the loss of model calculation accuracy, it is possible to determine whether the calibration method of the first calibration model meets the accuracy loss requirement based on the first quantization sensitivity, which is convenient for determining the target calibration method corresponding to some pseudo-quantization nodes. Further combined with the second calibration model, the target calibration method corresponding to each pseudo-quantization node can be effectively determined, so that the accuracy loss introduced to the model by the target calibration method adopted by each pseudo-quantization node is minimized, thereby achieving the quantization of the model on the basis of ensuring that the accuracy loss of the model is small.

[0084] In some optional embodiments, determining the first calibration model from each calibration model based on the second output data and the first output data corresponding to each calibration model in step 330 may include:

[0085] Based on the second output data and the first output data corresponding to each calibration model, a first average quantization error corresponding to each calibration model is determined; based on the first average quantization error corresponding to each calibration model, a first calibration model is determined from each calibration model.

[0086] The first average quantization error corresponding to any calibration model refers to the average quantization error of the calibration model relative to the first neural network model.

[0087] In some optional embodiments, for any calibration model, the second output data corresponding to the calibration model includes the second output data of each sample in the calibration data under the inference of the calibration model, and the first output data includes the first output data of each sample in the calibration data under the inference of the first neural network model. A first average quantization error corresponding to the calibration model can be determined based on the second output data and the first output data corresponding to each sample. Optionally, the first average quantization error can be determined based on one or more of the following: cosine similarity, mean square error, mean relative error, signal-to-noise ratio, etc., between the second output data and the first output data corresponding to each sample. A higher cosine similarity indicates a smaller first average quantization error, while a lower cosine similarity indicates a larger first average quantization error. A smaller mean square error indicates a smaller first average quantization error, while a larger mean square error indicates a larger first average quantization error. A smaller mean relative error indicates a smaller first average quantization error, while a larger mean relative error indicates a larger first average quantization error. A higher signal-to-noise ratio indicates a smaller first average quantization error, while a lower signal-to-noise ratio indicates a larger first average quantization error. Optionally, multiple metrics such as cosine similarity, mean square error, mean relative error, signal-to-noise ratio, etc. may be used and weighted by preconfigured weights to determine the first average quantization error.

[0088] In some optional embodiments, after obtaining the first average quantization errors corresponding to each calibration model, the calibration model with the smallest first average quantization error can be determined as the first calibration model based on the size relationship of the first average quantization errors corresponding to each calibration model.

[0089] In an embodiment of the present disclosure, the first calibration model is determined by the average quantization error of each calibration model relative to the first neural network model, so that the first calibration model is the calibration model with the smallest average quantization error, providing an effective benchmark calibration model for reducing the calculation amount of subsequent quantization sensitivity.

[0090] In some optional embodiments, based on any of the above embodiments, determining the first quantization sensitivity corresponding to each pseudo quantization node based on the calibration data and the first calibration model in step 340 may include:

[0091] For each pseudo-quantization node, based on the calibration data and the first calibration model, the third output data of the pseudo-quantization node is determined to be individually quantized; based on the third output data and the first output data, the second average quantization error corresponding to the pseudo-quantization node is determined as the first quantization sensitivity corresponding to the pseudo-quantization node.

[0092] Among them, the individual quantization of the pseudo-quantization node can refer to the above-mentioned embodiment. The third output data of the individual quantization of the pseudo-quantization node is the model output data obtained by processing the calibration data while only performing quantization processing on the pseudo-quantization node in the first calibration model and skipping quantization processing on other pseudo-quantization nodes. The specific principle of determining the second average quantization error is similar to the above-mentioned determination of the first average quantization error, that is, the second average quantization error corresponding to the pseudo-quantization node can be determined based on the cosine similarity, mean square error, average relative error, signal-to-noise ratio, etc. of the third output data and the first output data corresponding to each sample, which will not be elaborated here. The second average quantization error corresponding to each pseudo-quantization node is used as the first quantization sensitivity corresponding to the pseudo-quantization node.

[0093] In an embodiment of the present disclosure, the average quantization error corresponding to each pseudo-quantization node is determined by separately quantizing the third output data obtained from each pseudo-quantization node and the first output data of the original model, which is used as the first quantization sensitivity corresponding to the pseudo-quantization node. This effectively determines the first quantization sensitivity corresponding to the pseudo-quantization node and provides an effective sensitivity reference for determining the target calibration method of each pseudo-quantization node.

[0094] Figure 5 3 is a flowchart of a target calibration method according to another exemplary embodiment of the present disclosure.

[0095] In some optional embodiments, based on any of the above embodiments, Figure 5 As shown, step 350 of determining the target calibration mode corresponding to each pseudo quantization node based on the first quantization sensitivity corresponding to each pseudo quantization node and the second calibration model in each calibration model may include:

[0096] Step 3510: Based on the first quantization sensitivity corresponding to each pseudo quantization node, determine at least one first pseudo quantization node that meets a first preset condition from each pseudo quantization node.

[0097] The first preset condition may be k pseudo-quantization nodes with larger first quantization sensitivities, i.e., the TOP k pseudo-quantization nodes sorted from largest to smallest by first quantization sensitivity, or the first quantization sensitivity being greater than a first threshold, where k is a positive integer and may be a preset value or a user-set value. A higher first quantization sensitivity indicates a greater error introduced by the corresponding pseudo-quantization node. For these pseudo-quantization nodes, an optimal calibration method needs to be determined from multiple calibration methods as a target calibration method.

[0098] Step 3520 : Determine that the target calibration mode corresponding to the second pseudo quantization node among the pseudo quantization nodes is the calibration mode corresponding to the first calibration model, and the second pseudo quantization node is a pseudo quantization node that does not meet the first preset condition.

[0099] Among them, the second pseudo-quantization node is a pseudo-quantization node other than the first pseudo-quantization node in each pseudo-quantization node. The number of second pseudo-quantization nodes can be one or more, which is determined by the total number of pseudo-quantization nodes and the number of first pseudo-quantization nodes. The second pseudo-quantization node is a pseudo-quantization node that does not meet the first preset condition, that is, the model calculation accuracy loss introduced by the quantization processing of the second pseudo-quantization node is small. For these second pseudo-quantization nodes, since the error is small, in order to reduce the calculation amount of quantization sensitivity, the target calibration method corresponding to each second pseudo-quantization node can be determined to be the calibration method corresponding to the first calibration model. For example, if the first calibration model is a KL calibration model, the target calibration method corresponding to each second pseudo-quantization node is KL calibration.

[0100] Step 3530 : Determine the second quantization sensitivity of each first pseudo quantization node on each second calibration model based on the calibration data and the second calibration model.

[0101] Among them, since the first quantization sensitivity corresponding to each first pseudo-quantization node is relatively high, that is, the error introduced by the calibration method corresponding to the first calibration model of each first pseudo-quantization node is relatively large, therefore, it is necessary to further combine the calibration data and each second calibration model to determine the second quantization sensitivity of each first pseudo-quantization node on each second calibration model. The second quantization sensitivity of any first pseudo-quantization node on any second calibration model refers to the quantization sensitivity of the first pseudo-quantization node quantized separately in the second calibration model, which is similar to the first quantization sensitivity. The difference is that the calibration method of the first pseudo-quantization node in the second calibration model is different from that in the first calibration model, which will not be elaborated here. Based on this, the second quantization sensitivity of each pseudo-quantization node on each second calibration model can be determined.

[0102] Step 3540 : Determine a target calibration method corresponding to each first dummy quantization node based on the second quantization sensitivity of each first dummy quantization node on each second calibration model and the first quantization sensitivity corresponding to each first dummy quantization node.

[0103] Among them, for any first pseudo-quantization node, the target calibration method corresponding to the first pseudo-quantization node can be determined based on the magnitude relationship between the second quantization sensitivity of the first pseudo-quantization node on each second calibration model and the first quantization sensitivity of the first pseudo-quantization node on the first calibration model. For example, the calibration method of the calibration model corresponding to the quantization sensitivity that is the smallest among the second quantization sensitivities and the first quantization sensitivity corresponding to the first pseudo-quantization node is determined as the target calibration method corresponding to the first pseudo-quantization node. For example, the KL calibration model is the first calibration model, the second calibration models include the MAX calibration model and the Percentile calibration model, the first quantization sensitivity of the first pseudo-quantization node b on the KL calibration model is e1, the quantization sensitivities on the MAX calibration model and the Percentile calibration model are e2 and e3 respectively, and e2 < e3 < e1. Then the target calibration method corresponding to the pseudo-quantization node b is the calibration method of the MAX calibration model corresponding to e2, that is, the target calibration method is MAX calibration.

[0104] Step 3550, based on the target calibration method corresponding to the second pseudo-quantization node and the target calibration methods corresponding to each first pseudo-quantization node respectively, obtain the target calibration methods corresponding to each pseudo-quantization node respectively.

[0105] Among them, the second pseudo-quantization node is the pseudo-quantization node other than the first pseudo-quantization node among all pseudo-quantization nodes. Therefore, after determining the target calibration methods corresponding to each first pseudo-quantization node respectively and the target calibration methods corresponding to each second pseudo-quantization node respectively, the target calibration methods corresponding to each pseudo-quantization node respectively are obtained.

[0106] In the embodiments of the present disclosure, based on determining the optimal first calibration model, one or more first pseudo-quantization nodes with the largest quantization sensitivity are determined from all pseudo-quantization nodes based on the first quantization sensitivity of each pseudo-quantization node on the first calibration model. It is only necessary to determine the second quantization sensitivity of these first pseudo-quantization nodes on each second calibration model. For the second pseudo-quantization node, since the quantization sensitivity is relatively low when using the calibration method corresponding to the first calibration model, the calibration method of the first calibration model can be directly determined as the target calibration method corresponding to the second pseudo-quantization node, effectively reducing the calculation amount of the second quantization sensitivity and improving the search efficiency.

[0107] In some optional embodiments, determining the second quantization sensitivity of each first pseudo-quantization node on each second calibration model based on the calibration data and the second calibration model in step 3530 may include:

[0108] For each second calibration model and each first pseudo quantization node in each first pseudo quantization node, based on the calibration data and the second calibration model, determine that the second calibration model alone quantizes the fourth output data of the first pseudo quantization node; based on the second calibration model, the fourth output data and the first output data of the first pseudo quantization node are individually quantized, and the third average quantization error of the first pseudo quantization node on the second calibration model is determined as the second quantization sensitivity of the first pseudo quantization node on the second calibration model.

[0109] The specific operation of determining the fourth output data is similar to the specific operation of determining the third output data in the above embodiment, and the specific operation of determining the third average quantization error is similar to the specific operation of determining the second average quantization error, which will not be repeated here.

[0110] In an embodiment of the present disclosure, for each first pseudo-quantization node, the average quantization error of each first pseudo-quantization node on each second calibration model is further determined as the second quantization sensitivity based on the calibration data and each second calibration model, thereby providing effective sensitivity reference data for further determining the target calibration method corresponding to each first pseudo-quantization node.

[0111] In some optional embodiments, determining the target calibration mode corresponding to each first pseudo quantization node based on the second quantization sensitivity of each first pseudo quantization node on each second calibration model and the first quantization sensitivity corresponding to each first pseudo quantization node in step 3540 may include:

[0112] For each first pseudo quantization node, based on the second quantization sensitivity of the first pseudo quantization node on each second calibration model and the first quantization sensitivity of the first pseudo quantization node on the first calibration model, determine the calibration model corresponding to the minimum quantization sensitivity; and determine the calibration method of the calibration model corresponding to the minimum quantization sensitivity as the target calibration method of the first pseudo quantization node.

[0113] Among them, the minimum quantization sensitivity (i.e., the lowest) indicates that the loss of model calculation accuracy introduced by quantizing the first pseudo-quantization node alone is minimal. Based on this, the calibration method of the calibration model corresponding to the minimum quantization sensitivity is determined as the target calibration method of the first pseudo-quantization node, so that the loss of model calculation accuracy brought about by the target calibration method of the first pseudo-quantization node is minimized.

[0114] In an embodiment of the present disclosure, the calibration method corresponding to the minimum quantization sensitivity of the first pseudo-quantization node individually quantized in each calibration model is used as the target calibration method of the first pseudo-quantization node, so that the target calibration method adopted by the first pseudo-quantization node is the calibration method that minimizes the loss of model calculation accuracy, thereby ensuring the optimization of the target calibration method of each first pseudo-quantization node.

[0115] Figure 6 It is a flowchart of a quantization method of a neural network model provided by another exemplary embodiment of the present disclosure.

[0116] In some optional embodiments, based on any of the above embodiments, Figure 6 As shown, step 240 of determining the target calibration model corresponding to the second neural network model based on the target calibration modes corresponding to the pseudo-quantization nodes may include:

[0117] Step 2410: Determine a hybrid calibration model based on the second neural network model and the target calibration methods corresponding to each pseudo quantization node.

[0118] The calibration method of each pseudo-quantization node in the second neural network model can be updated to the target calibration method corresponding to the pseudo-quantization node, thereby obtaining a hybrid calibration model. That is, in the hybrid calibration model, each pseudo-quantization node is quantized using its own corresponding target calibration method. For example, the target calibration method corresponding to pseudo-quantization node a is KL calibration, the target calibration method corresponding to pseudo-quantization node b is MAX calibration, and the target calibration method corresponding to pseudo-quantization node c is Percentile calibration. This is only an example. In actual applications, the target calibration methods of any two pseudo-quantization nodes may be the same or different.

[0119] Step 2420 : Determine a fourth average quantization error of the hybrid calibration model based on the calibration data.

[0120] The fourth average quantization error is the average quantization error of the hybrid calibration model relative to the first neural network model. Alternatively, calibration data may be input into the hybrid calibration model to obtain output data of the hybrid calibration model, and then the fourth average quantization error of the hybrid calibration model may be determined based on the output data of the hybrid calibration model and the first output data. The specific operation for determining the fourth average quantization error is similar to that for the third average quantization error described above and is not further described here.

[0121] Step 2430 : Determine a target calibration model from the first calibration model and the hybrid calibration model based on the first average quantization error of the first calibration model and the fourth average quantization error of the hybrid calibration model.

[0122] In some optional embodiments, the hybrid calibration model may be used as the target calibration model.

[0123] In some optional embodiments, since the first calibration model is the optimal calibration model among the calibration models, the hybrid calibration model can be further compared with the first calibration model, that is, the first average quantization error corresponding to the first calibration model and the fourth average quantization error of the hybrid calibration model are compared. If the fourth average quantization error of the hybrid calibration model is less than or equal to the first average quantization error, it means that the loss of model calculation accuracy brought about by the hybrid calibration model as a whole is less than that of the first calibration model, and the hybrid calibration model can be used as the target calibration model. If the fourth average quantization error of the hybrid calibration model is greater than the first average quantization error, it means that the loss of model calculation accuracy brought about by the hybrid calibration model as a whole is greater than that of the first calibration model, and the first calibration model can be used as the target calibration model.

[0124] In the disclosed embodiments, a hybrid calibration model is determined based on the target calibration method corresponding to each pseudo-quantization node. This ensures that the target calibration method used by each pseudo-quantization node in the hybrid calibration model is the calibration method that introduces the least precision loss among the various calibration methods. This helps reduce model calculation precision loss and improves model calculation accuracy. Furthermore, the hybrid calibration model is further compared with the first calibration model to determine a target calibration model, thereby optimizing the target calibration model and further improving model calculation accuracy.

[0125] Figure 7 It is a flowchart of a quantization method of a neural network model provided by another exemplary embodiment of the present disclosure.

[0126] In some optional embodiments, based on any of the above embodiments, determining the target quantization parameters corresponding to each operator in the target calibration model in step 250 may include:

[0127] Step 2510 : Determine the quantization sensitivity of each operator in the target calibration model under at least one set of preset quantization parameters based on the calibration data.

[0128] Among them, any set of preset quantization parameters includes at least one of quantization accuracy, quantization granularity and quantization mode. The operator in the target calibration model is the operator to be quantized in the first neural network model, that is, the input data of the operator (that is, the output data of another operator) needs to be quantized according to a certain quantization accuracy, quantization granularity and quantization mode. The input data of the operator may include activation values ​​and weight values. The specific meanings of quantization accuracy, quantization granularity and quantization mode can be found in the above content and will not be elaborated here.

[0129] In some optional embodiments, the quantization precision may include a first quantization precision corresponding to the activation value and a second quantization precision corresponding to the weight value. The first quantization precision and the second quantization precision may be the same or different. For example, the activation value may use a higher first quantization precision, such as INT8, and the weight value may use a lower second quantization precision, such as INT4, to preserve more detailed features of the activation value as much as possible.

[0130] In some optional embodiments, one or more groups of preset quantization parameters can be pre-set, and any two groups of preset quantization parameters have at least one different parameter value. For example, at least one parameter value of the quantization accuracy, quantization granularity, and quantization mode in the two groups of preset quantization parameters is different. For example, one group of preset quantization parameters includes quantization accuracy INT8-INT4, quantization granularity is layer-by-layer quantization, and quantization mode is symmetric quantization; another group of preset quantization parameters includes quantization accuracy INT4-INT4, quantization granularity is layer-by-layer quantization, and quantization mode is symmetric quantization; another group of preset quantization parameters includes quantization accuracy INT8-INT4, quantization granularity is channel-by-channel quantization, and quantization mode is symmetric quantization, and so on.

[0131] In some optional embodiments, the quantization sensitivity of an operator under a preset quantization parameter refers to the sensitivity of the operator to the loss of model calculation accuracy, which is quantized based on the preset quantization parameter on the basis of a target calibration model. Optionally, for any operator and any set of preset quantization parameters, the preset quantization parameter can be used as the current quantization parameter corresponding to the operator, and the operator is quantized separately according to the preset quantization parameter in the target calibration model, skipping the quantization processing of other operators. On this basis, the calibration data is subjected to model inference through the target calibration model to obtain model output data. The model output data of the operator and the first output data of the first neural network model are quantized separately based on the target calibration model to determine the quantization sensitivity of the operator under the preset quantization parameter. Based on this, the quantization sensitivity of each operator under each preset quantization parameter is obtained.

[0132] Step 2520 : Determine target quantization parameters corresponding to respective operators in the target calibration model based on the target calibration model and the quantization sensitivity of respective operators under respective preset quantization parameters.

[0133] Among them, the quantization sensitivity of the operator under the preset quantization parameter represents the size of the quantization error introduced by the quantization of the operator. Based on the principle of error minimization, the target quantization parameter corresponding to each operator can be determined. That is, for each operator, the preset quantization parameter that minimizes the quantization sensitivity corresponding to the operator is determined from the preset quantization parameters as the target quantization parameter of the operator.

[0134] In some optional embodiments, each set of preset quantization parameters can be sorted in order of computational cost from small to large, and the search can be started from the preset quantization parameter corresponding to the minimum computational cost. Under the condition of satisfying the preset sensitivity constraint, a set of preset quantization parameters with the minimum computational cost is determined for the operator as the target quantization parameter corresponding to the operator. The preset sensitivity constraint may include a quantization sensitivity less than a specified threshold. For example, the quantization sensitivity of the operator under the preset quantization parameter is represented by the cosine similarity of the model output data and the first output data of the operator alone. The higher the cosine similarity, the lower the quantization sensitivity. The cosine similarity greater than the similarity threshold indicates that the quantization sensitivity is less than the specified threshold. The similarity threshold is, for example, 0.99 or other values. Optionally, if the computational cost corresponding to multiple sets of preset quantization parameters is the same, the preset quantization parameter with the minimum quantization sensitivity can be used as the target quantization parameter.

[0135] In some optional embodiments, when each set of preset quantization parameters only includes quantization precision, the quantization precision can be searched layer by layer according to the above steps to obtain the target quantization precision corresponding to each operator in the target calibration model. In this case, during the layer-by-layer search process, the quantization granularity and quantization method required for operator quantization can use the default quantization granularity and default quantization method, or use the user-set quantization granularity and user-set quantization method.

[0136] In some optional embodiments, when each set of preset quantization parameters only includes quantization granularity, the quantization granularity can be searched layer by layer according to the above steps to obtain the target quantization granularity corresponding to each operator in the target calibration model. In this case, during the layer-by-layer search process, the quantization precision and quantization mode required for operator quantization can use a default quantization precision and a default quantization mode, or a user-set quantization precision and a user-set quantization mode.

[0137] In some optional embodiments, when each set of preset quantization parameters only includes a quantization mode, the quantization mode can be searched layer by layer according to the above steps to obtain the target quantization mode corresponding to each operator in the target calibration model. In this case, during the layer-by-layer search process, the quantization precision and quantization granularity required for operator quantization can use a default quantization precision and a default quantization granularity, or a user-set quantization precision and quantization granularity.

[0138] The embodiments of the present disclosure determine the target quantization parameters corresponding to each operator by further determining the quantization sensitivity of each operator under one or more sets of preset quantization parameters based on the calibration data on the basis of the target calibration model, so as to minimize the model calculation accuracy loss caused by the target quantization parameters corresponding to each operator when used for operator quantization or meet the calculation accuracy loss requirements, thereby effectively reducing the model calculation accuracy loss and improving the model calculation accuracy.

[0139] Figure 8 It is a flowchart of determining the quantization sensitivity of an operator under a preset quantization parameter provided by an exemplary embodiment of the present disclosure.

[0140] In some optional embodiments, such as Figure 8 As shown, step 2510 of determining the quantization sensitivity of each operator in the target calibration model under at least one set of preset quantization parameters based on the calibration data may include:

[0141] Step 410: Determine a first quantization parameter that meets a second preset condition from at least one set of preset quantization parameters.

[0142] The second preset condition refers to a condition where the computational cost is relatively low, such as a minimum computational cost or a computational cost less than a preset cost threshold. The specific second preset condition can be set based on actual quantization requirements. A preset quantization parameter that satisfies the second preset condition is determined from each set of preset quantization parameters as the first quantization parameter.

[0143] Step 420 : For each of the operators, determine a third quantization sensitivity of the target calibration model for individually quantizing the operator based on the calibration data and the first quantization parameter.

[0144] Specifically, the target calibration model quantizes the operator individually, meaning that only the output data of the operator is quantized based on the first quantization parameter. The quantization process includes quantization and inverse quantization to simulate hardware quantization. Quantization of other operators is skipped, meaning that the output data of other operators maintains the accuracy of the original model. On this basis, the calibration data is input into the target calibration model to obtain a third quantization sensitivity for quantizing the operator individually. Similar to the first and second quantization sensitivities described above, the third quantization sensitivity can be determined based on the model output data and the first output data, and the details will not be repeated here.

[0145] Step 430 : Based on the third quantization sensitivity corresponding to each operator, determine a first operator that meets a third preset condition from each operator.

[0146] The third preset condition refers to the top N (N is a positive integer) operators with the largest third quantization sensitivity, or the third quantization sensitivity is greater than a preset threshold. The specific third preset condition is not limited. Based on the third quantization sensitivity corresponding to each operator and the third preset condition, one or more operators that meet the third preset condition are determined from each operator as the first operator. For example, the top N operators ranked in the third quantization sensitivity are determined from each operator as the first operator.

[0147] Step 440 : Determine a fourth quantization sensitivity of each first operator under each second quantization parameter based on the calibration data and the second quantization parameter among the preset quantization parameters.

[0148] The second quantization parameter is a preset quantization parameter other than the first quantization parameter in at least one set of preset quantization parameters. Since the third quantization sensitivity of the first operator is relatively high, indicating that the first quantization parameter used in quantizing the first operator results in a significant loss in model calculation accuracy, it is necessary to further combine the calibration data with each second quantization parameter other than the first quantization parameter to determine the fourth quantization sensitivity of each first operator under each second quantization parameter, thereby further determining the optimal quantization parameter corresponding to the first operator.

[0149] Step 450 : Determine the third quantization sensitivity corresponding to each operator and the fourth quantization sensitivity of each first operator under each second quantization parameter as the quantization sensitivity of each operator under at least one set of preset quantization parameters.

[0150] Among them, the third quantization sensitivity is the quantization sensitivity of the operator under the first quantization parameter, and the fourth quantization sensitivity is the quantization sensitivity of the first operator under the second quantization parameter. The third quantization sensitivity and the fourth quantization sensitivity are used as the quantization sensitivity of each operator under at least one set of preset quantization parameters to determine the target quantization parameters corresponding to each operator.

[0151] In the embodiment of the present disclosure, the first quantization parameter is determined by the second preset condition, and then the first operator with higher quantization sensitivity is determined in combination with the third preset condition, and the fourth quantization sensitivity of the first operator under the second quantization parameter is further determined. For the second operator with lower quantization sensitivity among each operator, the calculation of the fourth quantization sensitivity can be avoided, thereby effectively reducing the amount of quantization sensitivity calculation.

[0152] In some optional embodiments, step 2520 of determining the target quantization parameters corresponding to each operator in the target calibration model based on the target calibration model and the quantization sensitivity of each operator under each preset quantization parameter includes:

[0153] Based on the fourth quantization sensitivity of each first operator under each second quantization parameter and the third quantization sensitivity corresponding to each first operator, the target quantization parameter corresponding to each first operator is determined; the target quantization parameter corresponding to the second operator in each operator is determined to be the first quantization parameter; the second operator is the operator in each operator except the first operator.

[0154] Among them, for each first operator, the third quantization sensitivity of the first operator under the first quantization parameter can be compared with the fourth quantization sensitivities of the first operator under each second quantization parameter to determine the minimum quantization sensitivity, and the quantization parameter corresponding to the minimum quantization sensitivity is determined as the target quantization parameter corresponding to the first operator. For example, the third quantization sensitivity of operator A under the first quantization parameter is e5, the number of second quantization parameters is 3, which are the second quantization parameter d1, the second quantization parameter d2, and the third quantization parameter d3 respectively. The fourth quantization sensitivity of operator A under d1 is e6, the fourth quantization sensitivity of operator A under d2 is e7, and the fourth quantization sensitivity of operator A under d3 is e8. Since e6 < e7 < e8 < e5, the target quantization parameter corresponding to operator A is the second quantization parameter d1 corresponding to e6. The second operator among each operator except the first operator is an operator with a relatively low third quantization sensitivity. Since the third quantization sensitivity of the second operator under the first quantization parameter is relatively low and meets the preset sensitivity constraint condition, the first quantization parameter can be used as the target quantization parameter corresponding to the second operator, avoiding further calculation of the quantization sensitivity of the second operator under the second quantization parameter.

[0155] In an embodiment of the present disclosure, based on the third quantization sensitivity of the first operator under the first quantization parameter and the fourth quantization sensitivity of the first operator under the second quantization parameter, the target quantization parameter corresponding to the first operator is determined, improving the effectiveness of the target quantization parameter corresponding to the first operator. Moreover, for the second operator among each operator except the first operator, the calculation of the quantization sensitivity of the second operator under the second quantization parameter can be avoided, and the first quantization parameter can be directly used as the target quantization parameter corresponding to the second operator. On the basis that the target quantization parameter meets the preset sensitivity constraint condition, the sensitivity calculation amount is effectively reduced.

[0156] Figure 9 It is a schematic flowchart of determining the quantization sensitivity of an operator under preset quantization parameters provided by another exemplary embodiment of the present disclosure.

[0157] In some optional embodiments, on the basis of the above embodiments, as Figure 9 shown, for each operator among each operator in step 420, based on the calibration data and the first quantization parameter, determining the third quantization sensitivity of the target calibration model for separately quantifying the operator may include:

[0158] Step 4210, for each operator among each operator, based on the calibration data and the first quantization parameter, determine the fifth output data of the target calibration model for separately quantifying the operator.

[0159] Among them, the separate quantization of the operator by the target calibration model can be referred to the foregoing content. The fifth output data is the output data of the model obtained by processing the calibration data when the target calibration model separately quantifies the operator.

[0160] Step 4220: Based on the fifth output data and the first output data of the first neural network model, determine the fifth average quantization error of the operator under the first quantization parameter as the third quantization sensitivity of the operator under the first quantization parameter.

[0161] The specific operation of determining the fifth average quantization error is similar to the aforementioned first average quantization error, second average quantization error, etc., and reference may be made to the aforementioned content, which will not be elaborated herein.

[0162] In an embodiment of the present disclosure, the average quantization error of the individual quantization operator of the target calibration model is used as the third quantization sensitivity of the operator, so that the third quantization sensitivity can effectively characterize the accuracy loss introduced by the individual quantization operator to the model, thereby ensuring the effectiveness of the third quantization sensitivity.

[0163] Figure 10 This is a flowchart of determining the quantization sensitivity of an operator under a preset quantization parameter provided by yet another exemplary embodiment of the present disclosure.

[0164] In some optional embodiments, based on the above embodiments, Figure 10 As shown, step 440 of determining the fourth quantization sensitivity of each first operator under each second quantization parameter based on the calibration data and the second quantization parameter in each preset quantization parameter may include:

[0165] Step 4410 : For each first operator and each second quantization parameter, determine a target calibration model based on the calibration data and the second quantization parameter to separately quantize the sixth output data of the first operator.

[0166] The sixth output data is similar to the fifth and fourth output data described above and is not described in detail here. For each first operator and each second quantization parameter, the second quantization parameter is used as the quantization parameter corresponding to the first operator, and the target calibration model is used to quantize the first operator separately to obtain the sixth output data.

[0167] Step 4420: Based on the sixth output data and the first output data of the first neural network model, determine the sixth average quantization error of the first operator under the second quantization parameter as the fourth quantization sensitivity of the first operator under the second quantization parameter.

[0168] The sixth average quantization error is similar to the first average quantization error and the second average quantization error, and thus will not be described in detail herein.

[0169] In an embodiment of the present disclosure, the fourth quantization sensitivity of the first operator under the second quantization parameter is determined by separately quantizing the average quantization error of the first operator under the second quantization parameter, so that the fourth quantization sensitivity can effectively characterize the model calculation accuracy loss introduced by separately quantizing the first operator based on the second quantization parameter to the model, thereby ensuring the effectiveness of the fourth quantization sensitivity.

[0170] In some optional embodiments, the quantization parameters include at least two types of quantization accuracy, quantization granularity, and quantization mode.

[0171] Determining the target quantization parameters corresponding to each operator in the target calibration model in step 250 includes:

[0172] At least two types are taken as target types in turn in a preset order. Based on the calibration data, the quantization parameters of the target types corresponding to each operator in the target calibration model are determined by searching the quantization parameters of the target types layer by layer; based on the quantization parameters of each target type, the target quantization parameters corresponding to each operator are determined.

[0173] The types of quantization parameters can be set as needed. For example, the quantization parameters can include quantization precision and quantization granularity, or the quantization parameters can include quantization precision, quantization granularity, and quantization mode, or the quantization parameters can include quantization granularity and quantization mode, etc. The preset order can be any order. For example, if the quantization parameters include quantization precision, quantization granularity, and quantization mode, the preset order can be quantization precision-quantization granularity-quantization mode, or the preset order can be quantization granularity-quantization precision-quantization mode, or the preset order can be quantization mode-quantization granularity-quantization precision, etc., without specific limitation. Each type is traversed in the preset order, and the traversed type is used as the target type. Based on the calibration data, the quantization parameters of the target type corresponding to each operator in the target calibration model are determined by searching the quantization parameters of the target type layer by layer. That is, the layer-by-layer search function for the quantization parameters of each type can be executed serially. For example, the quantization precision is first searched layer by layer to obtain the target quantization precision corresponding to each operator, the quantization granularity is then searched layer by layer to obtain the target quantization granularity corresponding to each operator, and the quantization mode is then searched layer by layer to obtain the target quantization mode corresponding to each operator. For any target type, the specific operation of searching for the quantization parameters of the target type layer by layer can be found in the above embodiment, such as steps 2510 to 2520, which will not be described in detail here.

[0174] In some optional embodiments, the user may specify the type to be searched layer by layer according to the quantization requirements. For the type set by the user to be searched layer by layer, the quantization parameter of the type is determined by searching the quantization parameter of the type layer by layer. For the type not specified by the user to be searched layer by layer, the preset default quantization parameter may be used, or if the user has set the quantization parameter of the type, the quantization parameter set by the user may be used. For example, if the user sets the type to be searched layer by layer as quantization accuracy and quantization granularity, then if the user has set the quantization method, the quantization method set by the user may be used, for example, the user sets the asymmetric quantization method, or if the user has not specified the quantization method, the default quantization method may be used, for example, the default quantization method is symmetric quantization.

[0175] In an embodiment of the present disclosure, for multiple types of quantization parameters, a layer-by-layer search can be performed serially to determine the quantization parameters of each type corresponding to each operator. Since each type of quantization parameter can be searched layer by layer separately, the search space can be reduced, the search efficiency can be improved, and it is convenient for users to select the parameter types that can be searched layer by layer to meet the different needs of users.

[0176] In some optional embodiments, based on any of the above embodiments, determining the target calibration mode corresponding to each pseudo quantization node in step 230 includes:

[0177] Determine a search state of the calibration mode; in response to the calibration mode search state being a first state, determine a target calibration mode corresponding to each pseudo quantization node by searching the calibration mode layer by layer based on the calibration data.

[0178] In some optional embodiments, the quantization parameters include three types: quantization accuracy, quantization granularity, and quantization mode. Step 250 of determining the target quantization parameters corresponding to each operator in the target calibration model includes:

[0179] Traverse each type in a preset order, take the traversed type as the target type, and determine the search state corresponding to the target type; in response to the search state corresponding to the target type being the first state, determine the quantization parameters of the target type corresponding to each operator by searching the quantization parameters of the target type layer by layer based on the calibration data; based on the quantization parameters of each target type, determine the target quantization parameters corresponding to each operator.

[0180] Among them, the preset order can be an arbitrary order. The search state includes a first state and a second state, the first state indicates that a layer-by-layer search is required, and the second state indicates that a layer-by-layer search is not required. The search state of the target type can be a preconfigured search state. The preconfigured search state can be dynamically configured according to the user application, that is, set by the user, or the preconfigured search state can be a preset default state. For example, the default state corresponding to each type is the first state, and the user can dynamically update the search state according to needs. For example, the user can choose to skip the layer-by-layer search of the quantization granularity and quantization method according to needs, that is, the search state of the quantization accuracy is the first state, the search state of the quantization granularity is the second state, and the search state of the quantization method is the second state.

[0181] In some optional embodiments, in response to the search state corresponding to the quantization precision being in a first state, a target quantization precision corresponding to each operator is determined by searching for quantization precision layer by layer based on calibration data; or, in response to the search state corresponding to the quantization precision being in a second state, a preconfigured preset quantization precision is determined as the target quantization precision. In response to the search state corresponding to the quantization granularity being in a first state, a target quantization granularity corresponding to each operator is determined by searching for quantization granularity layer by layer based on calibration data; or, in response to the search state corresponding to the quantization granularity being in a second state, a preconfigured preset quantization granularity is determined as the target quantization granularity. In response to the search state corresponding to the quantization mode being in a first state, a target quantization mode corresponding to each operator is determined by searching for quantization modes layer by layer based on calibration data; or, in response to the search state corresponding to the quantization mode being in a second state, a preconfigured preset quantization mode is determined as the target quantization mode. Based on the target quantization precision, target quantization granularity, and target quantization mode corresponding to each operator, a target quantization parameter corresponding to each operator is obtained.

[0182] In the embodiments of the present disclosure, separate layer-by-layer searches of the calibration method, quantization accuracy, quantization granularity, and quantization method are supported, as well as the configuration of each search state, so that users can choose the type of layer-by-layer search required according to different quantization requirements, thereby meeting the needs of different users and improving search flexibility and user experience.

[0183] In some optional embodiments, Figure 11 FIG. 1 is a flowchart of a method for searching for a calibration threshold layer by layer provided by an exemplary embodiment of the present disclosure. Figure 11 As shown, the process of searching for the calibration threshold layer by layer (i.e., searching for the calibration layer by layer) may include the following steps:

[0184] Step 61: Determine the original model, which is the first neural network model to be quantized.

[0185] Step 621: Determine a KL calibration model based on the original model and a preset calibration method (ie, KL calibration). In this embodiment, the KL calibration model is used as a reference calibration model (ie, the first calibration model) as an example.

[0186] Step 622 : Determine a MAX calibration model based on the original model and another preset calibration method (ie, MAX calibration).

[0187] Step 623 : Determine a Percentile calibration model based on the original model and another preset calibration method (ie, Percentile calibration).

[0188] Step 63 : Calculate the first quantization sensitivity of all pseudo quantization nodes. That is, after obtaining each calibration model, assuming that the KL calibration model is determined as the first calibration model, calculate the first quantization sensitivity corresponding to each pseudo quantization node based on the calibration data and the KL calibration model.

[0189] Step 64 determines whether the number of TopK nodes is specified. If so, proceed to step 66. If not, proceed to step 65. This determines whether the user specifies the value of K in TopK or whether the value of K is preconfigured. K represents the number of first pseudo-quantization nodes.

[0190] Step 65 : Determine the number of TopK nodes based on the cosine similarity. That is, determine the value of K based on the cosine similarity between the third output data and the first output data of each pseudo-quantization node individually quantized and the similarity threshold. Then proceed to step 66 .

[0191] Step 66 : Calculate the second quantization sensitivity of the TopK pseudo quantization nodes (ie, the first pseudo quantization nodes) under each second calibration model. Figure 11 The second calibration model including the MAX calibration model and the Percentile calibration model is taken as an example.

[0192] Step 67 selects a target calibration method layer by layer based on the principle of minimum quantization sensitivity. Specifically, for the first pseudo-quantization node in the TopK, the calibration method corresponding to the calibration model with the minimum quantization sensitivity is determined as the target calibration method based on the first and second quantization sensitivities. For any second pseudo-quantization node other than the TopK, the target calibration method is determined to be the calibration method corresponding to the first calibration model. For example, the calibration method corresponding to the KL calibration model in the figure is KL calibration.

[0193] Step 68: Update the calibration threshold based on the target calibration method and construct a hybrid calibration model.

[0194] Step 69 : Evaluate the errors of the reference calibration model and the hybrid calibration model, that is, determine the target calibration model from the first calibration model and the hybrid calibration model based on the first average quantization error of the reference calibration model and the fourth average quantization error of the hybrid calibration model.

[0195] Step 610 : Obtain a target calibration model based on the evaluation result of step 69 .

[0196] The specific operations of steps 61 to 610 can be found in the aforementioned related embodiments and will not be described here one by one. It can be seen that if the second neural network model has a total of M pseudo-quantization nodes, when L calibration methods are selected for layer-by-layer search, the quantization sensitivity calculation involves a total of M+K*(L-1) model inferences, where K represents the number of first pseudo-quantization nodes, and K is less than M, that is, the baseline calibration model needs to be inferred M times, and the first quantization sensitivity of the M pseudo-quantization nodes under the baseline calibration model (i.e., the first calibration model) is calculated. For the TopK first pseudo-quantization nodes, it is necessary to calculate the second quantization sensitivity of the K first pseudo-quantization nodes under L-1 second calibration models, and each second calibration model needs to be inferred K times. Compared with calculating the quantization sensitivity of each pseudo-quantization node under each calibration model (requiring a total of M*L inferences), TopK can effectively reduce the number of model inferences, thereby improving search efficiency.

[0197] In some optional embodiments, Figure 12 FIG. 1 is a flowchart of a method for searching quantization accuracy layer by layer provided by an exemplary embodiment of the present disclosure, such as Figure 12 As shown, taking the preset quantization parameters including quantization precision as an example, the quantization granularity and quantization mode can be preset values. The search principle of other preset quantization parameters is similar to that of quantization precision. The process of searching quantization precision layer by layer may include the following steps:

[0198] Step 71: Determine the target calibration model.

[0199] Step 72: Determine multiple preset quantization accuracies. Figure 12The multiple preset quantization precisions include five quantization precisions from 721 to 725, namely int8-int4, int8-int8, int16-int8, float16-float16, and bfloat16-bfloat16. Each quantization precision includes a first quantization precision corresponding to the activation value and a second quantization precision corresponding to the weight value. For example, the first quantization precision corresponding to the activation value in int8-int4 is int8 (also known as INT8), and the second quantization precision corresponding to the weight value in int8-int4 is int4 (also known as INT4). Among them, int represents integer, float (also known as floating-point) represents single-precision floating point type, and bfloat (also known as brain floating point) represents a floating point type specially proposed for deep learning optimization.

[0200] Step 73: Calculate the third quantization sensitivity of all operators. In the figure, taking int8-int4 as the quantization accuracy in the first quantization parameter as an example, based on the calibration data and the first quantization parameter, determine the third quantization sensitivity of the target calibration model for quantizing the operator individually.

[0201] Step 74: Determine whether the TopN number is specified, where N represents the number of first operators. That is, determine whether the specified TopN number exists.

[0202] Step 75: If it is determined according to step 74 that the specified TopN number does not exist, the TopN number is determined based on the cosine similarity, that is, the value of N is determined based on the cosine similarity between the model output data of the individual quantization operator and the first output data and the similarity threshold.

[0203] Step 76 : If it is determined according to step 74 that the specified number of TopN exists, or the number of TopN is determined according to step 75 , the fourth quantization sensitivity of the first operator of the TopN under the second quantization parameter is calculated.

[0204] Step 77: Determine the minimum quantization precision that satisfies the sensitivity constraint. Specifically, for the first operator, the minimum quantization precision that satisfies the sensitivity constraint is determined from among the various quantization precisions based on the third and fourth quantization sensitivities of the first operator. For example, if multiple quantization precisions satisfy the sensitivity constraint, such as int8-int4 and int16-int8, then the minimum quantization precision among the multiple quantization precisions that satisfy the sensitivity constraint is used as the target quantization precision, such as int8-int4. For each second operator other than the first operator, the quantization precision in the first quantization parameter can be directly used as the target quantization precision of the second operator. Figure 12 The first quantization parameter is taken as int8-int4.

[0205] Step 78: Update the quantization precision and construct a mixed precision model. That is, based on the result of step 77, update the quantization precision of each operator in the target calibration model to the target quantization precision corresponding to each operator to obtain a mixed precision model.

[0206] Step 79: Obtain a quantized model based on the mixed precision model updated in step 78.

[0207] The specific operations of steps 71 to 79 can be found in the aforementioned related embodiments and will not be described in detail here. Similar to the layer-by-layer search calibration method, if the target calibration model has a total of S (S is a positive integer) operators, when T types of quantization precision are selected for layer-by-layer search, the quantization sensitivity involves a total of S+N*(T-1) model inferences, where N represents the number of first operators, and N is less than S. Compared to calculating the quantization sensitivity of each operator at each quantization precision (requiring a total of S*T model inferences), by using the first operator of TopN, the number of model inferences can be effectively reduced, thereby improving the search efficiency.

[0208] In some optional embodiments, Figure 13 FIG. 1 is a flowchart of a quantization method for a neural network model provided by an exemplary embodiment of the present disclosure. Figure 13 As shown, the method provided by the embodiment of the present disclosure includes:

[0209] S710: Determine the original model to be quantized (ie, the first neural network model).

[0210] S711: Determine whether to skip the calibration threshold search. If the calibration threshold search is skipped, i.e., the layer-by-layer search of the calibration mode is skipped, i.e., the search state corresponding to the calibration mode is the second state, then S712 is executed. If the calibration threshold search is not skipped, i.e., the search state corresponding to the calibration mode is the first state, then S713 is executed.

[0211] S712, obtain a pre-configured target calibration method, and execute S714.

[0212] S713, determine the target calibration method by searching the calibration threshold, and execute S714.

[0213] S714: Determine a target calibration model based on the target calibration method.

[0214] S715: Determine whether to skip the quantization precision search. If the quantization precision search is skipped, proceed to S716; if the quantization precision search is not skipped, proceed to S717. Based on the target calibration model, further determine whether to skip the quantization precision search. For example, if the search state corresponding to the quantization precision is the second state, determine to skip the quantization precision search. If the search state corresponding to the quantization precision is the first state, determine not to skip the quantization precision search.

[0215] S716: Obtain a pre-configured target quantization accuracy, and execute S718.

[0216] S717, determine the target quantization accuracy through quantization accuracy search (ie, search the quantization accuracy layer by layer), and execute S718.

[0217] S718, determining a quantization model based on the target quantization accuracy.

[0218] Figure 13 Only quantization accuracy is taken as an example. In actual applications, after determining the target quantization accuracy, we can continue to determine whether to skip the quantization granularity search, and then determine the target quantization granularity, determine whether to skip the quantization method search, determine the target quantization method, and then determine the quantization model based on the target quantization accuracy, target quantization granularity, and target quantization method.

[0219] In some optional embodiments, the quantization accuracy, quantization granularity, and quantization mode can each be treated as a separate search subspace, and searches can be performed separately according to the aforementioned layer-by-layer search process for quantization accuracy to determine the target quantization accuracy, target quantization granularity, and target quantization mode corresponding to each operator. For example, the calibration mode corresponds to one search subspace, the quantization accuracy corresponds to one search subspace, the quantization granularity corresponds to one search subspace, and the quantization mode corresponds to one search subspace. The search processes for each search subspace can be connected in series, and a skip function, i.e., a search state, can be set for each search subspace. This allows users to select the subspace to be searched layer by layer based on their needs, thereby meeting the diverse needs of various users.

[0220] The quantization method of the neural network model provided by the embodiment of the present disclosure is aimed at the fact that the global configuration of the same calibration method in the related art may not be friendly to the quantization of some layers. For example, for pseudo-quantization nodes with outliers, KL calibration may obtain a better calibration threshold than MAX calibration. The embodiment of the present disclosure determines the target calibration method of each layer by searching the calibration threshold layer by layer for different layers, and then determines the calibration threshold of each layer, so that each layer can use its own optimal calibration method to determine the calibration threshold, effectively improving the accuracy of the quantization model. Secondly, for the differences in the errors introduced by operators in the model, higher quantization accuracy can be used for more sensitive operators. For operators with lower sensitivity, even if low quantization accuracy is used, the impact on the accuracy of the model is still small and can be ignored. Based on this, for operators in different layers, the optimal quantization accuracy, quantization granularity, quantization method and other quantization parameters of each layer are determined by searching the quantization parameters layer by layer, so as to better balance the quantization accuracy and model performance. While ensuring that the sensitivity constraints are met, the calculation accuracy of the quantization model is effectively improved, and the accuracy loss of the quantization model relative to the original model is reduced. In addition, when a large number of adjustable quantization parameters are involved in the model quantization process, the size of the search space for layer-by-layer search grows exponentially with the number of quantization parameters. If the layer-by-layer search calibration method and the layer-by-layer search quantization parameters are fully performed, the time consumption is usually large. The embodiment of the present disclosure sets different search spaces and supports users to set the search status of each search space, which can effectively reduce the search space and search time consumption, and facilitates the search processes of different search spaces or search subspaces to be combined in series in the form of plug-ins, so that users can arbitrarily choose whether to perform a search on a certain subspace, thereby improving the versatility of the method of the embodiment of the present disclosure and enhancing the user experience.

[0221] The above-mentioned embodiments of the present disclosure may be implemented individually or in any combination without conflict. The specific configuration may be based on actual needs and is not limited by the present disclosure.

[0222] Any of the quantization methods for the neural network models provided in the embodiments of the present disclosure can be executed by any appropriate electronic device with data processing capabilities, including but not limited to electronic devices such as terminal devices and servers. Alternatively, any of the quantization methods for the neural network models provided in the embodiments of the present disclosure can be executed by a processor, such as a processor that executes any of the quantization methods for the neural network models mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.

[0223] Exemplary devices

[0224] Figure 14FIG. 1 is a schematic diagram of a quantization device for a neural network model provided by an exemplary embodiment of the present disclosure. The device of this embodiment can be used to implement the corresponding method embodiments of the present disclosure, such as Figure 14 The device shown may include: a first processing module 81 , a second processing module 82 , a third processing module 83 , a fourth processing module 84 , a fifth processing module 85 and a sixth processing module 86 .

[0225] The first processing module 81 is used to determine a first neural network model to be quantized.

[0226] The second processing module 82 is used to perform pseudo-quantization processing on the first neural network model to obtain a second neural network model, which includes pseudo-quantization nodes and operators.

[0227] The third processing module 83 is configured to determine a target calibration method corresponding to each pseudo quantization node.

[0228] The fourth processing module 84 is configured to determine a target calibration model corresponding to the second neural network model based on target calibration modes corresponding to the pseudo quantization nodes.

[0229] The fifth processing module 85 is used to determine the target quantization parameters corresponding to each operator in the target calibration model.

[0230] The sixth processing module 86 is used to determine the quantization model corresponding to the first neural network model based on the target calibration model and the target quantization parameters corresponding to each operator.

[0231] Figure 15 2 is a schematic structural diagram of a quantization device for a neural network model provided by another exemplary embodiment of the present disclosure.

[0232] In some optional embodiments, in the above Figure 14 Based on the embodiment shown, Figure 15 As shown, the third processing module 83 includes: an acquisition unit 831 , a first processing unit 832 , a second processing unit 833 and a third processing unit 834 .

[0233] The acquisition unit 831 is used to acquire calibration data for model quantization.

[0234] The first processing unit 832 is used to process the calibration data based on the first neural network model to obtain first output data and input data corresponding to each operator in the first neural network model.

[0235] The second processing unit 833 is configured to determine a calibration threshold corresponding to each pseudo quantization node in each preset calibration mode based on input data corresponding to each preset calibration mode and each operator in at least one preset calibration mode.

[0236] The third processing unit 834 is configured to determine a target calibration mode corresponding to each pseudo quantization node by searching the calibration threshold layer by layer based on the calibration threshold corresponding to each pseudo quantization node under each preset calibration mode and the first output data.

[0237] In some optional embodiments, the third processing unit 834 is specifically configured to:

[0238] Based on the calibration threshold corresponding to each pseudo-quantization node under each preset calibration method, a calibration model corresponding to each preset calibration method is determined. Based on the calibration data, second output data corresponding to each calibration model is determined. Based on the second output data and first output data corresponding to each calibration model, a first calibration model is determined from each calibration model. Based on the calibration data and the first calibration model, a first quantization sensitivity corresponding to each pseudo-quantization node is determined. Based on the first quantization sensitivity corresponding to each pseudo-quantization node and a second calibration model in each calibration model, a target calibration method corresponding to each pseudo-quantization node is determined. The second calibration model is a calibration model other than the first calibration model in each calibration model.

[0239] In some optional embodiments, the third processing unit 834 is specifically configured to:

[0240] Based on the second output data and the first output data corresponding to each calibration model, a first average quantization error corresponding to each calibration model is determined; based on the first average quantization error corresponding to each calibration model, a first calibration model is determined from each calibration model.

[0241] In some optional embodiments, based on any of the above embodiments, the third processing unit 834 is specifically configured to:

[0242] For each pseudo-quantization node, based on the calibration data and the first calibration model, the third output data of the pseudo-quantization node is determined to be individually quantized; based on the third output data and the first output data, the second average quantization error corresponding to the pseudo-quantization node is determined as the first quantization sensitivity corresponding to the pseudo-quantization node.

[0243] In some optional embodiments, based on any of the above embodiments, the third processing unit 834 is specifically configured to:

[0244] Based on the first quantization sensitivity corresponding to each pseudo quantization node, at least one first pseudo quantization node that meets the first preset condition is determined from each pseudo quantization node. The target calibration method corresponding to the second pseudo quantization node in each pseudo quantization node is determined to be the calibration method corresponding to the first calibration model. The second pseudo quantization node is a pseudo quantization node other than the first pseudo quantization node in each pseudo quantization node. Based on the calibration data and the second calibration model, the second quantization sensitivity of each first pseudo quantization node on each second calibration model is determined. Based on the second quantization sensitivity of each first pseudo quantization node on each second calibration model and the first quantization sensitivity corresponding to each first pseudo quantization node, the target calibration method corresponding to each first pseudo quantization node is determined. Based on the target calibration method corresponding to the second pseudo quantization node and the target calibration method corresponding to each first pseudo quantization node, the target calibration method corresponding to each pseudo quantization node is obtained.

[0245] In some optional embodiments, the third processing unit 834 is specifically configured to:

[0246] For each second calibration model and each first pseudo quantization node in each first pseudo quantization node, based on the calibration data and the second calibration model, determine that the second calibration model alone quantizes the fourth output data of the first pseudo quantization node; based on the second calibration model, the fourth output data and the first output data of the first pseudo quantization node are individually quantized, and the third average quantization error of the first pseudo quantization node on the second calibration model is determined as the second quantization sensitivity of the first pseudo quantization node on the second calibration model.

[0247] In some optional embodiments, the third processing unit 834 is specifically configured to:

[0248] For each first pseudo quantization node, based on the second quantization sensitivity of the first pseudo quantization node on each second calibration model and the first quantization sensitivity of the first pseudo quantization node on the first calibration model, determine the calibration model corresponding to the minimum quantization sensitivity; and determine the calibration method of the calibration model corresponding to the minimum quantization sensitivity as the target calibration method of the first pseudo quantization node.

[0249] In some optional embodiments, based on any of the above embodiments, Figure 15 As shown, the fourth processing module 84 includes: a first determining unit 841 , a second determining unit 842 and a third determining unit 843 .

[0250] The first determining unit 841 is configured to determine a hybrid calibration model based on the second neural network model and target calibration modes corresponding to each pseudo quantization node.

[0251] The second determining unit 842 is configured to determine a fourth average quantization error of the hybrid calibration model based on the calibration data.

[0252] The third determining unit 843 is configured to determine a target calibration model from the first calibration model and the hybrid calibration model based on the first average quantization error of the first calibration model and the fourth average quantization error of the hybrid calibration model.

[0253] In some optional embodiments, based on any of the above embodiments, Figure 15 As shown, the fifth processing module 85 includes: a fourth processing unit 851 and a fifth processing unit 852 .

[0254] The fourth processing unit 851 is configured to determine, based on the calibration data, the quantization sensitivity of each operator in the target calibration model under at least one set of preset quantization parameters.

[0255] Any set of preset quantization parameters includes at least one of quantization accuracy, quantization granularity and quantization mode.

[0256] The fifth processing unit 852 is configured to determine a target quantization parameter corresponding to each operator in the target calibration model based on the target calibration model and the quantization sensitivity of each operator under each preset quantization parameter.

[0257] In some optional embodiments, the fourth processing unit 851 is specifically configured to:

[0258] Determine a first quantization parameter that satisfies a second preset condition from at least one set of preset quantization parameters. For each operator in each operator, determine a third quantization sensitivity of the target calibration model for quantizing the operator individually based on the calibration data and the first quantization parameter. Determine a first operator that satisfies the third preset condition from each operator based on the third quantization sensitivity corresponding to each operator. Determine a fourth quantization sensitivity of each first operator under each second quantization parameter based on the calibration data and the second quantization parameter in each preset quantization parameter. The second quantization parameter is a preset quantization parameter other than the first quantization parameter in at least one set of preset quantization parameters. Determine the third quantization sensitivity corresponding to each operator and the fourth quantization sensitivity of each first operator under each second quantization parameter as the quantization sensitivity of each operator under at least one set of preset quantization parameters.

[0259] In some optional embodiments, the fifth processing unit 852 is specifically configured to:

[0260] Based on the fourth quantization sensitivity of each first operator under each second quantization parameter and the third quantization sensitivity corresponding to each first operator, the target quantization parameter corresponding to each first operator is determined; the target quantization parameter corresponding to the second operator in each operator is determined to be the first quantization parameter; the second operator is the operator in each operator except the first operator.

[0261] In some optional embodiments, the fourth processing unit 851 is specifically configured to:

[0262] For each of the operators, fifth output data of the operator is individually quantized using the target calibration model based on the calibration data and the first quantization parameter. A fifth average quantization error of the operator under the first quantization parameter is determined based on the fifth output data and the first output data of the first neural network model as a third quantization sensitivity of the operator under the first quantization parameter.

[0263] In some optional embodiments, the fourth processing unit 851 is specifically configured to:

[0264] For each first operator and each second quantization parameter, determine, based on the calibration data and the second quantization parameter, sixth output data of the first operator individually quantized by the target calibration model. Determine, based on the sixth output data and the first output data of the first neural network model, a sixth average quantization error of the first operator under the second quantization parameter as a fourth quantization sensitivity of the first operator under the second quantization parameter.

[0265] In some optional embodiments, the quantization parameters include at least two types of quantization precision, quantization granularity, and quantization mode. The fifth processing module 85 is specifically configured to: sequentially use the at least two types as target types in a preset order, determine the quantization parameters of the target types corresponding to each operator in the target calibration model by searching for the quantization parameters of the target types layer by layer based on the calibration data; and determine the target quantization parameters corresponding to each operator based on the quantization parameters of each target type.

[0266] In some optional embodiments, such as Figure 15 As shown, the fourth processing unit 851 and the fifth processing unit 852 can sequentially use at least two types as target types in a preset order, and determine the quantization parameters of the target types corresponding to each operator in the target calibration model based on the calibration data by searching the quantization parameters of the target types layer by layer, and determine the target quantization parameters corresponding to each operator based on the quantization parameters of each target type.

[0267] In some optional embodiments, the third processing module 83 is specifically used to: determine the search state of the calibration method; in response to the search state of the calibration method being the first state, based on the calibration data, determine the target calibration method corresponding to each pseudo-quantization node by searching the calibration method layer by layer.

[0268] In some optional embodiments, such as Figure 15As shown, the acquisition unit 831 is used to determine the search state of the calibration mode; in response to the calibration mode search state being the first state, calibration data is acquired. The first processing unit 832, the second processing unit 833, and the third processing unit 834 are used to determine the target calibration mode corresponding to each pseudo quantization node by searching the calibration mode layer by layer based on the calibration data.

[0269] In some optional embodiments, the quantization parameters include three types: quantization accuracy, quantization granularity, and quantization mode; the fifth processing module 85 is specifically used to: traverse each type in a preset order, take the traversed type as the target type, and determine the search state corresponding to the target type; in response to the search state corresponding to the target type being the first state, based on the calibration data, determine the quantization parameters of the target type corresponding to each operator by searching the quantization parameters of the target type layer by layer; based on the quantization parameters of each target type, determine the target quantization parameters corresponding to each operator.

[0270] The above-mentioned embodiments of the present disclosure may be implemented individually or in any combination without conflict. The specific configuration may be based on actual needs and is not limited by the present disclosure.

[0271] The beneficial technical effects corresponding to the exemplary embodiment of this device can be found in the corresponding beneficial technical effects of the above exemplary method part, which will not be repeated here.

[0272] Exemplary electronic devices

[0273] Figure 16 is a structural diagram of an electronic device provided by an embodiment of the present disclosure, including at least one processor 91 and a memory 92.

[0274] The processor 91 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 90 to perform desired functions.

[0275] The memory 92 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 91 may execute the one or more computer program instructions to implement the methods and / or other desired functions of the various embodiments of the present disclosure described above.

[0276] In one example, the electronic device 90 may further include an input device 93 and an output device 94 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0277] The input device 93 may also include, for example, a touch screen, a microphone, various sensors, and the like. Sensors may include, for example, image sensors (e.g., cameras, etc.), laser radars, millimeter-wave radars, ultrasonic radars, positioning sensors, pressure sensors, air quality sensors, temperature sensors, and the like. Image sensors, laser radars, millimeter-wave radars, ultrasonic radars, and the like may be used to perceive the surrounding environment, i.e., to detect dynamic and static objects in the surrounding environment. Dynamic and static objects may include, for example, static objects such as lane lines, curbs, arrows, signs, trees, and buildings, as well as dynamic objects such as surrounding vehicles, pedestrians, and cyclists. Positioning sensors are used to locate the movable device (e.g., a vehicle, robot, etc.) in which the electronic device is located. Positioning sensors may include, for example, an inertial measurement unit (IMU), a global positioning system (GPS), and the like. Pressure sensors may be used to detect seat pressure. Temperature sensors may be used to detect the temperature in the vehicle cabin. Air quality sensors may be used to detect the air quality in the vehicle cabin.

[0278] The output device 94 can output various information to the outside, and may include, for example, a display, a speaker, a communication network and a remote output device connected thereto, and the like.

[0279] Of course, to simplify, Figure 16 Only some of the components related to the present disclosure in the electronic device 90 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device 90 may further include any other appropriate components according to specific application scenarios.

[0280] Exemplary computer program products and computer-readable storage media

[0281] In addition to the above methods and devices, embodiments of the present disclosure may also provide a computer program product, including computer program instructions, which, when executed by a processor, enable the processor to perform the steps in the methods of various embodiments of the present disclosure described in the above "Exemplary Method" section.

[0282] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0283] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the steps in the method of various embodiments of the present disclosure described in the above “Exemplary Method” section.

[0284] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium is, for example, but not limited to, a system, device or component comprising electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0285] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be considered as essential to each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0286] Those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is intended to include these modifications and variations.

Claims

1. A quantization method for a neural network model, comprising: determining a first neural network model to be quantized; Performing pseudo-quantization processing on the first neural network model to obtain a second neural network model, wherein the second neural network model includes pseudo-quantization nodes and operators; Determining target calibration methods corresponding to the pseudo-quantization nodes; Determining a target calibration model corresponding to the second neural network model based on target calibration modes corresponding to the pseudo quantization nodes; Determining target quantization parameters corresponding to each operator in the target calibration model; Based on the target calibration model and the target quantization parameters corresponding to each of the operators, a quantization model corresponding to the first neural network model is determined.

2. The method according to claim 1, wherein The determining of the target calibration mode corresponding to each of the pseudo quantization nodes includes: Obtain calibration data for model quantization; Processing the calibration data based on the first neural network model to obtain first output data and input data corresponding to each operator in the first neural network model; Determining, based on input data corresponding to each of the preset calibration methods and each of the operators in at least one preset calibration method, a calibration threshold corresponding to each of the pseudo quantization nodes in each of the preset calibration methods; Based on the calibration threshold corresponding to each pseudo quantization node under each preset calibration mode and the first output data, a target calibration mode corresponding to each pseudo quantization node is determined by searching the calibration threshold layer by layer.

3. The method according to claim 2, wherein: The determining of the target calibration mode corresponding to each pseudo quantization node by searching the calibration threshold layer by layer based on the calibration threshold corresponding to each pseudo quantization node under each preset calibration mode and the first output data includes: Determining calibration models corresponding to each of the preset calibration modes based on calibration thresholds corresponding to each of the pseudo quantization nodes under each of the preset calibration modes; Determining second output data corresponding to each of the calibration models based on the calibration data; determining a first calibration model from each of the calibration models based on the second output data and the first output data respectively corresponding to each of the calibration models; Determining first quantization sensitivities corresponding to the pseudo quantization nodes based on the calibration data and the first calibration model; Based on the first quantization sensitivity corresponding to each pseudo quantization node and the second calibration model in each calibration model, the target calibration method corresponding to each pseudo quantization node is determined; the second calibration model is a calibration model in each calibration model except the first calibration model.

4. The method according to claim 3, wherein: The determining the first calibration model from each calibration model based on the second output data and the first output data respectively corresponding to each calibration model includes: Determining first average quantization errors corresponding to each calibration model based on the second output data and the first output data corresponding to each calibration model; The first calibration model is determined from the calibration models based on the first average quantization errors respectively corresponding to the calibration models.

5. The method according to claim 3, wherein The determining, based on the calibration data and the first calibration model, first quantization sensitivities corresponding to the pseudo-quantization nodes, includes: For each of the pseudo quantization nodes, determining third output data of the pseudo quantization node individually quantized based on the calibration data and the first calibration model; Based on the third output data and the first output data, a second average quantization error corresponding to the pseudo quantization node is determined as the first quantization sensitivity corresponding to the pseudo quantization node.

6. The method according to claim 3, wherein: The determining, based on the first quantization sensitivity corresponding to each of the pseudo quantization nodes and the second calibration model in each of the calibration models, a target calibration mode corresponding to each of the pseudo quantization nodes, includes: Based on the first quantization sensitivity corresponding to each of the pseudo quantization nodes, at least one first pseudo quantization node that meets a first preset condition is determined from each of the pseudo quantization nodes; determining that the target calibration mode corresponding to a second pseudo quantization node in each of the pseudo quantization nodes is the calibration mode corresponding to the first calibration model; the second pseudo quantization node is a pseudo quantization node in each of the pseudo quantization nodes other than the first pseudo quantization node; determining, based on the calibration data and the second calibration model, a second quantization sensitivity of each of the first pseudo quantization nodes on each of the second calibration models; Determining the target calibration mode corresponding to each of the first pseudo quantization nodes based on the second quantization sensitivity of each of the first pseudo quantization nodes on each of the second calibration models and the first quantization sensitivity corresponding to each of the first pseudo quantization nodes; Based on the target calibration mode corresponding to the second pseudo quantization node and the target calibration modes corresponding to each of the first pseudo quantization nodes, the target calibration modes corresponding to each of the pseudo quantization nodes are obtained.

7. The method according to claim 6, wherein: The determining, based on the calibration data and the second calibration model, the second quantization sensitivity of each first pseudo quantization node on each second calibration model, includes: For each second calibration model and each first dummy quantization node in each first dummy quantization node, determining, based on the calibration data and the second calibration model, that the second calibration model is used to separately quantize fourth output data of the first dummy quantization node; The fourth output data and the first output data of the first pseudo quantization node are separately quantized based on the second calibration model, and the third average quantization error of the first pseudo quantization node on the second calibration model is determined as the second quantization sensitivity of the first pseudo quantization node on the second calibration model.

8. The method according to claim 6, wherein: The determining, based on the second quantization sensitivity of each first pseudo quantization node on each second calibration model and the first quantization sensitivity corresponding to each first pseudo quantization node, the target calibration mode corresponding to each first pseudo quantization node includes: For each of the first dummy quantization nodes, determining a calibration model corresponding to a minimum quantization sensitivity based on the second quantization sensitivity of the first dummy quantization node in each of the second calibration models and the first quantization sensitivity of the first dummy quantization node in the first calibration model; The calibration mode of the calibration model corresponding to the minimum quantization sensitivity is determined as the target calibration mode of the first pseudo quantization node.

9. The method according to claim 3, wherein: The determining, based on the target calibration modes corresponding to the pseudo-quantization nodes, a target calibration model corresponding to the second neural network model includes: Determining a hybrid calibration model based on the second neural network model and the target calibration modes respectively corresponding to the pseudo quantization nodes; determining a fourth average quantization error of the hybrid calibration model based on the calibration data; The target calibration model is determined from the first calibration model and the hybrid calibration model based on a first average quantization error of the first calibration model and a fourth average quantization error of the hybrid calibration model.

10. The method according to any one of claims 1 to 9, wherein: Determining target quantization parameters corresponding to respective operators in the target calibration model includes: Determining, based on the calibration data, the quantization sensitivity of each operator in the target calibration model under at least one set of preset quantization parameters; any set of preset quantization parameters includes at least one of quantization accuracy, quantization granularity, and quantization mode; Based on the target calibration model and the quantization sensitivity of each operator under each preset quantization parameter, the target quantization parameter corresponding to each operator in the target calibration model is determined.

11. The method according to claim 10, wherein: The determining, based on the calibration data, the quantization sensitivity of each operator in the target calibration model under at least one set of preset quantization parameters includes: Determining a first quantization parameter that meets a second preset condition from the at least one set of preset quantization parameters; For each of the operators, determining, based on the calibration data and the first quantization parameter, a third quantization sensitivity of the target calibration model for individually quantizing the operator; Determining a first operator that meets a third preset condition from the operators based on the third quantization sensitivities respectively corresponding to the operators; determining, based on the calibration data and a second quantization parameter in each of the preset quantization parameters, a fourth quantization sensitivity of each of the first operators under each of the second quantization parameters, wherein the second quantization parameter is a preset quantization parameter in the at least one set of preset quantization parameters other than the first quantization parameter; The third quantization sensitivity corresponding to each of the operators and the fourth quantization sensitivity of each of the first operators under each of the second quantization parameters are determined as the quantization sensitivity of each of the operators under at least one set of preset quantization parameters.

12. The method according to claim 11, wherein The determining, based on the target calibration model and the quantization sensitivity of each operator under each preset quantization parameter, the target quantization parameter corresponding to each operator in the target calibration model includes: determining a target quantization parameter corresponding to each of the first operators based on the fourth quantization sensitivity of each of the first operators under each of the second quantization parameters and the third quantization sensitivity corresponding to each of the first operators; Determine that a target quantization parameter corresponding to a second operator among the operators is the first quantization parameter; the second operator is an operator among the operators except the first operator.

13. The method according to claim 11, wherein The determining, based on the calibration data and the first quantization parameter, a third quantization sensitivity of the target calibration model for quantizing the operator alone includes: determining, based on the calibration data and the first quantization parameter, that the target calibration model alone quantizes fifth output data of the operator; determining, based on the fifth output data and the first output data of the first neural network model, a fifth average quantization error of the operator under the first quantization parameter as the third quantization sensitivity of the operator under the first quantization parameter; and / or The determining, based on the calibration data and the second quantization parameter in each of the preset quantization parameters, the fourth quantization sensitivity of each of the first operators under each of the second quantization parameters includes: For each first operator and each second quantization parameter, determining, based on the calibration data and the second quantization parameter, the target calibration model to separately quantize sixth output data of the first operator; Based on the sixth output data and the first output data of the first neural network model, a sixth average quantization error of the first operator under the second quantization parameter is determined as the fourth quantization sensitivity of the first operator under the second quantization parameter.

14. The method according to any one of claims 1 to 9, wherein: The quantization parameters include at least two types of quantization accuracy, quantization granularity, and quantization mode; Determining target quantization parameters corresponding to respective operators in the target calibration model includes: Sequentially taking the at least two types as target types in a preset order, and determining the quantization parameters of the target types corresponding to the respective operators in the target calibration model by searching the quantization parameters of the target types layer by layer based on the calibration data; Based on the quantization parameters of each target type, the target quantization parameters corresponding to each operator are determined.

15. A quantization device for a neural network model, comprising: A first processing module, configured to determine a first neural network model to be quantized; a second processing module, configured to perform pseudo-quantization processing on the first neural network model to obtain a second neural network model, wherein the second neural network model includes pseudo-quantization nodes and operators; A third processing module is used to determine the target calibration mode corresponding to each of the pseudo quantization nodes; A fourth processing module, configured to determine a target calibration model corresponding to the second neural network model based on target calibration modes corresponding to the pseudo quantization nodes; a fifth processing module, configured to determine target quantization parameters corresponding to respective operators in the target calibration model; The sixth processing module is used to determine the quantization model corresponding to the first neural network model based on the target calibration model and the target quantization parameters corresponding to each of the operators.

16. A computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 14.

17. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method according to any one of claims 1 to 14.

Images