Neural network model quantification method and device, electronic equipment and storage medium

Abstract

The invention discloses a neural network model quantification method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, and the specific implementation scheme is as follows: obtaining a calibration data set and a to-be-quantized neural network model; performing model reasoning on the calibration data set by using a to-be-quantized neural network model to obtain a first model reasoning result of the to-be-quantized neural network model; determining a quantization parameter of each network layer in the neural network model to be quantized according to the first model reasoning result through a plurality of different quantization algorithms; and based on the quantization parameters of each network layer, generating a target quantization model corresponding to the neural network model to be quantized. With the adoption of the method and the device, the quantization parameters of each network layer obtained by different quantization algorithms are combined, so that the situation that a relatively large error is introduced by a single quantization scaling factor is avoided, the quantization error is reduced, and the accuracy of the target quantization model is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and particularly to a quantization method, apparatus, electronic device, and storage medium for a neural network model. Background Art

[0002] Currently, when a neural network model runs, it consumes a large amount of computing resources. Moreover, when the scale of the neural network model is large (such as a large model), the corresponding processing speed is slow, which may lead to a decline in the usage experience of the model service when computing resources are relatively scarce. Based on this, in order to reduce the system resources occupied by the operation of the neural network model and improve the usage experience of the neural network model, it is necessary to quantize the neural network model before deployment.

[0003] In the prior art, the quantization process is as follows: obtain a floating-point model, calculate the weight quantization scaling factor using the min-max method, calculate the activation value quantization scaling factor using the KL divergence (Kullback-Leibler Divergence) algorithm, then determine the optimal weight quantization scaling factor and the optimal activation value quantization scaling factor according to the cosine distance, and finally output the model with quantization information.

[0004] However, in the above prior art, a single quantization scaling factor may not be suitable for all network layers in the neural network model. Based on this, a single quantization scaling factor may introduce a large error and reduce the model accuracy. Summary of the Invention

[0005] The present disclosure provides a quantization method, apparatus, electronic device, and storage medium for a neural network model to at least solve the technical problem of large quantization error and low accuracy of the neural network model after quantization in the related art.

[0006] According to one aspect of the present disclosure, there is provided a quantization method for a neural network model, including:

[0007] Obtain a calibration data set and a neural network model to be quantized;

[0008] Use the neural network model to be quantized to perform model inference on the calibration data set to obtain a first model inference result of the neural network model to be quantized;

[0009] Determine the quantization parameters of each network layer in the neural network model to be quantized according to the first model inference result through multiple different quantization algorithms;

[0010] Generate a target quantization model corresponding to the neural network model to be quantized based on the quantization parameters of each network layer.

[0011] Optionally, determining the quantization parameters of each network layer in the neural network model to be quantized according to the first model inference result through a variety of different quantization algorithms includes:

[0012] Obtaining multiple initial truncation values corresponding to each network layer in the neural network model to be quantized according to the first model inference result through a variety of different quantization algorithms;

[0013] The neural network model to be quantized uses the multiple initial truncation values corresponding to each network layer to perform layer-by-layer model inference to determine the target truncation values corresponding to each network layer;

[0014] Determine the target truncation values corresponding to each network layer as the quantization parameters of each network layer in the neural network model to be quantized.

[0015] Optionally, the neural network model to be quantized uses the multiple initial truncation values corresponding to each network layer to perform layer-by-layer model inference to determine the target truncation values corresponding to each network layer, including:

[0016] Take the first network layer in the neural network model to be quantized as the current layer, and sequentially enable the quantization processing of the current layer through each initial truncation value of the current layer, without enabling the quantization processing of other network layers except the current layer, and sequentially generate the second model inference results corresponding to each initial truncation value of the current layer;

[0017] Based on the second model inference results corresponding to each initial truncation value of the current layer, determine the target truncation value corresponding to the current layer;

[0018] Repeat the above steps until the target truncation values corresponding to each network layer are determined.

[0019] Optionally, generating the target quantization model corresponding to the neural network model to be quantized based on the quantization parameters of each network layer includes:

[0020] Generating a first target quantization model corresponding to the neural network model to be quantized based on the quantization parameters of each network layer;

[0021] Perform a first quantization error analysis on the first target quantization model, and generate a second target quantization model based on the obtained first quantization error result;

[0022] Perform a second quantization error analysis on the second target quantization model, and generate a target quantization model based on the obtained second quantization error result.

[0023] Optionally, performing a first quantization error analysis on the first target quantization model, and generating a second target quantization model based on the obtained first quantization error result, includes:

[0024] Determine the first quantization error of the first target quantization model, and determine whether the first quantization error meets the first preset generation condition;

[0025] If the first quantization error meets the first preset generation condition, determine the first target quantization model as the second target quantization model;

[0026] If the first quantization error does not meet the first preset generation condition, adjust the quantization parameters of the first target quantization model, and generate a second target quantization model based on the adjusted quantization parameters.

[0027] Optionally, the adjusting the quantization parameters of the first target quantization model and generating a second target quantization model based on the adjusted quantization parameters includes:

[0028] Take the first network layer in the first target quantization model as the current layer, enable the quantization processing of the current layer, do not enable the quantization processing of other network layers except the current layer, and perform forward inference to generate the inference result of the first layer model of the first target quantization model;

[0029] Repeat the above steps until the inference results of all layer models of the first target quantization model are generated;

[0030] Based on the inference result of each layer model and the corresponding floating-point inference result, determine the first target network layer whose quantization parameters need to be adjusted;

[0031] Adjust the quantization parameters of the first target network layer, and generate a second target quantization model based on the adjusted quantization parameters.

[0032] Optionally, the performing a second quantization error analysis on the second target quantization model and generating a target quantization model based on the obtained second quantization error result includes:

[0033] Determine the second quantization error of the second target quantization model, and determine whether the second quantization error meets the second preset generation condition;

[0034] If the second quantization error meets the second preset generation condition, determine the second target quantization model as the target quantization model;

[0035] If the second quantization error does not meet the second preset generation condition, perform quantization-aware training on the second target quantization model to obtain a third target quantization model, and generate a target quantization model based on the third target quantization model.

[0036] Optionally, the generating a target quantization model based on the third target quantization model includes:

[0037] Determine the third quantization error of the third target quantization model, and determine whether the third quantization error meets the third preset generation condition;

[0038] If the third quantization error meets the third preset generation condition, determine the third target quantization model as the target quantization model;

[0039] If the third quantization error does not meet the third preset generation condition, adjust the quantization parameters of the third target quantization model, and repeat the quantization-aware training with the adjusted quantization parameters until the third quantization error meets the third preset generation condition.

[0040] Optionally, the adjustment of the quantization parameters of the third target quantization model includes:

[0041] Enable the weight dynamic quantization of all network layers of the third target quantization model;

[0042] Take the first network layer of the third target quantization model as the current layer, disable the weight dynamic quantization of the current layer, enable the weight dynamic quantization of other network layers except the current layer, and set the truncation value of the current layer to the truncation value of the same layer in the second target quantization model to obtain the third model inference result of the third target quantization model;

[0043] Repeat the above steps until the third model inference results of all network layers in the third target quantization model are generated;

[0044] Based on the third model inference result of each layer and the corresponding floating-point inference result, determine the second target network layer whose quantization parameters need to be adjusted in the third target quantization model, and adjust the quantization parameters of the second target network layer.

[0045] According to another aspect of the present disclosure, there is provided a quantization device for a neural network model, including:

[0046] An acquisition module, configured to acquire a calibration data set and a neural network model to be quantized;

[0047] An inference module, configured to use the neural network model to be quantized to perform model inference on the calibration data set to obtain a first model inference result of the neural network model to be quantized;

[0048] A determination module, configured to determine the quantization parameters of each network layer in the neural network model to be quantized according to the first model inference result through multiple different quantization algorithms;

[0049] A generation module, configured to generate a target quantization model corresponding to the neural network model to be quantized based on the quantization parameters of the respective network layers.

[0050] Optionally, the determination module is specifically configured to:

[0051] Obtain a plurality of initial truncation values corresponding to each network layer in the neural network model to be quantized according to the first model inference result through a variety of different quantization algorithms;

[0052] For the neural network model to be quantized, perform layer-by-layer model inference using the plurality of initial truncation values corresponding to each network layer to determine the target truncation value corresponding to each network layer;

[0053] Determine the target truncation value corresponding to each network layer as the quantization parameter of each network layer in the neural network model to be quantized.

[0054] Optionally, the determination module is further configured to:

[0055] Take the first network layer in the neural network model to be quantized as the current layer, and sequentially enable the quantization processing of the current layer through each initial truncation value of the current layer, without enabling the quantization processing for other network layers except the current layer, and sequentially generate the second model inference results corresponding to each initial truncation value of the current layer;

[0056] Based on the second model inference results corresponding to each initial truncation value of the current layer, determine the target truncation value corresponding to the current layer;

[0057] Repeat the above steps until the target truncation values corresponding to each network layer are determined.

[0058] Optionally, the generation module is specifically configured to:

[0059] Based on the quantization parameters of the respective network layers, generate a first target quantization model corresponding to the neural network model to be quantized;

[0060] Perform a first quantization error analysis on the first target quantization model, and generate a second target quantization model based on the obtained first quantization error result;

[0061] Perform a second quantization error analysis on the second target quantization model, and generate a target quantization model based on the obtained second quantization error result.

[0062] Optionally, the generation module is further configured to:

[0063] Determine the first quantization error of the first target quantization model, and determine whether the first quantization error meets the first preset generation condition;

[0064] If the first quantization error satisfies the first preset generation condition, the first target quantization model is determined as the second target quantization model;

[0065] If the first quantization error does not satisfy the first preset generation condition, the quantization parameters of the first target quantization model are adjusted, and a second target quantization model is generated based on the adjusted quantization parameters.

[0066] Optionally, the generation module is further configured to:

[0067] Take the first network layer in the first target quantization model as the current layer, enable the quantization processing of the current layer, do not enable the quantization processing for other network layers except the current layer, and perform forward inference to generate the first-layer model inference result of the first target quantization model;

[0068] Repeat the above steps until the inference results of all layers of the first target quantization model are generated;

[0069] Based on the inference result of each layer model and the corresponding floating-point inference result, determine the first target network layer whose quantization parameters need to be adjusted;

[0070] Adjust the quantization parameters of the first target network layer, and generate a second target quantization model based on the adjusted quantization parameters.

[0071] Optionally, the generation module is further configured to:

[0072] Determine the second quantization error of the second target quantization model, and determine whether the second quantization error satisfies the second preset generation condition;

[0073] If the second quantization error satisfies the second preset generation condition, the second target quantization model is determined as the target quantization model;

[0074] If the second quantization error does not satisfy the second preset generation condition, perform quantization-aware training on the second target quantization model to obtain a third target quantization model, and generate a target quantization model based on the third target quantization model.

[0075] Optionally, the generation module is further configured to:

[0076] Determine the third quantization error of the third target quantization model, and determine whether the third quantization error satisfies the third preset generation condition;

[0077] If the third quantization error satisfies the third preset generation condition, the third target quantization model is determined as the target quantization model;

[0078] If the third quantization error does not meet the third preset generation condition, adjust the quantization parameters of the third target quantization model, and repeat the quantization-aware training with the adjusted quantization parameters until the third quantization error meets the third preset generation condition.

[0079] Optionally, the generation module is further configured to:

[0080] Enable the weight dynamic quantization of all network layers of the third target quantization model;

[0081] Take the first network layer of the third target quantization model as the current layer, disable the weight dynamic quantization of the current layer, enable the weight dynamic quantization of other network layers except the current layer, and set the truncation value of the current layer to the truncation value of the same layer in the second target quantization model to obtain the third model inference result of the third target quantization model;

[0082] Repeat the above steps until the third model inference results of all network layers in the third target quantization model are generated;

[0083] Based on the third model inference result of each layer and the corresponding floating-point inference result, determine the second target network layer for which the quantization parameters of the third target quantization model need to be adjusted, and adjust the quantization parameters of the second target network layer.

[0084] According to another aspect of the present disclosure, there is provided an electronic device, including:

[0085] At least one processor; and

[0086] A memory communicatively connected to the at least one processor; wherein,

[0087] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of the foregoing aspects.

[0088] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method according to any one of the foregoing aspects.

[0089] According to another aspect of the present disclosure, there is provided a computer program product, including a computer program, where the computer program, when executed by a processor, implements the method according to any one of the foregoing aspects.

[0090] In one or more embodiments of the present disclosure, a calibration data set and a neural network model to be quantized are obtained; the calibration data set is used for model inference by the neural network model to be quantized to obtain a first model inference result of the neural network model to be quantized; according to the first model inference result, quantization parameters of each network layer in the neural network model to be quantized are determined through a variety of different quantization algorithms; and a target quantization model corresponding to the neural network model to be quantized is generated based on the quantization parameters of each network layer. Among them, by using a variety of different quantization algorithms to determine the quantization parameters suitable for each network layer according to the first model inference result, the quantization parameters of each network layer obtained by different quantization algorithms can be combined, thereby avoiding the situation of introducing large errors by a single quantization scaling factor, reducing the quantization error, and improving the accuracy of the target quantization model.

[0091] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] The drawings are used to better understand the solution and do not constitute a limitation to the present disclosure. Among them:

[0093] Figure 1 The flowchart showing the quantization method of the neural network model provided by an embodiment of the present disclosure;

[0094] Figure 2 The flowchart showing the quantization method of the neural network model provided by another embodiment of the present disclosure;

[0095] Figure 3 The flowchart showing the quantization method of the neural network model provided by an embodiment of the present disclosure;

[0096] Figure 4 The structural schematic diagram showing the quantization device of the neural network model provided by an embodiment of the present disclosure;

[0097] Figure 5 It is a block diagram of an electronic device for implementing the quantization method of the neural network model in the embodiments of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0098] The following describes exemplary embodiments of the present disclosure in conjunction with the drawings. Various details of the embodiments of the present disclosure are included to assist in understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, the description below omits the description of well-known functions and structures.

[0099] In the prior art, a single quantization scaling factor may introduce a large error, resulting in a poor image effect in the output of low-level tasks that focus on visual perception.

[0100] Figure 1 It is a schematic flowchart of a quantization method for a neural network model proposed by an embodiment of the present disclosure. As Figure 1 described, the quantization method for the neural network model includes:

[0101] S101, obtaining a calibration data set and a neural network model to be quantized.

[0102] Among them, the execution subject of the quantization method for the neural network model in this embodiment is a quantization device for the neural network model. This device can be implemented in a software and / or hardware manner and can be configured in an electronic device.

[0103] In addition, in some embodiments, the stored calibration data set and the target neural network model to be quantized can be obtained from a database. Among them, the calibration data set refers to the data set required to adjust the quantization parameters during the neural network quantization process and can be a subset of the validation data set. In addition, the neural network model to be quantized can be a high-precision floating-point model, and the neural network model to be quantized can include a convolutional layer and an activation layer.

[0104] S102, using the neural network model to be quantized to perform model inference on the calibration data set to obtain a first model inference result of the neural network model to be quantized.

[0105] In some embodiments, after obtaining the calibration data set and the neural network model to be quantized, the neural network model to be quantized can be used to perform model inference on the calibration data set to obtain a first model inference result of the neural network model to be quantized.

[0106] Specifically, in some embodiments, the calibration data set can be used for forward propagation model inference through each network layer of the neural network model to be quantized to obtain a data statistical histogram corresponding to each network layer, and the data statistical histogram corresponding to each network layer can be confirmed as the first model inference result.

[0107] S103, determining the quantization parameters of each network layer in the neural network model to be quantized according to the first model inference result through multiple different quantization algorithms.

[0108] In some embodiments, after obtaining the first model inference result through the above steps, the quantization parameters of each network layer in the neural network model to be quantized can be determined according to the first model inference result through multiple different quantization algorithms.

[0109] Among them, in some embodiments, the above-mentioned multiple different quantization algorithms include, but are not limited to, maximum-minimum, KL (Kullback Leibler) divergence quantization, and EQ (Easy Quant) quantization.

[0110] In addition, in some embodiments, the quantization parameters of the above-mentioned each network layer can be the truncation values of the activation values output by each network layer.

[0111] In some embodiments, the method for determining the quantization parameters of each network layer in the neural network model to be quantized according to the first model inference result through multiple different quantization algorithms may include: obtaining multiple initial truncation values corresponding to each network layer in the neural network model to be quantized through multiple different quantization algorithms according to the first model inference result; using the multiple initial truncation values corresponding to each network layer in the neural network model to be quantized to perform layer-by-layer model inference to determine the target truncation value corresponding to each network layer; and determining the target truncation value corresponding to each network layer as the quantization parameter of each network layer in the neural network model to be quantized. This part of the content will be introduced in detail in the subsequent embodiments.

[0112] In some embodiments, the above steps can determine the quantization parameters adapted to each network layer according to the first model inference result through multiple different quantization algorithms, so that the quantization parameters of each network layer obtained by different quantization algorithms can be combined, thereby avoiding the situation of introducing large errors by a single quantization scaling factor and reducing the quantization error.

[0113] S104. Generate a target quantization model corresponding to the neural network model to be quantized based on the quantization parameters of each network layer.

[0114] In some embodiments, after obtaining the quantization parameters of each network layer through the above steps, a target quantization model corresponding to the neural network model to be quantized can be generated based on the obtained quantization parameters of each network layer.

[0115] In some embodiments, the method for generating a target quantization model corresponding to the neural network model to be quantized based on the quantization parameters of each network layer may include: generating a first target quantization model corresponding to the neural network model to be quantized based on the quantization parameters of each network layer; performing a first quantization error analysis on the first target quantization model and generating a second target quantization model based on the obtained first quantization error result; and performing a second quantization error analysis on the second target quantization model and generating a target quantization model based on the obtained second quantization error result.

[0116] Among them, in some embodiments, after generating a first target quantization model corresponding to the neural network model to be quantized based on the quantization parameters of each network layer, perform quantization error analysis on the first target quantization model, and then adjust the quantization parameters of the first target quantization model based on the quantization error results to reduce the quantization error and improve the accuracy of the target quantization model.

[0117] In one or more embodiments of the present disclosure, obtain a calibration data set and a neural network model to be quantized; use the neural network model to be quantized to perform model inference on the calibration data set to obtain a first model inference result of the neural network model to be quantized; determine the quantization parameters of each network layer in the neural network model to be quantized according to the first model inference result through a variety of different quantization algorithms; generate a target quantization model corresponding to the neural network model to be quantized based on the quantization parameters of each network layer. Among them, by using a variety of different quantization algorithms according to the first model inference result, quantization parameters suitable for each network layer are determined, so that the quantization parameters of each network layer obtained by different quantization algorithms can be combined, thereby avoiding the situation of introducing large errors by a single quantization scaling factor, reducing the quantization error, and improving the accuracy of the target quantization model.

[0118] Figure 2 is a schematic flowchart of a quantization method for a neural network model proposed in an embodiment of the present disclosure, as Figure 2 described, the quantization method of the neural network model includes:

[0119] S201, obtain a calibration data set and a neural network model to be quantized.

[0120] S202, use the neural network model to be quantized to perform model inference on the calibration data set to obtain a first model inference result of the neural network model to be quantized.

[0121] For the description of S201-S202, specific reference can be made to the above embodiments and will not be elaborated here.

[0122] S203, obtain multiple initial truncation values corresponding to each network layer in the neural network model to be quantized according to the first model inference result through a variety of different quantization algorithms.

[0123] In some embodiments, a variety of different quantization algorithms can obtain multiple initial truncation values corresponding to each network layer according to the data statistical histogram corresponding to each network layer. Among them, the initial truncation values corresponding to each network layer for different quantization algorithms are different, and the initial truncation values corresponding to each network layer for the same quantization algorithm are also different.

[0124] Exemplarily, if there are two neural network layers in the neural network model to be quantized, the first neural network layer corresponds to initial truncation values 1, initial truncation values 2, ..., initial truncation values n; the second neural network layer corresponds to initial truncation values 11, initial truncation values 22, ..., initial truncation values mm, where n and mm are both the number of quantization algorithms.

[0125] S204. For the neural network model to be quantized, use the multiple initial truncation values corresponding to each network layer to perform layer-by-layer model inference, and determine the target truncation value corresponding to each network layer.

[0126] In some embodiments, for the neural network model to be quantized, use the multiple initial truncation values corresponding to each network layer to perform layer-by-layer model inference, and determine the target truncation value corresponding to each network layer from the multiple initial truncation values of each network layer according to the model inference results of the multiple initial truncation values of each network layer.

[0127] Specifically, in some embodiments, the method for the above-mentioned neural network model to be quantized to use the multiple initial truncation values corresponding to each network layer to perform layer-by-layer model inference and determine the target truncation value corresponding to each network layer may include the following steps:

[0128] S2041. Take the first network layer in the neural network model to be quantized as the current layer, sequentially enable the quantization process of the current layer through each initial truncation value of the current layer, do not enable the quantization process for other network layers except the current layer, and sequentially generate the second model inference results corresponding to each initial truncation value of the current layer.

[0129] S2042. Based on the second model inference results corresponding to each initial truncation value of the current layer, determine the target truncation value corresponding to the current layer.

[0130] S2043. Repeat the above steps until the target truncation values corresponding to each network layer are determined.

[0131] In some embodiments, after obtaining the second model inference results corresponding to each initial truncation value of each network layer through the above steps, the second model inference results corresponding to each initial truncation value may be compared with the corresponding floating-point inference results, and the initial truncation value corresponding to the second model inference result with the smallest error from the floating-point inference result is determined as the target truncation value corresponding to each network layer. Among them, the second model inference result with the smallest error from the floating-point inference result can be determined by manual identification or result comparison methods.

[0132] In some embodiments, the target truncation values corresponding to each network layer can be determined through the above steps, so that the quantization parameters of each network layer obtained by different quantization algorithms can be combined, thereby avoiding the situation of introducing large errors by a single quantization scaling factor and reducing the quantization error.

[0133] S205. Determine the target truncation values corresponding to each network layer as the quantization parameters of each network layer in the neural network model to be quantized.

[0134] S206. Generate a target quantization model corresponding to the neural network model to be quantized based on the quantization parameters of each network layer.

[0135] In one or more embodiments of the present disclosure, a calibration data set and a neural network model to be quantized are obtained; the calibration data set is used for model inference by the neural network model to be quantized to obtain a first model inference result of the neural network model to be quantized; quantization parameters of each network layer in the neural network model to be quantized are determined according to the first model inference result through a variety of different quantization algorithms; a target quantization model corresponding to the neural network model to be quantized is generated based on the quantization parameters of each network layer. Among them, by determining the quantization parameters adapted to each network layer according to the first model inference result through a variety of different quantization algorithms, the quantization parameters of each network layer obtained by different quantization algorithms can be combined, thereby avoiding the situation of introducing large errors by a single quantization scaling factor, reducing the quantization error, and improving the accuracy of the target quantization model.

[0136] Figure 3 is a schematic flowchart of a quantization method for a neural network model proposed in an embodiment of the present disclosure, as Figure 3 shown, the quantization method of the neural network model may include:

[0137] S301. Obtain a calibration data set and a neural network model to be quantized.

[0138] S302. Use the neural network model to be quantized to perform model inference on the calibration data set to obtain a first model inference result of the neural network model to be quantized.

[0139] S303. Determine the quantization parameters of each network layer in the neural network model to be quantized according to the first model inference result through a variety of different quantization algorithms.

[0140] For the descriptions of S301 - S303, specific reference can be made to the above embodiments, and details are not described herein again.

[0141] S304. Generate a first target quantization model corresponding to the neural network model to be quantized based on the quantization parameters of each network layer.

[0142] S305. Perform a first quantization error analysis on the first target quantization model and generate a second target quantization model based on the obtained first quantization error result.

[0143] In some embodiments, after obtaining the first target quantization model through the above steps, the first target quantization model can be subjected to a first quantization error analysis to obtain a first quantization error result, and a second target quantization model can be generated based on the obtained first quantization error result.

[0144] Specifically, in some embodiments, the method of performing a first quantization error analysis on the first target quantization model and generating a second target quantization model based on the obtained first quantization error result may include the following steps:

[0145] S3051, determine the first quantization error of the first target quantization model, and determine whether the first quantization error satisfies a first preset generation condition.

[0146] In some embodiments, the calibration data set can be used by the first target quantization model to perform model inference to obtain a model inference result of the first target quantization model, and the quantization error between the model inference result and the corresponding floating-point inference result can be determined as the first quantization error of the first target quantization model.

[0147] In some embodiments, the quantization error between the third model inference result and the corresponding floating-point inference result can be determined by manual discrimination or result comparison methods. Among them, when determining the quantization error between the third model inference and the corresponding floating-point inference result by manual discrimination, the quantization error between the third model inference and the corresponding floating-point inference result can be scored manually. The higher the score, the greater the quantization error, and the lower the score, the smaller the quantization error. Also, when determining the quantization error between the third model inference and the corresponding floating-point inference result by the result comparison method, the third model inference and the corresponding floating-point inference result can be input into the trained comparison model to obtain the difference value between the third model inference and the corresponding floating-point inference result. The higher the difference value, the greater the quantization error, and the lower the difference value, the smaller the quantization error.

[0148] In some embodiments, after determining the first quantization error of the first target quantization model, it can be determined whether the first quantization error satisfies a first preset generation condition. Among them, the first preset generation condition may be that the first quantization error corresponding to each data in the calibration data set is less than a first preset threshold, that is, the accuracy of the third model inference obtained through the first target quantization model meets the requirements. Among them, in some embodiments, if the first quantization errors are all less than the first preset threshold, it is determined that the first preset generation condition is satisfied; otherwise, it is determined that the first preset generation condition is not satisfied.

[0149] S3052, if the first quantization error satisfies the first preset generation condition, the first target quantization model is determined as the second target quantization model.

[0150] In some embodiments, if the first quantization error meets the first preset generation condition, that is, the first quantization error corresponding to each data in the calibration dataset is less than the first preset threshold, then the first target quantization model is determined as the second target quantization model.

[0151] S3053, if the first quantization error does not meet the first preset generation condition, then adjust the quantization parameters of the first target quantization model, and generate the second target quantization model based on the adjusted quantization parameters.

[0152] In some embodiments, if the first quantization error does not meet the first preset generation condition, that is, there is data in the calibration dataset whose first quantization error is greater than or equal to the first preset threshold. At this time, it indicates that there may be a relatively large quantization error in the quantization parameters of the first target quantization model. Then, it is necessary to adjust the quantization parameters of the first target quantization model, and generate the second target quantization model based on the adjusted quantization parameters.

[0153] Specifically, in some embodiments, the method of adjusting the quantization parameters of the first target quantization model and generating the second target quantization model based on the adjusted quantization parameters may include the following steps:

[0154] Step 1: Take the first network layer in the first target quantization model as the current layer, enable the quantization processing of the current layer, do not enable the quantization processing of other network layers except the current layer, and perform forward inference to generate the inference result of the first layer model of the first target quantization model.

[0155] Step 2: Repeat the above steps until the inference results of all layer models of the first target quantization model are generated.

[0156] Step 3: Based on the inference result of each layer model and the corresponding floating-point inference result, determine the first target network layer whose quantization parameters need to be adjusted.

[0157] Step 4: Adjust the quantization parameters of the first target network layer, and generate the second target quantization model based on the adjusted quantization parameters.

[0158] In some embodiments, through the above steps 1 - 3, the inference result of each layer model can be compared with the corresponding floating-point inference result, and the layer with the largest quantization error compared with the floating-point inference result is determined as the first target network layer whose quantization parameters need to be adjusted; or the network layers with the largest quantization error compared with the floating-point inference result are sorted in descending order, and the network layers of the preset number of layers at the front are determined as the first target network layer whose quantization parameters need to be adjusted, so as to accurately locate the origin of the quantization error. Among them, the method of comparing the inference result of each layer model with the corresponding floating-point inference result can refer to the detailed results in the above embodiments, and the embodiments of the present disclosure will not be elaborated here.

[0159] In some embodiments, after determining the first target network layer for which the quantization parameter needs to be adjusted, the truncation value can be scaled by the rounding error or truncation error to adjust the quantization parameter of the first target network layer, and a second target quantization model is generated based on the adjusted quantization parameter.

[0160] For example, in some embodiments, it is assumed that the neural network model to be quantized is a low-level task. Through the above single-point quantization method, layer-by-layer inference is performed and the corresponding inference images are output. The number of inference images should be the same as the number of network layers. Then, all the output images are manually compared with the floating-point inference results one by one to determine the layer with the largest difference from the floating-point result, and the first target network layer is determined. Then, the truncation value of the first target network layer is scaled according to the rounding error or truncation error. Based on this, the above method can, for tasks with a focus on low-level perception, determine the first target network layer layer by layer and perform truncation adjustment on it, so as to reduce the cumulative error and improve the image effect output in low-level tasks with a focus on perception.

[0161] In some embodiments, the above method adopts the method of automatically analyzing and adjusting the truncation value, and combines the truncation values generated by multiple different quantization algorithms through the corresponding inference effects, thereby reducing the quantization error and making it closer to the floating-point inference result.

[0162] S306. Perform a second quantization error analysis on the second target quantization model, and generate a target quantization model based on the obtained second quantization error result.

[0163] In some embodiments, after determining the second target quantization model through the above steps, a second quantization error analysis can be performed on the second target quantization model, and a target quantization model is generated based on the obtained second quantization error result to further improve the accuracy of the quantized model.

[0164] Specifically, in some embodiments, the method of performing a second quantization error analysis on the second target quantization model and generating a target quantization model based on the obtained second quantization error result may include the following steps:

[0165] S3061. Determine the second quantization error of the second target quantization model, and determine whether the second quantization error meets the second preset generation condition;

[0166] S3062. If the second quantization error meets the second preset generation condition, determine the second target quantization model as the target quantization model;

[0167] S3063. If the second quantization error does not meet the second preset generation condition, perform quantization-aware training on the second target quantization model to obtain a third target quantization model, and generate a target quantization model based on the third target quantization model.

[0168] In some embodiments, steps S3061 - S3062 may refer to the detailed introduction of the above steps S3051 - S3052. Among them, the first preset generation condition and the second preset generation condition may be the same or different. And when the first preset generation condition is different from the second preset generation condition, the first preset threshold in the first preset generation condition is different from the second preset threshold in the second preset generation condition.

[0169] In some embodiments, if the second quantization error does not meet the second preset generation condition, that is, there is data in the calibration dataset whose corresponding second quantization error is greater than or equal to the second preset threshold. At this time, it indicates that there may be a relatively large quantization error in the quantization parameters of the second target quantization model. Then, perform quantization-aware training on the second target quantization model to obtain a third target quantization model, and generate a target quantization model based on the third target quantization model. In some embodiments, the above quantization-aware training method is the same as the prior art, and the specific details can refer to the detailed introduction of the prior art. This disclosure embodiment will not elaborate here.

[0170] In some embodiments, after obtaining the third target quantization model through the above steps, a target quantization model can be generated based on the third target quantization model. Specifically, in some embodiments, the method for generating a target quantization model based on the third target quantization model may include the following steps:

[0171] Step 1: Determine the third quantization error of the third target quantization model, and determine whether the third quantization error meets the third preset generation condition;

[0172] Step 2: If the third quantization error meets the third preset generation condition, then determine the third target quantization model as the target quantization model;

[0173] Step 3: If the third quantization error does not meet the third preset generation condition, then adjust the quantization parameters of the third target quantization model, and repeat the quantization-aware training with the adjusted quantization parameters until the third quantization error meets the third preset generation condition.

[0174] In some embodiments, steps 1 - 2 may refer to the detailed introduction of the above steps S3051 - S3052. Among them, the third preset generation condition and the second preset generation condition may be the same or different. And when the third preset generation condition is different from the second preset generation condition, the third preset threshold in the third preset generation condition is different from the second preset threshold in the second preset generation condition.

[0175] In some embodiments, if the third quantization error does not meet the third preset generation condition, that is, there is data in the calibration dataset whose corresponding third quantization error is greater than or equal to the third preset threshold. At this time, it indicates that there may be a relatively large quantization error in the quantization parameters of the third target quantization model. Then, it is necessary to adjust the quantization parameters of the third target quantization model and repeat the quantization-aware training with the adjusted quantization parameters until the third quantization error meets the third preset generation condition.

[0176] Among them, in some embodiments, the method for adjusting the quantization parameters of the third target quantization model may include the following steps:

[0177] Step a: Enable the weight dynamic quantization of all network layers of the third target quantization model;

[0178] Step b: Take the first network layer of the third target quantization model as the current layer, disable the weight dynamic quantization of the current layer, enable the weight dynamic quantization of other network layers except the current layer, and set the truncation value of the current layer to the truncation value of the same layer in the second target quantization model to obtain the third model inference result of the third target quantization model;

[0179] Step c: Repeat the above steps until the third model inference results of all network layers in the third target quantization model are generated;

[0180] Step d: Based on the third model inference result of each layer and the corresponding floating-point inference result, determine the second target network layer whose quantization parameters need to be adjusted in the third target quantization model, and adjust the quantization parameters of the second target network layer.

[0181] In some embodiments, the model weight w is determined when training and generating the neural network model to be quantized. Among them, a neural network is composed of multiple neurons, and each neuron can be regarded as a function: y = w * x + b. Where x is the input, y is the output, w is the weight, the weight is a parameter in the neuron, b is the bias term, and * can include dense operations such as convolution (Conv), transposed convolution (Transpose Conv), and matrix multiplication (Linear).

[0182] And, in some embodiments, the above weight dynamic quantization is to dynamically calculate the truncation value of the weight, so that the truncation value of the weight is the maximum truncation value of the forward data this time. That is, the minmax method is used to calculate the truncation value of the weight of each network layer.

[0183] In some embodiments, steps b and c are to gradually disable the weight dynamic quantization and use the truncation value of the same layer in the second target quantization model for forward propagation to disable the dynamic quantization.

[0184] In addition, in some embodiments, after obtaining the third model inference results of all network layers in the third target quantization model, the third model inference result of each layer can be compared with the corresponding floating-point inference result, and the layer with the largest quantization error compared with the floating-point inference result is determined as the second target network layer whose quantization parameters need to be adjusted; or the quantization errors compared with the floating-point inference results are sorted in descending order, and the network layers of the preset number of layers at the front are determined as the second target network layers whose quantization parameters need to be adjusted, so as to locate the quantization node defects. Among them, the method of comparing the third model inference result of each layer with the corresponding floating-point inference result can refer to the detailed results in the above embodiments, and the embodiments of the present disclosure will not elaborate herein.

[0185] In some embodiments, after determining the second target network layer whose quantization parameters need to be adjusted, the truncation value can be scaled through the truncation error to adjust the quantization parameters of the third target network layer. It should be noted that in some embodiments, the above method can detect quantization nodes with truncation errors. If it is necessary to detect quantization nodes with rounding errors, it can be detected through the forward results of int16.

[0186] In one or more embodiments of the present disclosure, a calibration data set and a neural network model to be quantized are obtained; the calibration data set is used for model inference by the neural network model to be quantized to obtain the first model inference result of the neural network model to be quantized; according to the first model inference result, quantization parameters of each network layer in the neural network model to be quantized are determined through a variety of different quantization algorithms; based on the quantization parameters of each network layer, a target quantization model corresponding to the neural network model to be quantized is generated. Among them, according to the first model inference result, quantization parameters suitable for each network layer are determined through a variety of different quantization algorithms, so that the quantization parameters of each network layer obtained by different quantization algorithms can be combined, thereby avoiding the situation of introducing large errors by a single quantization scaling factor, reducing the quantization error, and improving the accuracy of the target quantization model.

[0187] Please refer to Figure 4 , which shows a schematic structural diagram of a quantization device for a neural network model provided by an embodiment of the present disclosure. The quantization device 400 for the neural network model includes an acquisition module 401, an inference module 402, a determination module 403, and a generation module 404, where:

[0188] The acquisition module 401 is configured to acquire a calibration data set and a neural network model to be quantized;

[0189] The inference module 402 is configured to use the neural network model to be quantized to perform model inference on the calibration data set to obtain the first model inference result of the neural network model to be quantized;

[0190] A determination module 403, configured to determine quantization parameters of each network layer in a neural network model to be quantized according to a first model inference result through multiple different quantization algorithms;

[0191] A generation module 404, configured to generate a target quantization model corresponding to the neural network model to be quantized based on the quantization parameters of each network layer.

[0192] Optionally, the above-mentioned determination module 403 is specifically configured to:

[0193] Obtain multiple initial truncation values corresponding to each network layer in the neural network model to be quantized according to the first model inference result through multiple different quantization algorithms;

[0194] For the neural network model to be quantized, perform layer-by-layer model inference using the multiple initial truncation values corresponding to each network layer to determine the target truncation value corresponding to each network layer;

[0195] Determine the target truncation value corresponding to each network layer as the quantization parameter of each network layer in the neural network model to be quantized.

[0196] Optionally, the above-mentioned determination module 403 is further configured to:

[0197] Take the first network layer in the neural network model to be quantized as the current layer, and sequentially enable the quantization processing of the current layer through each initial truncation value of the current layer, without enabling the quantization processing for other network layers except the current layer, and sequentially generate second model inference results corresponding to each initial truncation value of the current layer;

[0198] Based on the second model inference results corresponding to each initial truncation value of the current layer, determine the target truncation value corresponding to the current layer;

[0199] Repeat the above steps until the target truncation values corresponding to each network layer are determined.

[0200] Optionally, the above-mentioned generation module 404 is specifically configured to:

[0201] Generate a first target quantization model corresponding to the neural network model to be quantized based on the quantization parameters of each network layer;

[0202] Perform a first quantization error analysis on the first target quantization model, and generate a second target quantization model based on the obtained first quantization error result;

[0203] Perform a second quantization error analysis on the second target quantization model, and generate a target quantization model based on the obtained second quantization error result.

[0204] Optionally, the above-mentioned generation module 404 is further configured to:

[0205] Determine the first quantization error of the first target quantization model, and determine whether the first quantization error meets the first preset generation condition;

[0206] If the first quantization error meets the first preset generation condition, determine the first target quantization model as the second target quantization model;

[0207] If the first quantization error does not meet the first preset generation condition, adjust the quantization parameters of the first target quantization model, and generate a second target quantization model based on the adjusted quantization parameters.

[0208] Optionally, the above generation module 404 is further configured to:

[0209] Take the first network layer in the first target quantization model as the current layer, enable the quantization processing of the current layer, do not enable the quantization processing of other network layers except the current layer, and perform forward inference to generate the inference result of the first layer model of the first target quantization model;

[0210] Repeat the above steps until the inference results of all layer models of the first target quantization model are generated;

[0211] Based on the inference result of each layer model and the corresponding floating-point inference result, determine the first target network layer whose quantization parameters need to be adjusted;

[0212] Adjust the quantization parameters of the first target network layer, and generate a second target quantization model based on the adjusted quantization parameters.

[0213] Optionally, the above generation module 404 is further configured to:

[0214] Determine the second quantization error of the second target quantization model, and determine whether the second quantization error meets the second preset generation condition;

[0215] If the second quantization error meets the second preset generation condition, determine the second target quantization model as the target quantization model;

[0216] If the second quantization error does not meet the second preset generation condition, perform quantization-aware training on the second target quantization model to obtain a third target quantization model, and generate a target quantization model based on the third target quantization model.

[0217] Optionally, the above generation module 404 is further configured to:

[0218] Determine the third quantization error of the third target quantization model, and determine whether the third quantization error meets the third preset generation condition;

[0219] If the third quantization error meets the third preset generation condition, determine the third target quantization model as the target quantization model;

[0220] If the third quantization error does not meet the third preset generation condition, the quantization parameters of the third target quantization model are adjusted, and quantization-aware training is repeated with the adjusted quantization parameters until the third quantization error meets the third preset generation condition.

[0221] Optionally, the above-mentioned generation module 404 is further configured to:

[0222] Enable the weight dynamic quantization of all network layers of the third target quantization model;

[0223] Take the first network layer of the third target quantization model as the current layer, disable the weight dynamic quantization of the current layer, enable the weight dynamic quantization of other network layers except the current layer, and set the truncation value of the current layer to the truncation value of the same layer in the second target quantization model to obtain the third model inference result of the third target quantization model;

[0224] Repeat the above steps until the third model inference results of all network layers in the third target quantization model are generated;

[0225] Based on the third model inference result of each layer and the corresponding floating-point inference result, determine the second target network layer for which the third target quantization model needs to adjust the quantization parameters, and adjust the quantization parameters of the second target network layer.

[0226] Figure 5 FIG. shows a schematic block diagram of an exemplary electronic device 500 that can be used to implement the embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processing, cellular phones, smart phones, wearable electronic devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0227] As Figure 5 shown, the electronic device 500 includes a computing unit 501, which can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 502 or the computer program loaded from the storage unit 508 into the random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. The input / output (I / O) interface 505 is also connected to the bus 504.

[0228] Multiple components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disc, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the electronic device 500 to exchange information / data with other electronic devices through a computer network such as the Internet and / or various telecommunication networks.

[0229] The computing unit 501 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 executes the various methods and processes described above, such as the quantization method of a neural network model. For example, in some embodiments, the quantization method of a neural network model can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the quantization method of the neural network model described above can be executed. Alternatively, in other embodiments, the computing unit 501 can be configured to execute the quantization method of the neural network model by any other suitable means (e.g., by means of firmware).

[0230] The various embodiments of the systems and technologies described above in this article can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs, which can be executed and / or interpreted on a programmable system including at least one programmable processor, and the programmable processor can be a special or general-purpose programmable processor, and can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0231] The program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program codes cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0232] In the context of the present disclosure, a machine-readable medium may be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or electronic device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or electronic devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0233] In order to provide interaction with a user, the systems and techniques described herein may be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0234] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), the Internet, and blockchain network.

[0235] A computer system can include a client and a server. The client and the server are generally far from each other and usually interact through a communication network. The client-server relationship is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services ("Virtual Private Server", or simply "VPS"). The server can also be a server of a distributed system, or a server combined with blockchain.

[0236] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in this disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this is not limited herein.

[0237] The above specific embodiments do not constitute a limitation on the protection scope of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the protection scope of this disclosure.

Claims

1. A quantization method for a neural network model, characterized in that, Including: Obtain a calibration dataset and a neural network model to be quantized; Using the neural network model to be quantized, perform model inference on the calibration dataset to obtain a first model inference result of the neural network model to be quantized; According to the first model inference result, determine quantization parameters of each network layer in the neural network model to be quantized through multiple different quantization algorithms; Based on the quantization parameters of each network layer, generate a target quantization model corresponding to the neural network model to be quantized.

2. The method according to claim 1, characterized in that, The step of determining quantization parameters of each network layer in the neural network model to be quantized according to the first model inference result through multiple different quantization algorithms includes: According to the first model inference result, obtain multiple initial truncation values corresponding to each network layer in the neural network model to be quantized through multiple different quantization algorithms; The neural network model to be quantized performs layer-by-layer model inference using the multiple initial truncation values corresponding to each network layer to determine target truncation values corresponding to each network layer; Determine the target truncation values corresponding to each network layer as the quantization parameters of each network layer in the neural network model to be quantized.

3. The method according to claim 2, wherein The neural network model to be quantized performs layer-by-layer model inference using the multiple initial truncation values corresponding to each network layer to determine target truncation values corresponding to each network layer, including: Take the first network layer in the neural network model to be quantized as the current layer, and sequentially enable the quantization processing of the current layer through each initial truncation value of the current layer, without enabling the quantization processing of other network layers except the current layer, and sequentially generate second model inference results corresponding to each initial truncation value of the current layer; Based on the second model inference results corresponding to each initial truncation value of the current layer, determine the target truncation value corresponding to the current layer; Repeat the above steps until the target truncation values corresponding to each network layer are determined.

4. The method according to claim 1, wherein The step of generating a target quantization model corresponding to the neural network model to be quantized based on the quantization parameters of each network layer includes: Based on the quantization parameters of each network layer, generate a first target quantization model corresponding to the neural network model to be quantized; Perform a first quantization error analysis on the first target quantization model, and generate a second target quantization model based on the obtained first quantization error result; Perform a second quantization error analysis on the second target quantization model, and generate a target quantization model based on the obtained second quantization error result.

5. The method according to claim 4, wherein The step of performing a first quantization error analysis on the first target quantization model and generating a second target quantization model based on the obtained first quantization error result includes: Determine the first quantization error of the first target quantization model, and determine whether the first quantization error meets a first preset generation condition; If the first quantization error meets the first preset generation condition, determine the first target quantization model as the second target quantization model; If the first quantization error does not meet the first preset generation condition, adjust the quantization parameters of the first target quantization model, and generate a second target quantization model based on the adjusted quantization parameters.

6. The method according to claim 5, characterized in that, Adjusting the quantization parameters of the first target quantization model and generating a second target quantization model based on the adjusted quantization parameters includes: Taking the first network layer in the first target quantization model as the current layer, enabling the quantization processing of the current layer, disabling the quantization processing of other network layers except the current layer, and performing forward inference to generate the inference result of the first layer model of the first target quantization model; Repeating the above steps until the inference results of all layer models of the first target quantization model are generated; Determining the first target network layer whose quantization parameters need to be adjusted based on the inference result of each layer model and the corresponding floating-point inference result; Adjusting the quantization parameters of the first target network layer and generating a second target quantization model based on the adjusted quantization parameters.

7. The method according to claim 4, characterized in that Performing a second quantization error analysis on the second target quantization model and generating a target quantization model based on the obtained second quantization error result includes: Determining the second quantization error of the second target quantization model and determining whether the second quantization error meets the second preset generation condition; If the second quantization error meets the second preset generation condition, determining the second target quantization model as the target quantization model; If the second quantization error does not meet the second preset generation condition, performing quantization-aware training on the second target quantization model to obtain a third target quantization model, and generating a target quantization model based on the third target quantization model.

8. The method according to claim 7, wherein Generating a target quantization model based on the third target quantization model includes: Determining the third quantization error of the third target quantization model and determining whether the third quantization error meets the third preset generation condition; If the third quantization error meets the third preset generation condition, determining the third target quantization model as the target quantization model; If the third quantization error does not meet the third preset generation condition, adjusting the quantization parameters of the third target quantization model, and repeating the quantization-aware training with the adjusted quantization parameters until the third quantization error meets the third preset generation condition.

9. The method according to claim 8, characterized in that, Adjusting the quantization parameters of the third target quantization model includes: Enabling the weight dynamic quantization of all network layers of the third target quantization model; Taking the first network layer in the third target quantization model as the current layer, disabling the weight dynamic quantization of the current layer, enabling the weight dynamic quantization of other network layers except the current layer, and setting the truncation value of the current layer to the truncation value of the same layer in the second target quantization model to obtain the inference result of the third model of the third target quantization model; Repeating the above steps until the inference results of all network layers of the third target quantization model are generated; Based on the inference result of each layer of the third model and the corresponding floating-point inference result, determining the second target network layer of the third target quantization model whose quantization parameters need to be adjusted, and adjusting the quantization parameters of the second target network layer.

10. A quantization device for a neural network model, characterized in that, Including: An acquisition module for acquiring a calibration data set and a neural network model to be quantized; An inference module, configured to use the neural network model to be quantized to perform model inference on the calibration data set to obtain a first model inference result of the neural network model to be quantized; A determination module, configured to determine quantization parameters of each network layer in the neural network model to be quantized according to the first model inference result through a variety of different quantization algorithms; A generation module, configured to generate a target quantization model corresponding to the neural network model to be quantized based on the quantization parameters of each network layer.

11. An electronic device, characterized in that, Comprising: At least one processor; And A memory communicatively connected to the at least one processor, wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1-9.

12. A computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions; after the computer-executable instructions are executed by a processor, the method according to any one of claims 1-9 can be implemented.

Images