A neural network model quantization method, system, device and medium

Abstract

The application provides a neural network model quantification method, system, device and medium, first, an original neural network model to be quantified and a preset precision loss range are acquired; then all network layers in the original neural network model are quantified, and a quantified neural network model is acquired, which is denoted as a quantified neural network model; then a target picture is input into the original neural network model and the quantified neural network model respectively for identification, and a precision loss of the quantified neural network model relative to the original neural network model is acquired; finally, the precision loss is compared with the preset precision loss range, and a quantified neural network model is output according to a comparison result, or the quantified neural network model is iteratively quantified according to the comparison result. It can be known that the application designs a quantification standard of a quantification model which can be automatically optimized and adjusted according to a preset precision loss range; and an optimal mixed precision quantification model can also be returned without manual intervention.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, system, device and medium for quantizing neural network models. Background Technology

[0002] Neural network model quantization is a widely used acceleration algorithm in the deployment phase. Quantization is the process of representing a previous high-precision neural network model with a low-precision neural network model, or representing a previous high-bit neural network model with a low-bit neural network model, thereby enabling the neural network model to accelerate computation while reducing memory usage.

[0003] However, due to the limited range of low-precision or low-bit-level representation, quantization often involves performance loss. In deep neural networks, this quantization loss accumulates layer by layer, leading to a significant drop in the recognition performance of the final trained deep neural network model, sometimes even reaching unacceptable levels. Therefore, existing technologies have begun to employ mixed-precision quantization. This is because some layers of the network can tolerate lower precision compared to others; that is, using partial quantization of the network to achieve mixed-precision inference, thus improving speed while maintaining accuracy. However, when using mixed-precision quantization, there is currently no completely unified standard for selecting which network layers to quantize and which to retain high-precision mode. Furthermore, current quantization methods are primarily based on manual adjustment and testing, which is not only inconvenient but also fails to achieve the desired accuracy. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a method, system, device and medium for quantizing neural network models, so as to solve the problems existing in the prior art when quantizing neural network models.

[0005] To achieve the above and other related objectives, the present invention provides a method for quantizing neural network models, the method comprising the following steps:

[0006] Obtain the original neural network model to be quantized and the preset accuracy loss range;

[0007] Quantize all network layers in the original neural network model to obtain the quantized neural network model, denoted as the quantized neural network model.

[0008] The target image is input into the original neural network model and the quantized neural network model respectively for recognition, and the accuracy loss of the quantized neural network model relative to the original neural network model is obtained;

[0009] The precision loss is compared with the preset precision loss range, and the quantized neural network model is output according to a comparison result or the quantized neural network model is iteratively quantized according to the comparison result.

[0010] Optionally, the process of quantizing all network layers in the original neural network model comprises:

[0011] The precision of each network layer in the original neural network model is obtained, and the precision of each network layer is reduced.

[0012] Alternatively, the bit of the precision of each network layer in the original neural network model is obtained, and the bit of the precision of each network layer is reduced.

[0013] Optionally, the manner of reducing the precision of each network layer comprises at least one of reducing single-precision floating-point precision to half-precision floating-point precision, reducing single-precision floating-point precision to eight-bit integer precision, and reducing half-precision floating-point precision to eight-bit integer precision.

[0014] Optionally, if the precision loss is within the preset precision loss range, the quantized neural network model is output.

[0015] If the precision loss is not within the preset precision loss range, the quantized neural network model is iteratively quantized until the precision loss corresponding to a new quantized neural network model is within the preset precision loss range, the iterative quantization is terminated, and the corresponding new quantized neural network model is output as a final quantized neural network model.

[0016] Optionally, the process of iteratively quantizing the quantized neural network model comprises:

[0017] The target picture is input into the original neural network model and the quantized neural network model for identification, respectively, and the absolute value of the inference acceleration time difference of each network layer in the quantized neural network model and the corresponding network layer in the original neural network model and the information loss of each network layer in the quantized neural network model relative to the corresponding network layer in the original neural network model are obtained.

[0018] According to the absolute value of the inference acceleration time difference of each network layer in the quantized neural network model and the information loss, the information loss ratio of each network layer in the quantized neural network model is calculated.

[0019] The quantized neural network model is iteratively quantized according to the information loss ratio of each network layer.

[0020] Optionally, the process of iteratively quantizing the quantized neural network model according to the information loss ratio of each network layer comprises:

[0021] sorting the information loss ratios of all network layers in the quantized neural network model;

[0022] obtaining the precision of the network layer corresponding to the largest information loss ratio after sorting, denoted as the to-be-quantized precision;

[0023] improving the to-be-quantized precision to obtain a new quantized neural network model;

[0024] inputting the target picture into the original neural network model and the new neural network model respectively for recognition, obtaining the precision loss of the new neural network model relative to the original neural network model, denoted as the intermediate precision loss;

[0025] comparing the intermediate precision loss with the preset precision loss range;

[0026] if the intermediate precision loss is within the preset precision loss range, the current new quantized neural network model is taken as the final quantized neural network model;

[0027] if the intermediate precision loss is not within the preset precision loss range, the precision of the corresponding network layer is improved in turn according to the sorting result of the information loss ratio, and then iteratively quantized, until the precision loss of the final quantized neural network model is within the preset precision loss range, and the iterative quantization is terminated.

[0028] Optionally, the process of inputting the target picture into the original neural network model and the quantized neural network model respectively for recognition, and obtaining the information loss of each network layer in the quantized neural network model relative to the corresponding network layer in the original neural network model comprises:

[0029] inputting the target picture into each network layer of the original neural network model for recognition, and taking the obtained probability distribution result as the first recognition result;

[0030] inputting the target picture into each network layer of the quantized neural network model for recognition, and taking the obtained probability distribution result as the second recognition result;

[0031] calculating the distance between the first recognition result and the second recognition result, and determining the information loss of each network layer in the quantized neural network model relative to the corresponding network layer in the original neural network model according to the distance calculation result; wherein the distance comprises at least one of the following: cosine distance, Euclidean distance.

[0032] Optionally, the original neural network model to be quantized comprises at least one of: a neural network model with single-precision floating-point precision, and a neural network model with half-precision floating-point precision.

[0033] The application further provides a neural network model quantization system, comprising:

[0034] An acquisition module is configured to acquire an original neural network model to be quantized and a preset precision loss range;

[0035] An initial quantization module is configured to quantize all network layers in the original neural network model to obtain a quantized neural network model, denoted as a quantized neural network model;

[0036] A precision loss module is configured to input a target picture into the original neural network model and the quantized neural network model respectively for identification, and obtain a precision loss of the quantized neural network model relative to the original neural network model;

[0037] A comparison module is configured to compare the precision loss with the preset precision loss range;

[0038] An iterative quantization module is configured to perform iterative quantization on the quantized neural network model when the precision loss is not located in the preset precision loss range.

[0039] An output module is configured to output a corresponding quantized neural network model when the precision loss is located in the preset precision loss range.

[0040] The application further provides a computer device, comprising:

[0041] one or more processors; and

[0042] A computer readable medium having instructions stored thereon, which, when executed by the one or more processors, cause the device to perform the method of any one of the above.

[0043] The application further provides a computer readable medium having instructions stored thereon, which, when executed by the one or more processors, cause the device to perform the method of any one of the above.

[0044] As described above, this invention provides a method, system, device, and medium for quantizing neural network models, which has the following beneficial effects: First, this invention obtains the original neural network model to be quantized and a preset precision loss range; then, it quantizes all network layers in the original neural network model to obtain the quantized neural network model, denoted as the quantized neural network model; next, it inputs the target image into both the original neural network model and the quantized neural network model for recognition, and obtains the precision loss of the quantized neural network model relative to the original neural network model; finally, it compares the precision loss with the preset precision loss range, and outputs the quantized neural network model based on the comparison result, or iteratively quantizes the quantized neural network model based on the comparison result. Therefore, this invention designs a quantization standard that can automatically optimize and adjust the quantization model according to a preset precision loss range; it can also return the optimal hybrid precision quantization model without manual intervention. Essentially, this invention can automatically perform iterative quantization based on the set quantization parameters and adaptively search for a hybrid quantization model that meets the conditions. Attached Figure Description

[0045] Figure 1 A flowchart illustrating a neural network model quantization method provided in one embodiment;

[0046] Figure 2 A flowchart illustrating a neural network model quantization method provided in another embodiment;

[0047] Figure 3 A schematic diagram of the hardware structure of a neural network model quantization system provided in one embodiment;

[0048] Figure 4 A schematic diagram of the hardware structure of a terminal device provided in one embodiment;

[0049] Figure 5 A schematic diagram of the hardware structure of a terminal device provided in another embodiment.

[0050] Component designation explanation

[0051] 1100 Input Device

[0052] 1101 First Processor

[0053] 1102 Output device

[0054] 1103 First Memory

[0055] 1104 Communication Bus

[0056] 1200 processing components

[0057] 1201 Second Processor

[0058] 1202 second memory

[0059] 1203 communication component

[0060] 1204 power component

[0061] 1205 multimedia component

[0062] 1206 audio component

[0063] 1207 input / output interface

[0064] 1208 sensor component DETAILED DESCRIPTION

[0065] The present application is herein described, by way of example only, with reference to embodiments thereof. It is to be understood that variations and modifications of the application can be made based on the description set forth herein, without departing from the scope and spirit of the application. The embodiments set forth herein are to be construed as merely illustrative, and not a limitation of the scope of the application in any way. It is also to be understood that not all of the embodiments described herein need include all of the components described in the detailed description or the summary. Combinations of the embodiments described herein are also possible.

[0066] It is to be understood that the present application can assume various alternative embodiments, and that some of the features of the present application can be used to advantage without departing from the spirit and scope of the present application. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without these specific details. In other instances, well-known methods have not been described in detail in order not to unnecessarily obscure the present application.

[0067] Referring to Figure 1 The embodiment shown provides a neural network model quantization method, and the method comprises the following steps:

[0068] S100, an original neural network model to be quantized and a preset precision loss range are obtained. In the embodiment, the original neural network model to be quantized includes but is not limited to a neural network model with single-precision floating-point precision and a neural network model with half-precision floating-point precision. For example, the original neural network model to be quantized in the embodiment can be an fp32 type neural network model.

[0069] S200, all network layers in the original neural network model are quantized to obtain a quantized neural network model, denoted as a quantized neural network model;

[0070] S300, input the target picture into the original neural network model and the quantized neural network model respectively for recognition, and obtain the accuracy loss of the quantized neural network model relative to the original neural network model.

[0071] S400, compare the accuracy loss with the preset accuracy loss range, and output the quantized neural network model according to the comparison result, or iteratively quantize the quantized neural network model according to the comparison result. Specifically, if the accuracy loss in the embodiment is within the preset accuracy loss range, the quantized neural network model is output; if the accuracy loss in the embodiment is not within the preset accuracy loss range, the quantized neural network model is iteratively quantized until the accuracy loss corresponding to the new quantized neural network model is within the preset accuracy loss range, the iteration is terminated, and the corresponding new quantized neural network model is output as the final quantized neural network model. As an example, the preset accuracy loss range in the embodiment can be 3% to 5%; if the recognition accuracy of the target picture input into the original neural network model is 89%, and the recognition accuracy of the target picture input into the quantized neural network model is 83%, the accuracy loss of the quantized neural network model relative to the original neural network model is 6%, and the corresponding accuracy loss is not within the preset accuracy loss range.

[0072] Therefore, the embodiment designs a set of quantization standards that can automatically optimize and adjust the quantization model according to the preset accuracy loss range; without manual intervention, the optimal mixed precision quantization model can also be returned. The embodiment can automatically iteratively quantize according to the set quantization parameters, and adaptively search for a mixed quantization model that meets the conditions. In the embodiment, after the neural network model is quantized, a forward inference test is performed as a whole, and then the accuracy before quantization is compared to determine the accuracy loss. Therefore, the embodiment can input the picture used for verification as a target picture into the current quantized neural network model and the original neural network model, and then determine the accuracy loss of the current quantized neural network model. If the accuracy loss meets the preset accuracy loss range, the current quantized neural network model is directly used as the final quantized neural network model; if it does not meet the preset accuracy loss range, the current quantized neural network model needs to be iteratively quantized until the final quantized neural network model is obtained.

[0073] According to the above description, in an exemplary embodiment, the process of quantizing all network layers in the original neural network model includes: obtaining the precision of each network layer in the original neural network model, and reducing the precision of each network layer; or obtaining the bit of the precision of each network layer in the original neural network model, and reducing the bit of the precision of each network layer. The way of reducing the precision of each network layer includes but is not limited to: reducing the single-precision floating-point number precision to half-precision floating-point number precision, reducing the single-precision floating-point number precision to eight-bit integer precision, and reducing the half-precision floating-point number precision to eight-bit integer precision. As an example, in this embodiment, when quantizing all network layers in the original neural network model, the original neural network model can be pre-quantized first, which includes but is not limited to representing the previous single-precision floating-point number with half-precision floating-point number and 8-bit integer type or even lower bit; that is, representing the previous floating-point number expressed by fp32 with fp16, int8 or even lower bit. By pre-quantizing all network layers in the original neural network model first, this embodiment not only facilitates the speed improvement of the neural network model before and after quantization, but also facilitates the information loss ratio of each network layer before and after quantization of the neural network model.

[0074] According to the above description, in an exemplary embodiment, the process of iteratively quantizing the quantized neural network model includes: inputting the target picture into the original neural network model and the quantized neural network model respectively for identification, and obtaining the absolute value of the inference acceleration time difference value of each network layer in the quantized neural network model and the corresponding network layer in the original neural network model, and the information loss of each network layer in the quantized neural network model relative to the corresponding network layer in the original neural network model; calculating the information loss ratio of each network layer in the quantized neural network model according to the absolute value of the inference acceleration time difference value of each network layer in the quantized neural network model and the information loss; and iteratively quantizing the quantized neural network model according to the information loss ratio of each network layer. In this embodiment, the information loss ratio of each network layer = information loss of the corresponding network layer ÷ absolute value of the inference acceleration time difference value of the corresponding network layer; wherein the physical meaning of the information loss ratio is the degree of information loss per unit time. In this embodiment, the smaller the information loss ratio, the higher the cost performance of quantizing the corresponding network layer, and it is suitable to be quantized preferentially.

[0075] According to the above description, in an example embodiment, the target picture is input into the original neural network model and the quantized neural network model respectively for identification, and the specific process of obtaining the information loss of each network layer in the quantized neural network model relative to the corresponding network layer in the original neural network model includes: inputting the target picture into each network layer of the original neural network model for identification, and recording the obtained probability distribution result as a first identification result; inputting the target picture into each network layer of the quantized neural network model for identification, and recording the obtained probability distribution result as a second identification result; calculating the distance between the first identification result and the second identification result, and determining the information loss of each network layer in the quantized neural network model relative to the corresponding network layer in the original neural network model according to the distance calculation result; wherein the distance includes at least one of the following: cosine distance, Euclidean distance. In this embodiment, the test picture is input as the target picture, and the inference acceleration time and the information loss of each layer of the neural network model before and after quantization are sequentially counted; then the KL divergence is used to measure the distance between the probability distribution before and after quantization of each network layer of the neural network model, and the closer the distance, the smaller the information loss before and after quantization. The information loss is the relative entropy, i.e. the KL divergence. The KL divergence, i.e. the relative entropy, is used to measure the distance between two probability distributions. If the two probability distributions are the same, the relative entropy is 0, and the information loss is less at this time.

[0076] In an example embodiment, the process of iteratively quantizing the quantized neural network model according to the information loss ratio of each network layer comprises: sorting the information loss ratios of all network layers in the quantized neural network model; obtaining the precision of the network layer corresponding to the largest information loss ratio after sorting, denoted as the to-be-quantized precision; improving the to-be-quantized precision to obtain a new quantized neural network model; inputting the target picture into the original neural network model and the new neural network model respectively for identification, obtaining the precision loss of the new neural network model relative to the original neural network model, denoted as the intermediate precision loss; comparing the intermediate precision loss with the preset precision loss range; if the intermediate precision loss is within the preset precision loss range, the current new quantized neural network model is taken as the final quantized neural network model; if the intermediate precision loss is not within the preset precision loss range, the precision of the corresponding network layer is improved in turn according to the sorting result of the information loss ratio, and then iteratively quantized, until the precision loss of the final quantized neural network model is within the preset precision loss range, and the iterative quantization is terminated. According to the description of some embodiments above, the smaller the information loss ratio, the higher the performance-price ratio of the network layer for quantization, and the network layer is suitable for being quantized first. Therefore, in this embodiment, the network layers are sorted according to the information loss ratio from large to small, and then quantized. If the precision loss of the quantized neural network model meets the preset precision loss range, the process is directly exited; if it does not meet the requirement, the precision of the corresponding network layer is improved in turn according to the information loss ratio from large to small, and then iteratively quantized, i.e. the corresponding network layer is improved from int8 to fp32 in turn according to the information loss ratio from large to small, and then iteratively quantized, and then inference testing is performed. If the precision loss of the neural network model after one iteration does not meet the preset precision loss range, the iterative quantization is continued for the second time, until the precision loss of the neural network model after the iterative quantization meets the preset precision loss range, and the iterative quantization is terminated. Compared with this embodiment, the speed of the neural network model is sacrificed to obtain the corresponding precision, until the final precision meets the pre-set precision requirement.

[0077] As shown in Figure 2 In a specific embodiment, the present embodiment provides a neural network model quantization method, which comprises the following steps:

[0078] Step 1, preparation phase. Prepare the to-be-quantized model, and input the acceptable loss precision to facilitate the automatic process of the subsequent process.

[0079] Step 2, model pre-quantization. In order to facilitate the calculation of the speed improvement and the information loss ratio before and after quantization, all network layers need to be pre-quantized once, but the pre-quantized neural network model does not represent the final quantized neural network model.

[0080] Step 3, reasoning test precision. After quantization is completed, the embodiment needs to perform a forward reasoning test as a whole, and then compare the precision loss before quantization. If the precision loss is acceptable, the quantized model is directly used. If the precision loss is not acceptable, step 4 is entered.

[0081] Step 4, calculate information loss ratio. If the precision loss is not acceptable, enter the mixed precision quantization mode. At this time, the information loss ratio of each network layer in the quantized neural network model is calculated, and the network layer with the largest information loss ratio is not suitable for quantization deployment. Therefore, the embodiment selects the network layer corresponding to the largest loss ratio to restore to the original precision (for example, to fp32). Then, it is judged whether the precision of the neural network model meets the requirements. If it meets the requirements, the iteration is directly exited, and the corresponding neural network model is output as the final quantized neural network model. If the precision does not meet the requirements, return to step 3 for iteration. If the precision of the neural network model still cannot meet the requirements after one iteration, the information loss ratio is calculated again, and the network layer with the largest loss ratio is restored to the original precision again. The iteration is stopped until the precision of the last iteration meets the requirements. The mixed precision quantization mode refers to a method of performing different quantization reasoning on different network layers in model reasoning, that is, only part of the network layers are quantized, and the other network layers are reasoned with the original precision.

[0082] In summary, the present application provides a neural network model quantization method, first obtain the original neural network model to be quantized and the preset precision loss range; then quantize all network layers in the original neural network model, obtain the quantized neural network model, denoted as quantized neural network model; then input the target picture into the original neural network model and the quantized neural network model respectively for identification, and obtain the precision loss of the quantized neural network model relative to the original neural network model; finally, compare the precision loss with the preset precision loss range, and output the quantized neural network model according to the comparison result, or iteratively quantize the quantized neural network model according to the comparison result. This method can design a set of quantization standards that can automatically optimize and adjust the quantization model according to the pre-set precision loss range; without manual intervention, the optimal mixed precision quantization model can also be returned. This method can automatically iteratively quantize according to the set quantization parameters, and adaptively search for a mixed quantization model that meets the conditions. This method designs a set of adaptive model quantization standards based on information loss, that is, the difference in information entropy before and after quantization, which can be represented by relative entropy (KL divergence), thereby representing the difference between two probability distributions. Furthermore, the method also proposes the concept of loss ratio, which comprehensively analyzes the impact of single-layer quantization by combining the quantization loss of each layer and the acceleration time before and after quantization, and designs a scheme that can automatically optimize and adjust the quantization model according to the set precision without manual intervention, thereby returning the optimal mixed precision quantization model. The method proposes a measurement method based on information loss to reflect the information loss from the probability distribution. Current mainstream quantization algorithms are designed based on probability distribution, such as uniform quantization and pointer quantization. Therefore, it is more reasonable to use KL divergence information loss to measure the distance between before and after quantization. The method also proposes the concept of loss ratio, which considers whether to quantize the single-layer network by combining information loss and acceleration time, and tilts towards network structures with obvious acceleration effect. At the same time, the method can automatically calculate the loss ratio according to the set parameters, and adaptively search for a mixed quantization model that meets the conditions. In addition, the method based on more reasonable information loss ratio comprehensively measures the performance difference before and after quantization, ensures that the mixed precision quantization meets the inference acceleration while minimizing the precision loss. Moreover, the method uses an adaptive method to automatically search for a mixed quantization scheme that meets the conditions, saving labor costs. At the same time, the method uses the information loss ratio method, which is a comprehensive current quantization algorithm, measures the distance between the probability distributions before and after quantization based on the feature distribution consistency premise, is the most reasonable measurement method at present, and also considers the benefits brought by quantization speed. In addition, the mixed precision quantization algorithm used in the method is completely based on offline, and only the trained model is needed.If online training is used, a corresponding pseudo-quantization algorithm must be inserted into the training, and the training method of the model must be understood in depth, which is more time-consuming and laborious.

[0083] As shown in Figure 3 The application also provides a neural network model quantization system, which comprises:

[0084] The acquisition module M10 is configured to acquire an original neural network model to be quantized and a preset accuracy loss range. In this embodiment, the original neural network model to be quantized includes but is not limited to a neural network model with single-precision floating-point accuracy and a neural network model with half-precision floating-point accuracy. For example, the original neural network model to be quantized in this embodiment can be an fp32 type neural network model.

[0085] The initial quantization module M20 is configured to quantize all network layers in the original neural network model to obtain a quantized neural network model.

[0086] The accuracy loss module M30 is configured to input a target image into the original neural network model and the quantized neural network model respectively for identification, and to obtain an accuracy loss of the quantized neural network model relative to the original neural network model.

[0087] The comparison module M40 is configured to compare the accuracy loss with the preset accuracy loss range.

[0088] The iterative quantization module M50 is configured to perform iterative quantization on the quantized neural network model when the accuracy loss is not located in the preset accuracy loss range. If the accuracy loss in this embodiment is not located in the preset accuracy loss range, iterative quantization is performed on the quantized neural network model until the accuracy loss corresponding to a new quantized neural network model is located in the preset accuracy loss range, the iterative quantization is terminated, and the corresponding new quantized neural network model is output as the final quantized neural network model. For example, the preset accuracy loss range in this embodiment can be 3% to 5%. If the identification accuracy of the target image input into the original neural network model is 89%, and the identification accuracy of the target image input into the quantized neural network model is 83%, the accuracy loss of the quantized neural network model relative to the original neural network model is 6%, and the corresponding accuracy loss is not located in the preset accuracy loss range.

[0089] The output module M60 is configured to output the corresponding quantized neural network model when the precision loss is within the preset precision loss range. If the precision loss in the embodiment is within the preset precision loss range, the quantized neural network model is output. As an example, the preset precision loss range in the embodiment can be 3% to 5%. If the recognition accuracy of the target picture input into the original neural network model is 91%, and the recognition accuracy of the target picture input into the quantized neural network model is 87%, the precision loss of the quantized neural network model relative to the original neural network model is 4%, and the corresponding precision loss is within the preset precision loss range.

[0090] Therefore, the embodiment designs a set of quantization standards that can automatically optimize and adjust the quantized model according to the preset precision loss range. Without human manual intervention, the optimal mixed precision quantized model can also be returned. That is, the embodiment can automatically perform iterative quantization according to the set quantization parameters, and adaptively search for a mixed quantized model that meets the conditions. In the embodiment, after the neural network model is quantized, a forward inference test is performed on the whole, and then the precision before quantization is compared to determine the precision loss. Therefore, the embodiment can input the picture used for verification as a target picture into the current quantized neural network model and the original neural network model, respectively, and then determine the precision loss of the current quantized neural network model. If the precision loss meets the preset precision loss range, the current quantized neural network model is directly used as the final quantized neural network model. If the preset precision loss range is not met, the current quantized neural network model needs to be iteratively quantized until the final quantized neural network model is obtained.

[0091] According to the above description, in an exemplary embodiment, the process of quantizing all network layers in the original neural network model includes: obtaining the precision of each network layer in the original neural network model, and reducing the precision of each network layer; or obtaining the bit of the precision of each network layer in the original neural network model, and reducing the bit of the precision of each network layer. The way of reducing the precision of each network layer includes but is not limited to: reducing the single-precision floating-point number precision to half-precision floating-point number precision, reducing the single-precision floating-point number precision to eight-bit integer precision, and reducing the half-precision floating-point number precision to eight-bit integer precision. As an example, in this embodiment, when quantizing all network layers in the original neural network model, the original neural network model can be pre-quantized first, which includes but is not limited to representing the previous single-precision floating-point number with half-precision floating-point number and 8-bit integer type or even lower bit; that is, representing the previous floating-point number expressed by fp32 with fp16, int8 or even lower bit. By pre-quantizing all network layers in the original neural network model first, this embodiment not only facilitates the speed improvement of the neural network model before and after quantization, but also facilitates the information loss ratio of each network layer before and after quantization of the neural network model.

[0092] According to the above description, in an exemplary embodiment, the process of iteratively quantizing the quantized neural network model includes: inputting the target picture into the original neural network model and the quantized neural network model respectively for identification, and obtaining the absolute value of the inference acceleration time difference value of each network layer in the quantized neural network model and the corresponding network layer in the original neural network model, and the information loss of each network layer in the quantized neural network model relative to the corresponding network layer in the original neural network model; calculating the information loss ratio of each network layer in the quantized neural network model according to the absolute value of the inference acceleration time difference value of each network layer in the quantized neural network model and the information loss; and iteratively quantizing the quantized neural network model according to the information loss ratio of each network layer. In this embodiment, the information loss ratio of each network layer = information loss of the corresponding network layer ÷ absolute value of the inference acceleration time difference value of the corresponding network layer; wherein the physical meaning of the information loss ratio is the degree of information loss per unit time. In this embodiment, the smaller the information loss ratio, the higher the cost performance of quantizing the corresponding network layer, and it is suitable to be quantized preferentially.

[0093] According to the above description, in an example embodiment, the target picture is input into the original neural network model and the quantized neural network model respectively for identification, and the specific process of obtaining the information loss of each network layer in the quantized neural network model relative to the corresponding network layer in the original neural network model includes: inputting the target picture into each network layer of the original neural network model for identification, and recording the obtained probability distribution result as a first identification result; inputting the target picture into each network layer of the quantized neural network model for identification, and recording the obtained probability distribution result as a second identification result; calculating the distance between the first identification result and the second identification result, and determining the information loss of each network layer in the quantized neural network model relative to the corresponding network layer in the original neural network model according to the distance calculation result; wherein the distance includes at least one of the following: cosine distance, Euclidean distance. In this embodiment, the test picture is input as the target picture, and the inference acceleration time and the information loss of each layer of the neural network model before and after quantization are sequentially counted; then the KL divergence is used to measure the distance between the probability distribution before and after quantization of each network layer of the neural network model, and the closer the distance, the smaller the information loss before and after quantization. The information loss is the relative entropy, i.e. the KL divergence. The KL divergence, i.e. the relative entropy, is used to measure the distance between two probability distributions. If the two probability distributions are the same, the relative entropy is 0, and the information loss is less at this time.

[0094] In an example embodiment, the process of iteratively quantizing the quantized neural network model according to the information loss ratio of each network layer comprises: sorting the information loss ratios of all network layers in the quantized neural network model; obtaining the precision of the network layer corresponding to the largest information loss ratio after sorting, denoted as the to-be-quantized precision; improving the to-be-quantized precision to obtain a new quantized neural network model; inputting the target picture into the original neural network model and the new neural network model respectively for identification, obtaining the precision loss of the new neural network model relative to the original neural network model, denoted as the intermediate precision loss; comparing the intermediate precision loss with the preset precision loss range; if the intermediate precision loss is within the preset precision loss range, the current new quantized neural network model is taken as the final quantized neural network model; if the intermediate precision loss is not within the preset precision loss range, the precision of the corresponding network layer is improved in turn according to the sorting result of the information loss ratio, and then iteratively quantized, until the precision loss of the final quantized neural network model is within the preset precision loss range, and the iterative quantization is terminated. According to the description of some embodiments above, the smaller the information loss ratio, the higher the performance-price ratio of the network layer for quantization, and the network layer is suitable for being quantized first. Therefore, in this embodiment, the network layers are sorted according to the information loss ratio from large to small, and then quantized. If the precision loss of the quantized neural network model meets the preset precision loss range, the process is directly exited; if it does not meet the requirement, the precision of the corresponding network layer is improved in turn according to the information loss ratio from large to small, and then iteratively quantized, i.e. the corresponding network layer is improved from int8 to fp32 in turn according to the information loss ratio from large to small, and then iteratively quantized, and then inference testing is performed. If the precision loss of the neural network model after one iteration does not meet the preset precision loss range, the iterative quantization is continued for the second time, until the precision loss of the neural network model after the iterative quantization meets the preset precision loss range, and the iterative quantization is terminated. Compared with this embodiment, the speed of the neural network model is sacrificed to obtain the corresponding precision, until the final precision meets the pre-set precision requirement.

[0095] In a specific embodiment, the present embodiment provides a neural network model quantization system for performing the following steps:

[0096] Step 1, preparation phase. Prepare the to-be-quantized model, and input the acceptable loss precision to facilitate the automatic process of the subsequent process.

[0097] Step 2, model pre-quantization. In order to facilitate the calculation of the speed improvement and the information loss ratio before and after quantization, all network layers need to be pre-quantized once, but the pre-quantized neural network model does not represent the final quantized neural network model.

[0098] Step 3, reasoning test precision. After the quantization is completed, the embodiment needs to perform a forward reasoning test as a whole, and then compare the precision loss before and after quantization. If the precision loss is acceptable, the quantized model is directly used. If the precision loss is not acceptable, step 4 is entered.

[0099] Step 4, calculate information loss ratio. If the precision loss is not acceptable, enter the mixed precision quantization mode. At this time, the information loss ratio of each network layer in the quantized neural network model is calculated, and the network layer with a larger information loss ratio is less suitable for quantization deployment. Therefore, the embodiment selects the network layer corresponding to the maximum loss ratio to restore to the original precision (for example, to fp32); then it is judged whether the precision of the neural network model meets the requirements. If it meets the requirements, the iteration is directly exited, and the corresponding neural network model is output as the final quantized neural network model. If the precision does not meet the requirements, return to step 3 for iteration. If the precision of the neural network model still cannot meet the requirements after one iteration, the information loss ratio is calculated again, and the network layer with the largest loss ratio is restored to the original precision again. The iteration is stopped after the precision of the last iteration meets the requirements. The mixed precision quantization mode refers to a method of performing different quantization reasoning on different network layers during model reasoning, that is, only part of the network layers are quantized, and the other network layers are reasoned at the original precision.

[0100] In another specific embodiment, the embodiment also provides a neural network model quantization system, which comprises:

[0101] Loss ratio module: used to measure the information loss effect of a single network layer, and the lower the information loss ratio, the better. The information loss ratio is calculated by the formula of information loss before and after quantization divided by the acceleration time before and after quantization. The information loss is relative entropy, that is, KL divergence, which measures the distance between the probability distribution before and after quantization. The physical meaning of the information loss ratio is the loss degree of information per unit time.

[0102] Adaptive quantization module: used to automatically calculate the information loss ratio of each network layer according to the set minimum loss precision and quantized model, and gradually set the layer with a larger loss ratio to the fp32 precision mode. After each setting is completed, a forward reasoning test is performed to test the precision, and then whether the corresponding precision meets the precision requirements is used to automatically complete the mixed precision quantization.

[0103] In summary, the present application provides a neural network model quantization system, first obtain the original neural network model to be quantized and a preset precision loss range; then quantize all network layers in the original neural network model, obtain the quantized neural network model, denoted as a quantized neural network model; then input the target picture into the original neural network model and the quantized neural network model respectively for identification, and obtain the precision loss of the quantized neural network model relative to the original neural network model; finally, compare the precision loss with the preset precision loss range, and output the quantized neural network model according to the comparison result, or iteratively quantize the quantized neural network model according to the comparison result. The system can design a set of quantization standards that can automatically optimize and adjust the quantization model according to the pre-set precision loss range; without manual intervention, the optimal mixed precision quantization model can also be returned. The system can automatically iteratively quantize according to the set quantization parameters, and adaptively search for a mixed quantization model that meets the conditions. The system designs a set of adaptive model quantization standards based on information loss, that is, the difference in information entropy before and after quantization, which can be represented by relative entropy (KL divergence), thereby representing the difference between two probability distributions. The system also proposes the concept of loss ratio, which comprehensively analyzes the impact of single-layer quantization based on the quantization loss of each layer and the acceleration time before and after quantization, and uses this as a measurement standard to design a scheme that can automatically optimize and adjust the quantization model according to the set precision without manual intervention, and returns the optimal mixed precision quantization model. The system proposes a measurement method based on information loss to reflect the information loss from the probability distribution. The current mainstream quantization algorithms are designed based on probability distribution, such as uniform quantization and pointer quantization. Therefore, it is more reasonable to use KL divergence information loss to measure the distance between the pre-quantization and post-quantization. The system also proposes the concept of loss ratio, which combines information loss and acceleration time to comprehensively consider whether to quantize single-layer networks and tilt towards network structures with obvious acceleration effects. At the same time, the system can automatically iteratively calculate the loss ratio based on the set parameters, and adaptively search for a mixed quantization model that meets the conditions. In addition, based on the more reasonable information loss ratio, the system comprehensively measures the performance difference before and after quantization, ensuring that the mixed precision quantization meets the inference acceleration while minimizing the precision loss. Moreover, the system uses an adaptive method to automatically search for a mixed quantization scheme that meets the conditions based on the information loss ratio, saving labor costs. At the same time, the system uses the information loss ratio method, which is a comprehensive measurement of the current mainstream quantization algorithm based on the premise of feature distribution consistency, measures the distance between the probability distribution before and after quantization, and is currently the most reasonable measurement method. It also takes into account the benefits of quantization speed. In addition, the mixed precision quantization algorithm used in the system is completely based on an offline method, which only requires a trained model.If online training is used, a corresponding pseudo-quantization algorithm must be inserted into the training, and the training method of the model must be understood in depth, which is more time-consuming and laborious.

[0104] The embodiments of the present application also provide a computer device, which can include one or more processors, and one or more machine-readable media having instructions stored thereon, when executed by the one or more processors, cause the device to perform the method of any of the embodiments of the present application. Figure 1 In actual application, the device can be used as a terminal device or a server. Examples of the terminal device can include a smartphone, a tablet computer, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer, an on-board computer, a desktop computer, a set-top box, a smart television, a wearable device, and the like. The embodiments of the present application are not limited to a specific device.

[0105] The embodiments of the present application also provide a non-volatile readable storage medium, which stores one or more programs. When the one or more programs are applied to a device, the device can perform the method of any of the embodiments of the present application. Figure 1 The instructions of the method of data processing.

[0106] Figure 4 A hardware structure schematic diagram of a terminal device provided by an embodiment of the present application is shown in FIG. 1. As shown in the figure, the terminal device can include an input device 1100, a first processor 1101, an output device 1102, a first memory 1103, and at least one communication bus 1104. The communication bus 1104 is used to realize the communication connection between elements. The first memory 1103 can include a high-speed RAM memory, and can also include a non-volatile storage NVM, for example, at least one disk memory. The first memory 1103 can store various programs for completing various processing functions and realizing the method steps of the embodiments.

[0107] Optionally, the first processor 1101 can be implemented by a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor, or other electronic devices. The first processor 1101 is coupled to the input device 1100 and the output device 1102 by wire or wirelessly.

[0108] Optionally, the input device 1100 can include various input devices, such as at least one of a user interface facing a user, a device interface facing a device, a programmable interface of software, a camera, and a sensor. Optionally, the device interface facing a device can be a wired interface for data transmission between devices, and can also be a hardware insertion interface (such as a USB interface, a serial port, etc.) for data transmission between devices. Optionally, the user interface facing a user can be, for example, a control button facing a user, a voice input device for receiving voice input, and a touch sensing device (such as a touch screen with touch sensing function, a touchpad, etc.) for receiving touch input of a user. Optionally, the programmable interface of software can be, for example, an entry for a user to edit or modify a program, such as an input pin interface or an input interface of a chip, etc. The output device 1102 can include a display, a sound device, etc.

[0109] In this embodiment, the processor of the terminal device includes a function for executing each module of the voice recognition apparatus in each device. The specific function and technical effect can refer to the above-mentioned embodiments, which will not be described here.

[0110] Figure 5 A hardware structure diagram of a terminal device is provided for another embodiment of the present application. Figure 5 is to Figure 4 In one specific embodiment in the implementation process. As shown in the figure, the terminal device of the embodiment can include a second processor 1201 and a second memory 1202.

[0111] The second processor 1201 executes the computer program code stored in the second memory 1202 to realize the functions of the modules of the voice recognition apparatus in the above-mentioned embodiments. Figure 1 The method.

[0112] The second memory 1202 is configured to store various types of data to support the operation of the terminal device. Examples of these data include instructions for any application or method operating on the terminal device, messages, pictures, videos, etc. The second memory 1202 can contain a random access memory (RAM) and can also include a non-volatile memory such as at least one disk memory.

[0113] Optionally, the second processor 1201 is arranged in the processing assembly 1200. The terminal device can further include a communication assembly 1203, a power supply assembly 1204, a multimedia assembly 1205, an audio assembly 1206, an input / output interface 1207, and / or a sensor assembly 1208. The terminal device specifically contains components, etc. according to the actual demand setting, and the embodiment does not limit this.

[0114] The processing assembly 1200 generally controls the overall operation of the terminal device. The processing assembly 1200 can include one or more second processors 1201 to execute instructions to complete all or part of the steps of the above-mentioned methods. In addition, the processing assembly 1200 can include one or more modules to facilitate interaction between the processing assembly 1200 and other components. For example, the processing assembly 1200 can include a multimedia module to facilitate interaction between the multimedia assembly 1205 and the processing assembly 1200. Figure 1

[0115] The power supply assembly 1204 provides power for various components of the terminal device. The power supply assembly 1204 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the terminal device.

[0116] The multimedia assembly 1205 includes a display screen that provides an output interface between the terminal device and the user. In some embodiments, the display screen can include a liquid crystal display (LCD) and a touch panel (TP). If the display screen includes a touch panel, the display screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor can not only sense the boundaries of a touch or swipe action, but also detect the duration and pressure associated with the touch or swipe operation.

[0117] ​Audio component 1206 is configured to output and / or input voice signals. For example, audio component 1206 includes a microphone (MIC) configured to receive external voice signals when the terminal device is in an operating mode, such as a voice recognition mode. The received voice signals may be further stored in a second memory 1202 or transmitted via communication component 1203. In some embodiments, audio component 1206 also includes a speaker for outputting voice signals.

[0118] Input / output interface 1207 provides an interface between processing component 1200 and peripheral interface modules, such as click wheels, buttons, etc. These buttons may include, but are not limited to, volume buttons, start buttons, and lock buttons.

[0119] Sensor assembly 1208 includes one or more sensors for providing status assessments of various aspects of the terminal device. For example, sensor assembly 1208 can detect the on / off state of the terminal device, the relative positioning of components, and the presence or absence of user contact with the terminal device. Sensor assembly 1208 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact, including detecting the distance between the user and the terminal device. In some embodiments, sensor assembly 1208 may also include a camera, etc.

[0120] Communication component 1203 is configured to facilitate wired or wireless communication between the terminal device and other devices. The terminal device can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one embodiment, the terminal device may include a SIM card slot for inserting a SIM card, enabling the terminal device to log in to a GPRS network and establish communication with a server via the Internet.

[0121] As can be seen from the above, in Figure 5 The communication component 1203, audio component 1206, input / output interface 1207, and sensor component 1208 involved in the embodiment can all be used as... Figure 4 The implementation method of the input device in the embodiment.

[0122] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

[0123] It should be understood that, although the terms first, second, third, etc. can be adopted in the embodiments of the present application to describe the preset accuracy loss ranges, etc., the preset accuracy loss ranges should not be limited to these terms. These terms are only used to distinguish the preset accuracy loss ranges from each other. For example, without departing from the scope of the embodiments of the present application, the first preset accuracy loss range can also be referred to as the second preset accuracy loss range, and similarly, the second preset accuracy loss range can also be referred to as the first preset accuracy loss range.

Claims

1. A method for quantizing neural network models, characterized in that, The method includes the following steps: Obtain the original neural network model to be quantized and the preset accuracy loss range; Quantize all network layers in the original neural network model to obtain the quantized neural network model, denoted as the quantized neural network model. The target image is input into the original neural network model and the quantized neural network model respectively for recognition, and the accuracy loss of the quantized neural network model relative to the original neural network model is obtained; The accuracy loss is compared with the preset accuracy loss range, and the quantized neural network model is output based on the comparison result, or the quantized neural network model is iteratively quantized based on the comparison result. The iterative quantization process of the quantized neural network model includes: inputting the target image into the original neural network model and the quantized neural network model for recognition, and obtaining the absolute value of the difference in inference acceleration time between each network layer in the quantized neural network model and the corresponding network layer in the original neural network model, as well as the information loss of each network layer in the quantized neural network model relative to the corresponding network layer in the original neural network model; calculating the information loss ratio of each network layer in the quantized neural network model based on the absolute value of the difference in inference acceleration time and the information loss; and iteratively quantizing the quantized neural network model based on the information loss ratio of each network layer. The information loss ratio of each network layer is defined as: information loss of the corresponding network layer ÷ absolute value of the difference in inference acceleration time of the corresponding network layer. The physical meaning of the information loss ratio is the degree of information loss per unit time. The smaller the information loss ratio, the higher the cost-effectiveness of quantizing the corresponding network layer, and the more suitable it is to prioritize quantization.

2. The neural network model quantization method according to claim 1, characterized in that, The process of quantizing all network layers in the original neural network model includes: Obtain the accuracy of each network layer in the original neural network model, and reduce the accuracy of each network layer; Alternatively, obtain the precision bits of each network layer in the original neural network model and reduce the precision bits of each network layer.

3. The neural network model quantization method according to claim 2, characterized in that, The reduction of precision for each network layer can be achieved by at least one of the following: reducing single-precision floating-point precision to half-precision floating-point precision, reducing single-precision floating-point precision to eight-bit integer precision, or reducing half-precision floating-point precision to eight-bit integer precision.

4. The neural network model quantization method according to claim 1, characterized in that, If the accuracy loss is within the preset accuracy loss range, then the quantized neural network model is output; If the accuracy loss is not within the preset accuracy loss range, the quantized neural network model is iteratively quantized until the accuracy loss corresponding to the new quantized neural network model is within the preset accuracy loss range. Then, the iterative quantization is terminated, and the corresponding new quantized neural network model is output as the final quantized neural network model.

5. The neural network model quantization method according to any one of claims 1 to 4, characterized in that, The process of iteratively quantizing the quantized neural network model based on the information loss ratio of each network layer includes: The information loss ratios of all network layers in the quantized neural network model are ranked. The precision of the network layer corresponding to the maximum information loss ratio after sorting is denoted as the precision to be quantized. The precision of the quantization target is improved to obtain a new quantization neural network model; The target image is input into the original neural network model and the new neural network model respectively for recognition, and the accuracy loss of the new neural network model relative to the original neural network model is obtained, which is denoted as the intermediate accuracy loss; The intermediate accuracy loss is compared with the preset accuracy loss range; If the intermediate precision loss is within the preset precision loss range, then the current new quantized neural network model will be used as the final quantized neural network model. If the intermediate precision loss is not within the preset precision loss range, the precision of the corresponding network layer is improved sequentially according to the sorting result of the information loss ratio, and then iterative quantization is performed until the precision loss corresponding to the final quantized neural network model is within the preset precision loss range, at which point the iterative quantization terminates.

6. The neural network model quantization method according to any one of claims 1 to 4, characterized in that, The process of inputting the target image into the original neural network model and the quantized neural network model for recognition, and obtaining the information loss of each network layer in the quantized neural network model relative to the corresponding network layer in the original neural network model, includes: The target image is input into each network layer of the original neural network model for recognition, and the resulting probability distribution is recorded as the first recognition result. The target image is input into each network layer of the quantized neural network model for recognition, and the resulting probability distribution is recorded as the second recognition result. Calculate the distance between the first recognition result and the second recognition result, and determine the information loss of each network layer in the quantized neural network model relative to the corresponding network layer in the original neural network model based on the distance calculation result; wherein, the distance includes at least one of the following: cosine distance and Euclidean distance.

7. The neural network model quantization method according to claim 1, characterized in that, The original neural network model to be quantized includes at least one of the following: a neural network model with single-precision floating-point precision, or a neural network model with half-precision floating-point precision.

8. A neural network model quantization system, characterized in that, The system includes: The acquisition module is used to acquire the original neural network model to be quantized and the preset accuracy loss range; The initial quantization module is used to quantize all network layers in the original neural network model to obtain the quantized neural network model, denoted as the quantized neural network model. The accuracy loss module is used to input the target image into the original neural network model and the quantized neural network model respectively for recognition, and to obtain the accuracy loss of the quantized neural network model relative to the original neural network model. The comparison module is used to compare the accuracy loss with the preset accuracy loss range; The iterative quantization module is used to iteratively quantize the quantized neural network model when the accuracy loss is not within the preset accuracy loss range. This includes: inputting the target image into the original neural network model and the quantized neural network model for recognition, and obtaining the absolute value of the difference in inference acceleration time between each network layer in the quantized neural network model and the corresponding network layer in the original neural network model, as well as the information loss of each network layer in the quantized neural network model relative to the corresponding network layer in the original neural network model; calculating the information loss ratio of each network layer in the quantized neural network model based on the absolute value of the difference in inference acceleration time and the information loss; and iteratively quantizing the quantized neural network model based on the information loss ratio of each network layer. The information loss ratio of each network layer is defined as: information loss of the corresponding network layer ÷ absolute value of the difference in inference acceleration time of the corresponding network layer. The physical meaning of the information loss ratio is the degree of information loss per unit time. A smaller information loss ratio indicates a higher cost-effectiveness of quantizing the corresponding network layer, making it suitable for priority quantization. The output module is used to output the corresponding quantized neural network model when the accuracy loss is within the preset accuracy loss range.

9. A computer device, characterized in that, include: One or more processors; and A computer-readable medium storing instructions that, when executed by the one or more processors, cause the device to perform the method as described in any one of claims 1 to 7.

10. A computer-readable medium, characterized in that, It stores instructions that, when executed by one or more processors, cause the device to perform the method as described in any one of claims 1 to 7.

Images