Neural network quantization accuracy evaluation method, device, electronic device and storage medium

Abstract

The present application relates to a method, device, electronic device, and storage medium for evaluating the quantization accuracy of a neural network, wherein the method includes: obtaining an original onnx model, calling the onnx_graphsurgeon application programming interface (API), setting all onnx operators in the original onnx model as output nodes, generating a quantization threshold for each onnx operator, and converting the quantization threshold and the unquantized threshold of each onnx operator into a first target trt file and a second target trt file, respectively, and obtaining the first output tensor of each onnx operator in the quantized model and the second output tensor of each onnx operator in the unquantized model based on a preset TensorRT environment and output nodes, obtaining the quantization accuracy of each onnx operator, and generating a final evaluation result. Thus, the accuracy loss after quantization of each operator can be quickly and accurately determined.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device and storage medium for evaluating the quantitative accuracy of a neural network. Background Art

[0002] For a trained model, an evaluation method is needed to intuitively understand its performance.

[0003] In the related art, a method for determining quantization accuracy is provided. By performing quantization operations on processed data during the execution of the neural network based on the data format information used when the neural network is executed in the processing device, the network output result of the neural network is obtained, and the network output result is compared with the benchmark output result to obtain the corresponding quantization accuracy of the neural network executed in the processing device. The benchmark output result is the output result obtained by the neural network processing unquantized processed data, thereby realizing the measurement of the quantization accuracy of the hardware quantization implementation of the neural network.

[0004] However, related technologies can only evaluate the accuracy of the model's output layer and cannot obtain the quantitative accuracy of each operator in the model. Summary of the Invention

[0005] The present application provides a neural network quantization accuracy evaluation method, device, electronic device and storage medium to solve the problem that related technologies cannot evaluate the quantization accuracy of each operator in the model. It can quickly and accurately determine the accuracy loss of each operator after quantization, and can help model developers and quantization deployment personnel accurately locate operators with low quantization accuracy, so as to take corresponding measures.

[0006] A first aspect of the present application provides a method for evaluating neural network quantization accuracy, comprising the following steps: obtaining an original onnx (Open Neural Network Exchange) model and calling an application programming interface (API) of onnx_graphsurgeon to set all onnx operators in the original onnx model as output nodes; generating a quantization threshold for each onnx operator based on the relative entropy of the distribution of the output tensor of each onnx operator before and after quantization, converting the quantization threshold of each onnx operator into a first target trt file, and converting the unquantized threshold of each onnx operator into a second target trt file; and loading the first target trt file and the second target trt file based on a preset TensorRT environment and the output node to obtain a first output tensor of each onnx operator in the quantized model and a second output tensor of each onnx operator in the unquantized model, obtaining the quantization accuracy of each onnx operator based on the first output tensor and the second output tensor, and generating a final evaluation result based on the quantization accuracy of each onnx operator.

[0007] Based on the above technical means, this application can solve the problem that related technologies cannot evaluate the quantization accuracy of each operator in the model, can quickly and accurately determine the accuracy loss of each operator after quantization, and can help model developers and quantization deployment personnel accurately locate operators with low quantization accuracy, so as to take corresponding measures.

[0008] Optionally, in some embodiments, after obtaining the quantization accuracy of each ONNX operator according to the first output tensor and the second output tensor, it also includes: based on the quantization accuracy of each ONNX operator, screening out the ONNX operators to be processed with a quantization accuracy less than a preset quantization accuracy; and generating a target processing strategy according to the ONNX operators to be processed.

[0009] Based on the above technical means, this application can filter out operators that do not meet the accuracy requirements and provide processing strategies to facilitate targeted maintenance by developers.

[0010] Optionally, in some embodiments, the quantization accuracy is characterized by a first indicator and a second indicator, wherein the first indicator is a relative error, and the second indicator is a cosine value of the first output tensor and the second output tensor.

[0011] Based on the above technical means, this application can judge the level of quantization accuracy by the cosine value of the output tensor and filter out quantum that does not meet the accuracy requirements.

[0012] Optionally, in some embodiments, after generating the quantization threshold of each onnx operator based on the relative entropy of the distribution of the output tensor of each onnx operator before and after quantization, it also includes: outputting the quantization threshold as a target calib file, wherein the calib file saves the truncation threshold of the output tensor of each onnx operator in hexadecimal.

[0013] Based on the above technical means, this application can convert the target file into an applicable file for easy retrieval and use.

[0014] Optionally, in some embodiments, after setting all the onnx operators in the original onnx model as output nodes, the method further includes: placing the output tenors of all the onnx operators in an outputs list of a graph.

[0015] Based on the above technical means, the present application can solve the problem that related technologies cannot evaluate the quantization accuracy of each operator in the model, and can quickly and accurately determine the accuracy loss of each operator after quantization.

[0016] A second aspect of the present application provides a neural network quantization accuracy evaluation device, including: an acquisition module, used to obtain an original onnx model, and call the onnx_graphsurgeon application programming interface API to set all onnx operators in the original onnx model as output nodes; a conversion module, used to generate a quantization threshold for each onnx operator based on the relative entropy of the distribution of the output tensor of each onnx operator before and after quantization, and convert the quantization threshold of each onnx operator into a first target trt file, and convert the unquantized threshold of each onnx operator into a second target trt file; and a generation module, used to load the first target trt file and the second target trt file based on a preset TensorRT environment and the output node, obtain the first output tensor of each onnx operator in the quantized model and the second output tensor of each onnx operator in the unquantized model, obtain the quantization accuracy of each onnx operator according to the first output tensor and the second output tensor, and generate a final evaluation result according to the quantization accuracy of each onnx operator.

[0017] Optionally, in some embodiments, after obtaining the quantization accuracy of each ONNX operator according to the first output tensor and the second output tensor, the generation module further includes: screening out the ONNX operators to be processed with a quantization accuracy less than a preset quantization accuracy based on the quantization accuracy of each ONNX operator; and generating a target processing strategy according to the ONNX operators to be processed.

[0018] Optionally, in some embodiments, the quantization accuracy is characterized by a first indicator and a second indicator, wherein the first indicator is a relative error, and the second indicator is a cosine value of the first output tensor and the second output tensor.

[0019] Optionally, in some embodiments, after generating the quantization threshold of each onnx operator based on the relative entropy of the distribution of the output tensor of each onnx operator before and after quantization, it also includes: outputting the quantization threshold as a target calib file, wherein the calib file saves the truncation threshold of the output tensor of each onnx operator in hexadecimal.

[0020] Optionally, in some embodiments, after setting all the onnx operators in the original onnx model as output nodes, the method further includes: placing the output tenors of all the onnx operators in the outputs list of the graph.

[0021] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the neural network quantization accuracy evaluation method as described in the above embodiment.

[0022] The fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the neural network quantization accuracy evaluation method as described in the above embodiment.

[0023] Therefore, by obtaining the original onnx model and calling the onnx_graphsurgeon application programming interface API, all onnx operators in the original onnx model are set as output nodes, the quantization threshold of each onnx operator is generated, and the quantization threshold and the unquantization threshold of each onnx operator are converted into the first target trt file and the second target trt file respectively. Based on the preset TensorRT environment and output nodes, the first output tensor of each onnx operator in the quantized model and the second output tensor of each onnx operator in the unquantized model are obtained, and the quantization accuracy of each onnx operator is obtained, and the final evaluation result is generated, which solves the problem that related technologies cannot evaluate the quantization accuracy of each operator in the model. It can quickly and accurately determine the accuracy loss of each operator after quantization, which can help model developers and quantization deployment personnel accurately locate operators with low quantization accuracy and take corresponding measures.

[0024] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0026] Figure 1 This is a flowchart of a method for evaluating the quantization accuracy of a neural network according to an embodiment of the present application;

[0027] Figure 2 This is a flowchart of a neural network quantization accuracy evaluation method provided according to one embodiment of the present application;

[0028] Figure 3 Schematic diagram of a block diagram of a neural network quantization accuracy evaluation device provided according to an embodiment of the present application;

[0029] Figure 4 A schematic diagram of an electronic device provided according to an embodiment of the present application.

[0030] Explanation of the accompanying symbols: 10 - neural network quantization accuracy evaluation device, 100 - acquisition module, 200 - conversion module, 300 - evaluation module. DETAILED DESCRIPTION

[0031] The following describes in detail embodiments of the present application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0032] The following describes a neural network quantization accuracy evaluation method, device, electronic device, and storage medium according to an embodiment of the present application with reference to the accompanying drawings. In response to the problem that the related art mentioned in the background art cannot evaluate the quantization accuracy of each operator in the model, the present application provides a neural network quantization accuracy evaluation method. In this method, by obtaining the original onnx model and calling the onnx_graphsurgeon application programming interface API, all onnx operators in the original onnx model are set as output nodes, quantization thresholds for each onnx operator are generated, and the quantization thresholds and unquantized thresholds of each onnx operator are converted into first target trt files and second target trt files, respectively. Based on the preset TensorRT environment and output nodes, the first output tensor of each onnx operator in the quantized model and the second output tensor of each onnx operator in the unquantized model are obtained, and the quantization accuracy of each onnx operator is obtained, and a final evaluation result is generated, which solves the problem that the related art cannot evaluate the quantization accuracy of each operator in the model, can quickly and accurately determine the accuracy loss of each operator after quantization, and can help model developers and quantization deployment personnel accurately locate operators with low quantization accuracy, so as to take corresponding measures.

[0033] Specifically, Figure 1 A flowchart of a neural network quantization accuracy evaluation method provided in an embodiment of the present application.

[0034] like Figure 1 As shown, the neural network quantization accuracy evaluation method includes the following steps:

[0035] In step S101, the original onnx model is obtained, and the application programming interface API of onnx_graphsurgeon is called to set all onnx operators in the original onnx model as output nodes.

[0036] Optionally, in some embodiments, after setting all onnx operators in the original onnx model as output nodes, the method further includes: placing the output tenors of all onnx operators in the outputs list of the graph.

[0037] It should be noted that the embodiments of the present application only deal with the post-quantization situation, and the accuracy evaluation of quantization during training is not considered.

[0038] Specifically, after the model is trained using a deep learning framework, in order to deploy the trained model in various environments, the embodiment of the present application generally exports it as an onnx model. Generally, deep learning frameworks such as pytorch, tensorflow, etc. have corresponding APIs, which can be directly generated by calling these APIs.

[0039] It is important for those skilled in the art to understand that the data output by the onnx model will be separately marked as output nodes in its file, and these output nodes can obtain the calculated tensor in subsequent reasoning. The output node of the onnx file is the output data of the original neural network model. Using it for reasoning cannot obtain the calculated output of the intermediate node, and thus cannot compare the quantization accuracy of the intermediate operator. To solve this problem, it is necessary to modify the onnx operator so that it can output the calculated data of all nodes. This embodiment of the application is accomplished by using onnx_graphsurgeon. By calling the API of onnx_graphsurgeon and placing the output tensor of all operators in the outputs list of the graph, the output of all operators can be set as output nodes, thereby obtaining all operators and calculation results in subsequent operations.

[0040] In step S102, a quantization threshold of each onnx operator is generated according to the relative entropy of the distribution of the output tensor of each onnx operator before and after quantization, and the quantization threshold of each onnx operator is converted into a first target trt file, and the unquantized threshold of each onnx operator is converted into a second target trt file.

[0041] Optionally, in some embodiments, after generating the quantization threshold of each onnx operator based on the relative entropy of the distribution of the output tensor of each onnx operator before and after quantization, it also includes: outputting the quantization threshold as a target calib file, wherein the calib file saves the truncation threshold of the output tensor of each onnx operator in hexadecimal.

[0042] It should be noted that during the saturation quantization process, when the fp32 tensor is quantized to the int8 type, floating-point numbers exceeding this threshold will be directly set as the quantization threshold, that is, floating-point numbers exceeding the threshold will be directly truncated. In theory, the quantization threshold can be any value, but in order to obtain better quantization results, the threshold needs to be set appropriately.

[0043] Specifically, the threshold is determined by the relative entropy of the distribution of each tensor before and after quantization. The most appropriate threshold should minimize the relative entropy of the data distribution before and after quantization. In the embodiment of the present application, the quantization threshold can be output as a calib file, which saves the truncation threshold of each tensor in the onnx model in hexadecimal. In addition, in order to convert the onnx model into a runtime engine that can be called by TensorRT, the embodiment of the present application can directly use the trtexec executable file to generate, convert the quantization threshold of each onnx operator into the first target trt file, and convert the unquantized threshold of each onnx operator into the second target trt file. For the second target trt file of int quantization, it is necessary to manually set the int8 flag and generate the calib file.

[0044] In step S103, based on the preset TensorRT environment and output node, the first target trt file and the second target trt file are loaded to obtain the first output tensor of each onnx operator in the quantized model and the second output tensor of each onnx operator in the unquantized model. The quantization accuracy of each onnx operator is obtained according to the first output tensor and the second output tensor, and the final evaluation result is generated according to the quantization accuracy of each onnx operator.

[0045] Optionally, in some embodiments, the quantization accuracy is characterized by a first indicator and a second indicator, wherein the first indicator is a relative error and the second indicator is a cosine value of the first output tensor and the second output tensor.

[0046] Specifically, in an embodiment of the present application, the first target trt file and the second target trt file are loaded in the TensorRT environment, and the above two trt files are deserialized into an engine, and the same input data is used for inference. Based on step S101, we can obtain the output of each operator. By comparing the output tensor of each operator of the quantized model and the unquantized model, the quantization accuracy of each operator can be obtained. The accuracy can be characterized by two indicators. The first indicator is the relative error. The smaller the relative error, the higher the quantization accuracy; the second indicator is the cosine value of the two tensors. The value should be a floating point number between 0 and 1. The closer to zero, the higher the quantization accuracy. In this way, the quantization accuracy of the operator can be obtained and the final evaluation result can be generated.

[0047] Optionally, in some embodiments, after obtaining the quantization accuracy of each ONNX operator according to the first output tensor and the second output tensor, it also includes: based on the quantization accuracy of each ONNX operator, screening out the ONNX operators to be processed that are less than a preset quantization accuracy; and generating a target processing strategy according to the ONNX operators to be processed.

[0048] The preset quantization accuracy may be pre-set by relevant personnel and is not specifically limited here.

[0049] Specifically, in the embodiment of the present application, the onnx operator can be evaluated by setting a preset quantization accuracy. If the quantization accuracy is less than the preset quantization accuracy, it means that the quantization accuracy is too low, and relevant personnel need to perform targeted processing to generate a target processing strategy.

[0050] In order to enable those skilled in the art to further understand the neural network quantization accuracy evaluation method of the embodiment of the present application, it is described below in conjunction with specific embodiments.

[0051] Figure 2 This is a flowchart of a neural network quantization accuracy evaluation method proposed according to an embodiment of the present application.

[0052] S201, after the model training is completed, obtain the onnx model.

[0053] S202, modify the onnx model and set all operators as output.

[0054] S203, generating int8 quantization thresholds for each operator.

[0055] S204: Generate two trt files, one quantized and one unquantized.

[0056] S205: Run the two trt files, obtain the output data of each operator, and compare them.

[0057] S206: Targeted processing is performed on operators with large precision loss.

[0058] According to the neural network quantization accuracy evaluation method proposed in the embodiment of the present application, by obtaining the original onnx model and calling the onnx_graphsurgeon application programming interface API, all onnx operators in the original onnx model are set as output nodes, and quantization thresholds for each onnx operator are generated. The quantization thresholds and unquantized thresholds of each onnx operator are converted into first target trt files and second target trt files respectively. Based on the preset TensorRT environment and output nodes, the first output tensor of each onnx operator in the quantized model and the second output tensor of each onnx operator in the unquantized model are obtained, and the quantization accuracy of each onnx operator is obtained, and the final evaluation result is generated. This solves the problem that the related technology cannot evaluate the quantization accuracy of each operator in the model, and can quickly and accurately determine the accuracy loss of each operator after quantization. It can help model developers and quantization deployment personnel accurately locate operators with low quantization accuracy, so as to take corresponding measures.

[0059] Next, a neural network quantization accuracy evaluation device proposed according to an embodiment of the present application will be described with reference to the accompanying drawings.

[0060] Figure 3 Schematic diagram of a neural network quantization accuracy evaluation device according to an embodiment of the present application.

[0061] like Figure 3 As shown, the neural network quantization accuracy evaluation device 10 includes: an acquisition module 100, a conversion module 200 and an evaluation module 300.

[0062] Among them, the acquisition module 100 is used to obtain the original onnx model and call the application programming interface API of onnx_graphsurgeon to set all onnx operators in the original onnx model as output nodes; the conversion module 200 is used to generate a quantization threshold for each onnx operator based on the relative entropy of the distribution of the output tensor of each onnx operator before and after quantization, and convert the quantization threshold of each onnx operator into a first target trt file, and convert the unquantized threshold of each onnx operator into a second target trt file; and the generation module 300 is used to load the first target trt file and the second target trt file based on the preset TensorRT environment and output node, obtain the first output tensor of each onnx operator in the quantized model and the second output tensor of each onnx operator in the unquantized model, obtain the quantization accuracy of each onnx operator according to the first output tensor and the second output tensor, and generate the final evaluation result according to the quantization accuracy of each onnx operator.

[0063] Optionally, in some embodiments, after obtaining the quantization accuracy of each ONNX operator based on the first output tensor and the second output tensor, the generation module 300 further includes: based on the quantization accuracy of each ONNX operator, screening out the ONNX operators to be processed that are less than a preset quantization accuracy; and generating a target processing strategy according to the ONNX operators to be processed.

[0064] Optionally, in some embodiments, the quantization accuracy is characterized by a first indicator and a second indicator, wherein the first indicator is a relative error and the second indicator is a cosine value of the first output tensor and the second output tensor.

[0065] Optionally, in some embodiments, after generating the quantization threshold of each onnx operator based on the relative entropy of the distribution of the output tensor of each onnx operator before and after quantization, it also includes: outputting the quantization threshold as a target calib file, wherein the calib file saves the truncation threshold of the output tensor of each onnx operator in hexadecimal.

[0066] Optionally, in some embodiments, after setting all onnx operators in the original onnx model as output nodes, the method further includes: placing the output tenors of all onnx operators in the outputs list of the graph.

[0067] It should be noted that the aforementioned explanation of the embodiment of the neural network quantization accuracy evaluation method is also applicable to the neural network quantization accuracy evaluation device of this embodiment, and will not be repeated here.

[0068] According to the neural network quantization accuracy evaluation device proposed in the embodiment of the present application, by obtaining the original onnx model and calling the onnx_graphsurgeon application programming interface API, all onnx operators in the original onnx model are set as output nodes, and quantization thresholds for each onnx operator are generated. The quantization thresholds and unquantized thresholds of each onnx operator are converted into first target trt files and second target trt files respectively. Based on the preset TensorRT environment and output nodes, the first output tensor of each onnx operator in the quantized model and the second output tensor of each onnx operator in the unquantized model are obtained, and the quantization accuracy of each onnx operator is obtained, and the final evaluation result is generated. This solves the problem that the related technology cannot evaluate the quantization accuracy of each operator in the model, can quickly and accurately determine the accuracy loss of each operator after quantization, and can help model developers and quantization deployment personnel accurately locate operators with low quantization accuracy, so as to take corresponding measures.

[0069] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0070] Memory 401 , processor 402 , and computer programs stored in the memory 401 and executable on the processor 402 .

[0071] When the processor 402 executes the program, the neural network quantization accuracy evaluation method provided in the above embodiment is implemented.

[0072] Furthermore, the electronic device further includes:

[0073] The communication interface 403 is used for communication between the memory 401 and the processor 402 .

[0074] The memory 401 is used to store computer programs that can be run on the processor 402 .

[0075] The memory 401 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0076] If the memory 401, the processor 402, and the communication interface 403 are implemented independently, the communication interface 403, the memory 401, and the processor 402 can be connected to each other via a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0077] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0078] The processor 402 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0079] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned neural network quantization accuracy evaluation method.

[0080] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0081] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0082] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0083] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0084] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0085] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A neural network quantization accuracy evaluation method, characterized in that: The following steps are involved: Obtain the original onnx model and call the onnx_graphsurgeon application programming interface API to set all onnx operators in the original onnx model as output nodes; Generate a quantization threshold value for each onnx operator according to the relative entropy of the distribution of the output tensor before and after quantization of each onnx operator, convert the quantization threshold value of each onnx operator into a first target trt file, and convert the unquantized threshold value of each onnx operator into a second target trt file; as well as Based on the preset TensorRT environment and the output node, the first target trt file and the second target trt file are loaded, and the first target trt file and the second target trt file are deserialized into corresponding engines. The same input data is used for inference to obtain the first output tensor of each onnx operator in the quantized model and the second output tensor of each onnx operator in the unquantized model. The quantization accuracy of each onnx operator is obtained according to the first output tensor and the second output tensor, and the final evaluation result is generated according to the quantization accuracy of each onnx operator.

2. The method according to claim 1, characterized in that After obtaining the quantization accuracy of each onnx operator according to the first output tensor and the second output tensor, the method further includes: Based on the quantization precision of each onnx operator, filter out onnx operators to be processed that have a quantization precision less than a preset quantization precision; Generate a target processing strategy according to the onnx operator to be processed.

3. The method according to claim 2, characterized in that The quantization accuracy is characterized by a first indicator and a second indicator, wherein the first indicator is a relative error, and the second indicator is a cosine value of the first output tensor and the second output tensor.

4. The method according to claim 1, wherein After generating the quantization thresholds of the onnx operators according to the relative entropy of the distribution of the output tensors of the onnx operators before and after quantization, the method further includes: The quantization threshold is output as a target calib file, wherein the calib file stores the truncation threshold of the output tensor of each onnx operator in hexadecimal.

5. The method according to claim 1, wherein After setting all the onnx operators in the original onnx model as output nodes, the method further includes: Place the output tenors of all onnx operators in the outputs list of the graph.

6. A neural network quantization accuracy evaluation device, characterized in that: include: An acquisition module is used to obtain the original onnx model and call the onnx_graphsurgeon application programming interface API to set all onnx operators in the original onnx model as output nodes; A conversion module is used to generate a quantization threshold value of each onnx operator according to the relative entropy of the distribution of the output tensor of each onnx operator before and after quantization, and convert the quantization threshold value of each onnx operator into a first target trt file, and convert the unquantized threshold value of each onnx operator into a second target trt file; as well as A generation module is used to load the first target trt file and the second target trt file based on the preset TensorRT environment and the output node, and deserialize the first target trt file and the second target trt file into corresponding engines, use the same input data for inference, obtain the first output tensor of each onnx operator in the quantized model and the second output tensor of each onnx operator in the unquantized model, obtain the quantization accuracy of each onnx operator according to the first output tensor and the second output tensor, and generate a final evaluation result according to the quantization accuracy of each onnx operator.

7. The device according to claim 6, characterized in that After obtaining the quantization accuracy of each onnx operator according to the first output tensor and the second output tensor, the generation module further includes: Based on the quantization precision of each onnx operator, filter out onnx operators to be processed that have a quantization precision less than a preset quantization precision; Generate a target processing strategy according to the onnx operator to be processed.

8. The device according to claim 7, characterized in that The quantization accuracy is characterized by a first indicator and a second indicator, wherein the first indicator is a relative error, and the second indicator is a cosine value of the first output tensor and the second output tensor.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the neural network quantization accuracy evaluation method according to any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the neural network quantization accuracy evaluation method according to any one of claims 1 to 5.

Images