ONNX-based data acceleration reasoning method and device and medium
By replacing the custom operators in the quantized ONNX model with ONNX native operators and using GPU-accelerated inference, the problem of ONNX Runtime being unable to efficiently infer the quantized model is solved, shortening the model quantization cycle and improving data inference efficiency.
Patent Information
- Application Number
- CN202510442251.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-09
- Publication Date
- 2025-09-16
AI Technical Summary
ONNX Runtime cannot efficiently infer the quantized model, which makes accuracy verification time-consuming and reduces quantization efficiency.
Replace the quantized custom operators with ONNX native operators to generate an accelerated fixed-point model, and then use the GPU version for accelerated inference.
The model quantization cycle is shortened and data inference efficiency is improved.
Smart Images
Figure CN120654804A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of model reasoning technology, and more specifically, to a method, device, and medium for data acceleration reasoning based on ONNX. Background Art
[0002] With the increasing application of deep learning models in various fields, model deployment and inference efficiency have become key challenges. Neural network models generally have the characteristics of large number of model parameters, high computational complexity, high memory usage, and high model accuracy. When deploying neural network models, we want the network to be as small as possible, which requires compression methods such as model quantization. Quantization technology, as an effective model optimization method, can significantly reduce model size and computational complexity, accelerate calculations, maintain accuracy, save memory, energy, and reduce chip area, while maintaining high inference accuracy.
[0003] ONNX defines a set of standard formats that are independent of the environment and platform. ONNX enhances the interoperability of various AI models, making it easier for researchers and engineers to migrate models between different deep learning frameworks and hardware platforms. ONNX-based model quantization tools are particularly useful for deploying models on edge devices and other resource-constrained environments. After the model is quantized, the accuracy of the quantized model needs to be verified to ensure that the accuracy loss after quantization is within a reasonable range. If the loss is too large, a different quantization strategy needs to be used for re-quantization. Therefore, accuracy verification of the model after quantization is an important part of model quantization, and it is also a time-consuming part. For the current ONNX quantized fixed-point model, the quantization parameters and fixed-point related information are stored in the form of custom operators. However, ONNX Runtime does not support custom operator inference, which results in ONNX Runtime being unable to use the GPU to efficiently infer the quantized model. As a result, the accuracy verification link is very time-consuming, greatly reducing the quantization efficiency. Summary of the Invention
[0004] In response to the deficiencies of the prior art, the present invention provides a method, device, and medium for data acceleration reasoning based on ONNX.
[0005] According to one aspect of the present invention, a method for accelerating data reasoning based on ONNX is provided, comprising:
[0006] A quantized accelerated fixed-point model is used to accelerate inference on the processed data to obtain the data inference result. The generation process of the accelerated fixed-point model is as follows:
[0007] Quantize the original ONNX floating-point model to generate a quantized IR fixed-point model;
[0008] Replace the custom operators in the IR fixed-point model with ONNX native operators to generate an accelerated fixed-point model;
[0009] The validation data set is used to verify the accuracy of the accelerated fixed-point model, and the final accelerated fixed-point model is determined when the accuracy is met.
[0010] Optionally, replace the custom operators in the IR fixed-point model with ONNX native operators to generate an accelerated fixed-point model, including:
[0011] Get all custom operators in the IR fixed-point model;
[0012] According to the fixed-point rules of different operator types, the custom operator is replaced with the ONNX native operator to generate an accelerated fixed-point model.
[0013] Optionally, based on the fixed-point rules of different operator types, replace custom operators with ONNX native operators to generate accelerated fixed-point models, including:
[0014] Traverse all custom operators and obtain the operator type and quantization parameters of each custom operator;
[0015] Use the quantization parameters of each custom operator respectively, replace each custom operator with an ONNX native operator according to the fixed-point rules corresponding to the operator type of each custom operator, and generate an accelerated fixed-point model. The output name of the last ONNX native operator in the accelerated fixed-point model is consistent with the output name of the replaced custom operator.
[0016] Optionally, the quantization parameters include: operator output value range, quantization bit width ratio, input data type, output data type, input scale value, output scale value, and fixed-point value required by the operator.
[0017] Optionally, the generation process of the accelerated fixed-point model further includes: comparing and verifying the output results of each layer of the accelerated fixed-point model and the IR fixed-point model.
[0018] Optionally, perform a self-verification comparison on the output results of each layer of the accelerated fixed-point model and the IR fixed-point model, including:
[0019] Get the output name intersection of the accelerated fixed-point model and the IR fixed-point model;
[0020] Use the same input data to infer the IR fixed-point model and the accelerated fixed-point model respectively, and obtain the intersection of the inference output names;
[0021] Compare the output name intersection and the inference output name intersection layer by layer to see if they are aligned. If there is an unaligned output layer, check the accuracy of the fixed-point rule.
[0022] Optionally, the accuracy of the accelerated fixed-point model is verified using a calibration dataset, which also includes:
[0023] If the accuracy verification fails, adjust the quantization strategy and re-quantize the model.
[0024] According to another aspect of the present invention, a device for accelerating inference of data based on ONNX is provided, comprising:
[0025] The inference module is used to accelerate inference on the processed data using a quantized accelerated fixed-point model to obtain data inference results. The generation process of the accelerated fixed-point model is as follows:
[0026] The quantization submodule is used to quantize the original ONNX floating-point model and generate a quantized IR fixed-point model;
[0027] The replacement submodule is used to replace the custom operators in the IR fixed-point model with ONNX native operators to generate an accelerated fixed-point model;
[0028] The verification submodule is used to verify the accuracy of the accelerated fixed-point model using a verification data set, and determine the final accelerated fixed-point model when the accuracy is met.
[0029] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the method described in any one of the above aspects of the present invention.
[0030] According to another aspect of the present invention, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; and the processor for reading the executable instructions from the memory and executing the instructions to implement the method described in any one of the above aspects of the present invention.
[0031] Therefore, during the model quantization process, this invention replaces fixed-point operators with ONNX native operators based on the fixed-point schemes of different custom operators. Since the quantized model is composed entirely of ONNX native operators after the custom operators are replaced, the replaced fixed-point model can be accelerated using the ONNX Runtime GPU version, thereby speeding up the entire verification process, shortening the model quantization cycle, and further accelerating the data inference process. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] A more complete understanding of exemplary embodiments of the present invention may be obtained by referring to the following drawings:
[0033] Figure 1 1 is a flow chart of a method for accelerating inference of data based on ONNX provided by an exemplary embodiment of the present invention;
[0034] Figure 21 is another flowchart of a method for accelerating inference of data based on ONNX provided by an exemplary embodiment of the present invention;
[0035] Figure 3 is a schematic diagram of an IR fixed-point model provided by an exemplary embodiment of the present invention;
[0036] Figure 4 is a schematic diagram of an accelerated fixed-point model provided by an exemplary embodiment of the present invention;
[0037] Figure 5 1 is a flow chart of a method for comparing inference outputs of an IR fixed-point model and an accelerated fixed-point model provided by an exemplary embodiment of the present invention;
[0038] Figure 6 1 is a schematic diagram of a device for accelerating inference of data based on ONNX provided by an exemplary embodiment of the present invention;
[0039] Figure 7 This is a structure of an electronic device provided by an exemplary embodiment of the present invention. DETAILED DESCRIPTION
[0040] Below, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein.
[0041] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.
[0042] Those skilled in the art will understand that the terms "first" and "second" in the embodiments of the present invention are only used to distinguish different steps, devices or modules, and neither represent any specific technical meaning nor indicate the necessary logical order between them.
[0043] It should also be understood that, in the embodiments of the present invention, “a plurality of” may refer to two or more than two, and “at least one” may refer to one, two or more than two.
[0044] It should also be understood that any component, data or structure mentioned in the embodiments of the present invention can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.
[0045] In addition, the term "and / or" in this invention merely describes an association relationship between related objects, indicating that three possible relationships exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. Furthermore, the character " / " in this invention generally indicates that the related objects are in an "or" relationship.
[0046] It should also be understood that the description of the various embodiments of the present invention focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced with each other. For the sake of brevity, they will not be described one by one.
[0047] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.
[0048] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.
[0049] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.
[0050] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0051] Exemplary Methods
[0052] Figure 1 This is a flow chart of a method for accelerating inference based on ONNX data provided by an exemplary embodiment of the present invention. This embodiment can be applied to electronic devices, such as Figure 1 As shown, the method 100 for accelerating inference based on ONNX data includes the following steps:
[0053] Step 101: Use a quantized accelerated fixed-point model to perform accelerated inference on the data to be processed to obtain a data inference result. The generation process of the accelerated fixed-point model is as follows:
[0054] Quantize the original ONNX floating-point model to generate a quantized IR fixed-point model;
[0055] Replace the custom operators in the IR fixed-point model with ONNX native operators to generate an accelerated fixed-point model;
[0056] The validation data set is used to verify the accuracy of the accelerated fixed-point model, and the final accelerated fixed-point model is determined when the accuracy is met.
[0057] Specifically, in response to the phenomenon that accuracy verification takes a long time during the quantization process, this paper proposes an acceleration solution for the ONNX fixed-point model to solve the technical problem of long data inference time. Specifically, the fixed-point operators are replaced with ONNX native operators according to the fixed-point solutions of different custom operators. Since the quantization model is composed entirely of ONNX native operators after the custom operators are replaced, the ONNX Runtime GPU version can be used to accelerate the replaced fixed-point model, thereby speeding up the entire verification process and greatly shortening the model quantization cycle. Figure 2 As shown in the figure, the generation process of the accelerated fixed-point model is as follows:
[0058] S10: Prepare the original ONNX floating-point model and validation dataset to be quantized;
[0059] S20: Select the preset quantization strategy to quantize the original ONNX floating-point model and generate an IR fixed-point model in the ONNX format. The fixed-point of the IP fixed-point model is implemented based on ONNX custom operators, and the proportion of custom operators is very high.
[0060] S30: Convert the IR fixed-point model to an accelerated fixed-point model. The process is as follows:
[0061] 1) Load the IR fixed-point model;
[0062] 2) Get all custom operators of the IR fixed-point model;
[0063] 3) Different types of custom operators are replaced with different ONNX native operators according to their respective fixed-point rules. Specifically:
[0064] 3.1) Traverse the custom operator in step 2);
[0065] 3.2) Obtain the operator type and quantization parameters of the custom operator. The quantization parameters include some common properties of the custom operator, such as the min and max output value ranges of the operator after data calibration statistics, the quantization bit width ratio bit_width, the input data type input_bit, the output data type output_bit, the input scale value bottom_scale, and the output scale value top_scale. They also include the fixed-point values required by different operator types, such as q value, scale_int16, and bias_int16.
[0066] 3.3) Depending on the operator type, use the obtained quantization parameters combined with different fixed-point rules to replace the custom operator with an ONNX native operator equivalent, and ensure that the output name of the last native operator is consistent with the output name of the replaced custom operator.
[0067] 4) After completing the above steps, all custom operators in the IR fixed-point model are replaced with native ONNX operators, generating an accelerated ONNX fixed-point model. Because quantization schemes are sometimes modified to improve quantization accuracy or adapt to hardware, further comparison of the output results of each layer of the two models is necessary to detect these changes and ensure mathematical equivalence between the accelerated fixed-point model and the original fixed-point model.
[0068] Get the output name intersection of the accelerated fixed-point model and the IR fixed-point model, use the same input data to perform inference on the fixed-point accelerated model and the original fixed-point model respectively, and compare the output name intersection results layer by layer to see if they are aligned. If there are misaligned output layers, exit the program and check whether the quantization scheme has been adjusted, verify the accuracy of the fixed-point rules, and regenerate the accelerated model.
[0069] S40: Verify the accuracy of the fixed-point model. If the accuracy meets the requirements, the quantization is terminated. Otherwise, it is necessary to return to step S20 to adjust the quantization strategy and re-quantize until the accuracy requirements are met.
[0070] After completing the above steps, a GPU-accelerated fixed-point model is generated. This accelerated fixed-point model can independently verify quantization accuracy without the quantization framework, greatly shortening the model quantization cycle and accelerating the data inference process.
[0071] In one embodiment of the present invention, the implementation process of replacing the custom operator in the IR fixed-point model with the ONNX native operator is as follows:
[0072] The operator type of the custom operator is determined so that the custom operator can be replaced based on different fixed-point rules. For example, to replace the custom fixed-point operator Input of the 8-bit floating-point input quantization model:
[0073] The fixed-point rules for operators of type Input are:
[0074] clip(c_round(clip((X*scale_int16),-2**64,2**64-1) / 2*q_scale),
[0075] -2**bit_width,2**bit_width-1).
[0076] Where X is a floating-point input; scale_int16 is the fixed-point value of the input data scale; q_scale is the scale shift exponent; bit_width is the quantization bit width; clip is the truncation function; and c_round is a rounding rule, specifically sign(x)*floor(abs(x)+0.5).
[0077] These fixed-point information values can be obtained from the custom operator. Therefore, according to the above formula and fixed-point information values, the ONNX native operators are inserted into the new graph in sequence. The output name of the last ONNX native operator is the output name of the custom operator input_0_round_clip. The examples of the original IR fixed-point model and the accelerated fixed-point model are as follows Figure 3 and Figure 4 shown.
[0078] In one embodiment of the present invention, because the quantization scheme is sometimes modified to improve quantization accuracy or to adapt to hardware, in order to be able to perceive such changes and ensure that the accelerated fixed-point model is completely mathematically equivalent to the original fixed-point model, the present invention compares the inference output of the original IR fixed-point model and the accelerated fixed-point model to ensure the accuracy of the accelerated fixed-point model. Figure 5 As shown, the specific implementation process is as follows:
[0079] 1) First, insert the custom operator of the IR fixed-point model into the output node. Only by inserting the output node can the inference output of this layer be obtained.
[0080] 2) Insert an output node into the node of the accelerated fixed-point model that has the same output name as the original fixed-point model. According to the custom operator replacement implementation method, the outputs of the two nodes should be equivalent.
[0081] 3) Use the same data to infer on the two models and obtain the model output results, that is, the output results of the inserted output node set.
[0082] 4) Compare the output results of the two models in turn to see if they are consistent. If they are inconsistent, print the inconsistent output node names and locate the cause of the error. If all output results are consistent, end the process.
[0083] Therefore, during the model quantization process, this invention replaces fixed-point operators with ONNX native operators based on the fixed-point schemes of different custom operators. Since the quantized model is composed entirely of ONNX native operators after the custom operators are replaced, the replaced fixed-point model can be accelerated using the ONNX Runtime GPU version, thereby speeding up the entire verification process, significantly shortening the model quantization cycle, and further accelerating the data inference process.
[0084] Exemplary devices
[0085] Figure 6 FIG is a schematic diagram of a device for accelerating inference based on ONNX data provided by an exemplary embodiment of the present invention. Figure 6 As shown, the apparatus 600 includes:
[0086] The inference module 610 is configured to perform accelerated inference on the processed data using a quantized accelerated fixed-point model to obtain a data inference result. The generation process of the accelerated fixed-point model is as follows:
[0087] The quantization submodule is used to quantize the original ONNX floating-point model and generate a quantized IR fixed-point model;
[0088] The replacement submodule is used to replace the custom operators in the IR fixed-point model with ONNX native operators to generate an accelerated fixed-point model;
[0089] The verification submodule is used to verify the accuracy of the accelerated fixed-point model using a verification data set, and determine the final accelerated fixed-point model when the accuracy is met.
[0090] Optionally, replace submodules, including:
[0091] A first acquisition unit is used to acquire all custom operators in the IR fixed-point model;
[0092] The replacement unit is used to replace custom operators with ONNX native operators based on the fixed-point rules of different operator types to generate accelerated fixed-point models.
[0093] Optionally, the replacement unit comprises:
[0094] The traversal subunit is used to traverse all custom operators and obtain the operator type and quantization parameters of each custom operator;
[0095] The replacement subunit is used to use the quantization parameters of each custom operator respectively, replace each custom operator with an ONNX native operator according to the fixed-point rules corresponding to the operator type of each custom operator, and generate an accelerated fixed-point model, where the output name of the last ONNX native operator in the accelerated fixed-point model is consistent with the output name of the replaced custom operator.
[0096] Optionally, the quantization parameters include: operator output value range, quantization bit width ratio, input data type, output data type, input scale value, output scale value, and fixed-point value required by the operator.
[0097] Optionally, the generation process of the accelerated fixed-point model further includes: a self-verification submodule, which is used to compare and self-verify the output results of each layer of the accelerated fixed-point model and the IR fixed-point model.
[0098] Optionally, the self-verification submodule includes:
[0099] The second acquisition unit is used to obtain the output name intersection of the acceleration fixed-point model and the IR fixed-point model;
[0100] The inference unit is used to perform inference on the IR fixed-point model and the accelerated fixed-point model using the same input data and obtain the intersection of the inference output names;
[0101] The comparison unit is used to compare the output name intersection and the inference output name intersection layer by layer to see if they are aligned. If there is an unaligned output layer, the accuracy of the fixed-point rule is checked.
[0102] Optionally, the verification submodule uses a calibration data set to verify the accuracy of the accelerated fixed-point model, and further includes:
[0103] The adjustment unit is used to adjust the quantization strategy and re-quantize the model when the accuracy verification fails.
[0104] Exemplary electronic devices
[0105] Figure 7 This is the structure of an electronic device provided by an exemplary embodiment of the present invention. Figure 7 As shown, the electronic device 70 includes one or more processors 71 and a memory 72 .
[0106] The processor 71 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.
[0107] The memory 72 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may, for example, include read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 71 may execute the program instructions to implement the methods of the software programs of the various embodiments of the present invention described above and / or other desired functions. In one example, the electronic device may further include: an input device 73 and an output device 74, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown).
[0108] In addition, the input device 73 may also include, for example, a keyboard, a mouse, and the like.
[0109] The output device 74 can output various information to the outside. The output device 74 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto.
[0110] Of course, to simplify, Figure 7Only some of the components related to the present invention in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device may further include any other appropriate components according to specific application conditions.
[0111] Exemplary computer program products and computer-readable storage media
[0112] In addition to the above-mentioned methods and devices, an embodiment of the present invention may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to perform the steps of the method according to various embodiments of the present invention described in the above "Exemplary Method" section of this specification.
[0113] The computer program product may be written in any combination of one or more programming languages to implement the operations of embodiments of the present invention, including object-oriented programming languages such as Java, C++, and conventional procedural programming languages such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0114] In addition, an embodiment of the present invention may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present invention described in the above "Exemplary Method" section of this specification.
[0115] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, system or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.
[0116] The basic principles of the present invention have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in the present invention are merely illustrative and non-limiting, and should not be construed as necessarily possessed by each embodiment of the present invention. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, and are not intended to be limiting. These details do not necessarily limit the present invention to being implemented using these specific details.
[0117] Each embodiment in this specification is described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. References to the same or similar parts between the various embodiments are sufficient. For system embodiments, since they largely correspond to method embodiments, their description is relatively simple. For relevant parts, references to the description of the method embodiments are sufficient.
[0118] The block diagrams of the devices, systems, equipment, and systems involved in the present invention are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, systems, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "including," "comprising," "having," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.
[0119] The method and system of the present invention may be implemented in many ways. For example, the method and system of the present invention may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above sequence of steps for the method is for illustration only, and the steps of the method of the present invention are not limited to the sequence specifically described above, unless otherwise specified. In addition, in some embodiments, the present invention may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present invention. Thus, the present invention also covers recording media that store programs for executing the method according to the present invention.
[0120] It should also be noted that, in the system, device and method of the present invention, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present invention. The above description of the disclosed aspects is provided to enable any technician in this field to make or use the present invention. Various modifications to these aspects will be very obvious to those skilled in the art, and the general principles defined here can be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown here, but according to the widest scope consistent with the principles disclosed here and novel features.
[0121] The above description has been provided for the purpose of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present invention to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.
Claims
1. A method for accelerating data reasoning based on ONNX, characterized in that: include: A quantized accelerated fixed-point model is used to perform accelerated inference on the processed data to obtain a data inference result, wherein the generation process of the accelerated fixed-point model is as follows: Quantize the original ONNX floating-point model to generate a quantized IR fixed-point model; Replace the custom operators in the IR fixed-point model with ONNX native operators to generate an accelerated fixed-point model; The accuracy of the accelerated fixed-point model is verified using a validation data set, and the final accelerated fixed-point model is determined when the accuracy is met.
2. The method according to claim 1, characterized in that Replace the custom operators in the IR fixed-point model with ONNX native operators to generate an accelerated fixed-point model, including: Get all custom operators in the IR fixed-point model; According to the fixed-point rules of different operator types, the custom operator is replaced with the ONNX native operator to generate the accelerated fixed-point model.
3. The method according to claim 2, characterized in that According to the fixed-point rules of different operator types, the custom operator is replaced with the ONNX native operator to generate the accelerated fixed-point model, including: Traverse all custom operators and obtain the operator type and quantization parameters of each custom operator; Using the quantization parameters of each custom operator respectively, each custom operator is replaced with an ONNX native operator according to the fixed-point rule corresponding to the operator type of each custom operator to generate the accelerated fixed-point model, wherein the output name of the last ONNX native operator of the accelerated fixed-point model is consistent with the output name of the replaced custom operator.
4. The method according to claim 3, characterized in that The quantization parameters include: operator output value range, quantization bit width ratio, input data type, output data type, input scale value, output scale value and fixed-point value required by the operator.
5. The method according to claim 3, characterized in that The generation process of the accelerated fixed-point model further includes: comparing and verifying the output results of each layer of the accelerated fixed-point model and the IR fixed-point model.
6. The method according to claim 5, characterized in that Comparing and verifying the output results of each layer of the accelerated fixed-point model and the IR fixed-point model, including: Obtaining the intersection of output names of the acceleration fixed-point model and the IR fixed-point model; Using the same input data to perform inference on the IR fixed-point model and the accelerated fixed-point model respectively, and obtaining an intersection of inference output names; Compare the output name intersection and the inference output name intersection layer by layer to see if they are aligned. If there is an unaligned output layer, check the accuracy of the fixed-point rule.
7. The method according to claim 1, characterized in that Verifying the accuracy of the accelerated fixed-point model using a calibration data set also includes: If the accuracy verification fails, adjust the quantization strategy and re-quantize the model.
8. A device for accelerating inference of data based on ONNX, characterized in that: include: The inference module is used to use a quantized accelerated fixed-point model to perform accelerated inference on the processed data to obtain data inference results, wherein the generation process of the accelerated fixed-point model is as follows: The quantization submodule is used to quantize the original ONNX floating-point model and generate a quantized IR fixed-point model; The replacement submodule is used to replace the custom operator in the IR fixed-point model with the ONNX native operator to generate an accelerated fixed-point model; The verification submodule is used to verify the accuracy of the accelerated fixed-point model using a verification data set, and determine the final accelerated fixed-point model when the accuracy is met.
9. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and the computer program is used to execute the method according to any one of claims 1 to 7.
10. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method according to any one of claims 1 to 7.