A model deployment method and device, electronic equipment and storage medium

Abstract

The application provides a model deployment method and device, electronic equipment and storage medium, characterized in that the method comprises: quantizing an initial model after training to obtain a quantized model; obtaining an onnx model and a quantization calibration file of each module of the quantized model; correcting the onnx model and the quantization calibration file of each module; merging the onnx model and the quantization calibration file of each module to obtain a first onnx model and a quantization calibration file set; converting the quantization calibration file set into a readable file; generating a model based on the first onnx model and the readable file by a platform; and deploying the model generated by the platform. In the application, the onnx model of the quantized model is obtained, the fusion of operators by tensorRT is improved, the running efficiency is improved, the model is converted into a model of a specific platform, the compatibility of the model and the platform is better, each module is independent of each other, the deployment of the model can be quickly completed, and time for an engineering project is saved.

Description

Technical Field

[0001] This application relates to the field of access control technology, specifically to a model deployment method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the advancement of technology, the application of artificial intelligence is becoming increasingly widespread, which has had a significant impact on people's daily lives. As society develops, the demand for artificial intelligence is also increasing.

[0003] The development of artificial intelligence is highly complex, from model building and training to deployment and application, requiring a significant investment of time and expertise from technical personnel. The deployment process, in particular, is complex and time-consuming. Technical personnel must select appropriate frameworks and algorithms, experiment with the model's functionality through training and testing, and make necessary corrections. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the present invention provides a model deployment method, apparatus, electronic device and storage medium to solve the technical problem that the deployment of models on a specific platform is complex and time-consuming.

[0005] The model deployment method, apparatus, electronic device, and storage medium provided by this invention include:

[0006] The initial trained model is quantized to obtain a quantized model.

[0007] Obtain the ONNX model and quantization calibration file for each module of the quantization model;

[0008] Correct the ONNX model and quantization calibration file for each module;

[0009] The ONNX models and quantization calibration files of each module are merged to obtain a first ONNX model and quantization calibration file set;

[0010] Transcode the quantization calibration file set into a readable file;

[0011] The platform generates a model based on the first ONNX model and the readable file;

[0012] Deploy the model generated by the platform.

[0013] In one embodiment of the present invention, quantizing the trained initial model to obtain a quantized model includes: quantizing the trained initial model using a PTQ tool or a QAT tool to obtain a quantized model.

[0014] In one embodiment of the present invention, obtaining the ONNX model and quantization calibration file of each module of the quantization model includes: using the mqbench tool to obtain the ONNX model and quantization calibration file of each module of the quantization model.

[0015] In one embodiment of the present invention, the correction of the ONNX model and quantization calibration file for each module includes:

[0016] Remove redundant output nodes from the ONNX model of each module;

[0017] Set the same tensor names to be consistent in the ONNX models of each module;

[0018] Modify the ONNX model of each module according to the requirements of the deployment code.

[0019] In one embodiment of the present invention, the step of merging the ONNX models and quantization calibration files of each module to obtain a first ONNX model and quantization calibration file set includes:

[0020] The modified ONNX models of each module are merged into one to obtain the first ONNX model;

[0021] The quantization calibration files of each module are merged into one to obtain the quantization calibration file set.

[0022] In one embodiment of the present invention, the step of transcoding the quantization calibration file set into a readable file includes: using an ASCII encoding tool to convert the quantization calibration file set into a readable file.

[0023] In one embodiment of the present invention, the model generated by the platform based on the first ONNX model and the readable file includes: inputting the first ONNX model and the readable file into the target platform, and outputting a target platform model that runs at a fixed point by the target platform.

[0024] In one embodiment of the present invention, a model deployment apparatus is also provided, the apparatus comprising:

[0025] The quantization module is configured to quantize the trained model to obtain a quantized model.

[0026] The acquisition module is configured to acquire the ONNX models and quantization calibration files of each module of the quantization model;

[0027] The correction module is configured to correct the ONNX models and quantization calibration files of each module.

[0028] The merging module is configured to merge the ONNX models and quantization calibration files of each module to obtain the first ONNX model and quantization calibration file set;

[0029] The transcoding module is configured to transcode the quantization calibration file set into a readable file;

[0030] The generation module is configured to generate a model from the platform based on the ONNX model and the readable file.

[0031] The deployment module is configured to deploy models generated by the platform.

[0032] In one embodiment of the present invention, an electronic device is also provided, the electronic device comprising:

[0033] One or more processors;

[0034] A storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the model deployment method as described in any one of claims 1 to 7.

[0035] In one embodiment of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a computer processor, causes the computer to perform the model deployment method according to any one of claims 1 to 7.

[0036] The beneficial effects of this invention are: obtaining the ONNX model of the quantized model in this invention improves the fusion of operators in TensorRT, increases the running efficiency, and transforms it into a model for a specific platform, making the model and platform more compatible. Each module is independent of each other, and the model can be deployed quickly, saving time for engineering projects.

[0037] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0038] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0039] Figure 1 This is a schematic diagram illustrating the implementation environment of a model deployment method according to an exemplary embodiment of this application;

[0040] Figure 2 This is a flowchart illustrating a model deployment method in an exemplary embodiment of this application;

[0041] Figure 3 yes Figure 2 The flowchart of step S230 in the illustrated embodiment is shown in an exemplary embodiment;

[0042] Figure 4 yes Figure 2 The flowchart of step S240 in the illustrated embodiment is shown in an exemplary embodiment;

[0043] Figure 5 This is a block diagram illustrating an apparatus for a model deployment method, as shown in an exemplary embodiment of this application;

[0044] Figure 6 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0045] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0046] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0047] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0048] The Open Neural Network Exchange (ONNX) format is a standard for representing deep learning models, enabling models to be transferred between different frameworks.

[0049] ONNX is an open file format designed for machine learning, used to store trained models. It allows different AI frameworks (such as PyTorch and MXNet) to store model data in the same format and interact with each other. The ONNX specification and code were primarily developed by companies such as Microsoft, Amazon, Facebook, and IBM, and are hosted as open source on GitHub. Currently, the officially supported deep learning frameworks for loading ONNX models and performing inference include: Caffe2, PyTorch, MXNet, ML.NET, TensorRT, and Microsoft CNTK. TensorFlow also unofficially supports ONNX.

[0050] MQBench is a benchmark and framework for evaluating quantization algorithms on real-world hardware deployments. Integrating the latest features of PyTorch, MQBench can automatically trace full-precision models and convert them to quantized models. It provides researchers with a wealth of hardware and algorithms to benchmark the deployability and reproducibility of quantization.

[0051] Quantization refers to low-bit processing of the model's weights and activations, making the final generated network model more lightweight. This saves network model storage space, reduces transmission latency, and improves computational efficiency, thereby achieving the goal of performance improvement and optimization.

[0052] Post-training quantization (PTQ) directly converts a trained FP32 network into a fixed-point computation network without requiring any retraining of the original model. The quantization process involves adjusting only a few hyperparameters, allowing for more efficient inference. Quantized models use low-bit representations of parameters, reducing bandwidth requirements during data migration. Furthermore, hardware typically offers higher nominal computing power for low-bit integer data, thus optimizing quantization can significantly improve inference speed in most cases. This method has been widely adopted in numerous edge and cloud deployment scenarios.

[0053] Quantization-aware training (QAT) trains the network during the quantization process, allowing the network parameters to better adapt to the information loss caused by quantization. This method is more flexible, and therefore generally has higher accuracy than post-training quantization.

[0054] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating the implementation environment of a model deployment method as shown in an exemplary embodiment of this application. Figure 1The first smart terminal 110 shown can be any terminal device that supports model deployment preparation, such as an in-vehicle computer, tablet computer, laptop computer, or wearable device, but is not limited to this. Figure 1 The second smart terminal 120 shown can be any terminal device that supports deployment, such as an in-vehicle computer, tablet computer, laptop computer, or wearable device. The first smart terminal 110 can communicate with the second smart terminal 120 via wireless networks such as 3G (third-generation mobile information technology), 4G (fourth-generation mobile information technology), and 5G (fifth-generation mobile information technology), and this is not limited herein.

[0055] Please see Figure 2 , Figure 2 This is a flowchart illustrating a model deployment method in an exemplary embodiment of this application. This method can be applied to... Figure 1 The implementation environment shown is specifically executed by the smart terminal 110 within that implementation environment. It should be understood that this method can also be applied to other exemplary implementation environments and executed by devices in other implementation environments; this embodiment does not limit the implementation environment to which the method is applicable. In one exemplary embodiment, the model deployment method includes at least steps S210 to S270, detailed below:

[0056] Step S210: Quantize the trained initial model to obtain a quantized model.

[0057] In this embodiment, model quantization is performed using two methods: PTQ and QAT. PTQ offers high quantization efficiency, while QAT provides high accuracy. The appropriate quantization method should be selected based on the specific requirements. Both quantization methods are implemented within the mqbench framework.

[0058] Step S220: Obtain the ONNX model and quantization calibration file for each module of the quantization model.

[0059] In this embodiment, the mqbench convert deploy function is used to obtain the ONNX model and quantization calibration file of each module of the quantization model.

[0060] Step S230: Correct the ONNX model and the quantization calibration file of each module.

[0061] In this embodiment, the correction process is a prelude to the next step. After obtaining the ONNX models of each module, they still need to be corrected before they can be further merged.

[0062] Step S240: Merge the ONNX models and quantization calibration files of each module to obtain a first ONNX model and quantization calibration file set.

[0063] In this embodiment, the complete ONNX model is formed by merging the ONNX models of each module in the quantization model. The first ONNX model obtained by merging can decouple the model from the deployment code. After decoupling, changing the splitting method of the model will not affect the deployment code.

[0064] Step S250: Transcode the quantization calibration file set into a readable file.

[0065] In this embodiment, the quantization calibration file set is converted into a hexadecimal calib file, which can be read by the trtexec command.

[0066] Step S260: Based on the first ONNX model and the readable file, the platform generates a model.

[0067] In this embodiment, NVIDIA's GPU is used as the target platform. The first ONNX model and calib file are input into the trtexec tool provided by NVIDIA, and the output is a fixed-point running (int8 format) trt model.

[0068] Step S270: Deploy the model generated by the platform.

[0069] In this embodiment, deployment can be completed by deploying the model generated by the platform to the target platform.

[0070] Figure 3 yes Figure 2 The flowchart of step S230 in the illustrated embodiment is shown in an exemplary embodiment. (See attached flowchart.) Figure 3 As shown, the process of correcting the ONNX model and quantization calibration file of each module includes steps S310 to S330, which are described in detail below:

[0071] Step S310: Delete redundant output nodes in the ONNX model of each module.

[0072] In this embodiment, redundant parts of the original output nodes of each module's ONNX model framework are deleted, and only the nodes required by each module of the quantization model are retained.

[0073] Step S320: Set the names of the same tensors in the ONNX models of each module to be consistent.

[0074] In this embodiment, the same tensor names in each module are standardized for easier management.

[0075] Step S330: Modify the ONNX model of each module according to the requirements of the deployment code.

[0076] In this embodiment, the onnxgraphsurgeon tool is mainly used to modify the onnx model of each module according to the requirements of the deployment code, such as adding operators.

[0077] Figure 4 yes Figure 2 The flowchart of step S240 in the illustrated embodiment is shown in an exemplary embodiment. (See attached flowchart.) Figure 4 As shown, the process of merging the ONNX models and quantization calibration files of each module to obtain the first ONNX model and quantization calibration file set includes steps S410 to S420, which are described in detail below:

[0078] Step S410: Merge the modified ONNX models of each module into one to obtain the first ONNX model.

[0079] In this embodiment, the ONNX models of each module are merged into one to obtain the first ONNX model. This decouples the model from the deployment code, and changing the model splitting method does not affect the deployment code. At the same time, using a larger first ONNX model is more conducive to the fusion of model operators with TensorRT, thereby improving the running efficiency.

[0080] Step S420: Merge the quantization calibration files of each module into one to obtain the quantization calibration file set.

[0081] In this embodiment, in conjunction with merging the ONNX models of each module, the quantization calibration files of each module are also merged to obtain a quantization calibration file set.

[0082] Figure 5 This is a block diagram illustrating an apparatus for a model deployment method, as shown in an exemplary embodiment of this application. This apparatus can be applied to... Figure 1 The implementation environment shown is specifically configured in the smart terminal 110. This device can also be applied to other exemplary implementation environments and specifically configured in other devices. This embodiment does not limit the implementation environment to which the device is applicable.

[0083] like Figure 5 As shown, the exemplary model deployment apparatus includes: a quantization module 510, an acquisition module 520, a correction module 530, a merging module 540, a transcoding module 550, a generation module 560, and a deployment module 570.

[0084] The quantization module 510 is configured to quantize the trained model to obtain a quantized model; the acquisition module 520 is configured to acquire the ONNX models and quantization calibration files of each module of the quantized model; the correction module 530 is configured to correct the ONNX models and quantization calibration files of each module; the merging module 540 is configured to merge the ONNX models and quantization calibration files of each module to obtain a first ONNX model and quantization calibration file set; the transcoding module 550 is configured to transcode the quantization calibration file set into a readable file; the generation module 560 is configured to generate a model from the platform based on the ONNX model and the readable file; and the deployment module 570 is configured to deploy the model generated by the platform.

[0085] It should be noted that the model deployment device and the model deployment method provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs operations have been described in detail in the method embodiments and will not be repeated here. In practical applications, the model deployment device provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation here.

[0086] Embodiments of this application also provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the model deployment method provided in the above embodiments.

[0087] Figure 6 A schematic diagram of a computer system suitable for implementing the embodiments of this application is shown. It should be noted that... Figure 6 The computer system 600 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0088] like Figure 6As shown, the computer system 600 includes a Central Processing Unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 602 or programs loaded from storage portion 608 into Random Access Memory (RAM) 603, such as performing the methods described in the above embodiments. The RAM 603 also stores various programs and data required for system operation. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An Input / Output (I / O) interface 605 is also connected to the bus 604.

[0089] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0090] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs various functions defined in the system of this application.

[0091] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0092] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0093] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0094] Another aspect of this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer's processor, causes the computer to perform the model deployment method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0095] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the model deployment methods provided in the various embodiments described above.

[0096] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A model deployment method, characterized in that, The method includes: The initial trained model is quantized to obtain a quantized model. Obtain the ONNX model and quantization calibration file for each module of the quantization model; Correct the ONNX model and quantization calibration file for each module; The ONNX models and quantization calibration files of each module are merged to obtain a first ONNX model and quantization calibration file set; Transcode the quantization calibration file set into a readable file; The platform generates a model based on the first ONNX model and the readable file; Deploy the model generated by the platform; The ONNX models and quantization calibration files for each module that are being modified include: Remove redundant output nodes from the ONNX model of each module; Set the same tensor names to be consistent in the ONNX models of each module; Modify the ONNX model of each module according to the requirements of the deployment code; The merging of the ONNX models and quantization calibration files from each module to obtain the first set of ONNX models and quantization calibration files includes: The modified ONNX models of each module are merged into one to obtain the first ONNX model; The quantization calibration files of each module are merged into one to obtain the quantization calibration file set.

2. The model deployment method according to claim 1, characterized in that, The step of quantizing the initial model after training to obtain a quantized model includes: using the PTQ tool or the QAT tool to quantize the initial model after training to obtain a quantized model.

3. The model deployment method according to claim 1, characterized in that, The process of obtaining the ONNX models and quantization calibration files for each module of the quantization model includes: using the mqbench tool to obtain the ONNX models and quantization calibration files for each module of the quantization model.

4. The model deployment method according to claim 1, characterized in that, The step of transcoding the quantization calibration file set into a readable file includes: using an ASCII encoding tool to convert the quantization calibration file set into a readable file.

5. The model deployment method according to claim 1, characterized in that, Based on the first ONNX model and the readable file, the platform generates a model by: inputting the first ONNX model and the readable file into the target platform, and outputting a model of the target platform that runs at a fixed point.

6. A model deployment device, characterized in that, The device includes: The quantization module is configured to quantize the trained model to obtain a quantized model. The acquisition module is configured to acquire the ONNX models and quantization calibration files of each module of the quantization model; The correction module is configured to correct the ONNX models and quantization calibration files of each module. The merging module is configured to merge the ONNX models and quantization calibration files of each module to obtain the first ONNX model and quantization calibration file set; The transcoding module is configured to transcode the quantization calibration file set into a readable file; The generation module is configured to generate a model from the platform based on the ONNX model and the readable file. The deployment module is configured to deploy the models generated by the platform. The ONNX models and quantization calibration files for each module that are being modified include: Remove redundant output nodes from the ONNX model of each module; Set the same tensor names to be consistent in the ONNX models of each module; Modify the ONNX model of each module according to the requirements of the deployment code; The merging of the ONNX models and quantization calibration files from each module to obtain the first set of ONNX models and quantization calibration files includes: The modified ONNX models of each module are merged into one to obtain the first ONNX model; The quantization calibration files of each module are merged into one to obtain the quantization calibration file set.

7. An electronic device, characterized in that, The electronic device includes: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the model deployment method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the computer's processor, causes the computer to perform the model deployment method according to any one of claims 1 to 5.

Images