Model format conversion method and device
By determining and correlating the conversion function to generate a custom intermediate frame model operator in the field of autonomous driving, the problem of cumbersome and inconsistency of model format conversion in the existing technology is solved, and more efficient and more general The model format conversion simplifies the deployment process.
Patent Information
- Application Number
- CN202310604109.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-25
- Publication Date
- 2024-11-26
AI Technical Summary
In the field of autonomous driving, the existing technology has a low degree of automation during the deployment of lidar models, and requires manual construction and analysis of network structures. It is prone to errors and has a large workload, resulting in cumbersome and inconsistent conversion of model formats.
By determining the conversion function between the training framework model and the intermediate framework model and associating it with the operator of the training framework model, a custom intermediate framework model operator is generated to realize model format conversion and avoid being restricted by certain operators , improve conversion efficiency and versatility.
It improves the efficiency and versatility of model format conversion, reduces error rate and workload, realizes more efficient model deployment, and simplifies the problem investigation process.
Smart Images

Figure CN119026660A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of autonomous driving technology, and in particular to a model format conversion method and a device thereof. Background Art
[0002] LiDAR has a high-precision detection function, can describe the surrounding environment with a smaller error, more accurately reflect the surrounding environment, and outputs three-dimensional information, so it is widely used in the current field of autonomous driving.
[0003] Currently, based on deep learning technology, LiDAR technology solutions have made significant progress and development compared to the past. However, after obtaining a model with satisfactory accuracy, how to deploy the model on a vehicle-mounted embedded platform becomes a key step.
[0004] At present, in the laser deployment scheme, most of the methods adopted are that different vehicle deployment platforms mostly have their own set of deployment frameworks. You can use the corresponding interface functions to build the corresponding network structure, and then parse the weights of the training model into the corresponding structure in a certain way, and then convert the deployment framework into the model format required by the vehicle deployment platform. Its automation level is relatively low, and it is necessary to use the deployment platform related APIs to manually build a network model with the same structure as the training framework training, and then assign the network model obtained by the training framework to the network model obtained by the deployment platform through a certain parsing method, and then save it as the model file required by the deployment platform. This method is very flexible to operate, but it is relatively cumbersome and prone to errors. To troubleshoot problems, you need to analyze each network layer one by one, which is a relatively large workload. Summary of the invention
[0005] The purpose of the present invention is to provide a model format conversion method and device thereof to address the defects of the prior art, so that the conversion between the training framework model and the intermediate framework model is no longer limited to some operators and has higher versatility.
[0006] To achieve the above object, the present invention provides a model format conversion method in a first aspect, comprising:
[0007] Determine the conversion function between the training frame model and the intermediate frame model;
[0008] The conversion function is associated with the operator of the training framework model to obtain a customized intermediate framework model operator.
[0009] A second aspect of the present invention provides a model format conversion device, comprising:
[0010] A conversion function determination module, used to determine the conversion function between the training framework model and the intermediate framework model;
[0011] An association module is used to associate the conversion function with the operator of the training framework model to obtain a customized intermediate framework model operator.
[0012] According to a third aspect of the present invention, a computer-readable storage medium is provided, comprising a program or an instruction. When the program or the instruction is executed on a computer, the model format conversion method as described in any one of the first aspects is implemented.
[0013] According to a fourth aspect of the present invention, a computer program product is provided. When the computer program product is run on a computer, the computer is enabled to execute the model format conversion method as described in any one of the first aspects.
[0014] According to a fifth aspect of the present invention, a chip system is provided, comprising a processor, wherein the processor is coupled to a memory, wherein the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the model format conversion method described in any one of the first aspects is implemented.
[0015] A sixth aspect of the present invention provides a computer system, comprising a memory, and one or more processors communicatively connected to the memory;
[0016] The memory stores instructions that can be executed by the one or more processors, and the instructions are executed by the one or more processors so that the one or more processors implement the model format conversion method as described in any one of the first aspects.
[0017] The model format conversion method provided by the embodiment of the present invention, through the setting of the conversion function, makes the format conversion between the training framework model and the intermediate framework model no longer restricted to certain operators, and then through the generation of customized intermediate framework model operators, the model format conversion is more efficient and more universal. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 One of the flow charts of the model format conversion method provided by the embodiment of the present invention;
[0019] Figure 2 The second flow chart of the model format conversion method provided by the embodiment of the present invention;
[0020] Figure 3 A module structure diagram of a model format conversion device provided by an embodiment of the present invention;
[0021] Figure 4 A schematic diagram of the structure of a computer program product provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0022] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0023] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments.
[0024] The embodiment of the present invention provides a model format conversion method, which mainly includes the following steps: Figure 1 The steps shown in:
[0025] Step 110, determining a conversion function between the training framework model and the intermediate framework model.
[0026] Specifically, the training framework model can be a deep learning training framework, such as PyTorch or TensorFlow. The intermediate framework model can be a reasoning framework, such as ONNX. In this example, the technical solution of the present application is described with the training framework model being PyTorch, but this does not mean that the method of the present application is only applicable to the conversion between PyTorch and ONNX.
[0027] In a specific example, the voxelization and scatter operators that are often used in laser network models are not supported by the ONNX framework, so the onnx model containing such operators cannot be obtained smoothly, and then the onnx cannot be converted into the model file corresponding to the target deployment platform. Therefore, it is necessary to determine the conversion function between the training framework and the intermediate framework model. As a result, the corresponding on-board model file cannot be compiled through the corresponding command of the target device deployment platform. At this time, it is necessary to determine the conversion function between the training framework model and the intermediate framework model.
[0028] In this example, the conversion function is specifically a symbolic function, which can be regarded as a static method of the PyTorch operator class. The specific definition of the symbolic function is as follows:
[0029] defsymbolic(g:torch._C.Graph,input_0:torch._C.Value,input_1:torch._C.Value,...)
[0030] Among them, if PyTorch is implemented in C++, then torch._C.Graph and torch._C.Value both correspond to classes in C++. When PyTorch trains the model, it converts the entire network into a computational graph, in which data is calculated and flows, and the parameters in the graph are adjusted according to the error. The first parameter g is a fixed parameter, which represents the content related to the computational graph. Each subsequent parameter represents the input of the operator, which needs to be the same as the input of the operator's forward reasoning interface. The forward reasoning interface can be understood as a function interface included in pytorch but not in onnx.
[0031] By setting the conversion function, the format conversion between the training framework model and the intermediate framework model is no longer restricted by certain operators.
[0032] Before executing step 110, the present application also includes: obtaining an operator of a training framework model.
[0033] Specifically, you can search for the operator name corresponding to the conversion function in the training framework system path, and then obtain the operator corresponding to the operator name through the operator name, that is, obtain the function interface. The function interface is defined by the operator, including the function name (operator name) and parameters.
[0034] Step 120, associate the conversion function with the operator of the training framework model to obtain a customized intermediate framework model operator.
[0035] Specifically, call register_op in the PyTorch API to complete the association.
[0036] register_op('name',symbolic,",9)
[0037] Among them, the first parameter name represents the operator name in PyTorch, the second parameter symbolic represents the composite function, the third parameter represents the domain of the operator, and the fourth parameter represents which operator set version is registered.
[0038] That is to say, the parameter g in the symbolic function corresponds to a method, denoted by op, that is, g has a function method called op, where op is defined as follows:
[0039] defop(name:str,input_0:torch._C.Value,input_1:torch._C.Value,...)
[0040] Among them, the first parameter name represents the operator name in PyTorch.
[0041] When converting the model corresponding to the PyTorch training framework to the model corresponding to the ONNX framework, the symbolic function of each operator in PyTorch will be called in sequence to complete the conversion of operators in PyTorch to operators in ONNX. In other words, just use g.op() to map the operator input in PyTorch to the operator in ONNX one by one, and use the return value of g.op() as the return value of the symbolic function.
[0042] After completing the customized intermediate framework model operator, this application also includes the following Figure 2 The steps shown in:
[0043] Step S1, verifying the intermediate framework model operator.
[0044] The purpose of verification is to check whether there are any errors in the model format conversion process to prevent errors from being carried over to subsequent steps.
[0045] Specifically, you can use PyTorch to run the operator before conversion, and then use the inference engine (such as ONNXRuntime) to run the operator after conversion, that is, the intermediate framework model operator. Compare the results of the two runs. If the two results are consistent, it means that the verification is successful, that is, there is no problem with the conversion process, and you can proceed to step S2. Otherwise, you need to troubleshoot the problem and re-convert the format.
[0046] Step S2: converting the target device format model according to the intermediate framework model operator.
[0047] Specifically, the target device also has model deployment format requirements. The establishment of the intermediate framework model operator is equivalent to building a bridge between the model corresponding to the training framework and the target device format model, which makes the model format conversion more efficient, more universal and lower cost.
[0048] The model format conversion method provided by the embodiment of the present invention, through the setting of the conversion function, makes the format conversion between the training framework model and the intermediate framework model no longer restricted to certain operators, and then through the generation of customized intermediate framework model operators, the model format conversion is more efficient and more universal.
[0049] Embodiment 2
[0050] Figure 3 This is a module structure diagram of a model format conversion device provided in Embodiment 2 of the present invention. The device may be a device capable of implementing the method provided in Embodiment 1 of the present application, such as a central processing unit or server of an unmanned vehicle. The model format conversion device includes:
[0051] A conversion function determination module 10, used to determine a conversion function between a training framework model and an intermediate framework model;
[0052] The association module 20 is used to associate the conversion function with the operator of the training framework model to obtain a customized intermediate framework model operator.
[0053] The second embodiment of the present invention provides a model format conversion device, which can execute the method steps in the above method embodiment, the conversion function determination module 10 implements step 110, and the association module 20 implements step 120. The specific implementation principle and technical effect are similar and will not be repeated here.
[0054] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the conversion function determination module can be a separately established processing element, or it can be integrated in a chip of the above device for implementation. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in a processor element or instructions in the form of software.
[0055] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).
[0056] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The above computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The above-mentioned computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the above-mentioned computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, Bluetooth, microwave, etc.) methods. The above-mentioned computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The above-mentioned available medium can be a magnetic medium (such as a floppy disk, a hard disk, a tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.
[0057] Embodiment 3
[0058] Embodiment 3 of the present invention provides a computer-readable storage medium, including a program or an instruction. When the program or the instruction is executed on a computer, the model format conversion method as described in any one of the above embodiments 1 is implemented.
[0059] Embodiment 4
[0060] In some optional embodiments, any of the model format conversion methods provided in Embodiment 1 may be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or products. Figure 4 Schematically illustrates a conceptual partial view of an example computer program product arranged according to at least some of the embodiments presented herein, the example computer program product comprising a computer program for executing a computer process on a computing device. In one embodiment, the example computer program product is provided using a signal-bearing medium. The signal-bearing medium may include one or more program instructions, which when executed by one or more processors may provide any of the model format conversion methods provided in the first embodiment above to obtain an intermediate framework model operator. For example, Figure 1One or more features of steps 110 to 120 shown may be undertaken by one or more instructions associated with a signal-bearing medium. In some examples, the signal-bearing medium may include a computer-readable medium, such as, but not limited to, a hard drive, a compact disk (CD), a digital video disk (DVD), a digital tape, a memory, a ROM, or a RAM, etc. In some embodiments, the signal-bearing medium may include a computer-recordable medium, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, etc. In some embodiments, the signal-bearing medium may include a communication medium, such as, but not limited to, a digital and / or analog communication medium (e.g., a fiber optic cable, a waveguide, a wired communication link, a wireless communication link, etc.).
[0061] Embodiment 5
[0062] Embodiment 5 of the present invention provides a chip system, including a processor, the processor is coupled to a memory, the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the model format conversion method described in any one of the above embodiments is implemented.
[0063] Embodiment 6
[0064] Embodiment 6 of the present invention provides a computer system, including a memory, and one or more processors communicatively connected to the memory;
[0065] The memory stores instructions that can be executed by the one or more processors, and the instructions are executed by the one or more processors so that the one or more processors implement the model format conversion method as described in any one of the above-mentioned embodiments.
[0066] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0067] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0068] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A model format conversion method, characterized in that: include: Determine the conversion function between the training frame model and the intermediate frame model; The conversion function is associated with the operator of the training framework model to obtain a customized intermediate framework model operator.
2. The model format conversion method according to claim 1, characterized in that: The model conversion function is a sign function.
3. The model format conversion method according to claim 1, characterized in that: The training framework model is a deep learning training framework.
4. The model format conversion method according to claim 1, characterized in that: The intermediate framework model is a reasoning framework.
5. The model format conversion method according to claim 1, characterized in that: After obtaining the customized intermediate framework model operator, the method further includes: Verifying the intermediate framework model operator; When the verification is successful, the target device format model is converted according to the intermediate framework model operator.
6. The model format conversion method according to claim 1, characterized in that: Before determining the conversion function between the training framework model and the intermediate framework model, the method further includes: Get the operator of the training framework model.
7. A model format conversion device, characterized in that: include: A conversion function determination module, used to determine the conversion function between the training framework model and the intermediate framework model; An association module is used to associate the conversion function with the operator of the training framework model to obtain a customized intermediate framework model operator.
8. A computer-readable storage medium, characterized in that: The method comprises a program or an instruction, and when the program or the instruction is executed on a computer, the method for converting the model format as claimed in any one of claims 1 to 6 is implemented.
9. A chip system, characterized in that: The invention comprises a processor, the processor is coupled to a memory, the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the model format conversion method according to any one of claims 1 to 6 is implemented.
10. A computer system, characterized in that: comprising a memory, and one or more processors communicatively connected to the memory; The memory stores instructions that can be executed by the one or more processors, and the instructions are executed by the one or more processors so that the one or more processors implement the model format conversion method as described in any one of claims 1 to 6.
Citation Information
Patent Citations
Deep learning framework conversion method and system, storage medium and equipment
CN113379070A
Cloud deep learning model conversion method, system, device and medium
CN113780536A
