Model conversion method and device, electronic equipment and storage medium

Abstract

The invention provides a model conversion method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining the model information of a pre-converted intermediate model, generating a model structure description file according to header information, constructing a model conversion frame through the model structure description file, inputting constant information into the model conversion frame, and converting the constant information into a model conversion frame; determining an operator structure and an operator type in the computational graph information, mapping the operator structure into a target operator structure according to a preset operator mapping table and the operator type, and filling the model conversion framework with the target operator structure to obtain a target model. According to the embodiment of the invention, the operator mapping relation is utilized, accurate conversion from the Onnx model to the PyTorch model is realized, the readability of the converted model is ensured, and the deployment requirements of different types of front-end framework models on rear-end equipment are met.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model conversion method, device, electronic equipment and storage medium. Background Art

[0002] With the development of deep learning in recent years, many different formats of model representation methods have been generated, such as Caffe, Pytorch, Onnx, Tensorflow, etc. In actual engineering applications, these frameworks are called front-end frameworks. These front-end frameworks are mainly used by algorithm engineers, and the models they generate are not the final optimal performance models. If the model reasoning needs to perform at its best, the model needs to be run on a specified deep learning chip, which includes GPU, TPU, NPU, etc. This is the deployment of the model.

[0003] In the process of deploying deep learning models, different backend frameworks support different input model formats. Currently, most backend frameworks support frameworks such as Onnx, Pytorch, and Tensorflow. Onnx plays the role of the IR intermediate representation of the model. At present, the Onnx model is converted into the Pytorch python open source library. Since the library cannot generate the py file (source code file of the Python programming language) describing the model, the model cannot be debugged. After the Onnx operator conversion, the operator becomes more fragmented, resulting in poor readability of the converted model and failure to meet the model deployment requirements. Summary of the invention

[0004] In view of this, the present invention aims to propose a model conversion method, device, electronic device and storage medium to solve the problem that the model cannot be debugged during the current open source library conversion and the readability of the converted model is poor, which cannot meet the model deployment requirements.

[0005] According to a first aspect of the present invention, a model conversion method is provided, the method comprising:

[0006] Obtaining model information of the pre-converted intermediate model; wherein the model information includes header information, constant information, and computational graph information;

[0007] Generate a model structure description file according to the header information, construct a model conversion framework using the model structure description file, and input the constant information into the model conversion framework;

[0008] Determine the operator structure and operator type in the computation graph information, and map the operator structure to a target operator structure according to a preset operator mapping table and operator type;

[0009] The target operator structure is filled into the model conversion framework to obtain a target model.

[0010] Optionally, the obtaining model information of the pre-converted intermediate model includes:

[0011] Convert the initial model to be converted into an intermediate model through an intermediate file in advance;

[0012] The intermediate model is parsed to obtain header information, constant information, and computational graph information of the intermediate model.

[0013] Optionally, generating a model structure description file according to the header information, building a model conversion framework using the model structure description file, and inputting the constant information into the model conversion framework includes:

[0014] Generate a model structure description file according to the header information; the model structure description file is used to determine the import information;

[0015] Obtain a pre-created framework subclass, import the header information in the model structure description file into the framework subclass, and construct a model conversion framework;

[0016] The input and output data and data types in the constant information are input into the model conversion framework.

[0017] Optionally, determining the operator structure and the operator type in the computation graph information, and mapping the operator structure to a target operator structure according to a preset operator mapping table and the operator type, includes:

[0018] Traversing the computational graph information to determine an operator structure and an operator type; wherein the operator structure includes a subgraph and a single operator, and the subgraph is composed of at least two operators;

[0019] Determine a mapping mode of the operator structure according to the operator type and a preset operator mapping table; the mapping mode includes any one of operator to operator, operator to multiple operators, operator to subgraph, and subgraph to operator;

[0020] The operator structure is mapped to a target operator structure according to the mapping method.

[0021] Optionally, mapping the operator structure to a target operator structure according to the mapping manner includes:

[0022] Determining a preset mapping order of the operator structure;

[0023] The preset mapping order is used to perform mapping corresponding to the mapping method on the operator structure to obtain a target operator structure of the target model.

[0024] Optionally, filling the target operator structure into the model conversion framework to obtain the target model includes:

[0025] Obtaining the weight of the intermediate model;

[0026] The target operator structure is filled into the model conversion framework, and the weights of the intermediate model are converted into target model weights to obtain a target model.

[0027] Optionally, after filling the target operator structure into the model conversion framework to obtain the target model, the step further includes:

[0028] Inputting preset test samples into the intermediate model and the target model respectively, and obtaining the inference result of the intermediate model and the inference result of the target model;

[0029] The inference result of the intermediate model is compared with the inference result of the target model to obtain a model test result, and the target model is fine-tuned according to the model test result.

[0030] According to a second aspect of the present invention, there is provided a model conversion device, the device comprising:

[0031] An information acquisition module, used to acquire model information of the pre-converted intermediate model; wherein the model information includes header information, constant information and calculation graph information;

[0032] A conversion framework module, used to generate a model structure description file according to the header information, construct a model conversion framework using the model structure description file, and input the constant information into the model conversion framework;

[0033] An operator mapping module, used to determine the operator structure and operator type in the computation graph information, and map the operator structure to a target operator structure according to a preset operator mapping table and operator type;

[0034] The model conversion model is used to fill the target operator structure into the model conversion framework to obtain the target model.

[0035] According to another aspect of the present invention, there is also provided an electronic device, comprising:

[0036] processor;

[0037] a memory for storing instructions executable by the processor;

[0038] The processor is configured to execute the instructions to implement the model conversion method as described above.

[0039] According to another aspect of the present invention, a readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the model conversion method described above are implemented.

[0040] The model conversion method provided by the embodiment of the present invention obtains the model information of the pre-converted intermediate model, generates a model structure description file according to the header information, constructs a model conversion framework using the model structure description file, inputs the constant information into the model conversion framework, determines the operator structure and operator type in the calculation graph information, maps the operator structure to the target operator structure according to the preset operator mapping table and the operator type, fills the target operator structure into the model conversion framework, and obtains the target model. The embodiment of the present invention utilizes the operator mapping relationship to solve the ambiguity phenomenon in the operator translation, ensures the integrity of the operator translation, solves the poor performance problem caused by the operator being too fragmented, ensures the readability of the converted model, realizes the accurate conversion of the Onnx model to the PyTorch model, and can complete the automatic model conversion through accurate operator mapping, meeting the deployment requirements of different types of front-end framework models on the back-end devices.

[0041] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0043] Figure 1 is a flow chart of steps of a model conversion method provided by an embodiment of the present invention;

[0044] Figure 2 yes Figure 1 A flowchart of step 101 in the model conversion method provided in an embodiment of the present invention;

[0045] Figure 3 yes Figure 1 A flowchart of step 102 in the model conversion method provided in an embodiment of the present invention;

[0046] Figure 4 yes Figure 1 A flowchart of step 103 in the model conversion method provided in an embodiment of the present invention;

[0047] Figure 5 yes Figure 1 A flowchart of step 104 in the model conversion method provided in an embodiment of the present invention;

[0048] Figure 6 is a schematic diagram of a scenario architecture of a model conversion method provided by an embodiment of the present invention;

[0049] Figure 7 is a flowchart of another model conversion method provided by an embodiment of the present invention;

[0050] Figure 8 is a structural schematic diagram of a model conversion device provided by an embodiment of the present invention;

[0051] Fig. 9 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] To make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the embodiments of the present invention, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can be implemented. The division of the following embodiments is for the convenience of description, and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined and referenced with each other without contradiction.

[0053] Reference Figure 1 , shows a flow chart of the steps of a model conversion method provided by an embodiment of the present invention, the method may include:

[0054] Step 101, obtaining model information of the pre-converted intermediate model; wherein the model information includes header information, constant information and computational graph information.

[0055] In an embodiment of the present invention, in order to solve the problem that the model cannot be debugged during the conversion of the current open source library and the readability of the converted model is poor, and the model deployment requirements cannot be met, this embodiment uses Onnx as the middleware, first exports the model of other platforms into an onnx model, and then converts the onnx model into a Pytorch model. The operator classification mapping relationship is used to automatically convert the models of other frameworks into Pytorch models, which facilitates the use of powerful dynamic graphs, automatic differentiation and other functions in Pytorch to improve the performance of the model, and realizes the accurate conversion of the Onnx model to the PyTorch model, ensuring that the target model has the same performance as the intermediate model during reasoning, and improving the accuracy of the model conversion.

[0056] It should be noted that Onnx (Open Neural Network Exchange) is an open model format standard that aims to achieve model interoperability between different deep learning frameworks. Onnx defines a set of standard deep learning network operation methods. Other frameworks can unify the model formats under different frameworks by exporting them to Onnx. Pytorch is an open source machine learning library, mainly used for model training, which contains concepts such as Tensor and Module.

[0057] Specifically, the model information of the pre-converted intermediate model is obtained; wherein the model information includes header information, constant information and computational graph information, the header information includes the IR version, opset version, input and output name, input and output data type, input and output shape, etc. of the model, the constant information includes constant data such as weights and biases of the model, and the computational graph information includes the computational graph structure of the model, that is, the topological structure of the model and the operator type of each node.

[0058] Reference Figure 6 , showing a schematic diagram of the scenario architecture of the model conversion method provided by an embodiment of the present invention. The hardware setting of this embodiment includes a Linux host, and the software setting includes Onnx, onnxruntime, python, numpy, pytorch environment, etc. It is divided into three levels: conversion layer, mapping table layer, and system layer. The system layer represents the basic environment required for model conversion; the mapping table layer is an artificially established operator mapping relationship, which includes four categories: operator to operator, operator to multiple operators, operator to subgraph, and subgraph to operator; the conversion layer is a specific conversion code and implementation, including the most basic construction of the model, the logic of computational graph translation, and the accuracy verification link.

[0059] Step 102, generating a model structure description file according to the header information, constructing a model conversion framework using the model structure description file, and inputting constant information into the model conversion framework.

[0060] In an embodiment of the present invention, by generating a model structure description file according to the header information and using the file to build a model conversion framework, constant information can be input into the model conversion framework to complete the initialization of the model, and a model structure description file (such as model.py) is generated according to the header information. The file is used to define the basic structure and import information of the model. The model structure description file contains the input and output information, data type, shape, etc. of the model. This information is used to build the basic framework of the PyTorch model.

[0061] Specifically, the framework subclass is initialized using the header information in the model structure description file. When building the model conversion framework, the framework subclass will serve as the basic framework of the PyTorch model for subsequent operator mapping and model conversion. The constant information is converted into the Tensor format of PyTorch and filled into the model conversion framework. The input and output data and data types will be used to define the input and output interfaces of the model to ensure that the model conversion framework can correctly process the input and output data.

[0062] Step 103, determine the operator structure and operator type in the computation graph information, and map the operator structure to the target operator structure according to the preset operator mapping table and operator type.

[0063] In an embodiment of the present invention, the computational graph information is traversed to determine the operator structure and operator type of each node in the model. The operator structure includes a subgraph and a single operator, and the subgraph is composed of at least two operators. The operator is the basic computing unit in the computational graph, representing a specific mathematical operation or operation. The operator usually corresponds to a layer or operation in a neural network, such as convolution, activation function, pooling, normalization, etc. Each operator receives input data and outputs a result after calculation. A subgraph is a computing subnetwork composed of multiple operators. This operation cannot be completed by a single operator, but requires a combination of multiple operators. The subgraph is composed of at least two operators, which are connected according to a certain topological structure to form a complete computing logic.

[0064] Specifically, according to the operator type, the preset operator mapping table is queried to determine the mapping method of the operator structure. For different operator types, a suitable mapping method is selected to ensure that the operator structure can be correctly mapped to the target operator structure. The operator mapping table includes operator comparison relationships corresponding to different types of models performing the same operation, which can be obtained based on a large number of model configurations. Through the preset operator mapping table, a suitable mapping method can be selected according to different operator types, thereby enhancing the flexibility of model conversion. For different versions of Onnx operators, the operator mapping table can be adjusted to achieve compatibility conversion, thereby solving the compatibility problem between different opset versions. This embodiment does not make specific limitations on this.

[0065] Step 104, fill the target operator structure into the model conversion framework to obtain the target model. The model conversion method provided by the embodiment of the present invention obtains the model information of the pre-converted intermediate model, generates a model structure description file according to the header information, constructs a model conversion framework using the model structure description file, inputs the constant information into the model conversion framework, determines the operator structure and operator type in the calculation graph information, maps the operator structure to the target operator structure according to the preset operator mapping table and operator type, fills the target operator structure into the model conversion framework, and obtains the target model. The embodiment of the present invention utilizes the operator mapping relationship to solve the ambiguity in operator translation, ensure the integrity of operator translation, solve the poor performance problem caused by too fragmented operators, increase the readability of the model, and realize the accurate conversion of the Onnx model to the PyTorch model. The automatic model conversion can be completed through accurate operator mapping, meeting the deployment requirements of different types of front-end framework models on back-end devices.

[0066] Further, see Figure 2 , showing Figure 1 A flowchart of step 101 in a model conversion method is provided. The method is substantially the same as the model conversion method provided in the first embodiment of the present invention. Step 101 may include:

[0067] Step 1011, converting the initial model to be converted into an intermediate model through an intermediate file in advance.

[0068] Step 1012, parse the intermediate model to obtain header information, constant information, and computational graph information of the intermediate model.

[0069] It should be noted that in the embodiments of the present invention, in the process of deploying deep learning models, the format conversion of the model is a key link. Since different deep learning frameworks generate different model formats, and the back-end hardware (such as GPU, TPU, NPU, etc.) supports different model formats, it is necessary to convert the model from one format to another. In this embodiment, the initial model to be converted (such as TensorFlow's h5 model) is converted into an intermediate model through an intermediate file (such as Onnx). The format of the intermediate model is usually Onnx, because Onnx, as an intermediate representation format, is compatible with a variety of front-end frameworks (such as TensorFlow, PyTorch, Caffe, etc.) and back-end hardware (such as GPU, TPU, NPU, etc.)

[0070] Specifically, the initial model, such as the h5 model, can be converted into the Onnx model first. During the conversion process, the calculation graph and weight information of the initial model will be mapped to the calculation graph and weight information of Onnx. The converted intermediate model (Onnx model) is parsed to extract the model's header information, constant information and calculation graph information. The header information includes the model's IR version, opset version, input and output names, input and output data types, input and output shapes, etc. The constant information includes the model's weights, biases and other constant data. The calculation graph information includes the model's calculation graph structure, that is, the model's topological structure and the operator type of each node.

[0071] The embodiment of the present invention solves the problem of incompatibility of model formats between different frameworks by converting the initial model into an intermediate model and using Onnx as an intermediate representation format that is compatible with a variety of front-end frameworks and back-end hardware. The parsing and conversion steps of the intermediate model can be completed automatically, which significantly improves the efficiency of model conversion. Through the parsing of the intermediate model, the model can be further optimized and adjusted according to different needs, which simplifies the model conversion process and improves the flexibility of model conversion.

[0072] Further, see Figure 3 , showing Figure 1 A flowchart of step 102 in a model conversion method is provided. The method is substantially the same as the model conversion method provided in the first embodiment of the present invention. Step 102 may include:

[0073] Step 1021, generating a model structure description file according to the header information; the model structure description file is used to determine the import information.

[0074] Step 1022, obtain the pre-created framework subclass, import the header information in the model structure description file into the framework subclass, and build a model conversion framework.

[0075] Step 1023, input the input and output data and data types in the constant information into the model conversion framework.

[0076] It should be noted that, in an embodiment of the present invention, a model structure description file (such as model.py) is generated based on the header information. The file is used to determine the import information of the model. The model structure description file contains the input and output information, data type, shape, etc. of the model. This information is used to build the basic framework of the PyTorch model.

[0077] Specifically, the input and output definitions of the PyTorch model are generated according to the input and output names, data types, and shapes in the header information. The model structure description file also contains the imported library information for use in the subsequent construction of the model conversion framework. Get the pre-created framework subclass (such as the torch.nn.Module subclass of PyTorch), import the header information in the model structure description file into the framework subclass, build the model conversion framework, use the header information in the model structure description file to initialize the framework subclass, and when building the model conversion framework, the framework subclass will serve as the basic framework of the PyTorch model for subsequent operator mapping and model conversion. The input and output data and data types in the constant information are input into the model conversion framework. The constant information includes constant data such as the weight and bias of the model. These data will be used as the initialization parameters of the model. The constant information is converted into the Tensor format of PyTorch and filled into the model conversion framework. The input and output data and data types will be used to define the input and output interfaces of the model to ensure that the model conversion framework can correctly process the input and output data.

[0078] The embodiment of the present invention can flexibly adjust the input and output definitions and import information of the model according to different model requirements by generating a model structure description file. The model structure description file facilitates subsequent model conversion and debugging, enhances the maintainability of the model conversion framework, and significantly improves the efficiency and maintainability of the model conversion.

[0079] Further, see Figure 4 , showing Figure 1 A flowchart of step 103 in a model conversion method is provided. The method is substantially the same as the model conversion method provided in the first embodiment of the present invention. Step 103 may include:

[0080] Step 1031, traverse the computation graph information to determine the operator structure and operator type; wherein the operator structure includes a subgraph and a single operator, and the subgraph consists of at least two operators.

[0081] Step 1032, determining a mapping mode of the operator structure according to the operator type and a preset operator mapping table; the mapping mode includes any one of operator to operator, operator to multiple operators, operator to subgraph, and subgraph to operator.

[0082] Step 1031, mapping the operator structure to the target operator structure according to the mapping method.

[0083] It should be noted that, in an embodiment of the present invention, the computational graph information is traversed to determine the operator structure and operator type of each node. The operator structure includes a subgraph and a single operator. The subgraph consists of at least two operators. For a single operator, its operator type is directly obtained; for a subgraph, it is necessary to further parse the operator structure in the subgraph.

[0084] Specifically, the mapping method of the operator structure is determined according to the operator type and the preset operator mapping table. The mapping methods include the following four: operator to operator: Onnx operators are directly mapped to PyTorch operators; operator to multiple operators: Onnx operators are mapped to multiple PyTorch operators; operator to subgraph: Onnx operators are mapped to a PyTorch subgraph; subgraph to operator: Onnx subgraphs are mapped to a PyTorch operator.

[0085] This embodiment queries the preset operator mapping table according to the operator type, determines the mapping method of the operator structure, selects the appropriate mapping method for different operator types, ensures that the operator structure can be correctly mapped to the target operator structure, and converts the Onnx operator or subgraph into a PyTorch operator or subgraph according to the mapping method, fills the converted target operator structure into the model conversion framework, and completes the mapping of the operator structure.

[0086] It should be noted that the operator mapping table is generated based on the operation configuration between a large number of models. For different versions of Onnx operators, compatibility conversion can be achieved by adjusting the operator mapping table, which solves the compatibility problem between different opset versions. In this embodiment, operator mapping can be divided into four categories: operator to operator, operator to multiple operators, operator to subgraph, and subgraph to operator. Among them, operator to operator is a directly translatable operator, which means that the Onnx operator has the same function in Pytorch, which can be directly translated. A one-to-one mapping table can be established to associate operators, such as the addition operation of two Tensors, returning the maximum index in a Tensor, performing convolution operations on a Tensor, performing relu activation operations on a Tensor, and other operators. Operator-to-multi-operator refers to a basic operator in Onnx. There are many forms of implementation in Pytorch. The specific implementation to use needs to be judged according to the context, such as: implementing two-dimensional data normalization, implementing one-dimensional data normalization, flattening Tensor, transposing Tensor, etc. In Onnx operators, there is a type of operator that requires a series of operator combinations in Pytorch to achieve the same function. This series of Pytorch operators are combined into a subgraph, which constitutes the operator mapping relationship of the operator to the subgraph, such as: implementing HardSwish activation, implementing integer convolution, and other operations. In the onnx operator, the exported Onnx operator is fragmented due to version and other reasons. When converting to Pytorch, it needs to be converted into a complete operator. On the one hand, it can maintain the semantics of the model. On the other hand, a complete single operator has better performance than fragmented operators, which constitutes the operator mapping relationship between the subgraph and the operator. For example, there is no GELU implementation below opset20 in onnx. This type of operator requires subgraph matching of the Onnx calculation graph to find the subgraph object that needs to be converted.

[0087] The embodiment of the present invention traverses the computational graph information, determines the operator structure and operator type, and maps the operator structure to the target operator structure according to a preset operator mapping table, thereby ensuring the accuracy of the model conversion. Through a clear mapping method, ambiguity and errors in operator conversion are avoided. The appropriate mapping method can be selected according to different operator types, thereby significantly improving the efficiency of the model conversion.

[0088] Specifically, mapping the operator structure to a target operator structure according to the mapping method includes:

[0089] First, determine the preset mapping order of the operator structure.

[0090] Secondly, a preset mapping order is used to perform mapping corresponding to the mapping method on the operator structure to obtain a target operator structure of the target model.

[0091] It should be noted that in the above steps, in the conversion process of the deep learning model, the mapping of the operator structure is one of the key steps. By determining the preset mapping order of the operator structure and mapping the operator structure according to the preset order, the target operator structure of the target model is obtained, which can ensure that the operator structure in the target model is consistent with the operator structure in the intermediate model.

[0092] It should be noted that the preset mapping order of the operator structure is determined. The mapping order determines the construction order of the operator structure in the target model. The mapping order should be determined based on the dependency relationship between the operators and the topological structure of the computational graph to ensure that the operator structure in the target model is consistent with the operator structure in the intermediate model. The mapping order of the operator structure is determined specifically according to the dependency relationship. Operators with fewer dependencies are mapped first, and operators with more dependencies are mapped later. The mapping order in this embodiment can be set according to actual conversion requirements and is not specifically limited here.

[0093] Further, see Figure 5 , showing Figure 1 A flowchart of step 104 in a model conversion method is provided. The method is substantially the same as the model conversion method provided in the first embodiment of the present invention. Step 104 may include:

[0094] Step 1041, obtaining the weight of the intermediate model;

[0095] Step 1042, fill the target operator structure into the model conversion framework, and convert the weights of the intermediate model into the target model weights to obtain the target model.

[0096] It should be noted that in an embodiment of the present invention, the weight information of the intermediate model (Onnx model) is obtained, and the weight information includes parameters such as the model's weight and bias. These parameters will be used to initialize the target model. Specifically, the weight information of the intermediate model is obtained, and the weight information is converted into the Tensor format of PyTorch for subsequent filling into the target model. The target operator structure is filled into the model conversion framework, the basic structure of the target model is constructed, the weight of the intermediate model is converted into the weight of the target model, and filled into the target model, and finally the target model is obtained. The model can be directly used for reasoning or deployed to the back-end device.

[0097] Specifically, fill the target operator structure into the init function of the model conversion framework, build the basic structure of the target model, convert the weights of the intermediate model into the Tensor format of PyTorch, and fill them into the corresponding layers of the target model to ensure that the weights of the target model are consistent with those of the intermediate model to ensure the performance of the model during inference.

[0098] The embodiment of the present invention fills the target operator structure into the model conversion framework and automatically fills in the weights. The generated target model has a clear structure, realizes the accurate conversion of the Onnx model to the PyTorch model, ensures that the target model has the same performance as the intermediate model during reasoning, and improves the efficiency of model conversion.

[0099] Reference Figure 7 , shows a flow chart of steps of another model conversion method provided by an embodiment of the present invention, which method is basically the same as the model conversion method provided by the first embodiment of the present invention, except that the method may further include:

[0100] Step 101, obtaining model information of the pre-converted intermediate model; wherein the model information includes header information, constant information and computational graph information.

[0101] Step 102, generating a model structure description file according to the header information, constructing a model conversion framework using the model structure description file, and inputting constant information into the model conversion framework.

[0102] Step 103, determine the operator structure and operator type in the computation graph information, and map the operator structure to the target operator structure according to the preset operator mapping table and operator type.

[0103] Step 104, fill the target operator structure into the model conversion framework to obtain the target model.

[0104] The above steps 101 to 10,4 are described in the foregoing description and will not be repeated here.

[0105] Step 105, inputting the preset test samples into the intermediate model and the target model respectively, and obtaining the inference results of the intermediate model and the target model;

[0106] Step 106, compare the inference result of the intermediate model with the inference result of the target model to obtain a model test result, and fine-tune the target model according to the model test result.

[0107] In the embodiment of the present invention, a preset test sample is prepared, and the test sample should cover the input range of the model to ensure the comprehensiveness of the inference result. The test sample is input into the intermediate model (Onnx model) and the target model (PyTorch model) respectively to obtain the inference results of both. Specifically, the intermediate model is loaded using Onnx Runtime and inference is performed, and the target model is loaded using PyTorch and inference is performed, and the inference results of the intermediate model and the target model are compared.

[0108] Specifically, the inference results of the intermediate model are compared with the inference results of the target model, and the difference between the two is calculated. According to the comparison results, the accuracy of the target model is judged. If there is a large difference in the inference results, the target model needs to be fine-tuned to make its inference results consistent with those of the intermediate model. In this embodiment, the mean square error or other similarity measurement methods can be used to calculate the difference between the inference results of the intermediate model and the target model. If the difference is large, fine-tuning can be performed by adjusting the model weights, optimizer parameters, etc., and the inference results are verified again to ensure that the inference results of the target model are consistent with the intermediate model.

[0109] Compared with the prior art, the embodiments of the present invention, on the basis of achieving the beneficial effects brought by the first embodiment, respectively input preset test samples into the intermediate model and the target model, and compare the reasoning results of the two, so as to timely discover the differences between the target model and the intermediate model, and make fine adjustments to ensure the accuracy of model conversion, improve the reliability of model conversion, and realize the accurate conversion of Onnx model to PyTorch model.

[0110] In order to enable technical personnel in this field to more clearly understand the model conversion method described in the above embodiment, the following takes the conversion of the h5 format model into a Pytorch model through the Onnx intermediate model as an example for explanation: In this embodiment, the detection model is an h5 format model generated using Tensorflow, and there is another Cambrian MLU220 computing card. A detection model needs to be deployed on the MLU220 computing card. Since the tool chain of the MLU220 computing card does not support the h5 model but only supports the Pytorch model, the h5 model needs to be converted into a Pytorch model. This embodiment uses the Onnx intermediate model for conversion.

[0111] Specifically, use tools such as onnxsim to simplify the Onnx model, with the goal of using graph optimization and other techniques to reduce the model's nodes and improve the model's reasoning performance. Read the model information file of the Onnx model, including the supported ModelIR version, the used opset version, the model's input and output shapes, data types, etc., build the Module class of the Pytorch model, traverse the Onnx computational graph, match the convertible subgraphs, convert them into pytorch operators, add them to the Init function of the Pytorch model class, and find a pair of operators in the remaining subgraphs. After conversion, add them to the Init function of the Pytorch model class, find one-to-many operators, and convert them into Pytorch operators based on the context. Operator, add it to the Init function of the Pytorch model class, and finally add the directly translatable operator to the forward function of the Pytorch model class, and associate the nodes in Init according to the topological structure, convert the Onnx weights to Pytorch, test and obtain the Onnx inference results and Pytorch inference results and compare them for accuracy verification. After the verification is passed, the Pytorch model is deployed to the MLU device for MLU device reasoning, so as to realize the automatic conversion of the h5 model to the Pytorch model that can be configured on the MLU device.

[0112] Reference Figure 8 , shows a schematic diagram of the structure of a model conversion device provided by an embodiment of the present invention, the device comprising:

[0113] An information acquisition module 201 is used to acquire model information of a pre-converted intermediate model; wherein the model information includes header information, constant information, and computational graph information;

[0114] A conversion framework module 202, used to generate a model structure description file according to the header information, construct a model conversion framework using the model structure description file, and input the constant information into the model conversion framework;

[0115] An operator mapping module 203 is used to determine the operator structure and operator type in the computation graph information, and map the operator structure to a target operator structure according to a preset operator mapping table and operator type;

[0116] The model conversion model 204 is used to fill the target operator structure into the model conversion framework to obtain a target model.

[0117] Furthermore, the information acquisition module 201 includes:

[0118] A pre-conversion submodule, used to convert the initial model to be converted into an intermediate model through an intermediate file in advance;

[0119] The parsing submodule is used to parse the intermediate model to obtain header information, constant information and calculation graph information of the intermediate model.

[0120] Furthermore, the conversion framework module 202 includes:

[0121] A generating submodule, used to generate a model structure description file according to the header information; the model structure description file is used to determine the import information;

[0122] A construction submodule is used to obtain a pre-created framework subclass, import the header information in the model structure description file into the framework subclass, and construct a model conversion framework;

[0123] The input submodule is used to input the input and output data and data types in the constant information into the model conversion framework.

[0124] Furthermore, the operator mapping module 203 includes:

[0125] A traversal submodule, used to traverse the computational graph information to determine the operator structure and operator type; wherein the operator structure includes a subgraph and a single operator, and the subgraph is composed of at least two operators;

[0126] A determination submodule, used to determine a mapping mode of the operator structure according to the operator type and a preset operator mapping table; the mapping mode includes any one of operator to operator, operator to multiple operators, operator to subgraph, and subgraph to operator;

[0127] A mapping submodule is used to map the operator structure into a target operator structure according to the mapping method.

[0128] Furthermore, the mapping submodule includes:

[0129] A determining order unit, used to determine a preset mapping order of the operator structure;

[0130] A mapping unit is used to perform mapping corresponding to the mapping method on the operator structure using the preset mapping order to obtain a target operator structure of the target model.

[0131] Furthermore, the model conversion model 204 includes:

[0132] An acquisition submodule, used to acquire the weight of the intermediate model;

[0133] A filling submodule is used to fill the target operator structure into the model conversion framework, and convert the weight of the intermediate model into the target model weight to obtain the target model.

[0134] Furthermore, the device also includes:

[0135] A model testing module, used to input preset test samples into the intermediate model and the target model respectively, to obtain the inference result of the intermediate model and the inference result of the target model;

[0136] The model fine-tuning module is used to compare the inference result of the intermediate model with the inference result of the target model to obtain a model test result, and fine-tune the target model according to the model test result.

[0137] The model conversion device provided by the embodiment of the present invention obtains the model information of the pre-converted intermediate model, generates a model structure description file according to the header information, constructs a model conversion framework using the model structure description file, inputs the constant information into the model conversion framework, determines the operator structure and operator type in the calculation graph information, maps the operator structure to the target operator structure according to the preset operator mapping table and the operator type, fills the target operator structure into the model conversion framework, and obtains the target model. The embodiment of the present invention uses the operator mapping relationship to solve the ambiguity phenomenon in the operator translation, ensure the integrity of the operator translation, solve the poor performance problem caused by the operator being too fragmented, ensure the readability of the converted model, realize the accurate conversion of the Onnx model to the PyTorch model, and complete the automatic model conversion through accurate operator mapping, meeting the deployment requirements of different types of front-end framework models on the back-end devices.

[0138] Reference Fig. 9 , an embodiment of the present invention further provides an electronic device, such as Fig. 9 As shown, it includes a processor 301, a communication interface 302, a memory 303 and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304.

[0139] A processor 301, a memory 303 for storing processor executable instructions;

[0140] The processor 301 is configured to execute the instructions to implement the model conversion method as described above:

[0141] Obtaining model information of the pre-converted intermediate model; wherein the model information includes header information, constant information, and computational graph information;

[0142] Generate a model structure description file according to the header information, construct a model conversion framework using the model structure description file, and input the constant information into the model conversion framework;

[0143] Determine the operator structure and operator type in the computation graph information, and map the operator structure to a target operator structure according to a preset operator mapping table and operator type;

[0144] The target operator structure is filled into the model conversion framework to obtain a target model.

[0145] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0146] The communication interface is used for communication between the above terminal and other devices.

[0147] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0148] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0149] In another embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the model conversion method described in any of the above embodiments is implemented.

[0150] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0151] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0152] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0153] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A model conversion method, characterized in that: The method comprises: Obtaining model information of the pre-converted intermediate model; wherein the model information includes header information, constant information, and computational graph information; Generate a model structure description file according to the header information, construct a model conversion framework using the model structure description file, and input the constant information into the model conversion framework; Determine the operator structure and operator type in the computation graph information, and map the operator structure to a target operator structure according to a preset operator mapping table and operator type; The target operator structure is filled into the model conversion framework to obtain a target model.

2. The method according to claim 1, characterized in that The obtaining of model information of the pre-converted intermediate model includes: Convert the initial model to be converted into an intermediate model through an intermediate file in advance; The intermediate model is parsed to obtain header information, constant information, and computational graph information of the intermediate model.

3. The method according to claim 1, characterized in that The step of generating a model structure description file according to the header information, constructing a model conversion framework using the model structure description file, and inputting the constant information into the model conversion framework includes: Generate a model structure description file according to the header information; the model structure description file is used to determine the import information; Obtain a pre-created framework subclass, import the header information in the model structure description file into the framework subclass, and construct a model conversion framework; The input and output data and data types in the constant information are input into the model conversion framework.

4. The method according to claim 1, characterized in that: The determining the operator structure and the operator type in the computation graph information, and mapping the operator structure to a target operator structure according to a preset operator mapping table and the operator type, includes: Traversing the computational graph information to determine an operator structure and an operator type; wherein the operator structure includes a subgraph and a single operator, and the subgraph is composed of at least two operators; Determine a mapping mode of the operator structure according to the operator type and a preset operator mapping table; the mapping mode includes any one of operator to operator, operator to multiple operators, operator to subgraph, and subgraph to operator; The operator structure is mapped to a target operator structure according to the mapping method.

5. The method according to claim 4, characterized in that Mapping the operator structure to a target operator structure according to the mapping method includes: Determining a preset mapping order of the operator structure; The preset mapping order is used to perform mapping corresponding to the mapping method on the operator structure to obtain a target operator structure of the target model.

6. The method according to claim 1, characterized in that Filling the target operator structure into the model conversion framework to obtain the target model includes: Obtaining the weight of the intermediate model; The target operator structure is filled into the model conversion framework, and the weights of the intermediate model are converted into target model weights to obtain a target model.

7. The method according to claim 1, characterized in that After the target operator structure is filled into the model conversion framework to obtain the target model, the method further includes: Inputting preset test samples into the intermediate model and the target model respectively, and obtaining the inference result of the intermediate model and the inference result of the target model; The inference result of the intermediate model is compared with the inference result of the target model to obtain a model test result, and the target model is fine-tuned according to the model test result.

8. A model conversion device, characterized in that: The device comprises: An information acquisition module, used to acquire model information of the pre-converted intermediate model; wherein the model information includes header information, constant information and calculation graph information; A conversion framework module, used to generate a model structure description file according to the header information, construct a model conversion framework using the model structure description file, and input the constant information into the model conversion framework; An operator mapping module, used to determine the operator structure and operator type in the computation graph information, and map the operator structure to a target operator structure according to a preset operator mapping table and operator type; The model conversion model is used to fill the target operator structure into the model conversion framework to obtain the target model.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the instructions to implement the model conversion method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the model conversion method according to any one of claims 1 to 7 is implemented.

Images