Model architecture conversion method, device, equipment and storage medium

Abstract

The embodiments of the present disclosure disclose a model architecture conversion method, apparatus, device and storage medium, wherein the method comprises: obtaining an initial model of a first model architecture; the initial model comprises a registered custom operator; based on a mapping strategy between a first operator set of the first model architecture and a second operator set of the second model architecture, converting the initial model into an intermediate model under the second model architecture; the intermediate model comprises an intermediate operator corresponding to the custom operator; parsing the intermediate model to obtain a directed acyclic graph corresponding to the intermediate model; and constructing a target model of a third model architecture based on the directed acyclic graph.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of artificial intelligence, and in particular to a model architecture conversion method, device, equipment and storage medium. Background Art

[0002] Deep learning has provided great convenience for daily life. From taking photos to recognizing objects, to face payment and even photography enhancement, all rely on the deep learning technology behind it. With the expansion of the field of deep learning, the industry is increasingly using edge heterogeneous devices to provide artificial intelligence (AI) capabilities, thereby achieving the goal of reducing costs and increasing efficiency. Model deployment has also gradually expanded from running on general-purpose GPGPU graphics processors (General-purpose computing on graphics processing units, referred to as GPGPU) and CPUs to running on mobile GPUs, DSPs, NPUs and even a variety of AI processors. In this process, it is inevitable that the tools used in the field of deep algorithm research, such as PyTorch and TensorFlow, cannot be smoothly and efficiently migrated to the above heterogeneous devices. Summary of the invention

[0003] The embodiments of the present disclosure provide a model architecture conversion method, device, equipment and storage medium.

[0004] In a first aspect, a model architecture conversion method is provided, comprising:

[0005] Acquire an initial model of a first model architecture; the initial model includes a registered custom operator;

[0006] Based on a mapping strategy between a first operator set of the first model architecture and a second operator set of the second model architecture, converting the initial model into an intermediate model under the second model architecture; the intermediate model includes the intermediate operator corresponding to the custom operator;

[0007] Parsing the intermediate model to obtain a directed acyclic graph corresponding to the intermediate model;

[0008] A target model of a third model architecture is constructed based on the directed acyclic graph.

[0009] In some embodiments, the mapping strategy between the first set of operators of the first model architecture and the second set of operators of the second model architecture to convert the initial model into an intermediate model under the second model architecture includes:

[0010] Executing the reasoning process of the initial model to obtain a first calculation graph corresponding to the initial model; the first calculation graph includes at least one first operator in the first operator set and a connection relationship between the at least one first operator;

[0011] Based on a mapping strategy between the operator set of the first model architecture and the operator set of the second model architecture, determining in the second operator set a second operator corresponding to each of the at least one first operator;

[0012] Based on the connection relationship between the at least one first operator, determine the connection relationship between at least one second operator, and construct a second calculation graph corresponding to the intermediate model;

[0013] Based on the second computation graph, the intermediate model is constructed.

[0014] In some embodiments, the method further comprises:

[0015] In response to a registration request for the custom operator, adding the custom operator to the first operator set;

[0016] A mapping relationship between the custom operator and the intermediate operator is added to the mapping strategy to complete the registration process of the custom operator.

[0017] In some embodiments, the operator identifier of the custom operator is the same as the operator identifier of the intermediate operator, and the method further includes:

[0018] Searching for the intermediate operator in the second operator set based on the operator identifier of the custom operator;

[0019] When the second operator set does not include the intermediate operator, the definition of the intermediate operator is added to the second operator set; the semantics of the intermediate operator is the same as that of the user-defined operator.

[0020] In some embodiments, parsing the intermediate model to obtain a directed acyclic graph corresponding to the intermediate model includes:

[0021] Reading the intermediate model to obtain model structure data of the intermediate model;

[0022] Based on the model structure data, obtaining serialized graph data of the intermediate model;

[0023] Based on the serialized graph data, determining the operation node corresponding to the intermediate model and the node information corresponding to each of the operation nodes;

[0024] A directed acyclic graph corresponding to the intermediate model is constructed based on the node information corresponding to each of the operation nodes.

[0025] In some embodiments, parsing the intermediate model to obtain a directed acyclic graph corresponding to the intermediate model includes:

[0026] Parsing the intermediate model to obtain an initial directed acyclic graph corresponding to the intermediate model;

[0027] Based on a preset graph optimization strategy, the initial directed acyclic graph is optimized to obtain a directed acyclic graph corresponding to the intermediate model.

[0028] In some embodiments, the graph optimization strategy includes constant folding, and the initial directed acyclic graph is optimized based on a preset graph optimization strategy to obtain a directed acyclic graph corresponding to the intermediate model, including:

[0029] Acquire at least two consecutive linear change operation nodes in the initial directed acyclic graph;

[0030] The at least two linear change operation nodes are transformed into target operation nodes by constant folding, so as to obtain the directed acyclic graph including the target operation node.

[0031] In some embodiments, the graph optimization strategy includes dead code elimination, and the initial directed acyclic graph is optimized based on a preset graph optimization strategy to obtain a directed acyclic graph corresponding to the intermediate model, including:

[0032] Taking at least one output operation node in the initial directed acyclic graph as a first node, constructing a first list;

[0033] Traversing the first list, moving the current node as the second node to the second list, and adding the parent node of the current node as the first node to the first list until the first list is empty;

[0034] Compare the operation nodes of the second list and the initial directed acyclic graph, delete the invalid nodes in the initial directed acyclic graph, and obtain the directed acyclic graph; the invalid nodes exist in the initial directed acyclic graph and do not exist in the second list.

[0035] In some embodiments, the graph optimization strategy includes operator definition conversion, and the initial directed acyclic graph is optimized based on a preset graph optimization strategy to obtain a directed acyclic graph corresponding to the intermediate model, including:

[0036] Obtaining a preset node conversion table, wherein the node conversion table includes at least one node combination to be converted and a target node combination corresponding to each node combination to be converted; the node combination to be converted and the corresponding target node combination have the same semantics;

[0037] The initial directed acyclic graph is traversed, and the node combinations to be converted in the initial directed acyclic graph are converted into corresponding target node combinations to obtain the directed acyclic graph.

[0038] In some embodiments, the target model of the third model architecture is constructed based on the directed acyclic graph, including:

[0039] Deconstructing the directed acyclic graph to obtain a graph input corresponding to the directed acyclic graph, at least one graph node, and node connection information corresponding to the at least one graph node;

[0040] Converting the graph input into a network input of the third model architecture, and converting each of the at least one graph node into a network layer of the third model architecture;

[0041] Determine a connection relationship between at least one network layer based on node connection information corresponding to the at least one graph node;

[0042] Based on the network input of the third model architecture, the at least one network layer and the connection relationship between the at least one network layer, a target model of the third model architecture is constructed.

[0043] In some embodiments, the method further comprises:

[0044] Inputting test data into the initial model to obtain a first result corresponding to the test data output by the initial model;

[0045] Inputting the test data into the target model to obtain a second result output by the target model;

[0046] When the first result and the second result are the same, the target model is output.

[0047] In a second aspect, a model architecture conversion device is provided, comprising:

[0048] An acquisition module is used to acquire an initial model of the first model architecture; the initial model includes a registered custom operator.

[0049] A conversion module is used to convert the initial model into an intermediate model under the second model architecture based on a mapping strategy between the first operator set of the first model architecture and the second operator set of the second model architecture; the intermediate model includes the intermediate operator corresponding to the custom operator.

[0050] The parsing module is used to parse the intermediate model to obtain a directed acyclic graph corresponding to the intermediate model.

[0051] A construction module is used to construct a target model of a third model architecture based on the directed acyclic graph.

[0052] According to a third aspect, a model architecture conversion device is provided, comprising: a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements the steps of the above method when executing the computer program.

[0053] In a fourth aspect, a computer storage medium is provided, wherein the computer storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the above method.

[0054] In the embodiment of the present disclosure, an initial model of a first model architecture is obtained, wherein the initial model includes registered custom operators; based on a mapping strategy between a first operator set of the first model architecture and a second operator set of the second model architecture, the initial model is converted into an intermediate model under the second model architecture; wherein the intermediate model includes intermediate operators corresponding to the custom operators; the intermediate model is parsed to obtain a directed acyclic graph corresponding to the intermediate model; and a target model of a third model architecture is constructed based on the directed acyclic graph. The conversion of the initial model from the first model architecture to the third model architecture can be realized, which facilitates the user to deploy the initial model under the first model architecture to the platform corresponding to the third model architecture, without being limited to traditional GPGPU and CPU; at the same time, since the present disclosure supports the registration of custom operators, the algorithm deployment requirements can be enriched. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A flowchart of a model architecture conversion method provided in an embodiment of the present disclosure;

[0056] Figure 2 A flowchart of a model architecture conversion method provided in an embodiment of the present disclosure;

[0057] Figure 3 A flowchart of a model architecture conversion method provided in an embodiment of the present disclosure;

[0058] Figure 4 A flowchart of a model architecture conversion method provided in an embodiment of the present disclosure;

[0059] Figure 5 A flowchart of a model architecture conversion method provided in an embodiment of the present disclosure;

[0060] Figure 6 A flowchart of a model architecture conversion method provided in an embodiment of the present disclosure;

[0061] Figure 7 A flowchart of a model architecture conversion method provided by another embodiment of the present disclosure;

[0062] Figure 8 A schematic diagram of the structure of a model architecture conversion device provided in an embodiment of the present disclosure;

[0063] Fig. 9 A hardware entity schematic diagram of a model architecture conversion device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0064] The technical solution of the present disclosure will be described in detail below through embodiments and in conjunction with the accompanying drawings. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0065] It should be noted that in the examples of the present disclosure, "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe the order or precedence of the objects. In addition, the technical solutions recorded in the embodiments of the present disclosure can be arbitrarily combined without conflict.

[0066] Before further describing the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are described. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations.

[0067] 1) Computational graph is a way to describe the computational process using a graph structure. If the computation has obvious modularity and there are obvious temporal and logical dependencies between modules, a directed graph structure can usually be used to describe it. In practical applications, there are two basic elements of a graph structure, namely nodes and directed edges. In practical applications, neural networks are abstracted as directed graph structures composed of tensor data and operators. Nodes are also called operators.

[0068] 2) Model architecture: The deep learning framework is the first layer in the software stack of the artificial intelligence processor, which is used to communicate deep learning applications and various underlying deep learning computing platforms. Deep learning frameworks generally use computational graphs as the main data structure to describe neural network models. On this basis, the operator granularity or the cross-operator granularity is used to complete the mapping of computational graphs to underlying kernel functions. At the same time, deep learning frameworks can use methods including directly using programming languages ​​or calling underlying computing libraries to implement specific kernel functions.

[0069] 3) Open Neural Network Exchange (ONNX) is an open file format designed for machine learning and used to store trained models. It enables different artificial intelligence frameworks (such as PyTorch, MXNet) to store model data in the same format and interact with each other.

[0070] 4) The operator can be a commonly used algorithm in a neural network, such as a convolution operator, a fully connected operator, a pooling operator, or an activation operator, etc. By encapsulating different operators, the corresponding operator library can be obtained.

[0071] The embodiment of the present disclosure provides a model architecture conversion method, which can realize the conversion between different model architectures. The model architecture conversion method provided by the embodiment of the present disclosure is applied to electronic devices.

[0072] The following describes exemplary applications of the electronic device provided by the embodiments of the present disclosure. The electronic device provided by the embodiments of the present disclosure can be implemented as various types of user terminals (hereinafter referred to as terminals) such as laptop computers, tablet computers, desktop computers, set-top boxes, mobile devices (for example, mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), and can also be implemented as servers.

[0073] See also Figure 1 , Figure 1 is an optional flow chart of the model architecture conversion method provided by the embodiment of the present disclosure, which will be combined with Figure 1 The steps shown are explained.

[0074] S101. Obtain an initial model of a first model architecture; the initial model includes a registered custom operator.

[0075] In some embodiments, the initial model is a model weight file obtained through a model training process, and the initial model is obtained by loading the model weight file. The first model architecture can be PyTorch, TensorFlow, Keras, Horovod or PyTorch-on-Angel, etc. Among them, PyTorch is an open source Python machine learning library. TensorFlow is a symbolic mathematical system based on dataflow programming, which is widely used in the programming implementation of various machine learning algorithms. Tensor refers to tensors, and Flow refers to streams. Keras is an open source artificial neural network library written in Python. Horovod is a deep learning tool open sourced by Uber. PyTorch-on-Angel is a PyTorch project running on the Angel platform. Angel is a computing platform developed in Java and Scala languages, a high-performance distributed computing framework for machine learning.

[0076] Among them, the above-mentioned model training process may include obtaining an open source first model architecture, configuring training parameters, and the training parameters may include the number of network layers, gradient update weights, weight decay terms, maximum number of iterations, etc.; inputting sample data into the obtained first model architecture for training, and in each training process, the loss value is calculated based on the loss function, and the minimum value of the loss value is solved by gradient partial derivative operation. Then, the various related parameters are adjusted according to the loss value, and then training is performed again to obtain the minimum value of the loss value. By analogy, training is continuously performed until the minimum value of the loss value obtained after a certain parameter adjustment meets the preset convergence condition, and a converged model can be obtained. The files in the converged model include: network structure definition file, mean file, network parameter weight file, etc. Among them, taking the model as a convolutional neural network model as an example, the above-mentioned network structure definition file is a convolutional neural network structure definition file, and the above-mentioned network parameter weight file is a convolutional neural network parameter weight file.

[0077] In some embodiments, the custom operator has been registered in the first operator set corresponding to the first model architecture. The first operator set may include the original operator corresponding to the first model architecture, and may also include the custom operator developed by the user based on actual needs.

[0078] S102. Based on a mapping strategy between a first operator set of the first model architecture and a second operator set of the second model architecture, convert the initial model into an intermediate model of the second model architecture; the intermediate model includes an intermediate operator corresponding to the custom operator.

[0079] In some embodiments, the first operator set may include multiple first operators corresponding to the first model architecture, the second operator set may include multiple second operators corresponding to the second model architecture, and the mapping strategy between the first operator set and the second operator set may include at least one mapping operator pair, each mapping operator pair includes a first operator and a second operator that has a mapping relationship with the first operator. The mapping relationship between the first operator and the second operator in the mapping operator pair can be established by setting the same operator identifier.

[0080] In this embodiment, the mapping relationship between the user-defined operator and the intermediate operator can be obtained through the mapping strategy, and the intermediate operator can be obtained from the second operator set.

[0081] In some embodiments, the initial model may include at least one first operator in the first operator set, and the at least one first operator includes the above-mentioned custom operator. Based on the mapping strategy between the first operator set and the second operator set, the second operator corresponding to each first operator can be determined, and then an intermediate model of the second model architecture can be established based on the obtained at least one second operator, and the at least one second operator includes the intermediate operator.

[0082] In some embodiments, the second model architecture is ONNX. In this embodiment, after obtaining the ONNX model, the weight information corresponding to the second model can be stored based on the serialized data structure to obtain the corresponding model file. Since the model file is stored based on the serialized data structure, during the process of model architecture conversion, the conversion of multiple model architectures can be supported, that is, based on the currently required third model architecture, the corresponding target model is generated.

[0083] S103: Analyze the intermediate model to obtain a directed acyclic graph corresponding to the intermediate model.

[0084] In some embodiments, the intermediate model can be parsed using a corresponding parsing method based on the data structure of the intermediate model to obtain the node parameters of the existing operation nodes of the intermediate model and the connection relationship between the operation nodes, and establish a corresponding directed acyclic graph based on the connection relationship between the operation nodes.

[0085] S104. Construct a target model of a third model architecture based on the directed acyclic graph.

[0086] In some embodiments, at least one candidate model architecture can be displayed, and a user's selection instruction for a target candidate model architecture among the at least one candidate model architecture can be received, the target candidate model architecture can be used as the third model architecture, and a target model of the third model architecture can be constructed based on the directed acyclic graph.

[0087] Among them, the network layer corresponding to each operation node under the third model architecture can be obtained, and the network layer corresponding to the operation node can be configured based on each operation node and node parameters, and then the target model is constructed based on the connection relationship between the configured network layer and the operation node.

[0088] In the embodiment of the present disclosure, an initial model of a first model architecture is obtained, wherein the initial model includes registered custom operators; based on a mapping strategy between a first operator set of the first model architecture and a second operator set of the second model architecture, the initial model is converted into an intermediate model under the second model architecture; wherein the intermediate model includes intermediate operators corresponding to the custom operators; the intermediate model is parsed to obtain a directed acyclic graph corresponding to the intermediate model; and a target model of a third model architecture is constructed based on the directed acyclic graph. The conversion of the initial model from the first model architecture to the third model architecture can be realized, which facilitates the user to deploy the initial model under the first model architecture to the platform corresponding to the third model architecture, without being limited to traditional GPGPU and CPU; at the same time, since the present disclosure supports the registration of custom operators, the algorithm deployment requirements can be enriched.

[0089] See also Figure 2 , Figure 2 is an optional flow chart of the model architecture conversion method provided by the embodiment of the present disclosure, based on Figure 1 , Figure 1 S102 in can be updated to S201 to S204, which will be combined Figure 2 The steps shown are explained.

[0090] S201. Execute the reasoning process of the initial model to obtain a first calculation graph corresponding to the initial model; the first calculation graph includes at least one first operator in the first operator set and a connection relationship between the at least one first operator.

[0091] In some embodiments, during the process of reasoning the initial model, data flow information of the initial model during the reasoning process can be obtained, and the data flow information includes a connection relationship between at least one first operator and each of the first operators; based on the connection relationship between each of the first operators, the first computational graph is constructed.

[0092] Among them, if the initial model includes operator A and operator B, in the process of reasoning the initial model, the reasoning engine will obtain the resource to be reasoned, and find the operator function corresponding to operator A in the first operator set based on the operator identifier of operator A, and obtain the reasoning sub-result corresponding to the resource to be reasoned based on the operator function corresponding to operator A. If the reasoning engine finds the operator function of operator B in the first operator set based on the operator identifier of operator B, and inputs the reasoning sub-result obtained by operator A into the operator function of operator B, it can be determined that operator B is located before operator A. Correspondingly, in the process of constructing the calculation graph of the initial model, operator B and operator A can be connected, and a pointing relationship from operator A to operator B can be established.

[0093] S202. Based on a mapping strategy between the operator set of the first model architecture and the operator set of the second model architecture, determine in the second operator set a second operator corresponding to each of the at least one first operator.

[0094] In some embodiments, an operator identifier corresponding to each of the first operators may be obtained; and based on the operator identifier corresponding to each of the first operators, a second operator corresponding to each of the first operators may be obtained in the second operator set.

[0095] S203. Based on the connection relationship between the at least one first operator, determine the connection relationship between at least one second operator, and construct a second calculation graph corresponding to the intermediate model.

[0096] Exemplarily, based on the above example, taking the initial model including operator A and operator B as an example, the corresponding first calculation graph is operator A pointing to operator B. After obtaining operator C corresponding to operator A and operator D corresponding to operator B, it can be obtained that the connection relationship between the two second operators is still operator C pointing to operator D, and then the second calculation graph can be obtained.

[0097] S204: construct the intermediate model based on the second calculation graph.

[0098] In some embodiments, the intermediate model can be constructed based on the model construction function corresponding to the second computation graph and the second model architecture. Taking ONNX as an example, the corresponding intermediate model can be obtained based on the "make_model" function and the obtained second computation graph.

[0099] In some embodiments, the intermediate model can also be verified and saved. Taking the second model architecture as ONNX as an example, the intermediate model can be verified based on the "Check_model" function and saved based on the "Save_model" function.

[0100] In the embodiments of the present disclosure, by executing the reasoning process on the initial model, the accuracy of the calculation graph of the initial model can be improved. Afterwards, a mapping strategy between the operator set of the first model architecture and the operator set of the second model architecture can be established, which can effectively convert the first calculation graph under the first model architecture into the second calculation graph under the second model architecture, and then obtain the intermediate model under the second model architecture, thereby realizing the preliminary conversion of the model architecture.

[0101] See also Figure 3 , Figure 3 is an optional flow chart of the model architecture conversion method provided by the embodiment of the present disclosure, based on any of the above embodiments, based on Figure 1 For example, before S101, the method may further include S301 to S304, combining Figure 3 The steps shown are explained.

[0102] S301: In response to a registration request for the custom operator, add the custom operator to the first operator set.

[0103] In some embodiments, before the initial model needs to be converted, the operators in the initial model can be traversed, and each operator can be determined whether it exists in the first operator set. For each operator, it can be determined whether the operator exists in the first operator set based on the operator identifier of the operator. If the operator does not exist in the first operator set, the operator is determined to be a custom operator, and a registration request for the custom operator is generated.

[0104] Exemplarily, taking the first model architecture as PyTorch as an example, the operator file "torch / csrc / autograd / generated / VariableType.h" corresponding to PyTorch can be obtained as the first operator set. If the current operator does not exist in the operator file, the operator is determined to be a custom operator, and a registration request for the custom operator is generated.

[0105] In this embodiment, the process of adding the custom operator to the first operator set can be completed by declaring the operator function corresponding to the custom operator in the function class corresponding to the first model architecture. Taking the first model architecture as PyTorch as an example, the operator function corresponding to the custom operator can be declared in the corresponding PyTorch function class.

[0106] In some embodiments, in order to complete the conversion of the custom operator from the first model framework to the second model framework, not only the custom operator needs to exist in the first operator set, but also the intermediate operator corresponding to the custom operator needs to exist in the second operator set. Therefore, S302 to S303 can be executed to improve the accuracy of model conversion; in other embodiments, if the intermediate operator is the original operator under the second model framework, S302 to S303 can be skipped and S304 can be directly executed to improve the efficiency of model conversion.

[0107] S302: Search for the intermediate operator in the second operator set based on the operator identifier of the user-defined operator.

[0108] The operator identifiers of the user-defined operator and the intermediate operator are the same, so the intermediate operator can be directly searched in the second operator set based on the operator identifier of the user-defined operator. If the intermediate operator exists in the second operator set, S304 can be executed.

[0109] S303: When the second operator set does not include the intermediate operator, add the definition of the intermediate operator to the second operator set; the semantics of the intermediate operator is the same as that of the user-defined operator.

[0110] In some embodiments, since the intermediate operator does not exist in the second operator set, the intermediate operator written by the developer based on the second model architecture can be obtained and the definition of the intermediate operator can be added to the second operator set.

[0111] S304: Add a mapping relationship between the custom operator and the intermediate operator to the mapping strategy to complete the registration process of the custom operator.

[0112] In some embodiments, when the custom operator exists in the first operator set and an intermediate operator exists in the second operator set, the mapping relationship between the custom operator and the intermediate operator needs to be added to the mapping strategy. Only then is the registration process of the custom operator completed.

[0113] Exemplarily, taking the first model architecture as PyTorch and the second model architecture as ONNX as an example, the mapping strategy can be stored in the "torch / ONNX / symbolic.py" file, and the above-mentioned addition of the mapping relationship between the custom operator and the intermediate operator to the mapping strategy can be implemented by the following code: "def XXX(g,input,alpha,inplace=False):return g.op("XXX",input,alpha_f=_scalar(alpha))", where "XXX" is the operator identifier shared by the custom operator and the intermediate operator.

[0114] In the embodiments of the present disclosure, since the registration of custom operators is supported, the application scope of the model conversion method in the present disclosure can be improved, and at the same time, more abundant algorithm deployment requirements can be met.

[0115] See also Figure 4 , Figure 4 is an optional flow chart of the model architecture conversion method provided by the embodiment of the present disclosure, based on any of the above embodiments, based on Figure 1 For example, Figure 1 S103 in can be updated to S401 to S404, which will be combined with Figure 4 The steps shown are explained.

[0116] S401: Read the intermediate model to obtain model structure data of the intermediate model.

[0117] By reading the intermediate model, the model structure data (model structure) of the intermediate model can be obtained. Taking ONNX as an example, the second model architecture can be used to obtain the "modelproto structure", that is, the model structure data of the intermediate model, through the function "ONNX::ModelProto model_data; ReadProtoFromBinaryFile(ONNX,&model_data)".

[0118] S402: Based on the model structure data, obtain serialized graph data of the intermediate model.

[0119] The serialized graph data (graph structure) of the intermediate model can be accessed through the model structure. Taking ONNX as an example, the "graph structure", i.e., the serialized graph data of the intermediate model, can be obtained through the function "ONNX::GraphProto graph=model_data.graph()".

[0120] S403: Based on the serialized graph data, determine the operation nodes corresponding to the intermediate model and the node information corresponding to each of the operation nodes.

[0121] Among them, based on the graph structure, all the operation nodes in the intermediate model, such as the number of operation nodes, can be obtained; the node information corresponding to each of the operation nodes can also be obtained, such as node parameters (for example: stride, dilation, etc. of the conv layer), output parameters and output parameters; the input and output of the intermediate model can also be obtained.

[0122] S404: construct a directed acyclic graph corresponding to the intermediate model based on the node information corresponding to each of the operation nodes.

[0123] Among them, a connection relationship between operation nodes can be established based on the input parameters and output parameters of each operation node, and then a directed acyclic graph corresponding to the intermediate model can be obtained.

[0124] Exemplarily, when the input parameter of operation node A is the first parameter and the output parameter is the second parameter; the input parameter of operation node B is the second parameter and the output parameter is the third parameter, a connection relationship can be established from operation node A to operation node B, and then a corresponding directed acyclic graph can be obtained.

[0125] In the disclosed embodiment, the intermediate model can be parsed and the corresponding directed acyclic graph can be obtained to facilitate the implementation of subsequent graph optimization strategies.

[0126] See also Figure 5 , Figure 5 is an optional flow chart of the model architecture conversion method provided by the embodiment of the present disclosure, based on any of the above embodiments, based on Figure 1 For example, Figure 1 S103 in can also be updated to S501 to S502, which will be combined with Figure 5 The steps shown are explained.

[0127] S501, parsing the intermediate model to obtain an initial directed acyclic graph corresponding to the intermediate model.

[0128] S502: Based on a preset graph optimization strategy, the initial directed acyclic graph is optimized to obtain a directed acyclic graph corresponding to the intermediate model.

[0129] In some embodiments, the preset graph optimization strategy may include at least one optimization sub-strategy with a priority order. The above-mentioned optimization of the initial directed acyclic graph based on the preset graph optimization strategy to obtain the directed acyclic graph corresponding to the intermediate model may include: based on the order of priority of the optimization sub-strategies from high to low, sequentially obtaining the optimization sub-strategies to optimize the initial directed acyclic graph to obtain the directed acyclic graph corresponding to the intermediate model.

[0130] The at least one optimization sub-strategy includes but is not limited to constant folding, dead code elimination, and operator definition conversion. The priority of constant folding is higher than that of dead code elimination, and the priority of dead code elimination is higher than that of operator definition conversion.

[0131] In some embodiments, the graph optimization strategy includes constant folding, and the above-mentioned preset graph optimization strategy can be implemented through S5021 to S5022 to optimize the initial directed acyclic graph to obtain the directed acyclic graph corresponding to the intermediate model.

[0132] S5021. Obtain at least two consecutive linear change operation nodes in the initial directed acyclic graph.

[0133] Among them, the operation nodes in the initial directed acyclic graph can be used to represent the operations performed in the deep learning process. Common operations in deep learning algorithms may include convolution, full connection, activation, pooling, batch normalization, scaling, etc. These operations can be divided into two categories according to their specific implementation forms: linear transformation operations and nonlinear change operations. Among them, all linear transformation operations can be expressed in the form of vector or matrix multiplication and addition. Accordingly, operations that cannot be expressed in the form of vector or matrix multiplication and addition are nonlinear transformation operations. In an example, among the common operations of the deep learning algorithm listed above, convolution, full connection, batch normalization, and scaling operations are linear transformation operations, and pooling and activation are nonlinear transformation operations.

[0134] S5022. Transform the at least two linear change operation nodes into target operation nodes through constant folding to obtain the directed acyclic graph including the target operation node.

[0135] For example, if there is a linear change operation node A out = Conv (x, weight, bias), connected to the linear change operation node A is a linear change operation node B: out = out * scale + b, since scale and b are constants, they can be merged into the weight and bias of conv, reducing one linear calculation, and the target operation node C is out = Conv (x, weight_updated, bias_updated). It can be seen that the semantics before and after constant folding has not changed, and one linear calculation has been reduced.

[0136] In some embodiments, two consecutive linear transformation operation nodes can be simplified into one linear transformation operation node by linear transformation and constant folding. When there are three consecutive linear transformation operation nodes, the first two consecutive linear transformation operation nodes can be simplified into one linear transformation operation node by linear transformation and constant folding. Then, the simplified linear transformation operation node and the remaining linear transformation operation node are again simplified into one linear transformation operation node by linear transformation and constant folding. Similarly, when there are more consecutive linear transformation operation nodes, these linear transformation operation nodes can be merged and simplified into at least one linear transformation operation node by linear transformation and constant folding.

[0137] Through the above embodiments, the model data can be compressed while reducing the amount of calculation, which can reduce the storage overhead of the model data on the one hand, and reduce the amount of memory access during runtime on the other hand.

[0138] In some embodiments, the graph optimization strategy includes dead code elimination, and the above-mentioned preset graph optimization strategy can be implemented through S5023 to S5025 to optimize the initial directed acyclic graph to obtain the directed acyclic graph corresponding to the intermediate model:

[0139] S5023, taking at least one output operation node in the initial directed acyclic graph as a first node, and constructing a first list;

[0140] S5024, traversing the first list, moving the current node as the second node to the second list, and adding the parent node of the current node as the first node to the first list until the first list is empty;

[0141] S5025. Compare the operation nodes of the second list and the initial directed acyclic graph, delete the invalid nodes in the initial directed acyclic graph, and obtain the directed acyclic graph; the invalid nodes exist in the initial directed acyclic graph and do not exist in the second list.

[0142] Among them, the output node list O can be obtained and added to the queue to be searched Q; the queue to be searched Q is traversed in a loop; the current node is added to the queue Visited_Q, indicating that it has been visited; the parent node node_prev of the node node is obtained, and if it is not in the queue Q, it is added to the queue Q; if Q is empty, the loop ends; compare Visited_Q with the node list in the initial directed acyclic graph, delete the nodes that do not exist in Visited_Q from the initial directed acyclic graph, and obtain the directed acyclic graph.

[0143] Through the above embodiments, the operation nodes that have no effect on the model output can be analyzed and deleted, thereby reducing the amount of calculation of the model.

[0144] In some embodiments, the graph optimization strategy includes operator definition conversion, and the above-mentioned preset graph optimization strategy can be implemented through S5026 to S5027 to optimize the initial directed acyclic graph to obtain the directed acyclic graph corresponding to the intermediate model:

[0145] S5026. Obtain a preset node conversion table, wherein the node conversion table includes at least one node combination to be converted and a target node combination corresponding to each node combination to be converted; the node combination to be converted and the corresponding target node combination have the same semantics.

[0146] S5027. Traverse the initial directed acyclic graph, convert the node combinations to be converted in the initial directed acyclic graph into corresponding target node combinations, and obtain the directed acyclic graph.

[0147] In the disclosed embodiment, since the initial directed acyclic graph corresponding to the intermediate model is optimized, the amount of model calculation corresponding to the directed acyclic graph can be reduced, thereby improving the model reasoning efficiency.

[0148] See also Figure 6 , Figure 6 is an optional flow chart of the model architecture conversion method provided by the embodiment of the present disclosure, based on any of the above embodiments, based on Figure 1 For example, Figure 1 S106 in can be updated to S601 to S604, which will be combined with Figure 6 The steps shown are explained.

[0149] S601. Deconstruct the directed acyclic graph to obtain a graph input corresponding to the directed acyclic graph, at least one graph node, and node connection information corresponding to the at least one graph node.

[0150] S602: Convert the graph input into a network input of the third model architecture, and convert each of the at least one graph node into a network layer of the third model architecture.

[0151] S603: Determine a connection relationship between at least one network layer based on the node connection information corresponding to the at least one graph node.

[0152] S604: Construct a target model of the third model architecture based on the network input of the third model architecture, the at least one network layer, and the connection relationship between the at least one network layer.

[0153] Among them, taking caffe as the third model architecture as an example, the graph input can correspond to the network input in the Caffe model format, the graph nodes can correspond to the network layers in the Caffe model format, the graph connections can be converted into layer inputs and outputs in the Caffe model format, and then the obtained Caffe model file can be saved as the target model.

[0154] In some embodiments, in order to improve the accuracy of model architecture conversion, the converted target model can also be verified, and the method also includes: inputting test data into the initial model to obtain a first result corresponding to the test data output by the initial model; inputting the test data into the target model to obtain a second result output by the target model; and when the first result and the second result are the same, outputting the target model.

[0155] Among them, when the first result and the second result are different, the target model of the third model architecture can be reconstructed based on the original directed acyclic graph corresponding to the intermediate model, and the new target model can be verified. If the verification still fails, an alarm message is sent to the developer, and the alarm message includes the conversion log of the target model.

[0156] In the disclosed embodiment, since the converted target model is verified, it can be ensured that the output target model has the same semantics as the initial model, thereby improving the accuracy of model conversion.

[0157] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0158] Deep learning has now provided great convenience for daily life. From taking photos to identifying objects to face payment and even photography enhancement, all rely on the deep learning technology behind it. With the expansion of the field of deep learning, the industry is increasingly using edge heterogeneous devices to provide artificial intelligence (AI) capabilities, thereby achieving the goal of reducing costs and increasing efficiency. Model deployment has also gradually expanded from running on general-purpose GPGPU graphics processors (General-purpose computing on graphics processing units, referred to as GPGPU) and CPUs to running on mobile GPUs, DSPs, NPUs and even various AI processors. In this process, it is bound to face the problem that tools such as PyTorch and TensorFlow used in the field of deep algorithm research cannot be smoothly and efficiently migrated to the above-mentioned heterogeneous devices. Based on this, the disclosed embodiments can solve the problem of model deployment of the PyTorch framework, which occupies a dominant position in academia.

[0159] Among them, the PyTorch framework is a very mature model training framework in the industry, which supports the use of CPU and desktop GPU (NVIDIA GPU, AMD GPU) for model training. Currently, most newly published papers use PyTorch for research. Although PyTorch provides ONNX format and TorchScript export functions, due to the inconsistency of semantics between the two, such export is very limited. Some PyTorch-specific operations or user-defined operations are difficult to export to ONNX smoothly; on the other hand, the industry's support for the open format of ONNX is not complete, usually only supporting a small part, or only supporting the Caffe format.

[0160] In order to clearly illustrate the model deployment method in the embodiment of the present disclosure, the conversion of the PyTorch model to caffe is taken as an example for explanation below. Of course, the embodiment of the present disclosure can also convert the PyTorch model into models of other architectures, such as MXNet, ML.NET, TensorRT, and Microsoft CNTK.

[0161] See also Figure 7 , Figure 7 It is an optional flowchart of the PyTorch model deployment process provided by the embodiment of the present disclosure.

[0162] S701. Obtain the PyTorch model to be deployed.

[0163] Among them, the PyTorch model is a model weight file obtained through the model training process. By loading the model weight file, the corresponding operable object can be obtained.

[0164] Exemplarily, the loading of the PyTorch model can be completed through the "load" function.

[0165] In some embodiments, the model training process may include obtaining an open source PyTorch model framework and configuring training parameters. Among them, the training parameters may include the number of network layers, gradient update weights, weight decay terms, maximum number of iterations, etc. The sample data is input into the obtained PyTorch model framework for training. In each training process, the loss value is calculated based on the loss function, and the minimum value of the loss value is solved by gradient partial derivative operation. Then, the various relevant parameters are adjusted according to the loss value, and then the training is performed again to obtain the minimum value of the loss value. By analogy, the training is continuously performed until the minimum value of the loss value obtained by training after a certain parameter adjustment meets the preset convergence condition, and a converged PyTorch model can be obtained. The files in the converged PyTorch model include: network structure definition file, mean file, network parameter weight file, etc. Taking the PyTorch model as a convolutional neural network model as an example, the network structure definition file is a convolutional neural network structure definition file, and the network parameter weight file is a convolutional neural network parameter weight file.

[0166] In some embodiments, there are at least two implementations of loading a PyTorch model. When the model file includes the model structure and weight parameters, the loading of the PyTorch model can be completed through the "load" function, for example, the loading of the PyTorch model "test.pt" can be completed through the code "torch.load("test.pt")"; when the model file only includes weight parameters, the loading of the PyTorch model can be completed through the "load" function and the "selfmodel.load_state_dict" function, for example, the loading of the PyTorch model "test.pt" can be completed through the code "selfmodel.load_state_dict(torch.load("test.pt"))".

[0167] S702: Register a custom operator and create an operator set.

[0168] In some embodiments, the "torch.ONNX.export" provided by the PyTorch model architecture can complete the conversion process of standard operators, which may include addition operators, multiplication operators, matrix multiplication operators, matrix transposition operators, etc. Accordingly, since the PyTorch model to be deployed has user-defined operators, the custom operators are prone to conversion errors during the deployment process, or the reasoning efficiency of the converted custom operators is low. In order to avoid the above problems, it is necessary to register custom operators for the PyTorch model to be deployed.

[0169] The above standard operators are standardized operators / operators in ONNX. For any standard operator, the standard operator is an operator stored in the operator file corresponding to the ONNX model architecture. At the same time, the standard operator is also an operator stored in the operator file corresponding to the PyTorch model architecture. In some embodiments, the operator file corresponding to the ONNX model architecture can be an "Operators.md" file; the operator file corresponding to the PyTorch model architecture can be a "torch / csrc / autograd / generated / VariableType.h" file.

[0170] In some embodiments, the operator file corresponding to the PyTorch model architecture also includes a custom operator, which does not exist in the operator file corresponding to the ONNX model architecture, and the above-mentioned registration of the custom operator can register the custom operator corresponding to the PyTorch model architecture in the operator file corresponding to the ONNX model architecture. The above-mentioned step of registering the custom operator can be implemented in the following manner: defining the operator identifier and operator function corresponding to the custom operator in the operator file corresponding to the ONNX model architecture, the operator identifier corresponding to the custom operator in the operator file corresponding to the ONNX model architecture is the same as the operator identifier corresponding to the custom operator in the operator file corresponding to the PyTorch model architecture, and the operator semantics corresponding to the custom operator in the operator file corresponding to the ONNX model architecture is the same as the operator semantics corresponding to the custom operator in the operator file corresponding to the PyTorch model architecture.

[0171] S703: Based on the operator set and the preset conversion function, convert the PyTorch model to be deployed into an ONNX model.

[0172] In some embodiments, a dynamic tracking solution can be used to restore and convert the loaded PyTorch model. In particular, a reasoning process can be performed on the loaded PyTorch model to obtain data flow information of the PyTorch model during the reasoning process, and a computational graph of the PyTorch model can be constructed based on the data flow information; and an ONNX model can be constructed based on the computational graph.

[0173] In this embodiment, the operator set includes a first operator set corresponding to PyTorch and a second operator set corresponding to ONNX, wherein the first operator set includes at least one first operator and operator information corresponding to each of the first operators, and the operator information includes at least an operator function and an operator identifier; in some embodiments, the operator information also includes operator description information. Correspondingly, the second operator set includes at least one second operator and operator information corresponding to each of the second operators.

[0174] During the process of reasoning the first model, data flow information of the PyTorch model during the reasoning process can be obtained, and the data flow information includes a connection relationship between at least one first operator and each of the first operators; based on the connection relationship between each of the first operators, a computational graph of the first model is constructed.

[0175] Among them, if the first model includes operator A and operator B, in the process of reasoning the first model, the reasoning engine will obtain the resource to be reasoned, and find the operator function corresponding to operator A in the first operator set based on the operator identifier of operator A, and obtain the reasoning sub-result corresponding to the resource to be reasoned based on the operator function corresponding to operator A. If the reasoning engine finds the operator function of operator B in the first operator set based on the operator identifier of operator B, and inputs the reasoning sub-result obtained by operator A into the operator function of operator B, it can be determined that operator B is located before operator A. Correspondingly, in the process of constructing the calculation graph of the first model, operator B and operator A can be connected, and a pointing relationship from operator A to operator B can be established.

[0176] After obtaining the calculation graph of the first model, the calculation graph corresponding to the first model can be converted into the calculation graph corresponding to the second model based on the mapping relationship between the first operator in the first operator set and the second operator in the second operator set, and the second model can be constructed based on the calculation graph corresponding to the second model.

[0177] S704: parse the ONNX model to obtain a corresponding directed acyclic graph.

[0178] The ONNX model obtained in S703 is loaded into the memory to obtain the model structure data (model structure) corresponding to the ONNX model; based on the model structure data, the serialized graph data (graph) is obtained; based on the serialized graph data, multiple operation nodes of the ONNX model and node information corresponding to each operation node are obtained, and the node information includes node input, node output and node parameters. A directed acyclic graph is constructed based on the multiple operation nodes and the node information corresponding to each operation node.

[0179] S705: Optimize the directed acyclic graph based on a preset optimization strategy to obtain an optimized directed acyclic graph.

[0180] Among them, constant folding, dead code elimination, operator definition transformation (Gemm matrix multiplication (GEMM, General Matrix Multiplication) optimization extraction, HSwish extraction, computational graph equivalent transformation, etc. Pass (concepts in the compiler)) can be used in sequence until the optimization transformation can no longer be continued, and the final graph representation that can be used for direct deployment is obtained.

[0181] S706: Generate a model file in a target format based on the optimized directed acyclic graph.

[0182] In this embodiment, an optimized directed acyclic graph is obtained and deconstructed to obtain a corresponding graph input, at least one graph node and a graph connection corresponding to the at least one graph node; the graph input is converted into a network input in a target format; each graph node is converted into a network layer in a target format; and the graph connection corresponding to the at least one graph node is converted into input and output of the network layer.

[0183] Figure 8 A schematic diagram of the structure of a model architecture conversion device provided in an embodiment of the present disclosure is shown in FIG. Figure 8 As shown, the model architecture conversion device 800 includes:

[0184] The acquisition module 801 is used to acquire an initial model of the first model architecture; the initial model includes a registered custom operator.

[0185] The conversion module 802 is used to convert the initial model into an intermediate model under the second model architecture based on a mapping strategy between the first operator set of the first model architecture and the second operator set of the second model architecture; the intermediate model includes the intermediate operator corresponding to the custom operator.

[0186] The parsing module 803 is used to parse the intermediate model to obtain a directed acyclic graph corresponding to the intermediate model.

[0187] The construction module 804 is used to construct a target model of a third model architecture based on the directed acyclic graph.

[0188] In some embodiments, the conversion module 802 is further configured to:

[0189] Executing the reasoning process of the initial model to obtain a first calculation graph corresponding to the initial model; the first calculation graph includes at least one first operator in the first operator set and a connection relationship between the at least one first operator;

[0190] Based on a mapping strategy between the operator set of the first model architecture and the operator set of the second model architecture, determining in the second operator set a second operator corresponding to each of the at least one first operator;

[0191] Based on the connection relationship between the at least one first operator, determine the connection relationship between at least one second operator, and construct a second calculation graph corresponding to the intermediate model;

[0192] Based on the second computation graph, the intermediate model is constructed.

[0193] In some embodiments, the conversion module 802 is further configured to:

[0194] In response to a registration request for the custom operator, adding the custom operator to the first operator set;

[0195] A mapping relationship between the custom operator and the intermediate operator is added to the mapping strategy to complete the registration process of the custom operator.

[0196] In some embodiments, the operator identifier of the custom operator is the same as the operator identifier of the intermediate operator, and the conversion module 802 is further used to:

[0197] Searching for the intermediate operator in the second operator set based on the operator identifier of the custom operator;

[0198] When the second operator set does not include the intermediate operator, the definition of the intermediate operator is added to the second operator set; the semantics of the intermediate operator is the same as that of the user-defined operator.

[0199] In some embodiments, the parsing module 803 is further used to:

[0200] Reading the intermediate model to obtain model structure data of the intermediate model;

[0201] Based on the model structure data, obtaining serialized graph data of the intermediate model;

[0202] Based on the serialized graph data, determining the operation node corresponding to the intermediate model and the node information corresponding to each of the operation nodes;

[0203] A directed acyclic graph corresponding to the intermediate model is constructed based on the node information corresponding to each of the operation nodes.

[0204] In some embodiments, the parsing module 803 is further used to:

[0205] Parsing the intermediate model to obtain an initial directed acyclic graph corresponding to the intermediate model;

[0206] Based on a preset graph optimization strategy, the initial directed acyclic graph is optimized to obtain a directed acyclic graph corresponding to the intermediate model.

[0207] In some embodiments, the graph optimization strategy includes constant folding, and the parsing module 803 is further used to:

[0208] Acquire at least two consecutive linear change operation nodes in the initial directed acyclic graph;

[0209] The at least two linear change operation nodes are transformed into target operation nodes by constant folding, so as to obtain the directed acyclic graph including the target operation node.

[0210] In some embodiments, the graph optimization strategy includes dead code elimination, and the parsing module 803 is further used to:

[0211] Taking at least one output operation node in the initial directed acyclic graph as a first node, constructing a first list;

[0212] Traversing the first list, moving the current node as the second node to the second list, and adding the parent node of the current node as the first node to the first list until the first list is empty;

[0213] Compare the operation nodes of the second list and the initial directed acyclic graph, delete the invalid nodes in the initial directed acyclic graph, and obtain the directed acyclic graph; the invalid nodes exist in the initial directed acyclic graph and do not exist in the second list.

[0214] In some embodiments, the graph optimization strategy includes operator definition conversion, and the parsing module 803 is further used to:

[0215] Obtaining a preset node conversion table, wherein the node conversion table includes at least one node combination to be converted and a target node combination corresponding to each node combination to be converted; the node combination to be converted and the corresponding target node combination have the same semantics;

[0216] The initial directed acyclic graph is traversed, and the node combinations to be converted in the initial directed acyclic graph are converted into corresponding target node combinations to obtain the directed acyclic graph.

[0217] In some embodiments, the building module 804 is further used to:

[0218] Deconstructing the directed acyclic graph to obtain a graph input corresponding to the directed acyclic graph, at least one graph node, and node connection information corresponding to the at least one graph node;

[0219] Converting the graph input into a network input of the third model architecture, and converting each of the at least one graph node into a network layer of the third model architecture;

[0220] Determine a connection relationship between at least one network layer based on node connection information corresponding to the at least one graph node;

[0221] Based on the network input of the third model architecture, the at least one network layer and the connection relationship between the at least one network layer, a target model of the third model architecture is constructed.

[0222] In some embodiments, the building module 804 is further used to:

[0223] Inputting test data into the initial model to obtain a first result corresponding to the test data output by the initial model;

[0224] Inputting the test data into the target model to obtain a second result output by the target model;

[0225] When the first result and the second result are the same, the target model is output.

[0226] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present disclosure, please refer to the description of the method embodiment of the present disclosure for understanding.

[0227] It should be noted that in the embodiments of the present disclosure, if the above-mentioned model architecture conversion method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present disclosure is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a device to execute all or part of the methods of each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiments of the present disclosure are not limited to any combination of hardware and software of any target.

[0228] Fig. 9 A hardware entity diagram of a model architecture conversion device provided by an embodiment of the present disclosure, such as Fig. 9 As shown, the hardware entity of the model architecture conversion device 900 includes: a processor 901 and a memory 902, wherein the memory 902 stores a computer program that can be run on the processor 901, and the processor 901 implements the steps in the method of any of the above embodiments when executing the program.

[0229] The memory 902 stores a computer program that can be run on the processor. The memory 902 is configured to store instructions and applications executable by the processor 901. It can also cache data to be processed or processed by the processor 901 and each module in the model architecture conversion device 900 (for example, image data, audio data, voice communication data, and video communication data). It can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0230] When the processor 901 executes the program, the steps of any of the above-mentioned model architecture conversion methods are implemented. The processor 901 generally controls the overall operation of the model architecture conversion device 900.

[0231] An embodiment of the present disclosure provides a computer storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps of the model architecture conversion method in any of the above embodiments.

[0232] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of the present disclosure, please refer to the description of the method embodiments of the present disclosure for understanding.

[0233] The processor may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It is understandable that the electronic device that implements the functions of the processor may also be other, which is not specifically limited in the embodiments of the present disclosure.

[0234] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM) and the like; it can also be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0235] It should be understood that the "one embodiment" or "an embodiment" or "an embodiment of the present disclosure" or "the aforementioned embodiments" or "some embodiments" mentioned throughout the specification means that the target features, structures or characteristics related to the embodiment are included in at least one embodiment of the present disclosure. Therefore, "in one embodiment" or "in an embodiment" or "an embodiment of the present disclosure" or "the aforementioned embodiments" or "some embodiments" appearing in various places throughout the specification do not necessarily refer to the same embodiment. In addition, the features, structures or characteristics of these targets may be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present disclosure, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure. The serial numbers of the above-mentioned embodiments of the present disclosure are for description only and do not represent the advantages and disadvantages of the embodiments.

[0236] Unless otherwise specified, the model architecture conversion device executes any step in the embodiments of the present disclosure, and the processor of the model architecture conversion device may execute the step. Unless otherwise specified, the embodiments of the present disclosure do not limit the order in which the model architecture conversion device executes the following steps. In addition, the methods used to process data in different embodiments may be the same method or different methods. It should also be noted that any step in the embodiments of the present disclosure can be independently executed by the model architecture conversion device, that is, when the model architecture conversion device executes any step in the above embodiments, it may not rely on the execution of other steps.

[0237] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0238] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0239] In addition, all functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may be separately configured as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0240] The methods disclosed in several method embodiments provided in the present disclosure can be arbitrarily combined without conflict to obtain new method embodiments.

[0241] The features disclosed in several product embodiments provided in the present disclosure can be arbitrarily combined without conflict to obtain new product embodiments.

[0242] The features disclosed in several method or device embodiments provided in the present disclosure may be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0243] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0244] Alternatively, if the above-mentioned integrated unit of the present disclosure is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present disclosure can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a model architecture conversion device, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0245] In the embodiments of the present disclosure, the descriptions of the same steps and the same contents in different embodiments can refer to each other. In the embodiments of the present disclosure, the term "and" does not affect the sequence of the steps.

[0246] The above is only an embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A model architecture conversion method, characterized in that: The method comprises: Acquire an initial model of a first model architecture; the initial model includes a registered custom operator; Based on a mapping strategy between a first operator set of the first model architecture and a second operator set of the second model architecture, converting the initial model into an intermediate model under the second model architecture; the intermediate model includes the intermediate operator corresponding to the custom operator; Parsing the intermediate model to obtain a directed acyclic graph corresponding to the intermediate model; Constructing a target model of a third model architecture based on the directed acyclic graph; The step of parsing the intermediate model to obtain a directed acyclic graph corresponding to the intermediate model includes: Parsing the intermediate model to obtain an initial directed acyclic graph corresponding to the intermediate model; Based on a preset graph optimization strategy, the initial directed acyclic graph is optimized to obtain a directed acyclic graph corresponding to the intermediate model; the preset graph optimization strategy includes at least one optimization sub-strategy with a priority order; the optimization sub-strategy includes at least one of the following: constant folding, dead code elimination, and operator definition conversion.

2. The method according to claim 1, characterized in that The mapping strategy between the first operator set of the first model architecture and the second operator set of the second model architecture, converting the initial model into an intermediate model under the second model architecture, includes: Executing the reasoning process of the initial model to obtain a first calculation graph corresponding to the initial model; the first calculation graph includes at least one first operator in the first operator set and a connection relationship between the at least one first operator; Based on a mapping strategy between the operator set of the first model architecture and the operator set of the second model architecture, determining in the second operator set a second operator corresponding to each of the at least one first operator; Based on the connection relationship between the at least one first operator, determine the connection relationship between at least one second operator, and construct a second calculation graph corresponding to the intermediate model; Based on the second computation graph, the intermediate model is constructed.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: In response to a registration request for the custom operator, adding the custom operator to the first operator set; A mapping relationship between the custom operator and the intermediate operator is added to the mapping strategy to complete the registration process of the custom operator.

4. The method according to claim 3, characterized in that: The operator identifier of the custom operator is the same as the operator identifier of the intermediate operator, and the method further includes: Searching for the intermediate operator in the second operator set based on the operator identifier of the custom operator; When the second operator set does not include the intermediate operator, the definition of the intermediate operator is added to the second operator set; the semantics of the intermediate operator is the same as that of the user-defined operator.

5. The method according to claim 1 or 2, characterized in that: The step of parsing the intermediate model to obtain a directed acyclic graph corresponding to the intermediate model further includes: Reading the intermediate model to obtain model structure data of the intermediate model; Based on the model structure data, obtaining serialized graph data of the intermediate model; Based on the serialized graph data, determining the operation node corresponding to the intermediate model and the node information corresponding to each of the operation nodes; A directed acyclic graph corresponding to the intermediate model is constructed based on the node information corresponding to each of the operation nodes.

6. The method according to claim 1, characterized in that The step of optimizing the initial directed acyclic graph based on a preset graph optimization strategy to obtain a directed acyclic graph corresponding to the intermediate model includes: Acquire at least two consecutive linear change operation nodes in the initial directed acyclic graph; The at least two linear change operation nodes are transformed into target operation nodes by constant folding, so as to obtain the directed acyclic graph including the target operation node.

7. The method according to claim 1, characterized in that The method further comprises optimizing the initial directed acyclic graph based on a preset graph optimization strategy to obtain a directed acyclic graph corresponding to the intermediate model, and further comprising: Taking at least one output operation node in the initial directed acyclic graph as a first node, constructing a first list; Traversing the first list, moving the current node as the second node to the second list, and adding the parent node of the current node as the first node to the first list until the first list is empty; Compare the operation nodes of the second list and the initial directed acyclic graph, delete the invalid nodes in the initial directed acyclic graph, and obtain the directed acyclic graph; the invalid nodes exist in the initial directed acyclic graph and do not exist in the second list.

8. The method according to claim 1, characterized in that: The method further comprises optimizing the initial directed acyclic graph based on a preset graph optimization strategy to obtain a directed acyclic graph corresponding to the intermediate model, and further comprising: Obtaining a preset node conversion table, wherein the node conversion table includes at least one node combination to be converted and a target node combination corresponding to each node combination to be converted; the node combination to be converted and the corresponding target node combination have the same semantics; The initial directed acyclic graph is traversed, and the node combinations to be converted in the initial directed acyclic graph are converted into corresponding target node combinations to obtain the directed acyclic graph.

9. The method according to claim 1 or 2, characterized in that: The target model of the third model architecture is constructed based on the directed acyclic graph, comprising: Deconstructing the directed acyclic graph to obtain a graph input corresponding to the directed acyclic graph, at least one graph node, and node connection information corresponding to the at least one graph node; Converting the graph input into a network input of the third model architecture, and converting each of the at least one graph node into a network layer of the third model architecture; Determine a connection relationship between at least one network layer based on node connection information corresponding to the at least one graph node; Based on the network input of the third model architecture, the at least one network layer and the connection relationship between the at least one network layer, a target model of the third model architecture is constructed.

10. The method according to claim 1 or 2, characterized in that: The method further comprises: Inputting test data into the initial model to obtain a first result corresponding to the test data output by the initial model; Inputting the test data into the target model to obtain a second result output by the target model; When the first result and the second result are the same, the target model is output.

11. A model architecture conversion device, characterized in that: include: An acquisition module, used to acquire an initial model of the first model architecture; The initial model includes registered custom operators; A conversion module, configured to convert the initial model into an intermediate model under the second model architecture based on a mapping strategy between a first operator set of the first model architecture and a second operator set of the second model architecture; the intermediate model includes the intermediate operator corresponding to the custom operator; A parsing module, used for parsing the intermediate model to obtain a directed acyclic graph corresponding to the intermediate model; A construction module, used for constructing a target model of a third model architecture based on the directed acyclic graph; The parsing mode is also used to parse the intermediate model to obtain an initial directed acyclic graph corresponding to the intermediate model; based on a preset graph optimization strategy, the initial directed acyclic graph is optimized to obtain a directed acyclic graph corresponding to the intermediate model; the preset graph optimization strategy includes at least one optimization sub-strategy with a priority order; the optimization sub-strategy includes at least one of the following: constant folding, dead code elimination, and operator definition conversion.

12. A model architecture conversion device, characterized in that: include: Memory and processor, The memory stores a computer program executable on the processor. When the processor executes the computer program, the steps in the method according to any one of claims 1 to 10 are implemented.

13. A computer storage medium, characterized in that: The computer storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in any one of claims 1 to 10.

Images