Deep learning model conversion method and apparatus

Abstract

The embodiment of the application provides a deep learning model conversion method and device, the method comprises the following steps: processing the network structure code of a first model according to a function decorator and a capture function, obtaining a public layer, a custom layer and layer parameters of each layer in the first model, the first model being a model recognizable under a first deep learning framework; generating an intermediate representation format IR of the first model according to the public layer, the custom layer and the layer parameters of each layer, the intermediate representation format being a unified representation format for model conversion under multiple deep learning frameworks; and generating a second model recognizable under a second deep learning framework according to the intermediate representation format of the first model. The complete conversion of the custom layer can be realized, so that the correct conversion of the first model under different deep learning model frameworks is ensured.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer, and particularly relate to a deep learning model conversion method and device. BACKGROUND

[0002] Deep learning model is a kind of machine learning, and deep learning framework is generated for developing deep learning model. In some cases, it is needed to convert a deep learning model under one framework into a deep learning model under another framework to improve the operation efficiency of processor.

[0003] When converting deep learning model under different frameworks, the conversion of public layer and custom layer is usually involved. Among them, the public layer is supported by different frameworks, so the conversion is relatively easy, while the custom layer is a layer configured by user according to the function of the model, and the way of setting custom layer is different under different frameworks, so the conversion of custom layer is relatively complex. At present, the conversion of custom layer usually takes the custom layer as a series of specific business logic fine-grained operators, then traverses these operators and extracts corresponding parameters for conversion.

[0004] After the conversion of the above-mentioned scheme, an independent operator is obtained under another framework instead of a complete custom layer, so that the model conversion is prone to problems. SUMMARY

[0005] Embodiments of the present application provide a deep learning model conversion method and device to solve the problem that the custom layer is split into independent operators after conversion, which makes the model conversion prone to errors.

[0006] In a first aspect, embodiments of the present application provide a deep learning model conversion method, comprising:

[0007] processing the network structure code of the first model according to the function decorator and the capture function to obtain the public layer, the custom layer and the layer parameters of each layer in the first model, the first model being a model recognizable under a first deep learning framework;

[0008] generating the IR of the first model according to the public layer, the custom layer and the layer parameters of each layer, the IR being a unified representation format for model conversion under multiple deep learning frameworks;

[0009] generating a second model recognizable under a second deep learning framework according to the IR of the first model.

[0010] In a possible implementation, the network structure code of the first model is processed according to the function decorator and the capturing function, the common layer, the custom layer, and the layer parameters of each layer in the first model are obtained, including:

[0011] The first model is executed, and the operators in the first model are relocated according to the function decorator when the first model is executed, and the types of the layers in the first model are obtained, wherein each layer includes at least one operator;

[0012] The common layer and the custom layer are determined in the layers according to the types of the layers.

[0013] The layer parameters of each custom layer and each common layer are obtained according to the capturing function, and the layer parameters include hyperparameters and learning parameters.

[0014] In a possible implementation, the common layer and the custom layer are determined in the layers according to the types of the layers, including:

[0015] A preset layer list is obtained.

[0016] The common layer and the custom layer are determined in the layers according to the preset layer list and the types of the layers.

[0017] In a possible implementation, the common layer and the custom layer are determined in the layers according to the preset layer list and the types of the layers, including:

[0018] For any one of the layers, if it is determined according to the type of the layer that the layer belongs to a layer in the preset layer list, the layer is determined as the common layer.

[0019] If it is determined according to the type of the layer that the layer does not belong to a layer in the preset layer list, the layer is determined as the custom layer.

[0020] In a possible implementation, before the network structure code of the first model is processed according to the function decorator and the capturing function, the common layer, the custom layer, and the layer parameters of each layer in the first model are obtained, the method further includes:

[0021] It is determined that the operators included in each custom layer are located in the same function.

[0022] In a possible implementation, the IR of the first model is obtained according to the common layer, the custom layer, and the layer parameters of each layer, including:

[0023] For each public layer, a first conversion function corresponding to the public layer is obtained, and IR corresponding to the public layer is obtained according to the first conversion function and layer parameters of the public layer;

[0024] For each custom layer, a second conversion function corresponding to the custom layer is obtained, and IR corresponding to the custom layer is generated according to the second conversion function and layer parameters of the custom layer;

[0025] According to the IR corresponding to the public layer and the IR corresponding to the custom layer, the IR of the first model is generated.

[0026] In a possible implementation, the IR of the first model is obtained according to the IR corresponding to the public layer and the IR corresponding to the custom layer, including:

[0027] The connection relationship between each public layer and each custom layer in the first model is obtained;

[0028] According to the connection relationship, the IR corresponding to the public layer and the IR corresponding to the custom layer, the IR of the first model is generated.

[0029] In a possible implementation, the method further includes:

[0030] Receiving a request instruction sent by a client, the request instruction being used to request the second model with a first function, the first function being a function of the first model, wherein the first function is a target detection function and / or a target recognition function;

[0031] Sending the second model to the client.

[0032] In a second aspect, an embodiment of the present application provides a deep learning model conversion device, including:

[0033] An obtaining module is configured to process network structure code of a first model according to a function decorator and a capture function, and obtain public layers, custom layers and layer parameters of each layer in the first model, the first model being a model recognizable under a first deep learning framework;

[0034] A processing module is configured to generate IR of the first model according to the public layers, the custom layers and the layer parameters of each layer, the IR being a unified representation format for model conversion under multiple deep learning frameworks;

[0035] A conversion module is configured to generate a second model recognizable under a second deep learning framework according to the IR of the first model.

[0036] In a possible implementation, the obtaining module is specifically configured to:

[0037] perform the first model, and perform a repositioning process on operators in the first model according to the function decorator when the first model is performed, to obtain types of each layer in the first model, wherein any one layer includes at least one operator;

[0038] determine the common layer and the custom layer in the each layer according to the types of the each layer;

[0039] obtain layer parameters of each of the custom layer and each of the common layer according to the capture function, wherein the layer parameters include hyperparameters and learning parameters.

[0040] In a possible implementation, the obtaining module is specifically configured to:

[0041] obtain a preset layer list;

[0042] determine the common layer and the custom layer in the each layer according to the preset layer list and the types of the each layer.

[0043] In a possible implementation, the obtaining module is specifically configured to:

[0044] for any one layer in the each layer, if it is determined according to the type of the layer that the layer belongs to a layer in the preset layer list, the layer is determined as the common layer;

[0045] if it is determined according to the type of the layer that the layer does not belong to a layer in the preset layer list, the layer is determined as the custom layer.

[0046] In a possible implementation, the obtaining module is further configured to, before obtaining the common layer, the custom layer, and the layer parameters of each layer in the first model by processing network structure code of the first model according to the function decorator and the capture function:

[0047] determine that operators included in each of the custom layer are located in a same function.

[0048] In a possible implementation, the processing module is specifically configured to:

[0049] for each common layer, obtain a first conversion function corresponding to the common layer, and obtain IR corresponding to the common layer according to the first conversion function and the layer parameters of the common layer;

[0050] for each custom layer, obtain a second conversion function corresponding to the custom layer, and obtain IR corresponding to the custom layer according to the second conversion function and the layer parameters of the custom layer;

[0051] generate the IR of the first model according to the IR corresponding to the public layers and the IR corresponding to the custom layers.

[0052] In a possible implementation, the processing module is specifically configured to:

[0053] obtain a connection relationship between each of the public layers and each of the custom layers in the first model;

[0054] generate the IR of the first model according to the connection relationship, the IR corresponding to the public layers, and the IR corresponding to the custom layers.

[0055] In a possible implementation, the system further includes a transceiving module, which is configured to:

[0056] receive a request instruction sent by a client, the request instruction being used to request the second model with a first function, the first function being a function of the first model, wherein the first function is a target detection function and / or a target recognition function;

[0057] send the second model to the client.

[0058] In a third aspect, an embodiment of the present application provides a deep learning model conversion system, including a first processor and a second processor, wherein:

[0059] the first processor is any one of a central processing unit (CPU), a graphics processing unit (GPU), and an image signal processor (ISP), the second processor is any one of a CPU, a GPU, and an ISP, and the first processor is different from the second processor;

[0060] the first processor is configured to send a first model identifiable under a first deep learning framework to the second processor, a function of the first model being a target detection function and / or a target recognition function;

[0061] the second processor is configured to convert the first model identifiable under the first deep learning framework into a second model identifiable under a second deep learning framework according to the method in any one of the first aspect, a function of the second model being the same as the function of the first model.

[0062] In a fourth aspect, an embodiment of the present application provides a deep learning model conversion device, including:

[0063] a memory configured to store a program;

[0064] a processor configured to execute the program stored in the memory, when the program is executed, the processor is configured to execute the deep learning model conversion method in any one of the first aspect.

[0065] Fifthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the deep learning model conversion method as described in any of the first aspects.

[0066] In a sixth aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any one of the first aspects.

[0067] The deep learning model conversion method and apparatus provided in this application first process the network structure code of a first model according to function decorators and capturing functions to obtain the common layers, custom layers, and layer parameters of each layer in the first model. The first model is a model recognizable under a first deep learning framework. Then, based on the common layers, custom layers, and layer parameters of each layer, the Inverse Representation (IR) of the first model is obtained. Since the IR is a unified representation format for model conversion under multiple deep learning frameworks, a second model recognizable under a second deep learning framework can be generated based on the IR of the first model. The solution in this application, for custom layers, can identify the custom layers in the first model through function decorators and convert each custom layer as a whole functional layer, avoiding the conversion of a custom layer into individual operators, thereby achieving the correct conversion of the first model. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0070] Figure 2 A flowchart illustrating the deep learning model conversion method provided in this application embodiment;

[0071] Figure 3 A schematic diagram of the model conversion framework provided in the embodiments of this application;

[0072] Figure 4 A schematic diagram illustrating the process of obtaining layer type and layer parameters provided in an embodiment of this application;

[0073] Figure 5 A schematic diagram illustrating the process of obtaining a list of custom layers provided in an embodiment of this application;

[0074] Figure 6 A flowchart of the deep learning framework conversion provided by the embodiment of the present application is shown in the figure;

[0075] Figure 7 A schematic diagram of model conversion provided by the embodiment of the present application is shown in the figure;

[0076] Figure 8 A structural schematic diagram of a deep learning model conversion device provided by the embodiment of the present application is shown in the figure;

[0077] Figure 9 A structural schematic diagram of a deep learning model conversion system provided by the embodiment of the present application is shown in the figure;

[0078] Figure 10 A hardware structural schematic diagram of a deep learning model conversion device provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0079] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0080] First, the concepts involved in the embodiments of the present application are explained.

[0081] IR: intermediate representation, intermediate representation format, the intermediate representation format referred to in the embodiments of the present application is a unified representation format for model conversion under multiple deep learning frameworks, which can enable the model to be transferred between different frameworks.

[0082] Deep learning module: binary representation of the structure of a neural network based on deep learning.

[0083] Deep learning framework: a framework produced to improve the development efficiency of deep learning models, which includes the definition of many functional layers, and different functional layers can be called to obtain different deep learning models according to actual needs, so as to realize different functions. Common deep learning frameworks include pytorch, caffe, mxnet, etc.

[0084] Model conversion: the models of different deep learning frameworks are not compatible, and the model of one deep learning framework needs to be converted into a model that can be recognized under another deep learning framework, i.e. model conversion.

[0085] Custom layer: corresponding to the public layer, the public layer is a function layer defined under the deep learning framework, for example, the basic convolution, pooling and other layers contained in the neural network, and the algorithm personnel will also define their own calculation method according to the need, and the self-defined calculation method is the custom layer.

[0086] OP: operation, operator in deep learning network.

[0087] Function decoration: function decorator (decorators) is an important part of Python, and function decoration is a function that modifies the function of other functions.

[0088] Figure 1 An application scenario diagram provided by the embodiment of the application is shown in FIG. 1. Figure 1 As shown in FIG. 1, the application scenario diagram includes a processor 11 and a processor 12.

[0089] The processor 11 includes a first trained model, and the first model is a model identifiable under a first deep learning framework, for example, any one of the frameworks such as mxnet, caffe, pytorch and TensorFlow, or other frameworks.

[0090] When the first deep learning framework is not suitable for the hardware structure characteristics of the processor 12, if the first model is directly run on the processor 12, the operation efficiency of the processor 12 will be low, and therefore the conversion of the first model under different deep learning frameworks is involved, and the first model is converted into a second model under a second deep learning framework suitable for the processor 12.

[0091] The first model can be a model with face detection function, a model with vehicle detection function, a model with behavior detection function, or a model with other functions. After the first model is converted into the second model, the second model and the first model have the same function, only the corresponding framework is different. The processor 11 and the processor 12 can be CPU, GPU or other types of processors, and in some cases, the processor 11 and the processor 12 can also be integrated in one device.

[0092] When the model conversion is performed, three steps are mainly included: first, the model under the to-be-converted framework is converted into the corresponding IR; second, the IR is converted into the model data saving format (npy) under the target framework and the target framework model generation code (py); and finally, the target framework model generation code py file and npy are used to generate the model storage file (such as prototxt and caffemodel) of the corresponding framework.

[0093] In the conversion of the first model, the public layer in the first model can include a public layer and a custom layer, and the public layer is supported by multiple deep learning frameworks. Therefore, the conversion of the public layer is relatively easy. The custom layer is relatively complex to convert because the configurations of the custom layer are different in different frameworks. Currently, the model conversion is mainly achieved by constructing a computation graph, traversing the OP in the computation graph, and extracting parameters.

[0094] For the conversion of the custom layer, there are currently two processing methods. The first processing method is to regard the custom layer as a series of specific business logic fine-grained operators. In the conversion, the custom layer is converted into an independent operator.

[0095] For example, the first model includes a custom layer, and the custom layer includes two OPs, namely matrix addition and matrix multiplication. After conversion by this method, the matrix addition and matrix multiplication are split to obtain two corresponding layers instead of a complete whole.

[0096] Although this processing method can ensure that the converted first model can normally run on hardware that supports fine-grained small operators, this method cannot support hardware running of fine-grained operators such as matrix multiplication. On the other hand, the concept of the custom layer is lost, which does not meet the actual demand and cannot achieve the correct conversion of the first model.

[0097] The second processing method is to regard the custom layer as a whole composed of many OPs. In the model conversion, all OPs belonging to the custom layer need to be associated and processed, which is relatively complex to implement.

[0098] Further, this processing method cannot be converted when the network of the custom layer is not written in a standard manner. Even if the non-standard writing is adjusted, the parameter names in the adjusted first model and the parameter names in the original first model do not match, the parameter extraction fails, and the model conversion fails, which requires retraining the model and then converting.

[0099] Based on the above problems, the embodiment of the present application provides a deep learning model conversion scheme.

[0100] Figure 2 The flowchart of the deep learning model conversion method provided by the embodiment of the present application is shown in Figure 2 As shown in the figure, the method can include:

[0101] S21, processing the network structure code of the first model according to the function decorator and the capture function, obtaining the public layer, the custom layer, and the layer parameters of each layer in the first model, the first model being a model recognizable in a first deep learning framework.

[0102] The model identifiable under different deep learning frameworks is different, which is caused by different storage and analysis of data information in different deep learning frameworks. Therefore, the first model identifiable under the first deep learning framework may become unidentifiable under the second deep learning framework, which is the problem of model incompatibility of different deep learning frameworks.

[0103] In the embodiment of the application, the first model identifiable under the first deep learning framework needs to be converted into a model identifiable under another deep learning framework. The first model includes a plurality of functional layers, and the plurality of functional layers include a public layer and a custom layer. The public layer is a layer supported by a plurality of deep learning frameworks, and the public layer is an open source supported layer. The difference between the public layer in different deep learning frameworks is only that the storage and analysis of data in the public layer are different under different deep learning frameworks. The custom layer is a set of operations defined according to actual needs or functions to be implemented by the first model, and the custom layer can be one or more. The public layer and the custom layer are sequentially connected to form the first model.

[0104] For the public layer, since the public layer is supported by different deep learning frameworks, the conversion of the public layer is relatively easy. The custom layer is a layer composed of a set of operations defined according to needs. When converting, if it is regarded as a series of specific business logic fine-grained operators, the custom layer is converted into independent operators after conversion, instead of an entire custom functional layer, which loses the meaning of custom layer conversion. If it is regarded as a whole, the implementation logic is relatively complex.

[0105] In the embodiment of the application, the network structure code of the first model is processed by a function decorator and a capture function. The function decorator is a function that modifies the function of other functions. The function decorator is used to decorate all operators in the first model, so as to obtain each layer in the first model and determine the custom layer and the public layer in each layer. After the custom layer and the public layer are determined, the capture function can obtain the layer parameters of each layer, wherein the layer parameters of each layer can include hyperparameters and learning parameters.

[0106] S22, generating an IR of the first model according to the public layer, the custom layer and the layer parameters of each layer. The IR is a unified representation format for model conversion under a plurality of deep learning frameworks.

[0107] For each layer, each layer is composed of a plurality of operators, and each layer has corresponding layer parameters. After each layer and the corresponding layer parameters are obtained, the composition of the first model is determined. Then, the IR of the first model can be generated respectively for the custom layer and the layer parameters of each custom layer, the public layer and the layer parameters of each public layer.

[0108] IR is a universal representation format, a unified representation format for model conversion across various deep learning frameworks. The IR of the first model expresses all the layers included in the first model, including all common layers and custom layers, as well as the connection relationships between the layers and the layer parameters of each layer in the first model.

[0109] S23, Based on the IR of the first model, generate a second model that can be recognized under the second deep learning framework.

[0110] Intermediate Representation (IR) enables model transfer between different frameworks. An intermediate mode defines a model representation format that is universally applicable across various deep learning frameworks, thus facilitating model conversion between them. IRs can be predefined or open-source IRs can be used. For example, one possible IR is the Open Neural Network Exchange (ONNX) format, a standard for representing deep learning models that enables model conversion between different deep learning frameworks. It is understood that the ONNX format is merely one example of IR and does not constitute a limitation on IR.

[0111] After obtaining the IR of the first model, since the IR can be recognized by various deep learning frameworks, a second model that can be recognized by the second deep learning framework can be generated based on the IR of the first model.

[0112] Figure 3 This is a schematic diagram of the model conversion framework provided in the embodiments of this application, such as... Figure 3 As shown, it includes a first deep learning framework, an IR (Integrated Learning Framework), and a second deep learning framework. The first deep learning framework can be, for example, Caffe, MXNet, PyTorch, Keras, etc., and the second deep learning framework can also be one of these frameworks. That is, the first deep learning framework can be one of several deep learning frameworks, while the second deep learning framework is a different deep learning framework from the first. Figure 3 The deep learning framework shown is merely an example, and the embodiments of this application do not impose any particular limitations on it.

[0113] According to the above method, the first model under the first deep learning framework is transformed into the second model under the second deep learning framework, so that the second model can run on a processor adapted to the second deep learning framework.

[0114] The deep learning model conversion method provided in this application first processes the network structure code of a first model according to function decorators and capturing functions to obtain the common layers, custom layers, and layer parameters of each layer in the first model. The first model is a model recognizable under a first deep learning framework. Then, based on the common layers, custom layers, and layer parameters of each layer, the Inverse Representation (IR) of the first model is obtained. Since the IR is a unified representation format for model conversion under multiple deep learning frameworks, a second model recognizable under a second deep learning framework can be generated based on the IR of the first model. The solution in this application, for custom layers, can identify the custom layers in the first model through function decorators and convert each custom layer as a whole functional layer, avoiding the conversion of a custom layer into individual operators, thereby achieving the correct conversion of the first model.

[0115] The solution of this application will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0116] Before performing the conversion of the first model, it is necessary to first identify the type of each layer in the first model and obtain the layer parameters of each layer. Figure 4 This is a flowchart illustrating the process of obtaining layer type and layer parameters provided in an embodiment of this application, as shown below. Figure 4 As shown, it includes:

[0117] S41, execute the first model, and when the first model is executed, perform relocation processing on the operators in the first model according to the function decorator to obtain the type of each layer in the first model, wherein any layer includes at least one operator.

[0118] Before the first model is executed, it is impossible to know which layers it includes. In the solution of this application embodiment, each OP in the first model is decorated with a function. When the first model is loaded and an OP is executed, since the OP has been decorated, the function decorator corresponding to that OP is actually executed, that is, the decorator function. This is the relocation processing of operators in the first model.

[0119] S42, Based on the type of each layer, determine the common layer and the custom layer in each layer.

[0120] Through relocation processing, the types of each layer in the first model were obtained. At this point, it is still impossible to determine whether each layer belongs to a common layer or a custom layer.

[0121] Since the public layer is a layer supported by multiple deep learning frameworks, the structure of the public layer is fixed, and thus a preset layer list can be obtained, and the layers in the preset layer list are all public layers, and the public layer and the custom layer are determined according to the preset layer list and the types of the layers in the first model.

[0122] Specifically, for any one layer in the first model, the layer is compared with the layers in the preset layer list. If it is determined according to the type of the layer that the layer belongs to the layers in the preset layer list, the layer is determined to be a public layer. Conversely, if it is determined according to the type of the layer that the layer does not belong to the layers in the preset layer list, the layer is determined to be a custom layer.

[0123] Through the above method, the layers in the first model can be divided into public layers and custom layers, and thus the custom layer list can be obtained. The obtaining of the custom layer list will be further described below. Figure 5

[0124] Figure 5 The flowchart for obtaining the custom layer list provided by the embodiments of the present application is shown in FIG. 1, which includes the following steps. Figure 5

[0125] S501, decorating the OP.

[0126] First, the OP in the first model needs to be decorated by a function decorator. The OP to be decorated includes the OP in each functional layer, that is, the OP in the public layer and the OP in the custom layer.

[0127] S502, loading the first model.

[0128] S503, constructing the network.

[0129] S504, starting the forward inference.

[0130] The forward inference is to execute the first model according to the network structure code of the first model. In the process of network construction, each layer in the first model is captured.

[0131] S505, OP execution.

[0132] After starting the forward inference, each OP in the first model is executed in turn.

[0133] S506, redirecting the decorated function.

[0134] Since the OP is decorated by the decorated function in S501, when a certain OP is executed, the corresponding decorated function is actually executed, so that the OP in the first model is redirected.

[0135] ​​The granularity of the captured is the layer, rather than the individual operators, through the decoration function. The following is an example.

[0136] There is an OP A in the first model, which is composed of OP a and OP b. When each OP in the first model is decorated through the function decorator, OP A, OP a and OP b are all decorated. When the network structure code executes to OP A, since OP A is decorated, the decoration function of OP A is entered first. A global mark is recorded in the decoration function, indicating whether the OP is executed.

[0137] When the decoration function of OP A is entered, the mark variable is recorded as false, indicating that OP A has not been executed. Since OP A includes OP a and OP b, when OP a is executed, the decoration function of OP a is entered. However, since the global mark is false at this time, it indicates that the current OP is executed in a nested manner, and the type acquisition and parameter extraction are not performed on OP a. Similarly, for OP b, the global mark false indicates that OP b is executed in a nested manner. Only after OP A is executed, the global mark is changed to indicate that the OP is executed, so that only the layer OP A is extracted, and OP a and OP b are not extracted and converted as operators in OP A. In this way, the granularity of the captured is the layer OP A, rather than the fine-grained operators OP a and OP b.

[0138] S507, the layer type is extracted in the decoration function.

[0139] Through the relocation processing of the OP in the first model in the forward reasoning process, the layers in the first model are captured, and the layers included in the first model are determined.

[0140] S508, it is judged whether the last layer OP is executed, if yes, S510 is executed, if not, S509 is executed.

[0141] S509, the next OP is switched, and S505 is executed.

[0142] For all OPs in the first model, the above processing is required, until all OPs are processed.

[0143] S510, the layers extracted from the decoration function are compared with the preset layer list.

[0144] After all the OPs in the first model are processed, the layers in the first model can be captured through the decoration function, but it is not known which layers are custom layers and which layers are public layers.

[0145] Therefore, it is necessary to compare the captured layers with the preset layer list, wherein the layers included in the preset layer list are public layers, and other layers included in the first model which are not included in the preset layer list are custom layers.

[0146] S511, obtaining the custom layer list.

[0147] After the comparison with the preset layer list is completed, the custom layer list in the first model can be determined.

[0148] S43, obtaining layer parameters of each custom layer and each public layer according to the capture function, wherein the layer parameters include hyperparameters and learning parameters.

[0149] After the custom layers and the public layers in the first model are determined, each layer is processed according to the capture function, and the layer parameters of each layer can be obtained. The layer parameters include hyperparameters and learning parameters, wherein the hyperparameters are pre-set parameters, and the learning parameters are parameters obtained by model training.

[0150] Figure 6 A flowchart of the deep learning framework conversion provided by the embodiment of the present application is shown in FIG. 6. Figure 6 As shown in FIG. 6, the flowchart includes:

[0151] S601, encapsulating the custom layer as a class.

[0152] This step is mainly performed in the case of non-standard network writing of the custom layer. Before the deep learning framework conversion, it is necessary to determine that the operators included in each custom layer are located in the same function, that is, the logic of the related custom layer is in a function and is defined as a class. If this requirement is not met, the corresponding adjustment needs to be made to encapsulate the custom layer as a class according to the standard.

[0153] The standard of the custom layer includes but is not limited to: (1) the logic of the related custom layer is in a function and is defined as a class; (2) the hyperparameters of the custom layer must be the member variables of the custom layer in the _init_.

[0154] If the operators included in the custom layer are not located in the same function, the logic of the related custom layer cannot be encapsulated as a class, and in this case, the network writing of the custom layer is not standard. For example, when some OPs in the custom layer are placed in the main function of the code, the custom layer class cannot be directly called, and needs to be adjusted. For example, in the case of non-standard writing of each OP in the custom layer, each OP in the custom layer can be encapsulated in the same function, and after the encapsulation is completed, the network writing of the custom layer is adjusted to be standard. At this time, the custom logic call can be converted to the custom layer class call.

[0155] The following is an example code that does not conform to the specification of the custom layer encapsulation into a class, and the last two lines are custom layer OP logic:

[0156]

[0157] Since the custom layer needs to be encapsulated into a class, the encapsulated code of the above non-compliant example code is as follows, where Mylayer is a custom layer class object:

[0158]

[0159] As can be seen, after custom layer encapsulation, the custom layer OP is encapsulated in Mylayer. At this time, the custom layer is written in a standard manner. In the subsequent main code of the first model, the custom logic call can be converted to a custom layer class call. An example of the main code is as follows:

[0160]

[0161]

[0162] For any non-standard custom layer, the above processing can be performed to complete the custom layer encapsulation into a class, and then the first model conversion is performed.

[0163] S602, common OP and custom OP decoration.

[0164] Same as S501, the OP in the first model is decorated by the decoration function, and the OP includes the OP of the common layer and the OP of the custom layer.

[0165] S603, common OP and custom OP mapping capture function.

[0166] Since the first model includes multiple layers, when converting the model under different deep learning frameworks, not only the layer parameters of the layers under the model need to be converted, but also the layer parameters are obtained by the capture function. Therefore, after the OP in the first model is decorated by the decoration function, the parameters of the OP in the first model also need to be captured, that is, the corresponding capture function is set.

[0167] S604, load the first model.

[0168] S605, build the network.

[0169] S606, start forward inference.

[0170] Same as S504, forward inference is to execute the first model, and in the process of network construction, each layer in the first model and the layer parameters of each layer are captured.

[0171] S607, the OP is executed.

[0172] After starting the forward reasoning, each OP in the first model is executed in turn.

[0173] S608, the decoration function is redirected.

[0174] Before this, the OP is decorated by the decoration function, so when a certain OP is executed, the corresponding decoration function is actually executed, thereby realizing the redirection of the OP in the first model.

[0175] S609, the capture function is redirected.

[0176] In the actual execution of the forward reasoning process, not only the decoration function is redirected, but also the capture function is redirected, and the layer parameter extraction is performed in the capture function. In view of the problem that the writing of the custom layer is not standardized, after the standard adjustment, the parameter name can be mapped to the original parameter in the capture function, so as to ensure that the layer parameter can be correctly extracted without retraining the model.

[0177] S610, the OP parameter extraction in the capture function generates the OP in the IR.

[0178] In the above steps, each layer in the first model is extracted by the decoration function, and the layer parameters of each layer are extracted by the capture function, and then the IR conversion can be performed.

[0179] Specifically, for each public layer, a first conversion function corresponding to the public layer is obtained, and an IR corresponding to the public layer is obtained according to the first conversion function and the layer parameters of the public layer.

[0180] For each custom layer, a second conversion function corresponding to the custom layer is obtained, and an IR corresponding to the custom layer is obtained according to the second conversion function and the layer parameters of the custom layer.

[0181] S611, it is judged whether the last layer OP is executed, if yes, S613 is executed, and if not, S612 is executed.

[0182] S612, the next OP is switched, and S607 is executed.

[0183] For all OPs in the first model, the above processing is required, until all OPs are processed.

[0184] S613, the IR is generated.

[0185] After performing the above processing on all OPs in the first model, the first model can be converted into the corresponding IR. That is, the IR of the first model is generated based on the IR corresponding to the common layer and the IR corresponding to the custom layer.

[0186] Specifically, the connection relationships between each common layer and each custom layer in the first model can be obtained, and then the IR of the first model can be generated based on the connection relationships, the IR corresponding to the common layer and the IR corresponding to the custom layer.

[0187] It should be noted that the acquisition of the custom layer list and the capture of layer parameters for each layer were both obtained during the forward inference process of the first model. Figure 5 The example only illustrates the steps to obtain a custom layer list; therefore, it does not explain the process of obtaining layer parameters through capture functions. Figure 6 The example shows the complete steps for converting the first model into the corresponding IR.

[0188] After converting the first model into the corresponding IR (Integrated Resource Identifier), the IR can call the corresponding layer generation code of the target framework based on the target framework code. This process is repeated layer by layer, adding the corresponding target framework layer generation code for each layer to generate the generation code for the second deep learning framework. Finally, running the generation code and the corresponding model parameters under the second deep learning framework generates a second, recognizable model within that framework.

[0189] If a client supports a second deep learning framework and needs to implement the first function of the first model, the client can send a request command to request a second model with the first function. The first function can be either object detection or object recognition. The server can then send the second model to the client. Since the second model has the same function as the first model, only the corresponding deep learning framework is different, the client can successfully load the second model and implement its first function.

[0190] Figure 7 A schematic diagram of the model conversion provided in the embodiments of this application, such as Figure 7 As shown, it includes a client 71 and a server 72. The server 72 supports the Caffe deep learning framework, while the client 71 supports the PyTorch deep learning framework.

[0191] Server 72 includes a first model recognizable by the Caffe framework. This first model possesses a first function, which is an object detection function and / or an object recognition function. The object detection function may include, for example, face detection, vehicle detection, etc., and the object recognition function may include, for example, face recognition, vehicle recognition, text recognition, speech recognition, image recognition, action recognition, etc.

[0192] Taking the first function as an example of face recognition, the client 71 needs to obtain and load a face recognition model, and the client 71 can perform face recognition after the face recognition model is loaded on the client 71. The first model on the server 72 is a face recognition model, so the client 71 can send a request instruction to the server 72, and the request instruction is used to request a second model with a face recognition function.

[0193] After the server 72 receives the request instruction, because the deep learning frameworks supported by the server 72 and the client 71 are different, the server 72 does not directly send the first model to the client 71, but first converts the first model under the Caffe framework into a second model under the Pytorch framework. The functions of the first model and the second model are both face recognition, and the effects achieved are the same. Then, the server 72 sends the second model to the client 71. Because the second model is a recognizable model under the Pytorch framework, the client 71 can successfully load the second model and achieve the function of face recognition.

[0194] The deep learning model conversion method provided by the embodiment of the application first processes the network structure code of the first model according to the function decorator and the capturing function, obtains the public layer, the custom layer, and the layer parameters of each layer in the first model, wherein the first model is a recognizable model under a first deep learning framework, and then obtains the IR of the first model according to the public layer, the custom layer, and the layer parameters of each layer. Because the IR is a unified representation format for model conversion under multiple deep learning frameworks, the second model recognizable under a second deep learning framework can be generated according to the IR of the first model. The scheme of the embodiment of the application can recognize the custom layer in the first model through the function decorator, and convert each custom layer as a whole functional layer, thereby avoiding converting a custom layer into an independent operator, so as to realize correct conversion of the first model. Further, the custom layer can also be standardized and adjusted, and the parameters of the custom layer can be correctly extracted through the capturing function without retraining the model, thereby reducing the workload of model conversion.

[0195] Figure 8 A structural schematic diagram of a deep learning model conversion device provided by the embodiment of the application is shown in FIG. 1, which includes: Figure 8

[0196] The obtaining module 81 is configured to process the network structure code of the first model according to the function decorator and the capturing function, obtain the public layer, the custom layer, and the layer parameters of each layer in the first model, and the first model is a recognizable model under a first deep learning framework.​

[0197] a processing module 82, configured to acquire an IR of the first model according to the common layer, the custom layer, and the layer parameters of each layer, the IR being a unified representation format for model conversion under a plurality of deep learning frameworks;

[0198] a conversion module 83, configured to generate a second model identifiable under a second deep learning framework according to the IR of the first model.

[0199] In a possible implementation, the acquisition module 81 is specifically configured to:

[0200] execute the first model, and perform a relocation process on operators in the first model according to the function decorator when the first model is executed, to acquire types of each layer in the first model, wherein each layer includes at least one operator;

[0201] determine the common layer and the custom layer in the each layer according to the types of the each layer;

[0202] acquire layer parameters of each of the custom layers and each of the common layers according to the capture function, the layer parameters including hyperparameters and learning parameters.

[0203] In a possible implementation, the acquisition module 81 is specifically configured to:

[0204] acquire a preset layer list;

[0205] determine the common layer and the custom layer in the each layer according to the preset layer list and the types of the each layer.

[0206] In a possible implementation, the acquisition module 81 is specifically configured to:

[0207] for any one of the each layer, if it is determined according to the type of the layer that the layer belongs to a layer in the preset layer list, the layer is determined as the common layer;

[0208] if it is determined according to the type of the layer that the layer does not belong to a layer in the preset layer list, the layer is determined as the custom layer.

[0209] In a possible implementation, the acquisition module 81 is further configured to, before acquiring the common layer, the custom layer, and the layer parameters of each layer in the first model by processing network structure code of the first model according to the function decorator and the capture function:

[0210] determine that operators included in each of the custom layers are located in a same function.

[0211] In one possible implementation, the processing module 82 is specifically used for:

[0212] For each common layer, obtain the first transformation function corresponding to the common layer, and obtain the IR corresponding to the common layer based on the first transformation function and the layer parameters of the common layer;

[0213] For each of the custom layers, obtain the second transformation function corresponding to the custom layer, and obtain the IR corresponding to the custom layer based on the second transformation function and the layer parameters of the custom layer;

[0214] The IR of the first model is obtained based on the IR corresponding to the common layer and the IR corresponding to the custom layer.

[0215] In one possible implementation, the processing module 82 is specifically used for:

[0216] Obtain the connection relationships between each of the common layers and each of the custom layers in the first model;

[0217] Based on the connection relationship, the IR corresponding to the common layer, and the IR corresponding to the custom layer, the IR of the first model is obtained.

[0218] In one possible implementation, a transceiver module is further included, the transceiver module being used for:

[0219] The system receives a request instruction sent by a client, the request instruction being used to request a second model having a first function, the first function being the function of the first model, wherein the first function is a target detection function and / or a target recognition function;

[0220] The second model is sent to the client.

[0221] The apparatus provided in this application embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effects are similar, and will not be repeated here.

[0222] Figure 9 This is a schematic diagram of the structure of the deep learning model conversion system provided in the embodiments of this application, as shown below. Figure 9 As shown, it includes a first processor 91 and a second processor 92, wherein:

[0223] The first processor 91 is any one of a central processing unit (CPU), a graphics processing unit (GPU) and an image signal processor (ISP), the second processor 92 is any one of a CPU, a GPU and an ISP, and the first processor 91 is different from the second processor 92.

[0224] The first processor 91 is configured to send a first model identifiable under a first deep learning framework to the second processor 92, and the function of the first model is target detection and / or target recognition.

[0225] The second processor 92 is configured to convert the first model identifiable under the first deep learning framework into a second model identifiable under a second deep learning framework according to the method in the above embodiments, and the function of the second model is the same as that of the first model.

[0226] The present application also provides a computer program product, which comprises a computer program stored in a readable storage medium, and at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to enable the electronic device to execute the scheme provided in any one of the above embodiments.

[0227] Figure 10 The hardware structure diagram of the deep learning model conversion device provided in the embodiments of the present application is shown in FIG. 1. Figure 10 As shown in FIG. 1, the deep learning model conversion device comprises at least one processor 101 and a memory 102. The processor 101 and the memory 102 are connected through a bus 103.

[0228] Optionally, the model determination further comprises a communication component. For example, the communication component can comprise a receiver and / or a transmitter.

[0229] In the specific implementation process, the at least one processor 101 executes the computer execution instructions stored in the memory 102, so that the at least one processor 101 executes the deep learning model conversion method as described above.

[0230] The specific implementation process of the processor 101 can refer to the above method embodiments, which have similar implementation principles and technical effects, and will not be described here in detail.

[0231] In the above Figure 10In the illustrated embodiment, it is to be understood that the processor can be a central processing unit (CPU), and can also be other general purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the methods disclosed in conjunction with the application can be directly embodied for execution by a hardware processor, or be executed by a combination of hardware and software modules in the processor.

[0232] The memory can include a high-speed RAM memory, and can also include a non-volatile storage NVM, such as at least one disk memory.

[0233] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0234] The present application also provides a computer readable storage medium, the computer readable storage medium stores computer execution instructions, when the processor executes the computer execution instructions, the deep learning model conversion method as described above is realized.

[0235] The computer readable storage medium described above, the readable storage medium can be realized by any type of volatile or non-volatile storage device or their combination, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0236] An example readable storage medium is coupled to the processor such that the processor can read information from the readable storage medium and can write information to the readable storage medium. Of course, the readable storage medium can also be a part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0237] The division of the units is only a logical function division, and in actual implementation, another division manner can be used, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0238] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0239] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0240] If the functions are realized in the form of software function units and sold or used as independent products, the functions can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0241] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The foregoing program can be stored in a computer readable storage medium. The program executes to perform the steps of the above-mentioned method embodiments; and the foregoing storage medium includes various storage media that can store program codes, such as ROM, RAM, magnetic disk or optical disk.

[0242] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A deep learning model conversion method, characterized in that, The method comprises the following steps: processing network structure code of a first model according to a function decorator and a capture function, to obtain public layers, custom layers, and layer parameters of each layer in the first model, the first model being a model recognizable under a first deep learning framework; the first model being a face detection model or a vehicle detection model; for each public layer, obtaining a first conversion function corresponding to the public layer, and obtaining IR corresponding to the public layer according to the first conversion function and the layer parameters of the public layer; for each custom layer, obtaining a second conversion function corresponding to the custom layer, and obtaining IR corresponding to the custom layer according to the second conversion function and the layer parameters of the custom layer; generating an intermediate representation format IR of the first model according to the IR corresponding to the public layer and the IR corresponding to the custom layer, the IR being a unified representation format for model conversion under multiple deep learning frameworks; generating a second model recognizable under a second deep learning framework according to the IR of the first model; the second model has the same function as the first model.

2. The method of claim 1, wherein, The method comprises the following steps: processing network structure code of a first model according to a function decorator and a capture function, to obtain public layers, custom layers, and layer parameters of each layer in the first model, the first model being a model recognizable under a first deep learning framework; the first model being a face detection model or a vehicle detection model; executing the first model, and repositioning operators in the first model according to the function decorator when the first model is executed, to obtain the types of each layer in the first model, wherein any one layer includes at least one operator; determining the public layers and the custom layers in the layers according to the types of the layers; 3. The method of claim 2, wherein, obtaining layer parameters of each custom layer and each public layer according to the capture function, the layer parameters including hyperparameters and learning parameters. The method comprises the following steps: obtaining a preset layer list; 4. The method of claim 3, wherein, determining the public layers and the custom layers in the layers according to the preset layer list and the types of the layers. The method comprises the following steps: for any one layer in the layers, if it is determined according to the type of the layer that the layer belongs to a layer in the preset layer list, the layer is determined to be the public layer; 5. The method according to any one of claims 1 to 4, characterized in that, if it is determined according to the type of the layer that the layer does not belong to a layer in the preset layer list, the layer is determined to be the custom layer. Before processing network structure code of a first model according to a function decorator and a capture function, to obtain public layers, custom layers, and layer parameters of each layer in the first model, the method further comprises the following steps:

6. The method according to any one of claims 1 to 4, characterized in that, determining that operators included in each custom layer are located in the same function. The method comprises the following steps: obtaining a connection relationship between each public layer and each custom layer in the first model; According to the connection relationship, the IR corresponding to the public layer, and the IR corresponding to the custom layer, an IR of the first model is generated.

7. The method according to any one of claims 1 to 4, characterized in that, The method further includes: receiving a request instruction sent by a client, the request instruction being used to request the second model having a first function, the first function being a function of the first model, wherein the first function is a target detection function and / or a target recognition function; sending the second model to the client.

8. A deep learning model conversion system, comprising: comprising a first processor and a second processor, wherein: the first processor is any one of a central processing unit (CPU), a graphics processing unit (GPU), and an image signal processor (ISP), the second processor is any one of a CPU, a GPU, and an ISP, and the first processor is different from the second processor; the first processor is configured to send a first model identifiable under a first deep learning framework to the second processor, the function of the first model being a target detection function and / or a target recognition function; the second processor is configured to convert the first model identifiable under the first deep learning framework into a second model identifiable under a second deep learning framework according to the method of any one of claims 1-7, the function of the second model being the same as that of the first model. 9.A deep learning model conversion apparatus, characterized by, comprising: a obtaining module configured to process network structure code of a first model according to a function decorator and a capturing function, to obtain public layers, custom layers, and layer parameters of each layer in the first model, the first model being a model identifiable under a first deep learning framework; the first model being a face detection model or a vehicle detection model; a processing module configured to, for each public layer, obtain a first conversion function corresponding to the public layer, and obtain an IR corresponding to the public layer according to the first conversion function and the layer parameters of the public layer; for each custom layer, obtain a second conversion function corresponding to the custom layer, and obtain an IR corresponding to the custom layer according to the second conversion function and the layer parameters of the custom layer; generate an IR of the first model according to the IR corresponding to the public layer and the IR corresponding to the custom layer, the IR being a unified representation format for model conversion under multiple deep learning frameworks; a conversion module configured to generate a second model identifiable under a second deep learning framework according to the IR of the first model; the second model having the same function as the first model. 10.A device for converting a deep learning model, the device comprising: comprising: a memory configured to store a program; a processor configured to execute the program stored in the memory, when the program is executed, the processor is configured to execute the deep learning model conversion method of any one of claims 1-7.

11. A computer readable storage medium, characterized in that, instructions that, when executed on a computer, cause the computer to perform the deep learning model conversion method of any one of claims 1-7.

Images