Model deployment methods, devices, storage media and electronic devices

Abstract

This application provides a model deployment method, apparatus, storage medium, and electronic device. The method includes: acquiring an original training model, wherein the original training model is a neural network model to be deployed; converting the original training model into a first model file and a second model file, wherein the first model file needs to call a preset operator library to complete operator operations during inference, while the second model file does not need to call the preset operator library to complete operator operations during inference; packaging the first model file and the second model file into a target model file, wherein the target model file has the same format as the first model file; deploying the target model according to the target model file, performing inference, and outputting the target inference result. This application solves the problem of low model deployment efficiency in related technologies.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to a model deployment method, apparatus, storage medium, and electronic device. Background Technology

[0002] Neural network model deployment consists of two stages: model conversion and model inference. The model conversion stage transforms the trained model into an inference model, while model inference is performed on the deployment device by running operators to complete the inference task.

[0003] Currently, when deploying neural network models, related technologies either adopt a client / server service structure or a multi-process mode. However, this is not feasible on embedded devices due to limited resources. Alternatively, the model deployment method may only support the deployment of models within a single framework, or the deployment scheme may not consider all factors, making it impossible to meet the needs of complex network environments.

[0004] There is currently no effective solution to the problem of low model deployment efficiency in related technologies. Summary of the Invention

[0005] This application provides a model deployment method, apparatus, storage medium, and electronic device to at least address the problem of low model deployment efficiency in related technologies.

[0006] According to one embodiment of this application, a model deployment method is provided, comprising: obtaining an original training model, wherein the original training model is a neural network model to be deployed; converting the original training model into a first model file and a second model file, wherein the first model file needs to call a preset operator library to complete operator operations during inference, while the second model file does not need to call the preset operator library to complete operator operations during inference; packaging the first model file and the second model file into a target model file, wherein the target model file has the same format as the first model file; deploying the target model according to the target model file, performing inference, and outputting the target inference result.

[0007] According to another embodiment of this application, a model deployment apparatus is provided, comprising:

[0008] The system comprises the following modules: an acquisition module for acquiring the original training model, wherein the original training model is a neural network model to be deployed; a first processing module for converting the original training model into a first model file and a second model file, wherein the first model file needs to call a preset operator library to complete operator operations during inference, while the second model file does not need to call the preset operator library to complete operator operations during inference; a second processing module for packaging the first model file and the second model file into a target model file, wherein the target model file has the same format as the first model file; and a third processing module for deploying the target model according to the target model file, performing inference, and outputting the target inference result.

[0009] Optionally, the apparatus is further configured to: parse the original training model, determine a set of performance parameters for each network layer running in a first inference framework and a second inference framework, wherein the first inference framework is used to infer the first model file, the second inference framework is used to infer the second model file, and the set of performance parameters includes the performance parameters of each network layer; generate the first model file and the second model file according to the set of performance parameters, wherein the first model file includes N network layers of the original training model, the second model file includes M network layers of the original training model, and the original training model includes N+M network layers, where N and M are both positive integers.

[0010] Optionally, the apparatus is further configured to: determine a first performance parameter and a second performance parameter of the target network layer according to the set of performance parameters, wherein the first performance parameter represents the performance parameter for selecting the target network layer to use the first inference framework, and the second performance parameter represents the performance parameter for selecting the target network layer to use the second inference framework, and the original training model includes the target network layer; generate a first model file and a second model file according to the first performance parameter and the second performance parameter, wherein, if the first performance parameter and the second performance parameter satisfy a first preset condition, the target network layer belongs to the N network layers, and if the first performance parameter and the second performance parameter satisfy a second preset condition, the target network layer belongs to the M network layers.

[0011] Optionally, the apparatus is further configured to: parse the target model file using the first inference framework to obtain the first model file and the second model file; initialize the model based on the first model file using the first inference framework to obtain a target model, wherein the target model needs to call the preset operator library during the inference process; load the second model file using the second inference framework to determine a target dynamic library, wherein the target dynamic library includes operators to be used during the inference process using the second model file.

[0012] Optionally, the apparatus is further configured to: upon receiving initial data to be inferred, input the initial data into a first network layer to obtain a first inference result, and input the first inference result into a second network layer to obtain the target inference result, wherein the first network layer is used to call the target model and the preset operator library to complete the inference, and the second network layer is used to call the target dynamic library to complete the inference; or upon receiving initial data to be inferred, input the initial data into a second network layer to obtain a second inference result, and input the second inference result into a first network layer to obtain the target inference result, wherein the first network layer is used to call the target model and the preset operator library to complete the inference, and the second network layer is used to call the target dynamic library to complete the inference.

[0013] Optionally, the apparatus is further configured to: parse the first model file using the first inference framework to obtain a first data pointer and a first data length; and initialize the model using the first data pointer and the first data length using the first inference framework to obtain the target model.

[0014] Optionally, the apparatus is further configured to: preload a memory file system, wherein the memory file system is used to determine the target dynamic library; parse the second model file using the first inference framework to obtain a second data pointer and a second data length; use the second inference framework to concatenate the second data pointer and the second data length to obtain a dynamic library path; and determine the target dynamic library in the memory file system through the dynamic library path.

[0015] According to yet another embodiment of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0016] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0017] This application provides a method for model deployment that involves obtaining an original training model, converting it into a first model file and a second model file, packaging the first and second model files into a target model file, deploying the target model based on the target model file, performing inference, and outputting the target inference result. This method solves the problem of low model deployment efficiency in related technologies, thereby improving model deployment efficiency, simplifying the model deployment method, and enhancing model inference performance. Attached Figure Description

[0018] Figure 1 This is a hardware structure block diagram of a mobile terminal for a model deployment method according to an embodiment of this application;

[0019] Figure 2 This is a flowchart of a model deployment method according to an embodiment of this application;

[0020] Figure 3 This is a flowchart of another model deployment method according to an embodiment of this application;

[0021] Figure 4 This is a flowchart of another model deployment method according to an embodiment of this application;

[0022] Figure 5 This is a flowchart of another model deployment method according to an embodiment of this application;

[0023] Figure 6 This is a flowchart of another model deployment method according to an embodiment of this application;

[0024] Figure 7 This is a structural block diagram of a model deployment device according to an embodiment of this application. Detailed Implementation

[0025] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0027] The methods and embodiments provided in this application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a model deployment method according to an embodiment of this application. For example... Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0028] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the model deployment method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thus implementing the aforementioned method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0029] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0030] This embodiment provides a model deployment method. Figure 2 This is a flowchart of a model deployment method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0031] S202, Obtain the original training model, where the original training model is the neural network model to be deployed;

[0032] Optionally, in this embodiment, the original training model may include, but is not limited to, feedforward neural network models such as convolutional neural networks (CNN), fully connected neural networks (FCN), and generative adversarial networks (GAN) that have not yet been deployed, and feedback neural network models such as recurrent neural networks (RNN), long short-term memory networks (LSTM), Hopfield networks, and Boltzmann machines that have not yet been deployed.

[0033] S204, convert the original training model into a first model file and a second model file. The first model file needs to call the preset operator library to complete operator operations during inference, while the second model file does not need to call the preset operator library to complete operator operations during inference.

[0034] Optionally, in this embodiment, the first model file may include, but is not limited to, a file containing only network structure and weight data. During network inference, the first model file relies on the inference library to parse the model and calls the operator library to complete operator operations. The second model file may include, but is not limited to, a model file that contains not only network structure and weight data, but also the operator implementation required for network operation. The second model file is usually released in the form of a dynamic library. During network inference, it already has operator implementations, so it does not need to rely on the operator library to complete operator operations.

[0035] S206, Package the first model file and the second model file into a target model file, wherein the target model file has the same format as the first model file;

[0036] Optionally, in this embodiment, the target model file may include, but is not limited to, a model file obtained by packaging the first model file and the second model file.

[0037] It should be noted that the above target model file has the same format as the above first model file, but the above target model file has a different format than the above second model file.

[0038] S208: Deploy the target model based on the target model file, perform inference, and output the target inference result.

[0039] This application's embodiments employ a method that first converts the original training model into a first model file and a second model file, then packages the first and second model files into a target model file, and finally deploys the target model based on the target model file for inference. This yields the target inference result. By merging the first and second model files, the goal of maintaining a single model file during model deployment is achieved, solving the problem of low model deployment efficiency in related technologies. This results in improved model deployment efficiency, simplified model deployment methods, and enhanced model inference performance.

[0040] In an exemplary embodiment, converting the original training model into a first model file and a second model file includes: parsing the original training model and determining a set of performance parameters for each network layer running in a first inference framework and a second inference framework, wherein the first inference framework is used to infer the first model file, the second inference framework is used to infer the second model file, and the set of performance parameters includes the performance parameters of each network layer; generating the first model file and the second model file based on the set of performance parameters, wherein the first model file includes N network layers of the original training model, the second model file includes M network layers of the original training model, and the original training model includes N+M network layers, where N and M are both positive integers.

[0041] Optionally, in this embodiment, the first inference framework may include, but is not limited to, an inference framework in which the model file only contains network structure and weight data, and the second inference framework may include, but is not limited to, an inference framework in which the model file not only contains network structure and weight data, but also contains the operator implementation required for the operation of this network.

[0042] It should be noted that the mainstream reasoning frameworks currently include, but are not limited to:

[0043] (1) TensorRT: TensorRT has advantages that other frameworks do not have when it comes to NVIDIA series graphics cards. If it runs on an NVIDIA graphics card, TensorRT is generally the fastest inference framework. Most mainstream training frameworks such as TensorFlow and PyTorch can be converted into TensorRT-runnable models.

[0044] (2) ONNXRuntime is an inference framework that can run on multiple platforms. It accepts model input in ONNX format and supports inference on both GPU and CPU. The only drawback is that ONNX nodes are fine-grained, and the inference speed is sometimes lower than other inference frameworks.

[0045] (3) OpenVINO is an inference framework from Intel that is friendly to Intel CPUs and GPUs. It can also interface with different training frameworks.

[0046] (4) Unlike the three inference frameworks mentioned above, NCNN and MNN are designed for mobile deployment. NCNN's advantage is that it was open source earlier and has a very stable community.

[0047] Optionally, in this embodiment, the above-mentioned performance parameters may include, but are not limited to, parameters that can be used to measure the effect of the inference framework, such as running speed and running time, and the above-mentioned set of performance parameters may include, but are not limited to, a set of running time parameters for each network layer.

[0048] It should be noted that the above-mentioned generation of the first model file and the second model file based on the set of performance parameters can be understood as follows: the model tool that selects the network layer using the first inference framework can generate the first model file based on the set of performance parameters, and the model tool that selects the network layer using the second inference framework can generate the second model file.

[0049] In an exemplary embodiment, generating a first model file and a second model file based on a set of performance parameters includes: determining a first performance parameter and a second performance parameter of a target network layer based on the set of performance parameters, wherein the first performance parameter represents the performance parameter for selecting a first inference framework for the target network layer, and the second performance parameter represents the performance parameter for selecting a second inference framework for the target network layer, and the original training model includes the target network layer; generating a first model file and a second model file based on the first performance parameter and the second performance parameter, wherein if the first performance parameter and the second performance parameter satisfy a first preset condition, the target network layer belongs to N network layers, and if the first performance parameter and the second performance parameter satisfy a second preset condition, the target network layer belongs to M network layers.

[0050] Optionally, in this embodiment, the first performance parameter may include, but is not limited to, the runtime of the target network layer selecting the first inference framework, and the second performance parameter may include, but is not limited to, the runtime of the target network layer selecting the second inference framework.

[0051] Optionally, in this embodiment, the first preset condition may include, but is not limited to, conditions set in advance by relevant technical personnel based on prior experience. For example, when the performance parameter is runtime, the first preset condition may be set as follows: the first performance parameter is less than the second performance parameter. When the runtime of the target network layer using the first inference framework is less than the runtime of the target network layer using the second inference framework, the first preset condition is met, and the model file is generated using the model tool of the first inference framework. The target network layer belongs to N network layers.

[0052] Optionally, in this embodiment, the above-mentioned second preset condition may include, but is not limited to, conditions set in advance by relevant technical personnel based on prior experience. It should be noted that, for example, when the performance parameter is the running time, if the second preset condition is that the first performance parameter is greater than the second performance parameter, and the first performance parameter and the second performance parameter satisfy the second preset condition, the target network level belongs to M network levels.

[0053] It should be noted that the above method of selecting which inference framework to use to generate the model file based on whether the first performance parameter is greater than the second performance parameter meets specific preset conditions has the benefit of improving the model's inference performance. This model can be applied to scenarios with high requirements for model inference performance, including but not limited to:

[0054] (1) In the field of public safety, real-time facial recognition in video surveillance needs to have real-time display capabilities to facilitate tracking and locating relevant personnel.

[0055] (2) In the field of Internet applications, e-commerce websites and content applications require real-time personalized recommendations that can respond quickly. The lag in recommendations will directly affect the shopping or content acquisition experience.

[0056] (3) In the banking sector, real-time identification of abnormal transactions in electronic payments is also crucial. Any abnormal transaction needs to be quickly identified and intercepted, while normal transactions should not be affected.

[0057] (4) In the financial field, the millisecond-level trading judgment output of quantitative models is crucial for traders on Wall Street.

[0058] This application's embodiments employ a method that determines the first and second performance parameters of the target network layer based on a set of performance parameters, and then generates a first and second model file based on these parameters. Each network layer can choose to use a model tool from a first inference framework to generate the first model file, or it can choose to use a model tool from a second inference framework to generate the second model file, thus achieving the technical effect of improving model inference performance.

[0059] In an exemplary embodiment, before deploying the target model according to the target model file, performing inference, and outputting the target inference result, the method further includes: parsing the target model file using a first inference framework to obtain a first model file and a second model file; using the first inference framework to initialize the model based on the first model file to obtain the target model, wherein the target model needs to call a preset operator library during the inference process; and using a second inference framework to load the second model file and determine the target dynamic library, wherein the target dynamic library includes operators to be used during the inference process using the second model file.

[0060] Optionally, in this embodiment, the target model may include, but is not limited to, the model obtained after initialization using the first inference framework based on the first model file.

[0061] Optionally, in this embodiment, the above-mentioned operator may include, but is not limited to, a computing unit. Deep learning algorithms are composed of computing units, which are operators. In the network model, the operator corresponds to the computing logic in the layer. For example, a convolutional layer is an operator, and the weight summation process in a fully connected layer is also an operator. Each operator in the network implements the matching logic according to the operator type. There may be many operators of the same type in a network.

[0062] Optionally, in this embodiment, the aforementioned preset operator library may include, but is not limited to, a collection of multiple different types of computing units. During the reasoning process of the target model, the type of computing unit needed can be directly called from the preset operator library.

[0063] It should be noted that the model itself does not contain the operators required for the inference process, so the target model needs to call the preset operator library to complete the inference process.

[0064] Optionally, in this embodiment, the target dynamic library may include, but is not limited to, operators used in the inference process using the second model file. Therefore, the target dynamic library does not need to call the operator library during the inference process.

[0065] In this embodiment, a first inference framework is used to parse the target model file, resulting in a first model file and a second model file. The first inference framework is then used to initialize the model based on the first model file, yielding the target model. Finally, a second inference framework is used to load the second model file, determining the method for loading the target dynamic library. This allows for model initialization, preparing the model for subsequent deployment.

[0066] In an exemplary embodiment, the method further includes: upon receiving initial data to be inferred, inputting the initial data into a first network layer to obtain a first inference result, and inputting the first inference result into a second network layer to obtain a target inference result, wherein the first network layer is used to call a target model and a preset operator library to complete inference, and the second network layer is used to call a target dynamic library to complete inference; or upon receiving initial data to be inferred, inputting the initial data into a second network layer to obtain a second inference result, and inputting the second inference result into a first network layer to obtain a target inference result, wherein the first network layer is used to call a target model and a preset operator library to complete inference, and the second network layer is used to call a target dynamic library to complete inference.

[0067] Optionally, in this embodiment, the initial data mentioned above may include, but is not limited to, data in the form of images, numbers, voice, etc.

[0068] Optionally, in this embodiment, the first network layer may include, but is not limited to, a network layer that completes inference by calling the target model and a preset operator library, and the inference result it outputs is the first inference result or the target network result.

[0069] Optionally, in this embodiment, the second network layer may include, but is not limited to, a network layer that can complete inference by calling a dynamic library without calling a preset operator library, and its output result is the second inference result or the target inference result.

[0070] It should be noted that the model inference process may include, but is not limited to, multiple network layers. The first network layer mentioned above can be a network layer located in the first layer of the model inference, and the second network layer can be a network layer located in the second layer of the model. In this case, when performing model inference, the first network layer receives initial data and obtains the first inference result, which is then input into the second network layer to obtain the target inference result. Alternatively, the second network layer can also be a network layer located in the first layer of the model inference, and the first network layer can be a network layer located in the second layer of the model inference. In this case, the initial data is input into the second network layer, and the result is the second inference result, which is then input into the first network layer to obtain the target inference result.

[0071] For example, an image can be input as initial data into the first network layer. The first network layer then calls the target model and a preset operator library to perform inference and obtain the first inference result. The first inference result is then input into the second network layer, which calls the target dynamic library to complete the inference and obtain the target inference result.

[0072] Through the embodiments of this application, the method involves, upon receiving initial data to be inferred, inputting the initial data into a first network layer to obtain a first inference result, and then inputting the first inference result into a second network layer to obtain a target inference result; or, upon receiving initial data to be inferred, inputting the initial data into a second network layer to obtain a second inference result, and then inputting the second inference result into a first network layer to obtain a target inference result. Model inference can be used to perform inference at two network layers based on the initial data to be inferred to obtain the target inference result.

[0073] In an exemplary embodiment, initializing a model based on a first model file using a first inference framework to obtain a target model includes: parsing the first model file using the first inference framework to obtain a first data pointer and a first data length; and initializing the model using the first data pointer and the first data length using the first inference framework to obtain the target model.

[0074] Optionally, in this embodiment, the aforementioned first data pointer may include, but is not limited to, a pointer that can be used to indicate the data address of the first model file. The aforementioned first data may include, but is not limited to, data that can be used to represent the data size of the first model file. The aforementioned parsing of the first model file using the first inference framework to obtain the first data pointer and the first data length can be understood as obtaining the data address and data length of the first model file after parsing the first model file.

[0075] Through the embodiments of this application, the first model file is parsed using the first inference framework to obtain the first data pointer and the first data length. Then, the first inference framework is used to initialize the model using the first data pointer and the first data length to obtain the target model. The first model file can be parsed and initialized to obtain the target model.

[0076] In an exemplary embodiment, loading a second model file using a second inference framework to determine a target dynamic library includes: preloading a memory file system, wherein the memory file system is used to determine the target dynamic library; parsing the second model file using a first inference framework to obtain a second data pointer and a second data length; concatenating the second data pointer and the second data length using the second inference framework to obtain a dynamic library path; and determining the target dynamic library in the memory file system using the dynamic library path.

[0077] Optionally, in this embodiment, the aforementioned memory file system may include, but is not limited to, memfs, whose file data is stored in memory. When the system starts, it executes `insmod memfs.ko` (insmod is the driver installation command for the Linux operating system, and memfs.ko is the driver file), and this file system is registered to the path ` / memfs`. The filename (excluding the extension) must be in the format: `<memory address>_<data length>`, such as `0xffe12899_1341221.so`, `0xffe12899_1341221`, or `0xffe12899_1341221.txt`, all of which are valid formats.

[0078] It should be noted that the aforementioned memory file system is loaded during device initialization but before model initialization, and the memory file system is used to determine the target dynamic library.

[0079] Optionally, in this embodiment, the second data pointer may include, but is not limited to, a data pointer of the form addr_B, which can be used to represent the data address of the second model file, and the second data length may include, but is not limited to, a data pointer used to represent the data size of the second model file.

[0080] Optionally, in this embodiment, the above-mentioned use of the second inference framework to concatenate the second data pointer and the second data length to obtain the dynamic library path can be understood as follows: for example, when the second data pointer is 0xffe12899 and the second data length is 1341221, the dynamic library path is obtained by concatenating the second data pointer and the second data length using the second inference framework: / memfs / 0xffe12899_1341221.so, thereby determining the target dynamic library in the memory file system through the dynamic library path.

[0081] This application's embodiments employ a method of pre-loading a memory file system, then using a first inference framework to parse the second model file to obtain a second data pointer and a second data length. The second inference framework then concatenates the second data pointer and the second data length to obtain the dynamic library path. The target dynamic library is then determined within the memory file system using this dynamic library path. For scenarios with limited file systems, the second model file can be directly loaded from the second data pointer via the memory file system, saving unnecessary memory usage and improving memory utilization.

[0082] Obviously, the embodiments described above are only some embodiments of this application, and not all embodiments.

[0083] The present application will be described in detail below with reference to specific embodiments:

[0084] This application process comprises two parts: model transformation and model inference. It involves two inference frameworks (inference framework A and inference framework B) and one file system (memfs). The characteristics of the inference frameworks and file system are as follows:

[0085] Inference Framework A (corresponding to the first inference framework mentioned above): The model file for this framework only contains the network structure and weight data. The model file is typically distributed as a binary (.bin) file. During network inference, the model is parsed using an inference library, and operator operations are performed by calling an operator library.

[0086] Inference Framework B (corresponding to the second inference framework mentioned above): This framework's model file contains not only the network structure and weight data, but also the operator implementations required for the network to run. The model file is typically released as a dynamic library. Since the network already has operator implementations during inference, it does not need to rely on an operator library to perform operator operations.

[0087] MemFS file system: MemFS is a memory-based file system where file data is stored in memory. During system startup, `insmod memfs.ko` registers this file system on the path ` / memfs`. Filenames (excluding extensions) must be in the format: `<memory address>_<data length>`, such as `0xffe12899_1341221.so`, `0xffe12899_1341221`, or `0xffe12899_1341221.txt` are all valid formats.

[0088] The overall process for this application is as follows:

[0089] 1. Model conversion, the overall steps are as follows: (e.g.) Figure 3 (As shown)

[0090] S302, Input the original training model;

[0091] S304, convert the original model into model_A.bin (corresponding to the first model file mentioned above) and model_B.so (corresponding to the second model file mentioned above);

[0092] S306, Package model_A.bin and model_B.so into a single model model.bin (corresponding to the target model file mentioned above);

[0093] S308, outputs the inference model model.bin.

[0094] 2. Model inference, the overall steps are as follows: (e.g.) Figure 4 (As shown)

[0095] S402, Input the inference model model.bin;

[0096] S404, during device initialization, the memory file system memfs is loaded;

[0097] S406, During model initialization, inference framework A first loads model.bin, obtaining model_A.bin and model_B.so;

[0098] S408, Inference framework A (corresponding to the first inference framework above) uses model_A.bin to perform network initialization operations; Inference framework B (corresponding to the second inference framework above) uses model_B.so to perform network initialization operations;

[0099] S410, output the reasoning result.

[0100] The model conversion process involves transforming the trained model into two inference framework model files, which are then merged and packaged into a single inference model file. The key focus here is allocating the inference execution of each layer of the trained model to inference frameworks A and B based on their performance characteristics. The specific process is as follows: Figure 5 As shown.

[0101] The inference framework B generates a dynamic library file model_B.so, which can be treated as a regular file; and the model file model_A.bin is written into the model.bin file.

[0102] During the model inference phase, the operation of each operator in the model is similar to other frameworks. The main difference in this approach lies in the model initialization phase. Initialization includes the initialization of the memory file system and the dynamic library model_B.so. The process is as follows: Figure 6 As shown.

[0103] After registering the file system memfs to the directory / memfs, when dlopen opens the dynamic library file " / memfs / 0xffe12899_1341221.so", the system calls the memfs file system to parse the file and obtain the file data address 0xffe12899 and data length 1341221. Then, dlopen completes the loading operation of the dynamic library and returns a handle. dlsym uses the handle and the operator function name to find the address of the corresponding operator. This completes the loading and initialization of the dynamic library mem_B.so.

[0104] The subsequent inference process is similar to other frameworks, involving calling operator interfaces to complete the inference task.

[0105] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0106] This embodiment also provides a model deployment apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0107] Figure 7 This is a structural block diagram of a model deployment apparatus according to an embodiment of this application, such as... Figure 6 As shown, the device includes:

[0108] The acquisition module 702 is used to acquire the original training model, wherein the original training model is a neural network model to be deployed;

[0109] The first processing module 704 is used to convert the original training model into a first model file and a second model file. The first model file needs to call a preset operator library to complete operator operations during the inference process, while the second model file does not need to call the preset operator library to complete operator operations during the inference process.

[0110] The second processing module 706 is used to package the first model file and the second model file into a target model file, wherein the target model file has the same format as the first model file;

[0111] The third processing module 708 is used to deploy the target model according to the target model file, perform inference, and output the target inference result.

[0112] In an exemplary embodiment, the above-described apparatus is further configured to: parse the original training model, determine a set of performance parameters for each network layer running in a first inference framework and a second inference framework, wherein the first inference framework is used to infer a first model file, the second inference framework is used to infer a second model file, and the set of performance parameters includes the performance parameters of each network layer; generate a first model file and a second model file based on the set of performance parameters, wherein the first model file includes N network layers of the original training model, the second model file includes M network layers of the original training model, and the original training model includes N+M network layers, where N and M are both positive integers.

[0113] In an exemplary embodiment, the above-described apparatus is further configured to: determine a first performance parameter and a second performance parameter of the target network layer based on a set of performance parameters, wherein the first performance parameter represents the performance parameter for selecting the target network layer to use a first inference framework, and the second performance parameter represents the performance parameter for selecting the target network layer to use a second inference framework, and the original training model includes the target network layer; generate a first model file and a second model file based on the first performance parameter and the second performance parameter, wherein, if the first performance parameter and the second performance parameter satisfy a first preset condition, the target network layer belongs to N network layers, and if the first performance parameter and the second performance parameter satisfy a second preset condition, the target network layer belongs to M network layers.

[0114] In an exemplary embodiment, the above apparatus is further configured to: parse the target model file using a first inference framework to obtain a first model file and a second model file; initialize the model based on the first model file using the first inference framework to obtain a target model, wherein the target model needs to call a preset operator library during the inference process; load the second model file using a second inference framework to determine a target dynamic library, wherein the target dynamic library includes operators to be used during the inference process using the second model file.

[0115] In an exemplary embodiment, the above-described apparatus is further configured to: upon receiving initial data to be inferred, input the initial data into a first network layer to obtain a first inference result, and input the first inference result into a second network layer to obtain a target inference result, wherein the first network layer is used to call the target model and a preset operator library to complete the inference, and the second network layer is used to call the target dynamic library to complete the inference; or

[0116] Upon receiving the initial data to be inferred, the initial data is input into the second network layer to obtain the second inference result. The second inference result is then input into the first network layer to obtain the target inference result. The first network layer is used to call the target model and the preset operator library to complete the inference, while the second network layer is used to call the target dynamic library to complete the inference.

[0117] In an exemplary embodiment, the above apparatus is further configured to: parse a first model file using a first inference framework to obtain a first data pointer and a first data length; and initialize a model using the first data pointer and the first data length using the first inference framework to obtain a target model.

[0118] In an exemplary embodiment, the above apparatus is further configured to: preload a memory file system, wherein the memory file system is used to determine a target dynamic library; parse a second model file using a first inference framework to obtain a second data pointer and a second data length; use the second inference framework to concatenate the second data pointer and the second data length to obtain a dynamic library path; and determine the target dynamic library in the memory file system using the dynamic library path.

[0119] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0120] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0121] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0122] Embodiments of this application also provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0123] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0124] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0125] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0126] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A model deployment method, characterized in that, include: Obtain the original training model, wherein the original training model is the neural network model to be deployed; The original training model is converted into a first model file and a second model file. The first model file needs to call the preset operator library to complete operator operations during the inference process, while the second model file does not need to call the preset operator library to complete operator operations during the inference process. The first model file and the second model file are packaged into a target model file, wherein the target model file has the same format as the first model file; Deploy the target model based on the target model file, perform inference, and output the target inference result; Before deploying the target model based on the target model file, performing inference, and outputting the target inference result, the method further includes: parsing the target model file using a first inference framework to obtain the first model file, wherein the first inference framework includes an inference framework containing network structure and weight data in the model file; initializing the model based on the first model file using the first inference framework to obtain the target model, wherein the target model needs to call the preset operator library during the inference process; loading the second model file using a second inference framework to determine the target dynamic library, wherein the second inference framework includes an inference framework containing the network structure, the weight data, and operators required for network operation in the model file, the second inference framework is used to infer the second model file, and the target dynamic library includes operators to be used during the inference process using the second model file.

2. The method according to claim 1, characterized in that, Converting the original trained model into a first model file and a second model file includes: The original training model is analyzed to determine the set of performance parameters for each network layer running in the first inference framework and the second inference framework. The first inference framework is used to infer the first model file, and the second inference framework is used to infer the second model file. The set of performance parameters includes the performance parameters of each network layer. The first model file and the second model file are generated based on the set of performance parameters. The first model file includes N network layers of the original training model, and the second model file includes M network layers of the original training model. The original training model includes N+M network layers, where N and M are both positive integers.

3. The method according to claim 2, characterized in that, The step of generating the first model file and the second model file based on the performance parameter set includes: The first performance parameter and the second performance parameter of the target network layer are determined according to the set of performance parameters, wherein the first performance parameter represents the performance parameter of the target network layer when selecting to use the first inference framework, and the second performance parameter represents the performance parameter of the target network layer when selecting to use the second inference framework, and the original training model includes the target network layer; The first model file and the second model file are generated based on the first performance parameter and the second performance parameter. When the first performance parameter and the second performance parameter meet the first preset condition, the target network layer belongs to the N network layers. When the first performance parameter and the second performance parameter meet the second preset condition, the target network layer belongs to the M network layers.

4. The method according to claim 1, characterized in that, The method further includes: Upon receiving initial data to be inferred, the initial data is input into a first network layer to obtain a first inference result. The first inference result is then input into a second network layer to obtain the target inference result. The first network layer is used to call the target model and the preset operator library to complete the inference, and the second network layer is used to call the target dynamic library to complete the inference; or Upon receiving initial data to be inferred, the initial data is input into the second network layer to obtain a second inference result. The second inference result is then input into the first network layer to obtain the target inference result. The first network layer is used to call the target model and the preset operator library to complete the inference, and the second network layer is used to call the target dynamic library to complete the inference.

5. The method according to claim 1, characterized in that, The step of initializing the model based on the first model file using the first inference framework to obtain the target model includes: The first model file is parsed using the first inference framework to obtain the first data pointer and the first data length; The target model is obtained by initializing the model using the first inference framework with the first data pointer and the first data length.

6. The method according to claim 1, characterized in that, The step of loading the second model file using the second inference framework and determining the target dynamic library includes: A memory file system is preloaded, wherein the memory file system is used to determine the target dynamic library; The second model file is parsed using the first inference framework to obtain the second data pointer and the second data length; The dynamic library path is obtained by concatenating the second data pointer and the second data length using the second inference framework. The target dynamic library is determined in the memory file system through the dynamic library path.

7. A model deployment device, characterized in that, include: An acquisition module is used to acquire the original training model, wherein the original training model is a neural network model to be deployed; The first processing module is used to convert the original training model into a first model file and a second model file. The first model file needs to call a preset operator library to complete operator operations during the inference process, while the second model file does not need to call the preset operator library to complete operator operations during the inference process. The second processing module is used to package the first model file and the second model file into a target model file, wherein the target model file has the same format as the first model file; The third processing module is used to deploy the target model according to the target model file, perform inference, and output the target inference result; Before deploying the target model according to the target model file, performing inference, and outputting the target inference result, the device is further configured to: parse the target model file using a first inference framework to obtain the first model file, wherein the first inference framework includes an inference framework containing network structure and weight data in the model file; initialize the model based on the first model file using the first inference framework to obtain the target model, wherein the target model needs to call the preset operator library during the inference process; load the second model file using a second inference framework to determine the target dynamic library, wherein the second inference framework includes an inference framework containing the network structure, the weight data, and operators required for network operation in the model file, the second inference framework is used to infer the second model file, and the target dynamic library includes operators to be used during the inference process using the second model file.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of claims 1 to 6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 6.

Images