Model deployment system, method and product

Abstract

The embodiment of the invention provides a model deployment system, method and product. The system comprises a hardware basic function module, a deep learning algorithm module and a memory manager, the hardware basic function module provides a model file loading interface and is used for generating a deep learning model instance and providing the deep learning model instance to the deep learning algorithm module for model reasoning; the deep learning algorithm module provides a model reasoning calling interface, responds to a calling request of an upper-layer application, calls a deep learning model instance to submit a complete reasoning task to a reasoning flow for model reasoning, and synchronously outputs an analysis result to a target thread after obtaining and outputting the analysis result; and the memory manager provides a memory management interface, responds to the memory request and allocates the memory from the maintained memory pool. The invention aims to improve model deployment and operation efficiency.

Description

Technical Field

[0001] The present application relates to the field of intelligent model deployment technology, and specifically to a model deployment system, method and product. Background Art

[0002] A deep learning model will be deployed on a variety of different SoCs (common SoCs generally contain CPUs, GPUs, NPUs, and some other dedicated acceleration cores) to meet the needs of different vehicle models. However, due to the lack of unified standards for inference hardware and software interfaces, the GPU and NPU hardware and software interfaces of various manufacturers are different. In the process of migrating deep learning models to different SoCs, a lot of adaptation work is often required to drive the model on the new SoC. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a model deployment system, method and product, aiming to improve the model deployment and operation efficiency.

[0004] A first aspect of an embodiment of the present application provides a model deployment system, the system comprising: the system comprising: a hardware basic function module, a deep learning algorithm module and a memory manager;

[0005] The hardware basic function module provides a model file loading interface for generating a deep learning model instance and providing it to the deep learning algorithm module for model reasoning;

[0006] The deep learning algorithm module provides a model reasoning calling interface, responds to the call request of the upper-layer application, calls the deep learning model instance to submit the complete reasoning task to the reasoning flow for model reasoning, and after obtaining the output parsing result, synchronizes the output parsing result to the target thread;

[0007] The memory manager provides a memory management interface and allocates memory from a maintained memory pool in response to a memory request.

[0008] Optionally, the hardware basic function module further includes: a memory allocation interface, a memory flushing interface, a memory copy interface and a memory pool allocation memory interface;

[0009] The memory allocation interface is used to allocate memory of a specified type and size and save it in a memory block buffer;

[0010] The memory flushing interface is used to flush a memory block buffer from one hardware to another hardware;

[0011] The memory copy interface is used to copy the memory block buffer;

[0012] The memory pool allocates a memory interface, and is used to call the memory manager to perform memory allocation.

[0013] Optionally, the deep learning algorithm module includes: a model input and output module, a preprocessing module, a deep learning model instance and an output parser;

[0014] The model input and output module is used to store a first variable for input data of the deep learning model, a second variable for output data of the deep learning model, and a numbered variable for storing data, wherein the first variable, the second variable, and the numbered variable are public variables that can be directly accessed;

[0015] The preprocessing module is used to perform preprocessing tasks on the input data of the deep learning model to obtain corresponding output tensors;

[0016] The deep learning model instance is used to perform reasoning tasks on the obtained input tensor and save the reasoning results as output tensors;

[0017] The output parser is used to perform an output parsing task on the output tensor to obtain output data.

[0018] Optionally, the deep learning model instance includes: a model initialization interface, a model destruction interface, an inference interface and a synchronization interface;

[0019] The model initialization interface is used to allocate memory for input tensors and output tensors, load configuration file information of the deep learning model, read the model file, and generate a deep learning model instance;

[0020] The model destruction interface is used to release the memory and deep learning model instance allocated by the model initialization interface when model inference is not needed;

[0021] The reasoning interface is used to submit a complete reasoning task to the reasoning flow to execute the complete reasoning task;

[0022] The synchronization interface is used to wait in a blocking manner for the inference task in the complete inference task to be completed, and to perform the next step of processing after the inference task in the complete inference task is completed.

[0023] Optionally, the output parser includes: a parser initialization interface, a parser destruction interface, a device-side processing interface, a host-side processing interface and an enable judgment interface;

[0024] The parser initialization interface and the parser destruction interface are used to allocate and release resources required by the output parser;

[0025] The device-side processing interface is used to perform output parsing on the device side;

[0026] The host-side processing interface is used to perform output parsing on the host side;

[0027] The enabling judgment interface is used by the deep learning algorithm module to obtain the enabling status of the parser.

[0028] Optionally, the model reasoning calling interface includes: an output parser registration interface, an algorithm initialization interface, a destruction interface, a preprocessing interface and a model reasoning interface;

[0029] The output parser registration interface is used to register the output parser to the deep learning algorithm module;

[0030] The algorithm initialization interface is used to read the model file of the corresponding deep learning model from the configuration file, and call the model file loading interface of the hardware basic function module to load the model file and generate a deep learning model instance, and is used to call the initialization interface of each module belonging to the deep learning algorithm module;

[0031] The destruction interface is used to release the resources allocated by the algorithm initialization interface;

[0032] The preprocessing interface is used to call the preprocessing module;

[0033] The model inference interface is used to sequentially call the preprocessing interface, the inference interface of the deep learning model instance, the device-side processing interface of the registered output parser, the synchronization interface of the deep learning model instance, and the host-side processing interface of the registered output parser to perform a complete inference task.

[0034] Optionally, the reasoning flow includes: obtaining a hardware-related reasoning flow instance interface and a reasoning flow synchronization interface;

[0035] The interface for obtaining hardware-related reasoning flow instance is used to obtain the reasoning flow instance corresponding to the current hardware, and use the reasoning flow instance to submit a complete reasoning task through the hardware-related interface implementation;

[0036] The reasoning flow synchronization interface is used to wait for the complete reasoning task on the reasoning flow to be completed, and then synchronize the obtained output parsing results to the target thread.

[0037] Optionally, the memory management interface includes: a memory request interface, an interface for obtaining the total memory size of the current memory pool, and an interface for obtaining the free memory size;

[0038] The request memory interface is used to return the corresponding memory block buffer to the corresponding module in response to the memory request including the memory type and the memory size;

[0039] The interface for obtaining the total memory size of the current memory pool and the interface for obtaining the free memory size are used for debugging and log printing.

[0040] Optionally, the hardware basic function module, deep learning model instance, reasoning flow and memory manager are first elements; the first element includes a first basic part that is independent of specific hardware and a backend part that is related to hardware, and the first basic part is defined as a base class;

[0041] The first basic part is used to provide hardware-general interfaces and hardware-independent interface implementations, and define interfaces related to the backend part as virtual functions;

[0042] The backend part is used to develop and create corresponding inheritance classes when deploying a model that supports new hardware. The inheritance classes inherit the base class of the first basic part and implement the corresponding virtual functions defined in the base class.

[0043] Optionally, the deep learning algorithm module, the model input and output module, the preprocessing module, and the output parser are the second elements; the second elements include a second basic part that is not related to a specific model and a model-related part that is related to the model;

[0044] The second basic part is used to provide a general interface and an implementation that is independent of the model, and to define the interface related to the model as a virtual function, and the second basic part is defined as a base class;

[0045] The model-related part is used to develop and create a corresponding inheritance class when deploying a new model. The inheritance class inherits the base class of the second basic part and implements the corresponding virtual interface defined in the base class.

[0046] Optionally, the system also includes: a unified log interface, which is used to connect to the log interface of the upper-layer application and provide it to the deep learning algorithm module for log recording and printing.

[0047] A second aspect of an embodiment of the present application provides a model deployment method, the method comprising:

[0048] Get the model file and configuration file of the model to be deployed;

[0049] Based on the model deployment system described in the first aspect of the present application, model deployment is performed on the model file and the configuration file;

[0050] Execute the corresponding complete reasoning task based on the deployed deep learning model, and send the output data obtained by reasoning to the upper-level application.

[0051] A third aspect of an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and running on the processor, wherein when the computer program is executed by the processor, the steps in a model deployment method provided in the second aspect of the present application are implemented.

[0052] The fourth aspect of an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in a model deployment method provided in the second aspect of the present application are implemented.

[0053] The model deployment system provided by this application has the following advantages:

[0054] A model deployment system provided by an embodiment of the present application includes a hardware basic function module, a deep learning algorithm module and a memory manager; the hardware basic function module provides a model file loading interface for generating a deep learning model instance, which is provided to the deep learning algorithm module for model reasoning; the deep learning algorithm module provides a model reasoning call interface, responds to the call request of the upper application, calls the deep learning model instance to submit the complete reasoning task to the reasoning flow for model reasoning, and after obtaining the output parsing result, synchronously outputs the parsing result to the target thread; the memory manager provides a memory management interface, responds to the memory request, and allocates memory from the maintained memory pool. Model deployment through this system can improve deployment efficiency. At the same time, this application introduces a memory manager, which provides a memory pool interface to the outside through the memory manager. When each element in the deployment system needs to use memory, it directly applies to the memory pool. When there is a memory block of the right size in the memory pool, the memory pool will assign the pointer of the memory in the memory pool to the element applying for memory, which can reduce frequent memory allocation and release. Unified memory management by the memory pool can also largely avoid waste caused by independent memory application by each element. At the same time, this application introduces an inference flow, submits the preprocessing tasks, inference tasks and result parsing tasks to the inference flow for processing, and performs synchronization only once the result parsing is completed. This can ensure to the greatest extent that the tasks on the inference flow run independently of the CPU thread, reduce the interaction between the inference flow and the CPU, and thus reduce the model reasoning's dependence on the CPU. When the CPU resources on the embedded end are limited and the CPU load is extremely high, this method can reduce the difficulty of CPU scheduling and improve the stability of delay. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor.

[0056] Figure 1 A schematic diagram of a model deployment system shown as an embodiment of the present application;

[0057] Figure 2 A schematic diagram of a model deployment system in which an inference flow and a CPU thread run in parallel is shown as an embodiment of the present application;

[0058] Figure 3 Another schematic diagram of a model deployment system shown as an embodiment of the present application;

[0059] Figure 4 A flowchart of a model deployment method shown in one embodiment of the present application;

[0060] Figure 5 A reasoning flowchart of a deployment model in a model deployment method shown in one embodiment of the present application. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0062] refer to Figure 1 , Figure 1 This is a schematic diagram of a model deployment system shown in one embodiment of the present application. Figure 1 As shown, the system 100 includes: a hardware basic function module 101, a deep learning algorithm module 102 and a memory manager 103; the hardware basic function module provides a model file loading interface for generating a deep learning model instance, which is provided to the deep learning algorithm module for model reasoning; the deep learning algorithm module provides a model reasoning calling interface, responds to a call request from an upper-layer application, calls the deep learning model instance to submit the complete reasoning task to the reasoning flow for model reasoning, and after obtaining the output parsing result, synchronizes the output parsing result to the target thread; the memory manager provides a memory management interface, responds to a memory request, and allocates memory from a maintained memory pool.

[0063] In this embodiment, the model deployment system provided by the present application is implemented using an object-oriented programming language. The model deployment system includes a hardware basic function module 101, a deep learning algorithm module 102 and a memory manager 103. The hardware basic function module provides a model file loading interface, through which the model file is loaded to generate a deep learning model instance, so as to perform model inference through the generated deep learning model instance.

[0064] In this embodiment, the system also includes a deep learning algorithm module, which provides a model reasoning call interface. The deep learning algorithm module responds to the call request of the upper-level application, and calls the generated deep learning model instance through the provided model reasoning call interface to submit the complete reasoning task to the reasoning flow for model reasoning. Among them, the complete reasoning task includes preprocessing, reasoning and output analysis in model reasoning. In the field of intelligent driving, the preprocessing process generally includes image scaling, cropping, color space conversion, and laser radar point cloud voxelization; model reasoning uses the preprocessing result as input to run the deep learning model to obtain the original output of the model; output analysis sorts, filters, softmax and other operations on the original output to obtain the final output of the model. Common output analysis includes filtering, decoding, NMS of target detection boxes, argmax, softmax of classification results, and pooling and extreme value extraction of key point detection heat maps. Different hardware has quite different implementations of inference streams. For example, on the NVIDIA platform, the inference stream instance refers to cudaStream_t, and on OpenCL, the inference stream instance refers to cl_command_queue. On the CPU side, the asynchronous execution function of the inference stream can also be achieved through multi-threaded parallelism and equal locks. However, the core is to submit the preprocessing task, inference task and output parsing task of the complete inference task to the inference stream through the inference stream instance for model inference when performing a complete inference task based on the deep learning model instance. After the preprocessing task, inference task and output parsing task are completed, a synchronization is performed, that is, the output parsing result obtained by the final execution of the output parsing task is synchronized to the target thread. The target thread is the CPU thread running the inference interface in the model inference call interface provided by the deep learning algorithm module. The target thread and the inference stream that executes the complete inference task execute tasks asynchronously and in parallel, such as Figure 2 As shown, the complete reasoning task on the reasoning flow (such as Figure 2 Task A, Task B, and Task C in the inference process together form a complete inference task) can be run on the GPU, NPU, or another CPU. When using an inference flow, the target thread submits a complete inference task to the inference flow, and the inference flow starts working after receiving the complete inference task. After the target thread successfully submits the task, it can execute tasks asynchronously and in parallel with the inference flow, executing unrelated tasks (such as Figure 2Task D) in the process, after execution, waits for the complete reasoning task on the reasoning flow to be completed, and then synchronizes the output parsing result obtained after the reasoning flow executes the complete reasoning task to the target thread. Through this implementation, the preprocessing, reasoning, and output parsing included in a complete reasoning process will all be submitted to the reasoning flow for processing, and only one synchronization will be performed after the output parsing is completed. This can ensure that the tasks on the reasoning flow are independent of the target thread to the greatest extent, reduce the interaction between the reasoning flow and the CPU where the target thread is located, thereby reducing the dependence of the model reasoning on the CPU. When the CPU resources on the embedded end are limited and the CPU load is extremely high, this method can reduce the difficulty of CPU scheduling and improve the stability of delay. Among them, the reasoning flow instance is an instance of a C++ class, which is a specific implementation of the reasoning flow. The reasoning flow can be made into a FIFO queue when implemented. The CPU submits multiple tasks to the queue, and the GPU takes the tasks out of the queue one by one for execution.

[0065] In this embodiment, the system also includes a memory manager, which provides a memory management interface, through which memory requests in the model reasoning process are responded to, and memory is allocated from a memory pool maintained by the memory manager based on the request. When each module in the system needs to use memory, it can directly apply to the memory manager. When there happens to be a memory block of the right size in the memory pool, the memory manager will assign a pointer to the memory block in the memory pool to the module that applied for memory, which can reduce frequent memory allocation and release. Unified memory management by the memory pool can also largely avoid waste caused by independent memory applications by each module.

[0066] A model deployment system provided by an embodiment of the present application includes a hardware basic function module, a deep learning algorithm module and a memory manager; the hardware basic function module provides a model file loading interface for generating a deep learning model instance, which is provided to the deep learning algorithm module for model reasoning; the deep learning algorithm module provides a model reasoning call interface, responds to the call request of the upper application, calls the deep learning model instance to submit the complete reasoning task to the reasoning flow for model reasoning, and after obtaining the output parsing result, synchronously outputs the parsing result to the target thread; the memory manager provides a memory management interface, responds to the memory request, and allocates memory from the maintained memory pool. Model deployment through this system can improve deployment efficiency. At the same time, this application introduces a memory manager, which provides a memory pool interface to the outside through the memory manager. When each element in the deployment system needs to use memory, it directly applies to the memory pool. When there is a memory block of the right size in the memory pool, the memory pool will assign the pointer of the memory in the memory pool to the element applying for memory, which can reduce frequent memory allocation and release. Unified memory management by the memory pool can also largely avoid waste caused by independent memory application by each element. At the same time, this application introduces an inference flow, submits the preprocessing tasks, inference tasks and result parsing tasks to the inference flow for processing, and performs synchronization only once the result parsing is completed. This can ensure to the greatest extent that the tasks on the inference flow run independently of the CPU thread, reduce the interaction between the inference flow and the CPU, and thus reduce the model reasoning's dependence on the CPU. When the CPU resources on the embedded end are limited and the CPU load is extremely high, this method can reduce the difficulty of CPU scheduling and improve the stability of delay.

[0067] In combination with the above embodiments, in one implementation, the embodiment of the present application also provides a model deployment system. In the model deployment system, the hardware basic function module also includes: a memory allocation interface, a memory flush interface, a memory copy interface, and a memory pool allocation memory interface; the memory allocation interface is used to allocate memory of a specified type and size and save it to a memory block buffer; the memory flush interface is used to flush a memory block buffer from one hardware to another; the memory copy interface is used to copy a memory block buffer; the memory pool allocation memory interface is used to call the memory manager for memory allocation.

[0068] In this embodiment, the hardware basic function module represents the basic functions related to model deployment, which are bound to the underlying hardware. The hardware basic function module includes a memory allocation interface, a memory flush interface, a memory copy interface, and a memory pool allocation memory interface. The model file loading interface loadModel() is used to generate a deep learning model instance and provide it to the deep learning algorithm module for model reasoning. The memory allocation interface allocBuffer() is used to allocate memory of a specified type and size and save it to a memory block buffer. Since the buffer uses C++ shared pointers for memory lifecycle management, it will be automatically released when the reference count is zero. Therefore, the hardware basic function module does not provide a release interface. The memory flushing interface flushBuffer() is used to flush the memory block buffer from one hardware to another hardware so that it can be accessed on the other hardware later. For example, when using shared memory for model inference, the memory belongs to the inference device (GPU or BPU) after the model inference is completed. At this time, if you want to perform subsequent processing on the CPU, some chips (such as NVIDIA and Horizon) require that the memory be flushed to the CPU before it can be accessed from the CPU. The memory copy interface copyBuffer() is used to copy the memory block buffer. The memory copy interface supports both copying between the same type of memory and copying between different types of memory. The memory pool allocation memory interface allocBufferManaged() is used to call the memory manager for memory allocation. The specific memory manager used is specified by the configuration file. Users can also implement their own memory manager and use this memory pool allocation memory interface for memory allocation.

[0069] In combination with the above embodiments, in one implementation, the embodiment of the present application also provides a model deployment system. In the model deployment system, the deep learning algorithm module includes: a model input and output module, a preprocessing module, a deep learning model instance and an output parser; the model input and output module is used to save the first variable of the input data of the deep learning model, the second variable of the output data of the deep learning model and the numbered variable of the data, and the first variable, the second variable and the numbered variable are public variables that can be directly accessed; the preprocessing module is used to perform preprocessing tasks on the input data of the deep learning model to obtain the corresponding output tensor; the deep learning model instance is used to perform reasoning tasks on the obtained input tensor and save the reasoning results as output tensors; the output parser is used to perform output parsing tasks on the output tensor to obtain output data.

[0070] In this embodiment, if Figure 3As shown, the deep learning algorithm module includes: a model input and output module, a preprocessing module, a deep learning model instance and an output parser. The model input and output module is used to save the first variable of the input data of the deep learning model, the second variable of the output data of the deep learning model and the number variable of the data. The input data is structured data related to the model function, such as input pictures, point clouds, output detection boxes, categories, segmentation maps, etc. The model input and output module does not provide an external function interface, and its internal member variables are public variables that can be accessed directly. The first variable is input_data, the second variable is output_data, and the number variable is frame_id. Input_data uses an associative array to save multiple input data, and output_data also uses an associative array to save multiple output data, using the model input and output name string as the key value. Frame_id can be the frame number of the picture, or other user-defined numbers, which are unique and are mainly used for log printing and debugging.

[0071] In this embodiment, if Figure 3 As shown, the deep learning algorithm module includes a preprocessing module, which is used to perform preprocessing tasks on the input data of the deep learning model to obtain the corresponding output tensor. The preprocessing tasks performed by the preprocessing module will be submitted to the reasoning flow, which will be executed serially together with the reasoning tasks and output parsing tasks. Preprocessing refers to the processing of model input data, such as scaling, cropping, color space conversion, normalization of model input images, and voxelization of point clouds. Preprocessing outputs one or more tensors, which are directly used as input to the deep learning model.

[0072] In this embodiment, the deep learning algorithm module is responsible for completing the complete model reasoning process and is the main interface for interacting with upper-layer applications. Figure 3 As shown, the deep learning algorithm module includes a deep learning model instance, which is created by the basic hardware basic function module (Device) and is used to perform reasoning tasks based on the obtained input tensor, and save the reasoning result as an output tensor, which is then given to the output parser (such as Figure 3 Output analysis is performed using the result analysis in .

[0073] In this embodiment, if Figure 3 As shown, the deep learning algorithm module includes an output parser (such as Figure 3), the output parser is used to perform output parsing tasks on the output tensors obtained by deep learning model inference to obtain the corresponding output data. The output parser parses one or more output tensors to obtain structured and meaningful model output data. Common parsing operations include target detection result filtering, segmentation result softmax, heat map extraction key points, etc. An output parser is only responsible for the parsing of one task. The output of a model can be achieved through multiple output parsers, and there is no correlation between the parsers. Output parsing can generally be divided into two steps, including device-side processing and host-side processing. Device-side processing mainly uses acceleration devices to process and filter large batches of data; host-side processing mainly uses the CPU to convert and transfer the filtered data to facilitate encapsulation into messages for processing by downstream modules. Synchronization is required between device-side processing and host-side processing.

[0074] In this embodiment, the input and output of the deep learning model (Model) are multiple tensors, each of which contains dimension information, data type information, data arrangement information, etc., and also contains a memory block buffer for storing tensor data. Dimension information is saved in the form of a multidimensional array, which can be one-dimensional or multi-dimensional. Data type information refers to the number system of the data stored in the tensor (Tensor), which can be float, double, int32, uint8, etc., and data arrangement information can be chw, hwc, chw32, c32hw32, hwc8, etc.

[0075] In this embodiment, a large number of memory block caches are required for data interaction and storage in the model reasoning process. The memory type may be host memory accessible only to the CPU, device memory accessible only to the heterogeneous acceleration core, or shared memory accessible to both the host and the device. All of this memory-related information is stored in the memory block cache. The memory block cache also stores the address of the memory block and the size of the memory block in the form of a shared pointer.

[0076] In combination with the above embodiments, in one implementation, the embodiment of the present application also provides a model deployment system. In the model deployment system, the deep learning model instance includes: a model initialization interface, a model destruction interface, an inference interface, and a synchronization interface; the model initialization interface is used to allocate memory for input tensors and output tensors, load configuration file information of the deep learning model, read the model file, and generate a deep learning model instance; the model destruction interface is used to release the memory and deep learning model instance allocated by the model initialization interface when model inference is not required; the inference interface is used to submit a complete inference task to the inference flow and execute the complete inference task; the synchronization interface is used to wait in a blocking manner for the inference task in the complete inference task to end, and after the inference task in the complete inference task is completed, proceed to the next step of processing.

[0077] In this embodiment, the deep learning model instance provides multiple types of interfaces, including a model initialization interface, a model destruction interface, an inference interface, and a synchronization interface. The model initialization interface init() is used to allocate memory for input tensors and output tensors, load configuration file information of the deep learning model, read model files, and generate deep learning model instances; the model destruction interface teardown() is used to release the memory and deep learning model instance allocated by the model initialization interface when model inference is not required; the inference interface inference() is used to submit a complete inference task to the inference flow so that the inference flow can execute the complete inference task. The inference interface will not wait for the complete inference to end before submitting a new task, so that the caller can perform other work on the CPU or other inference flows or submit other inference tasks. The synchronization interface sync() is used to wait in a blocking manner for the inference task in the complete inference task to end, and after the inference task in the complete inference task is completed, the next step of processing is to execute the output parsing task.

[0078] In this embodiment, there is a state machine inside the deep learning model instance, which is initially in the NotReady state. After calling the model initialization interface, the deep learning model instance changes to the Ready state, at which time the inference interface or the model destruction interface can be called. In the Ready state, after calling the inference interface, the deep learning model instance changes to the Running state, and only the synchronization interface can be called, after which the deep learning model instance returns to the Ready state. In the Ready state, calling the model destruction interface will cause the deep learning model instance to enter the NotReady state.

[0079] In combination with the above embodiments, in one implementation, the embodiment of the present application further provides a model deployment system. In the model deployment system, the output parser includes: a parser initialization interface, a parser destruction interface, a device-side processing interface, a host-side processing interface, and an enable judgment interface; the parser initialization interface and the parser destruction interface are used to allocate and release resources required by the output parser; the device-side processing interface is used to perform output parsing on the device side; the host-side processing interface is used to perform output parsing on the host side; the enable judgment interface is used for the deep learning algorithm module to obtain the enable status of the parser.

[0080] In this embodiment, the output parser provides multiple types of interfaces, including: parser initialization interface, parser destruction interface, device-side processing interface, host-side processing interface and enable judgment interface. Parser initialization interface init() and parser destruction interface teardown() are used to allocate and release resources required by the output parser; device-side processing interface processOnDevice() is used to perform output parsing on the device side; host-side processing interface processOnHost() is used to perform output parsing on the host side; enable judgment interface isEnabled() is used for the deep learning algorithm module to obtain the enable status of the parser, and returning true means enabled. The deep learning algorithm module will only call the corresponding interface for output parsing when the output parser is enabled.

[0081] In combination with the above embodiments, in one implementation, the embodiment of the present application also provides a model deployment system. In the model deployment system, the model reasoning call interface includes: an output parser registration interface, an algorithm initialization interface, a destruction interface, a preprocessing interface and a model reasoning interface; the output parser registration interface is used to register the output parser to the deep learning algorithm module; the algorithm initialization interface is used to read the model file of the corresponding deep learning model from the configuration file, and call the model file loading interface of the hardware basic function module to load the model file and generate a deep learning model instance, and to call the initialization interface of each module belonging to the deep learning algorithm module; the destruction interface is used to release the resources allocated by the algorithm initialization interface; the preprocessing interface is used to call the preprocessing module; the model reasoning interface is used to call the preprocessing interface, the reasoning interface of the deep learning model instance, the device-side processing interface of the registered output parser, the synchronization interface of the deep learning model instance and the host-side processing interface of the registered output parser in sequence to perform a complete reasoning task.

[0082] In this embodiment, the model reasoning calling interface provided by the deep learning algorithm module includes: output parser registration interface, algorithm initialization interface, destruction interface, preprocessing interface and model reasoning interface.

[0083] In this embodiment, the output parser registration interface addDecoder() is used to register the output parser to the deep learning algorithm module. The output parser registration interface needs to be called before the algorithm initialization interface. The registered output parser will be called in sequence by the deep learning algorithm module during reasoning. The algorithm initialization interface init() is used to read the model file of the corresponding deep learning model from the configuration file, and call the model file loading interface of the hardware basic function module to load the model file and generate a deep learning model instance. It is also used to call the initialization interface of each module belonging to the deep learning algorithm module, such as the parser initialization interface of each registered output parser and the model initialization interface of the deep learning model instance. The destruction interface teardown() is used to release the resources allocated by the algorithm initialization interface when model reasoning is not required. The preprocessing interface preProecss() is used to call the preprocessing module to execute preprocessing tasks. The model inference interface process() is used to sequentially call the preprocessing interface preProecss(), the inference interface inference() of the deep learning model instance, the device-side processing interface processOnDevice() of the registered output parser, the synchronization interface sync() of the deep learning model instance, and the host-side processing interface processOnHost() of the registered output parser to perform a complete inference task. All steps from calling the preprocessing interface preProecss() to calling the synchronization interface sync() of the deep learning model instance are processed on the inference flow.

[0084] In combination with the above embodiments, in one implementation, the embodiment of the present application also provides a model deployment system. In the model deployment system, the reasoning flow includes: obtaining a hardware-related reasoning flow instance interface and a reasoning flow synchronization interface; the hardware-related reasoning flow instance acquisition interface is used to obtain a reasoning flow instance corresponding to the current hardware, and use the reasoning flow instance to submit a complete reasoning task through a hardware-related interface implementation; the reasoning flow synchronization interface is used to wait for the completion of the complete reasoning task on the reasoning flow, and then synchronize the obtained output parsing results to the target thread.

[0085] In this embodiment, the model reasoning process consists of three steps: preprocessing, reasoning, and output parsing. Each step contains one or more functions and can run on a heterogeneous acceleration chip or a CPU. During the execution of these three steps, the model reasoning process does not interact with the upper-level application, so that the model reasoning and the upper-level application can be executed completely asynchronously. Therefore, this application introduces the concept of reasoning flow, and puts each function in the model reasoning process on the reasoning flow for execution, which can ensure that the operations on the reasoning flow are serial, and the reasoning flow and the upper-level application can be parallel. After the reasoning is completed, the reasoning flow is synchronized to ensure that the upper-level application can obtain the final output parsing result. Figure 2 Describes the asynchronous operation between the target thread and the inference flow, where the target thread is the thread running the model inference interface of the deep learning algorithm module in this application, and the tasks on the inference flow can be run on the GPU, NPU or another CPU. When using the inference flow, the target thread submits a complete inference task to the inference flow (such as Figure 2 Task A, Task B, and Task C together constitute a complete reasoning task). The reasoning flow starts working after receiving the complete reasoning task. After the target thread successfully submits the task, it can asynchronously parallelize with the reasoning flow and execute unrelated tasks (such as Figure 2 Task D in). After the execution is completed, the reasoning flow synchronization is performed, and the task on the reasoning flow is waited for to be executed. The interface of the reasoning flow includes: obtaining the hardware-related reasoning flow instance interface and the reasoning flow synchronization interface, obtaining the hardware-related reasoning flow instance interface getStream(), which is used to obtain the reasoning flow instance corresponding to the current hardware, and submit the complete reasoning task by using the reasoning flow instance in the hardware-related interface implementation. In the deployment system of the present application, because the implementation of the reasoning flow by different hardware is quite different, the general interface of the reasoning flow does not include an interface for submitting a complete reasoning task to the reasoning flow. When it is necessary to submit a complete reasoning task to the reasoning flow, it is necessary to first call the hardware-related reasoning flow instance interface to obtain the reasoning flow instance corresponding to the current hardware (for example, cudaStream_t on the NVIDIA platform, or cl_command_queue on OpenCL), and then use the reasoning flow instance in the hardware-related interface implementation to submit the complete reasoning task. When the task submission is completed, the reasoning flow synchronization interface synchronize() can be called for synchronization, and after waiting for the complete reasoning task on the reasoning flow to be completed, the output parsing result is synchronized to the target thread.

[0086] In combination with the above embodiments, in one implementation, the embodiment of the present application further provides a model deployment system. In the model deployment system, the memory management interface includes: a request memory interface, an interface for obtaining the total memory size of the current memory pool, and an interface for obtaining the free memory size; the request memory interface is used to respond to a memory request including a memory type and a memory size, and return the corresponding memory block buffer to the corresponding module; the interface for obtaining the total memory size of the current memory pool and the interface for obtaining the free memory size are used for debugging and log printing.

[0087] In this embodiment, the memory used in the model reasoning process is large in size and has a variety of special properties, such as memory that can only be accessed on the host side (CPU), memory that can only be accessed on the device side (GPU or NPU, etc.), and memory shared by the host side and the device side. These memories are generally not suitable for frequent allocation and release, so this application introduces a memory manager, uses memory pools and other technologies to manage memory, and reduces the overhead of memory allocation and release. The memory manager in this application maintains a memory pool internally. When an element requests a certain size of memory, the memory manager first searches the memory pool to see if there is a piece of memory that is greater than or equal to the requested size. If so, it returns its pointer to the user directly in the form of a Buffer, and at the same time changes the destruction method of the shared pointer inside the Buffer to return the memory to the memory pool to avoid the direct release of this memory. When the reference count of the shared pointer inside the Buffer returns to zero, the memory will be automatically returned to the memory pool. When there is no memory of a suitable size in the memory pool, or all suitable memory has been used by other elements, the memory manager will reallocate a piece of memory for the element to use. In this implementation, if multiple elements need to frequently use the same size of memory, the memory manager can directly take out pre-allocated memory from the memory pool and give it to the elements for use, avoiding frequent memory allocation and release. The memory manager also provides an idle memory release function. When a memory block has not been used for a certain period of time, the memory will be released to reduce memory usage. The interfaces provided by the memory manager include: a memory request interface, an interface for obtaining the total memory size of the current memory pool, and an interface for obtaining the free memory size. The request memory interface requestMem() is used to respond to a memory request including a memory type and a memory size, and return the corresponding memory block buffer to the corresponding module; the interface for obtaining the total memory size of the current memory pool getTotalSize() and the interface for obtaining the free memory size getTotalFreeSize() are mainly used for debugging and log printing.

[0088] In combination with the above embodiments, in one implementation, the embodiment of the present application further provides a model deployment system. In the model deployment system, the system further includes: a unified log interface for connecting to the log interface of the upper layer application and providing it to the deep learning algorithm module for log recording and printing.

[0089] In this embodiment, the reasoning function requires the log function to print error or debugging information. The reasoning function is a submodule of the upper application, and the upper application will have its own log module. Therefore, the log interface designed in the deployment system provided by this application has the main function of docking the log interface of the upper application and providing a cross-platform, cross-application unified log interface to the reasoning function. The unified log interface logprint() needs to provide the text and log level information of the log when calling. The unified log interface will call the corresponding back end to print the log (such as glog or spdlog) according to the log back end selected at compile time according to the current deployment system version.

[0090] In this embodiment, during the development of the model reasoning function, scenarios requiring performance analysis may be encountered. Therefore, the deployment system provided in this application includes a timer for analyzing the time consumption of model reasoning. The timer supports CPU-side time statistics and heterogeneous core time statistics, and supports the time consumption of operations on statistical reasoning flows. The interface of the timer includes: startRecordTime(), endRecordTime(), and getElapsedTime(). The timer can receive parameters when it is created, and choose to use the Device-side timer or the Host-side Timer. When the Device-side timer is used, since the operation to be timed is run on the Device side, the startRecordTime() will receive an inference flow as input. When the Host-side timer is used, there is no need to provide an inference flow instance when starting the timing, because the operation to be timed runs on the CPU side. When the timing ends, the end timing stopRecordTime() can be called to stop the timing. After stopping the timing, you can call getElapsedTime() to get the time consumption. If the device-side timer is used, you need to call getElapsedTime() after the inference flow is synchronized. Otherwise, the operation on the inference flow may not be completed and the correct time cannot be obtained.

[0091] In combination with the above embodiments, in one implementation, the embodiment of the present application also provides a model deployment system. In the model deployment system, the hardware basic function module, deep learning model instance, reasoning flow and memory manager are the first elements; the first element includes a first basic part that is independent of specific hardware and a backend part that is related to hardware, and the first basic part is defined as a base class; the first basic part is used to provide hardware-general interfaces and hardware-independent interface implementations, and define interfaces related to the backend part as virtual functions; the backend part is used to develop and create corresponding inherited classes when deploying models that support new hardware, and the inherited classes inherit the base class of the first basic part and implement the corresponding virtual functions defined in the base class.

[0092] In this embodiment, the model deployment system provided by this application includes the following elements: hardware basic function (Device) module, deep learning model (Model) instance, deep learning algorithm (Algorithm) module, model input and output (Task) module, preprocessing (Preprocess) module, output parser (Decoder), reasoning stream (Stream), timer (Timer), log interface (Log), tensor (Tensor), memory block (Buffer), memory manager (MemManager). There is data and information interaction between these elements, and they collaborate to complete the model reasoning task.

[0093] In this embodiment, a part of all the elements in the model deployment system are related to hardware, namely, hardware basic function modules, deep learning model instances, reasoning flows, and memory managers. For the convenience of subsequent description, each of these hardware-related elements belongs to a first element. Each first element includes a basic part that is independent of specific hardware and a back-end part that is related to hardware. The basic part that is independent of specific hardware is the first basic part. The first basic part of the first element is defined by a base class, and the first basic part provides a universal interface across hardware and an interface implementation that is independent of hardware, and at the same time defines the interface related to the back-end part of the first element as a virtual function. When the model deployment system is to support model deployment of new hardware, only the back-end part of the first element is developed and implemented. Specifically, when the model deployment system is to support model deployment of new hardware, a corresponding inheritance class is developed and created, and the inheritance class will inherit the base class of the first basic part to implement the virtual function defined in the base class corresponding to the inheritance class, thereby implementing the virtual interface to provide reasoning support for new hardware.

[0094] In combination with the above embodiments, in one implementation, the embodiment of the present application also provides a model deployment system. In the model deployment system, the deep learning algorithm module, the model input and output module, the preprocessing module, and the output parser are the second elements; the second elements include a second basic part that is independent of a specific model and a model-related part that is related to the model; the second basic part is used to provide a general interface and an implementation that is independent of the model, and to define the interface related to the model as a virtual function, and the second basic part is defined as a base class; the model-related part is used to develop and create a corresponding inheritance class when deploying a new model, and the inheritance class inherits the base class of the second basic part and implements the corresponding virtual interface defined in the base class.

[0095] In this embodiment, part of all the elements in the model deployment system are related to the model, namely, the deep learning algorithm module, the model input and output module, the preprocessing module, and the output parser. For the convenience of subsequent description, each of these elements related to the model belongs to a second element. Each second element includes a basic part that is not related to a specific model and a model-related part that is related to the model. The basic part that is not related to the specific model is the second basic part. The second basic part for the second element is also defined by a base class. The second basic part provides a general interface and an interface implementation that is not related to the model, and defines the interface related to the model as a virtual function. When the model deployment system is to deploy a new model, only the model-related part of the second element is developed and implemented. Specifically, when the model deployment system is to deploy a new model, a corresponding inheritance class is developed and created. The inheritance class will inherit the base class of the second basic part to implement the virtual function defined in the base class corresponding to the inheritance class, thereby realizing the virtual interface to achieve the purpose of deploying the model. For example, the preprocessing interface of the deep learning algorithm module is a virtual function in the base class of the deep learning algorithm module. When the user deploys a new model, he develops and creates a corresponding inheritance class to inherit the corresponding virtual function in the base class of the deep learning algorithm module for implementation.

[0096] In this embodiment, the unified log interface is independent of the model or hardware, but is related to a specific log backend. When the model is deployed on different applications, different log backends may be used according to the requirements of the application, such as glog, spdlog, etc. The log printing interfaces of these log backends are different, so the deployment system provides a unified log interface. Each element within the deployment system uses the system's unified log interface for printing, and the unified log interface selects the appropriate log backend for log printing based on the configuration. The code related to log printing is often distributed in various locations in the software. If the log interface is not abstracted, the workload of switching the log backend will be very cumbersome, and the deployment system provides a unified log interface to avoid the occurrence of this problem.

[0097] In this embodiment, the purpose of designing the above-mentioned elements related to hardware, models, logs, etc. is that when deploying new hardware or a new model, it is only necessary to inherit and implement the relevant parts of the elements related to the hardware or model, and the code of other elements can be reused, which can maximize the code reuse rate and reduce the workload of development, testing, and maintenance. For all the elements involved in the deployment system mentioned above in this application, the remaining elements after removing the elements related to the hardware and the elements related to the model are other elements. These other elements can be reused directly without the need for redevelopment due to different deployed models and / or different deployed hardware.

[0098] Based on the same inventive concept, the second aspect of the present application provides a model deployment method, which includes: obtaining the model file and configuration file of the model to be deployed; based on the model deployment system of the first aspect of the present application, performing model deployment on the model file and configuration file; executing the corresponding complete reasoning task based on the deployed deep learning model, and sending the output data obtained by reasoning to the upper-level application.

[0099] In this embodiment, if Figure 4 He Ru Figure 5 As shown, first obtain the model file and configuration file of the model to be deployed, and then based on the obtained model file and configuration file of the model to be deployed, the model deployment system provided in the first aspect of this application is used to deploy the model file and configuration file, perform the corresponding complete reasoning task based on the deployed deep learning model, and send the output data obtained by reasoning to the upper-level application. After the model is deployed using the model deployment system provided by this application, the entire process of model reasoning is realized through the interaction between the elements in the system. These interactions are fully expressed by the base class of each element, and the inherited class cannot change the interaction relationship. These interactions are divided into four stages:

[0100] Initialization phase: Before inference begins, the upper-layer application is responsible for creating a deep learning algorithm (Algorithm) instance (i.e., the deep learning algorithm module mentioned above) and initializing it. During initialization, the model file of the deep learning model needs to be provided. The deep learning algorithm module will call the hardware basic function module to read the model file, generate a deep learning model instance, and save it inside the deep learning algorithm module.

[0101] Input and output data preparation phase: During runtime, the request for model inference is initiated by the upper-layer application. The upper-layer application first prepares the input and output data (Task) required for inference, allocates memory, and puts the valid input data into the Task.

[0102] Model inference phase: After the Task is prepared, the upper-layer application inputs the Task to the deep learning algorithm module for inference. After receiving the Task, the deep learning algorithm module first performs preprocessing, then calls the interface of the deep learning model instance for inference, and then calls the output parser one by one to parse the results. Data is exchanged between preprocessing, inference, and output parsing through tensors. The entire processing flow of the deep learning algorithm module is placed on the inference stream. The deep learning algorithm module performs a synchronization at the end of the processing and returns to the upper-layer application after synchronization. The upper-layer application obtains the output of model inference through the Task and performs subsequent processing.

[0103] Resource release phase: When the upper-layer application no longer needs to perform model inference, it calls the release interface of the deep learning algorithm module to release the memory and resources allocated to deep learning model instances, tensors, memory blocks, memory managers, and other instances.

[0104] In this embodiment, the interaction mode between elements and elements in the model deployment system provided by this application can take into account both versatility and model deployment performance. Specifically, the preprocessor and output parser can directly access the input tensors and output tensors used by the deep learning model, thereby realizing zero-copy data processing. The memory manager provides a memory pool interface. When the elements of the deployment system need to use memory, they can directly apply to the memory manager. When there is a memory block of the right size in the memory pool managed by the memory manager, the pointer of the memory in the memory pool will be assigned to the element applying for memory, which can reduce frequent memory allocation and release. The unified memory management by the memory pool can also largely avoid the waste caused by independent memory application by each element. Preprocessing, reasoning and output parsing are all submitted to the reasoning flow for processing, and synchronization is performed only once after the output parsing is completed. This can maximize the guarantee that the tasks on the reasoning flow run independently of the CPU thread, reduce the interaction between the reasoning flow and the CPU, and thus reduce the dependence of the model reasoning on the CPU. When the CPU resources on the embedded end are limited and the CPU load is extremely high, this method can reduce the difficulty of CPU scheduling and improve the stability of delay.

[0105] Based on the same inventive concept, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and running on the processor. When the computer program is executed by the processor, it implements the steps in a model deployment method described in the second aspect of the present application.

[0106] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in a model deployment method described in the second aspect of the present application are implemented.

[0107] As for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0108] It should be noted that, for the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0109] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0110] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the embodiments of the present application may adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0111] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0112] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0113] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0114] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0115] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0116] The above is a detailed introduction to a model deployment system, method and product provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A model deployment system, characterized in that: The system includes: a hardware basic function module, a deep learning algorithm module and a memory manager; The hardware basic function module provides a model file loading interface for generating a deep learning model instance and providing it to the deep learning algorithm module for model reasoning; The deep learning algorithm module provides a model reasoning calling interface, responds to the call request of the upper-layer application, calls the deep learning model instance to submit the complete reasoning task to the reasoning flow for model reasoning, and after obtaining the output parsing result, synchronizes the output parsing result to the target thread; The memory manager provides a memory management interface and allocates memory from a maintained memory pool in response to a memory request.

2. The model deployment system according to claim 1, characterized in that: The hardware basic function module also includes: a memory allocation interface, a memory flash interface, a memory copy interface and a memory pool allocation memory interface; The memory allocation interface is used to allocate memory of a specified type and size and save it in a memory block buffer; The memory flushing interface is used to flush a memory block buffer from one hardware to another hardware; The memory copy interface is used to copy the memory block buffer; The memory pool allocates a memory interface, and is used to call the memory manager to perform memory allocation.

3. The model deployment system according to claim 1, characterized in that: The deep learning algorithm module includes: a model input and output module, a preprocessing module, a deep learning model instance and an output parser; The model input and output module is used to store a first variable for input data of the deep learning model, a second variable for output data of the deep learning model, and a numbered variable for storing data, wherein the first variable, the second variable, and the numbered variable are public variables that can be directly accessed; The preprocessing module is used to perform preprocessing tasks on the input data of the deep learning model to obtain corresponding output tensors; The deep learning model instance is used to perform reasoning tasks on the obtained input tensor and save the reasoning results as output tensors; The output parser is used to perform an output parsing task on the output tensor to obtain output data.

4. The model deployment system according to claim 3, characterized in that: The deep learning model instance includes: a model initialization interface, a model destruction interface, an inference interface and a synchronization interface; The model initialization interface is used to allocate memory for input tensors and output tensors, load configuration file information of the deep learning model, read the model file, and generate a deep learning model instance; The model destruction interface is used to release the memory and deep learning model instance allocated by the model initialization interface when model inference is not needed; The reasoning interface is used to submit a complete reasoning task to the reasoning flow to execute the complete reasoning task; The synchronization interface is used to wait in a blocking manner for the inference task in the complete inference task to be completed, and to perform the next step of processing after the inference task in the complete inference task is completed.

5. The model deployment system according to claim 3, characterized in that: The output parser includes: a parser initialization interface, a parser destruction interface, a device-side processing interface, a host-side processing interface and an enable judgment interface; The parser initialization interface and the parser destruction interface are used to allocate and release resources required by the output parser; The device-side processing interface is used to perform output parsing on the device side; The host-side processing interface is used to perform output parsing on the host side; The enabling judgment interface is used by the deep learning algorithm module to obtain the enabling status of the parser.

6. The model deployment system according to claim 1, characterized in that: The model reasoning calling interface includes: an output parser registration interface, an algorithm initialization interface, a destruction interface, a preprocessing interface and a model reasoning interface; The output parser registration interface is used to register the output parser to the deep learning algorithm module; The algorithm initialization interface is used to read the model file of the corresponding deep learning model from the configuration file, and call the model file loading interface of the hardware basic function module to load the model file and generate a deep learning model instance, and is used to call the initialization interface of each module belonging to the deep learning algorithm module; The destruction interface is used to release the resources allocated by the algorithm initialization interface; The preprocessing interface is used to call the preprocessing module; The model inference interface is used to sequentially call the preprocessing interface, the inference interface of the deep learning model instance, the device-side processing interface of the registered output parser, the synchronization interface of the deep learning model instance, and the host-side processing interface of the registered output parser to perform a complete inference task.

7. The model deployment system according to claim 1, characterized in that: The reasoning flow includes: obtaining a hardware-related reasoning flow instance interface and a reasoning flow synchronization interface; The interface for obtaining hardware-related reasoning flow instance is used to obtain the reasoning flow instance corresponding to the current hardware, and use the reasoning flow instance to submit a complete reasoning task through the hardware-related interface implementation; The reasoning flow synchronization interface is used to wait for the complete reasoning task on the reasoning flow to be completed, and then synchronize the obtained output parsing results to the target thread.

8. The model deployment system according to claim 1, characterized in that: The memory management interface includes: a memory request interface, an interface for obtaining the total memory size of the current memory pool, and an interface for obtaining the free memory size; The request memory interface is used to return the corresponding memory block buffer to the corresponding module in response to the memory request including the memory type and the memory size; The interface for obtaining the total memory size of the current memory pool and the interface for obtaining the free memory size are used for debugging and log printing.

9. The model deployment system according to claim 3, characterized in that: The hardware basic function module, deep learning model instance, reasoning flow and memory manager are the first elements; the first elements include a first basic part that is independent of specific hardware and a backend part that is related to hardware, and the first basic part is defined as a base class; The first basic part is used to provide hardware-general interfaces and hardware-independent interface implementations, and define interfaces related to the backend part as virtual functions; The backend part is used to develop and create corresponding inheritance classes when deploying a model that supports new hardware. The inheritance classes inherit the base class of the first basic part and implement the corresponding virtual functions defined in the base class.

10. The model deployment system according to claim 3, characterized in that: The deep learning algorithm module, model input and output module, preprocessing module, and output parser are the second elements; the second elements include a second basic part that is not related to a specific model and a model-related part that is related to the model; The second basic part is used to provide a general interface and an implementation that is independent of the model, and to define the interface related to the model as a virtual function, and the second basic part is defined as a base class; The model-related part is used to develop and create a corresponding inheritance class when deploying a new model. The inheritance class inherits the base class of the second basic part and implements the corresponding virtual interface defined in the base class.

11. The model deployment system according to claim 1, characterized in that: The system also includes: a unified log interface, which is used to connect to the log interface of the upper-layer application and provide it to the deep learning algorithm module for log recording and printing.

12. A model deployment method, characterized in that: The method comprises: Get the model file and configuration file of the model to be deployed; Based on the model deployment system according to any one of claims 1 to 11, model deployment is performed on the model file and the configuration file; Execute the corresponding complete reasoning task based on the deployed deep learning model, and send the output data obtained by reasoning to the upper-level application.

13. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and running on the processor, wherein when the computer program is executed by the processor, the steps in the model deployment method as claimed in claim 12 are implemented.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps in the model deployment method according to claim 12 are implemented.

Images