Model deployment method, model operation method

Abstract

The invention relates to a model deployment method and a model running method. The model deployment method includes: obtaining a pipeline application; obtaining a basic docker image, and packaging the pipeline application as a docker image to be deployed based on the basic docker image; importing the docker image to be deployed into an edge smart device, so that the edge smart device can create a container for running the pipeline application from the docker image to be deployed, wherein the pipeline application determines the execution order of multiple plug-ins and the flow direction of data based on the pipeline when it is run. Through this application, the problem that the model deployment method in the related art is difficult to apply to complex business scenarios is solved, and flexible deployment of multiple models is achieved.

Description

Technical Field

[0001] The invention relates to the field of data processing technology, and in particular to a model deployment method and a model operation method. Background Art

[0002] The model deployment method in related technologies first compiles and generates SDK libraries such as image processing and model reasoning. When deploying the model, the interfaces in these libraries are called to implement the implementation of intelligent business. However, since the interfaces provided by the SDK library are often highly coupled, the calls between the modules are not independent of each other, resulting in the model deployed using the SDK library can only achieve a very simple business scenario, which is difficult to apply to complex business scenarios. Summary of the invention

[0003] The model deployment method, model operation method and non-transitory machine-readable medium provided by the embodiments of the present invention at least solve the problem that the model deployment method in the related art is difficult to apply to complex business scenarios.

[0004] A model deployment method, applied to a development server, includes:

[0005] Obtain a pipeline application, wherein the pipeline application includes a plug-in library and a pipeline corresponding to the model to be deployed, wherein the plug-in library includes plug-ins for multimedia data preprocessing, reasoning, and post-processing, and the pipeline includes a combination of multiple plug-ins in the plug-in library;

[0006] Obtain a basic Docker image, and package the pipeline application into a Docker image to be deployed based on the basic Docker image, wherein the basic Docker image includes an operating environment required to run the pipeline application;

[0007] The docker image to be deployed is imported into the edge intelligent device so that the edge intelligent device can create a container for running the pipeline application from the docker image to be deployed. When the pipeline application is run, the execution order of the multiple plug-ins and the flow direction of data are determined based on the pipeline.

[0008] In some of the embodiments, before obtaining the pipeline application, the method further includes:

[0009] Determine, according to the model to be deployed, the link relationship between the plug-ins used by the model to be deployed and the parameter configuration of the plug-ins;

[0010] The pipeline is obtained by combining the link relationship between the plug-ins used by the model to be deployed and the parameter configuration of the plug-ins.

[0011] In some embodiments, the plug-in library includes the following plug-ins: data source plug-in, decoding plug-in, inference plug-in, ID generation plug-in, event rule plug-in, OSD plug-in, encoding plug-in, event linkage plug-in, code stream forwarding plug-in and file storage plug-in; wherein,

[0012] The data source plug-in is used to obtain code stream data;

[0013] The decoding plug-in is used to decode the code stream data to obtain decoded data; the configuration parameters of the decoding plug-in include decoding parameters;

[0014] The inference plug-in is used to perform model inference on the decoded data using the inference model to obtain an inference result; the configuration parameters of the inference plug-in include the type and path of the inference model;

[0015] The ID generation plug-in is used to generate a unique identifier for the detected object;

[0016] The event rule plug-in is used to filter the external parameters of the operation of the inference result, and the external parameters include external rules and encoding parameters;

[0017] The OSD plug-in is used to overlay information on the display screen, and the configuration parameters of the OSD plug-in include the type of overlay information;

[0018] The encoding plug-in is used to encode YUV data, and the configuration parameters of the encoding plug-in include encoding parameters;

[0019] The event linkage plug-in is used to report the processing result of the multimedia data to the server;

[0020] The code stream forwarding plug-in is used to convert the code stream data into a media stream of a real-time streaming protocol or a hypertext transfer protocol;

[0021] The file storage plug-in is used to save the processing results of the multimedia data locally.

[0022] In some of the embodiments, the basic docker image also includes a remote access service, which includes a Jupyterlab service and / or an SSHD service; and the plug-in library includes user-defined plug-ins.

[0023] A model operation method, applied to an edge intelligent device, comprising:

[0024] Obtain a docker image to be deployed, wherein the docker image to be deployed is obtained by packaging a pipeline application program with a basic docker image, wherein the basic docker image includes an operating environment required to run the pipeline application program, wherein the pipeline application program includes a plug-in library and a pipeline corresponding to the model to be deployed, wherein the plug-in library includes plug-ins for multimedia data preprocessing, reasoning, and post-processing, and the pipeline includes a combination of multiple plug-ins in the plug-in library;

[0025] A container for running the pipeline application is created from the docker image to be deployed, the container is loaded to run the pipeline application, and the running result of the model to be deployed is obtained. When the pipeline application is run, the execution order of the multiple plug-ins and the flow direction of data are determined based on the pipeline.

[0026] In some embodiments, loading the container to run the pipeline application includes:

[0027] The master of the edge intelligent device loads the container, binds the container to the slave chip of the edge intelligent device, and initializes the slave chip so that the slave chip loads hardware resources and software resources, wherein the software resources include the pipeline and model resources.

[0028] In some embodiments, the method further comprises:

[0029] The slave parses the pipeline to obtain the parameter configuration of the plug-in and the link relationship between the plug-ins used by the model to be deployed, loads the parameter configuration into the model resource corresponding to the plug-in, executes the task, and receives and sends data according to the link relationship between the plug-ins used by the model to be deployed.

[0030] In some embodiments, the base Docker image further includes a remote access service, and a mapping port is configured between the container and the edge smart device so that the remote access service of the container is accessible on the edge smart device; the method further includes:

[0031] Receive remote access requests from users;

[0032] Based on the mapped port configured between the container and the edge intelligent device, the remote access service of the container is accessed, so that the container modifies and reconstructs the pipeline based on the remote access request.

[0033] In some embodiments, the plug-in library includes the following plug-ins: data source plug-in, decoding plug-in, inference plug-in, ID generation plug-in, event rule plug-in, OSD plug-in, encoding plug-in, event linkage plug-in, code stream forwarding plug-in and file storage plug-in; wherein,

[0034] The data source plug-in is used to obtain code stream data;

[0035] The decoding plug-in is used to decode the code stream data to obtain decoded data; the configuration parameters of the decoding plug-in include decoding parameters;

[0036] The inference plug-in is used to perform model inference on the decoded data using the inference model to obtain an inference result; the configuration parameters of the inference plug-in include the type and path of the inference model;

[0037] The ID generation plug-in is used to generate a unique identifier for the detected object;

[0038] The event rule plug-in is used to filter the external parameters of the operation of the inference result, and the external parameters include external rules and encoding parameters;

[0039] The OSD plug-in is used to overlay information on the display screen, and the configuration parameters of the OSD plug-in include the type of overlay information;

[0040] The encoding plug-in is used to encode YUV data, and the configuration parameters of the encoding plug-in include encoding parameters;

[0041] The event linkage plug-in is used to report the processing result of the multimedia data to the server;

[0042] The code stream forwarding plug-in is used to convert the code stream data into a media stream of a real-time streaming protocol or a hypertext transfer protocol;

[0043] The file storage plug-in is used to save the processing results of the multimedia data locally.

[0044] A non-transitory machine-readable medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the above method.

[0045] The model deployment method, model operation method and non-transitory machine-readable medium provided by the embodiments of the present invention are obtained by obtaining a pipeline application, the pipeline application includes a plug-in library and a pipeline corresponding to the model to be deployed, the plug-in library includes plug-ins for multimedia data preprocessing, reasoning and post-processing, and the pipeline includes a combination of multiple plug-ins in the plug-in library; obtaining a basic docker image, and packaging the pipeline application as a docker image to be deployed based on the basic docker image, the basic docker image includes the operating environment required to run the pipeline application; importing the docker image to be deployed into an edge intelligent device so that the edge intelligent device can create a container for running the pipeline application from the docker image to be deployed, and when the pipeline application is run, the execution order of multiple plug-ins and the flow direction of data are determined based on the pipeline, thereby solving the problem that the model deployment method in the related art is difficult to apply to complex business scenarios and realizing flexible deployment of multiple models. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other embodiments can be obtained based on these drawings without creative work.

[0047] Figure 1 It is a flowchart of the model deployment method of this embodiment.

[0048] Figure 2 Schematic diagram of the algorithm of the target detection model of this embodiment.

[0049] Figure 3 Schematic diagram of the algorithm of the classification detection model of this embodiment.

[0050] Figure 4 Schematic diagram of the algorithm of the semantic segmentation model of this embodiment.

[0051] Figure 5 It is a flow chart of the model operation method of this embodiment. DETAILED DESCRIPTION

[0052] The embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0053] Edge devices are devices that run at the "edge" of the network. These devices interact directly with end users or the physical world, collect data, process data, and send data to other central nodes when necessary. Edge computing refers to pushing computing power and data storage capabilities to the edge of the network, close to where data is generated and consumed. The goal of edge computing is to reduce the latency of data transmission, reduce the burden on centralized computing resources, and improve data security and privacy protection.

[0054] To realize edge computing, it is necessary to deploy computing models on edge smart devices. Related technologies usually implement computing models by calling interfaces provided by SDK libraries. SDKs usually include multiple closely related libraries and tools that work together to provide a complete set of services. This tight integration allows developers to use the functions provided by SDKs more conveniently, but it also leads to a high degree of coupling between modules, limiting the free combination of modules. If you want to use SDKs to implement complex business scenarios, you need to customize and modify the SDK for the complex business scenarios, which makes it difficult to implement complex business scenarios.

[0055] To this end, a model deployment method is provided in this embodiment, which is applied to a development server to deploy a model to be deployed to an edge intelligent device. Figure 1 is a flow chart of the model deployment method of this embodiment. Figure 1 As shown, the process includes the following steps:

[0056] Step S101: The development server obtains the pipeline application.

[0057] The pipeline application of this embodiment includes a plug-in library and a pipeline corresponding to the model to be deployed. The plug-in library includes various types of plug-ins for multimedia data preprocessing, reasoning and post-processing. These plug-ins can be pre-set plug-ins or user-defined plug-ins written by users according to their own needs. The pipeline includes a combination of multiple plug-ins in the plug-in library.

[0058] The plug-in library is a set of pluggable components, each component (plug-in) is responsible for completing a specific task. Plug-ins usually follow certain interface specifications so that they can be used in combination in different contexts. The plug-in library is designed to improve the flexibility and scalability of the software system, allowing developers to add new functions or replace existing functions without modifying the core code. The plug-in library of this embodiment contains a set of plug-ins for implementing specific functions, each of which can be dynamically loaded and used by the pipeline application. The difference between the high degree of coupling between the API interface of the SDK is that the plug-in interface is designed to be loosely coupled, and the plug-in interface provides a high degree of flexibility, allowing developers to freely combine and replace plug-ins. The plug-in library encapsulates pipelines, i.e. pipelines. Pipelines are used to describe the link relationship between the plug-ins used by the model to be deployed, thereby coordinating the work between multiple plug-ins, ensuring that data can be passed from one plug-in to another in a predetermined order and logic, forming a complete data processing flow. In addition, the parameter configuration related to the plug-in is also described by pipelines.

[0059] Step S102: The development server obtains a basic Docker image, and packages the pipeline application into a Docker image to be deployed based on the basic Docker image.

[0060] A basic Docker image is provided in the development server. The basic Docker image includes the operating environment required to run the pipeline application, thereby reducing the number of software packages that need to be downloaded and installed in building the Docker image to be deployed and shortening the build time.

[0061] The basic Docker image of this embodiment includes but is not limited to the following operating environment related content:

[0062] 1) Python 3.8 development environment and necessary installation packages such as pip3, opencv-headless, matplotlib, etc. for the development of Python examples.

[0063] 2) GCC / G++ compilation environment, which can be used to compile C / C++ sample demos and generate executable programs.

[0064] 3) Typical demo examples and development guides, including Python, C / C++, and ipynb examples.

[0065] In addition, the basic docker image also includes remote access services, including but not limited to Jupyterlab services and / or SSHD services. JupyterLab is a web-based interactive development environment for writing and running code, creating and editing notebooks, managing files, visualizing data, etc. It supports multiple programming languages ​​​​such as Python, R, Julia, etc., and integrates multiple tools such as code editors, terminals, file browsers, etc. SSHD (SecureShell Daemon) is a daemon process that provides secure remote login services through the SSH (Secure Shell) protocol. The SSH protocol provides an encrypted network connection, allowing users to safely perform remote login, file transfer, and other operations in an insecure network environment.

[0066] The Jupyterlab service can map ports to enable remote browser access to the basic docker image (or the docker image to be deployed) container resources. The SSHD service can map ports to enable the terminal to use the SSH protocol to remotely access the basic docker image (or the docker image to be deployed) container resources.

[0067] In step S103, the development server imports the docker image to be deployed to the edge intelligent device, so that the edge intelligent device can create a container for running the pipeline application from the docker image to be deployed. When the pipeline application is run, the execution order of multiple plug-ins and the flow direction of data are determined based on the pipeline.

[0068] On the development server, use the docker command to obtain the docker image to be deployed, and save the docker image to be deployed as a tar package. Then import the tar package onto the edge smart device. The edge smart device of this embodiment uses a master-slave architecture to run the pipeline application. When importing the tar package file, one or more slaves of the edge smart device (used to run the smart engine) can be selected to bind the smart engine to the imported docker image to be deployed. After creating a container from the docker image to be deployed on the edge smart device, the container is associated with the bound smart engine. While the container is running, the smart engine will also be initialized and run to wait for tasks distributed by the master control.

[0069] Through the above steps, the plug-in provided by the plug-in library is used instead of the SDK, and the pipeline is used to organize the flow of data between plug-ins and the parameter configuration of plug-ins, so as to adapt to the logical requirements of various single-intelligent algorithm reasoning and multi-intelligent algorithm serial and parallel reasoning, thereby realizing the model deployment of various complex business scenarios based on the basic modules implemented by the plug-in. In view of the performance differences of various edge intelligent devices, the performance of edge intelligent devices can be fully utilized by increasing or decreasing the type or number of serial and parallel plug-ins.

[0070] In some of the embodiments, before obtaining the pipeline application, a pipeline application is written, and the use of Python and C / C++ programming languages ​​is supported. The above-mentioned plug-in library is integrated in the written pipeline application, and the pipeline encapsulated by the library can realize pipeline creation, pipeline module creation, pipeline module parameter configuration, module series connection, dynamic parameter modification, pipeline startup, pipeline destruction and other functions. At the same time, the library is associated with the intelligent engine management, and when the program is running, the intelligent engine initialization can be realized so that it is bound to the created pipeline. In the plug-in library, various modules that implement specific functions are independently created in the form of plug-ins, and each module can be combined with each other in series and parallel to realize data flow links. When writing the pipeline application, according to the model to be deployed, the link relationship between the plug-ins used by the model to be deployed and the parameter configuration of the plug-in are determined; based on the link relationship between the plug-ins used by the model to be deployed and the parameter configuration of the plug-in, the pipeline is combined.

[0071] Taking the case where the number of plug-ins in the plug-in library is 5 as an example, the link relationship between the plug-ins used by the model to be deployed is represented by a 5×5 link relationship table, as shown in Table 1.

[0072]

[0073] Table 1 Link relationship table of serial relationship

[0074] The above √ indicates that there is data interaction, and × indicates that there is no data interaction. As shown in the link relationship table in Table 1, the link relationship between the plug-ins is: plug-in 1>>plug-in 2>>plug-in 3>>plug-in 4>>plug-in 5.

[0075] Still taking the example of 5 plug-ins in the plug-in library, assuming that there is a parallel relationship in the model to be deployed, the link relationship between the plug-ins used by the model to be deployed is represented by a 5×5 link relationship table, as shown in Table 2.

[0076]

[0077] Table 2 Link relationship table with parallel relationship

[0078] As shown in the link relationship table in Table 1, the link relationship between the plug-ins is: plug-in 1 >> plug-in 2 >> plug-in 3 >> plug-in 4 >> plug-in 5, and plug-in 1 >> plug-in 3; that is, the data output by plug-in 1 not only flows directly into plug-in 2, but also flows directly into plug-in 3.

[0079] It can be seen that by utilizing the series-parallel link relationship, the functions of various plug-ins can be flexibly combined to realize various complex business scenarios.

[0080] In some of the embodiments, the plug-in library includes the following plug-ins: data source plug-in, decoding plug-in, inference plug-in, ID generation plug-in, event rule plug-in, OSD plug-in, encoding plug-in, event linkage plug-in, stream forwarding plug-in and file storage plug-in.

[0081] The data source plug-in is used to obtain code stream data to obtain local videos or pictures, network streams, and NVR streams.

[0082] The decoding plug-in is used to decode the code stream data according to the decoding parameters configured by the user to obtain the decoded data; the configuration parameters of the decoding plug-in include the decoding parameters.

[0083] The inference plug-in is used to use the model inference type and model path configured by the user to perform model inference on the decoded data and obtain the inference result; the configuration parameters of the inference plug-in include the type and path of the inference model.

[0084] The ID generation plug-in is used to generate a unique identifier for the detected object, thereby uniquely identifying each detected object and achieving target recognition.

[0085] The event rule plug-in is used to filter the external parameters of the operation reasoning results. The event rule module supports multi-channel multi-rule settings, multi-channel OSD overlay settings, etc., algorithm result external conversion, etc. External parameters include external rules and encoding parameters.

[0086] The OSD (On-Screen Display) plug-in is used to overlay information on the display screen, including overlay category labels, target IDs, confidence levels, etc. The configuration parameters of the OSD plug-in include the type of overlay information.

[0087] The encoding plug-in is used to encode YUV data into jpg format, raw stream (Raw Stream) or ps (Program Stream) data stream. The configuration parameters of the encoding plug-in include encoding parameters. Raw stream refers to a video data stream that has not been encapsulated or has a low degree of encapsulation. The raw stream usually only contains the original compressed encoded data, and does not contain additional container format information or metadata. PS data stream is a variant of MPEG-2 Transport Stream (TS), which is usually used to encapsulate multiple media streams (such as video, audio, subtitles, etc.) together and contains metadata such as timestamps. PS data stream is part of the MPEG-2 standard.

[0088] The event linkage plug-in is used to report the processing results of multimedia data to the server, such as reporting the alarm information and image data generated after algorithm analysis to the server.

[0089] The stream forwarding plug-in is used to convert the stream data into a media stream of the real-time streaming protocol or the hypertext transfer protocol, thereby providing a preview function.

[0090] The file storage plug-in is used to save the processing results of multimedia data locally.

[0091] In addition to the above plug-ins, developers can also import user-defined plug-ins in the plug-in library to implement customized function modules.

[0092] The following will be based on the plug-in library in the above embodiment, including data source plug-in, decoding plug-in, reasoning plug-in, ID generation plug-in, event rule plug-in, OSD plug-in, encoding plug-in, event linkage plug-in, code stream forwarding plug-in and file storage plug-in, to explain the link relationship of target detection, classification detection and semantic segmentation models. In the pipeline application example to be described below, the analysis data all comes from the data source plug-in. Each pipeline application example calls the plug-in interface provided by the plug-in library through the constructed pipeline to configure the real-time stream, historical stream and image stream address to the data source plug-in. After the data source plug-in obtains the stream, it gives it to other plug-ins described in the pipeline for data analysis and processing.

[0093] Figure 2 It is a schematic diagram of the algorithm of the target detection model of this embodiment, such as Figure 2As shown in the figure, the target detection model uses data source plug-in, decoding plug-in, reasoning plug-in (target detection reasoning and attribute reasoning), ID generation plug-in, event rule plug-in, OSD plug-in, encoding plug-in, stream forwarding plug-in, file storage plug-in and event linkage plug-in. Among them, the output data of the decoding plug-in is used for target detection reasoning, and the output data of the ID generation plug-in is used for attribute reasoning information, which is determined by the pipeline's description of the plug-in parameter configuration. Figure 2 The link relationship table of the target detection model shown is shown in Table 3.

[0094]

[0095] Table 3 Link relationship table of target detection model

[0096] Figure 3 is a schematic diagram of the algorithm of the classification detection model of this embodiment, such as Figure 3 As shown in the figure, the classification detection model uses a data source plug-in, a decoding plug-in, an inference plug-in (target classification inference, such as cat and dog classification), an event rule plug-in, an OSD plug-in, an encoding plug-in, a stream forwarding plug-in, a file storage plug-in, and an event linkage plug-in.

[0097] Figure 4 is a schematic diagram of the algorithm of the semantic segmentation model of this embodiment, such as Figure 4 As shown in the figure, the semantic segmentation model uses a data source plug-in, a decoding plug-in, an inference plug-in (instance segmentation and attribute inference), an ID generation plug-in, an event rule plug-in, an OSD plug-in, an encoding plug-in, a stream forwarding plug-in, a file storage plug-in, and an event linkage plug-in.

[0098] As can be seen from the above three pipeline application examples, by using pipeline and plug-in library and connecting plug-ins in series and parallel, various complex business scenarios can be implemented conveniently and quickly.

[0099] In an embodiment where a remote access service is also built into the base docker image, the remote access service can also be used to modify the pipeline application that has been deployed to the edge smart device. For example, after the container running the pipeline application is started, the JupyterLab service can be used to remotely access container resources, including resources such as the provided typical demo examples. Developers can be supported to modify typical demo examples in the JupyterLab service, or to rewrite demo examples and compile C / C++demo examples. The SSHD service allows developers to access container resources in the terminal, and use the vim editor to also modify and reconstruct typical demo examples.

[0100] This embodiment also provides a model operation method applied to an edge intelligent device. Figure 5 is a flow chart of the model operation method of this embodiment, such as Figure 5 As shown, the process includes the following steps:

[0101] Step S501: The edge smart device obtains the docker image to be deployed.

[0102] In step S501, the docker image to be deployed is obtained by packaging the pipeline application from the basic docker image. The basic docker image includes the operating environment required to run the pipeline application. The pipeline application includes a plug-in library and a pipeline corresponding to the model to be deployed. The plug-in library includes plug-ins for multimedia data preprocessing, reasoning and post-processing. The pipeline includes a combination of multiple plug-ins in the plug-in library.

[0103] In step S502, the edge intelligent device creates a container for running the pipeline application from the docker image to be deployed, loads the container to run the pipeline application, and obtains the running result of the model to be deployed.

[0104] In step S502, when the pipeline application is run, the execution order of multiple plug-ins and the flow direction of data are determined based on the pipeline.

[0105] In some of the embodiments, the edge intelligent device applies a master-slave architecture, where the master performs data sending, forwarding, and storage, while the slave chip (running an intelligent engine) performs encoding and decoding, model reasoning, post-processing (such as the processing of the above-mentioned event rule plug-in), and other functions.

[0106] In some of the embodiments, step S502 includes: the master of the edge smart device loads the container, binds the container to the slave of the edge smart device, and initializes the slave so that the slave loads hardware resources and software resources, and the software resources include pipelines and model resources. In the plug-in library of this embodiment, various modules required for the business are created, and a data structure connected front and back is constructed to realize the construction of the pipeline. While building the pipeline, the intelligent engine is also initialized, and the intelligent engine resources (including pipelines, model resources, etc.) are loaded. When the pipeline is successfully built, the intelligent engine is also initialized. The data source plug-in will be loaded on the master to obtain data, and the file storage plug-in, the code stream forwarding plug-in, and the event linkage plug-in will be loaded at the same time to store, forward, and link events for the images and data encoded by the model reasoning. At the same time, the master will send the constructed pipeline together with the model resources to the intelligent engine for processing.

[0107] On the slave chip, the slave chip parses the pipeline to obtain the parameter configuration of the plug-in and the link relationship between the plug-ins used in the model to be deployed, loads the parameter configuration into the model resource corresponding to the plug-in, executes the task, and receives and sends data according to the link relationship between the plug-ins used in the model to be deployed, thereby opening up the data reception and processing link.

[0108] In some of the embodiments, the basic Docker image also includes a remote access service, and a mapping port is configured between the container and the edge smart device, so that the remote access service of the container is accessible on the edge smart device. The user can use the remote access service on the edge smart device to access the container resources of the edge smart device: the edge smart device receives the user's remote access request; the edge accesses the remote access service of the container based on the mapping port configured between the container and the edge smart device, so that the container modifies and reconstructs the pipeline based on the remote access request.

[0109] In some of the embodiments, the plug-in library includes the following plug-ins: data source plug-in, decoding plug-in, inference plug-in, ID generation plug-in, event rule plug-in, encoding plug-in, OSD plug-in, event linkage plug-in, stream forwarding plug-in and file storage plug-in.

[0110] The data source plug-in is used to obtain code stream data to obtain local videos or pictures, network streams, and NVR streams.

[0111] The decoding plug-in is used to decode the code stream data according to the decoding parameters configured by the user to obtain the decoded data; the configuration parameters of the decoding plug-in include the decoding parameters.

[0112] The inference plug-in is used to use the model inference type and model path configured by the user to perform model inference on the decoded data and obtain the inference result; the configuration parameters of the inference plug-in include the type and path of the inference model.

[0113] The ID generation plug-in is used to generate a unique identifier for the detected object, thereby uniquely identifying each detected object and achieving target recognition.

[0114] The event rule plug-in is used to filter the external parameters of the operation reasoning results. The event rule module supports multi-channel multi-rule settings, multi-channel osd overlay settings, algorithm result external conversion, etc. External parameters include external rules and encoding parameters.

[0115] The OSD (On-Screen Display) plug-in is used to overlay information on the display screen, including overlay category labels, target IDs, confidence levels, etc. The configuration parameters of the OSD plug-in include the type of overlay information.

[0116] The encoding plug-in is used to encode YUV data into jpg format, raw stream or ps (Program Stream) data stream. The configuration parameters of the encoding plug-in include encoding parameters. Raw stream refers to a video data stream that has not been encapsulated or has a low degree of encapsulation. The raw stream usually only contains the original compressed encoded data, and does not contain additional container format information or metadata. PS data stream is a variant of MPEG-2 Transport Stream (TS), which is usually used to encapsulate multiple media streams (such as video, audio, subtitles, etc.) together and contains metadata such as timestamps. PS data stream is part of the MPEG-2 standard.

[0117] The event linkage plug-in is used to report the processing results of multimedia data to the server, such as reporting the alarm information and image data generated after algorithm analysis to the server.

[0118] The stream forwarding plug-in is used to convert the stream data into a media stream of the real-time streaming protocol or the hypertext transfer protocol, thereby providing a preview function.

[0119] The file storage plug-in is used to save the processing results of multimedia data locally.

[0120] In addition to the above plug-ins, developers can also import user-defined plug-ins in the plug-in library to implement customized modules.

[0121] This embodiment also provides a non-transitory machine-readable medium storing computer instructions, which are used to enable a computer to execute the above-mentioned model deployment method or model operation method.

[0122] In summary, the model deployment method and model operation method provided by the above-mentioned embodiments of the present application, for edge intelligent devices, make each functional module independent, reduce the coupling between modules, and open up the data flow receiving and forwarding link through serial and parallel methods to realize the model reasoning function. At the same time, the pipeline application that realizes modular linking is inserted into the basic docker image to obtain the docker image to be deployed, which can be freely imported or exported to the edge intelligent device. The Python3 development environment, GCC / G++ compilation environment, Jupyterlab service, and SSHD service are integrated in the docker image to be deployed. Developers can develop and compile and debug pipeline applications in the docker image to be deployed. This application is based on the master-slave architecture of edge intelligent devices, and can realize modular offline deployment of open source models on edge intelligent devices by linking business function pipeline links and importing docker image packages. In addition, the model deployment method provided in this embodiment can adapt to the logical requirements of multiple single intelligent algorithm reasoning and multi-intelligent algorithm serial and parallel reasoning, and can use target detection, target recognition, classification detection, segmentation detection, key point detection and other reasoning models as reasoning modules, and construct reasoning units in a serial and parallel manner, so as to fully utilize the reasoning performance of edge intelligent devices.

[0123] It should be noted that the term "including" and its variations used in the embodiments of the present invention are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "multiple" mentioned in the embodiments of the present invention are illustrative and not restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0124] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0125] The various steps described in the method implementation methods provided by the embodiments of the present invention can be performed in different orders and / or in parallel. In addition, the method implementation methods may include additional steps and / or omit the steps shown. The scope of protection of the present invention is not limited in this respect.

[0126] The term "embodiment" in this specification refers to specific features, structures or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of the invention. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. The various embodiments in this specification are described in a related manner, and the same and similar parts between the various embodiments refer to each other. In particular, for the device, equipment, and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts refer to the partial description of the method embodiment.

[0127] The above-described embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of protection. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the attached claims.

Claims

1. A model operation method, applied to edge intelligent devices, characterized in that include: Obtain a docker image to be deployed, wherein the docker image to be deployed is obtained by packaging a pipeline application program with a basic docker image, wherein the basic docker image includes an operating environment required to run the pipeline application program, wherein the pipeline application program includes a plug-in library and a pipeline corresponding to the model to be deployed, wherein the plug-in library includes plug-ins for multimedia data preprocessing, reasoning, and post-processing, and the pipeline includes a combination of multiple plug-ins in the plug-in library; Creating a container for running the pipeline application from the Docker image to be deployed, loading the container to run the pipeline application, and obtaining the running result of the model to be deployed, wherein the pipeline application determines the execution order of the multiple plug-ins and the flow direction of data based on the pipeline when the pipeline application is run; Among them, loading the container to run the pipeline application includes: the master control of the edge intelligent device loads the container and binds the container to the slave chip of the edge intelligent device; while building the pipeline, the slave chip is initialized so that the slave chip loads hardware resources and software resources, and the software resources include the pipeline and model resources; the master control loads the data source plug-in to obtain data, and at the same time loads the file storage plug-in, the code stream forwarding plug-in, and the event linkage plug-in to store, forward and perform event linkage operations on the images and data encoded by the model inference, and send the pipeline and the model resources to the slave chip; the slave chip parses the pipeline to obtain the parameter configuration of the plug-in and the link relationship between the plug-in used by the model to be deployed, loads the parameter configuration into the model resource corresponding to the plug-in, executes the task, and receives and sends data according to the link relationship between the plug-ins used by the model to be deployed.

2. The method according to claim 1, characterized in that: The basic Docker image also includes a remote access service, and a mapping port is configured between the container and the edge smart device so that the remote access service of the container is accessible on the edge smart device; the method also includes: Receive remote access requests from users; Based on the mapped port configured between the container and the edge intelligent device, the remote access service of the container is accessed, so that the container modifies and reconstructs the pipeline based on the remote access request.

3. The method according to claim 1, characterized in that The plug-in library includes the following plug-ins: data source plug-in, decoding plug-in, inference plug-in, ID generation plug-in, event rule plug-in, OSD plug-in, encoding plug-in, event linkage plug-in, code stream forwarding plug-in and file storage plug-in; wherein, The data source plug-in is used to obtain code stream data; The decoding plug-in is used to decode the code stream data to obtain decoded data; the configuration parameters of the decoding plug-in include decoding parameters; The inference plug-in is used to perform model inference on the decoded data using the inference model to obtain an inference result; the configuration parameters of the inference plug-in include the type and path of the inference model; The ID generation plug-in is used to generate a unique identifier for the detected object; The event rule plug-in is used to filter the external parameters of the operation of the inference result, and the external parameters include external rules and encoding parameters; The OSD plug-in is used to overlay information on the display screen, and the configuration parameters of the OSD plug-in include the type of overlay information; The encoding plug-in is used to encode YUV data, and the configuration parameters of the encoding plug-in include encoding parameters; The event linkage plug-in is used to report the processing result of the multimedia data to the server; The code stream forwarding plug-in is used to convert the code stream data into a media stream of a real-time streaming protocol or a hypertext transfer protocol; The file storage plug-in is used to save the processing results of the multimedia data locally.

4. A non-transitory machine-readable medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 3.

Images