Application deployment method based on swan-gap equipment, swan-gap equipment and storage medium

Abstract

The invention provides an application deployment method based on a swan-gap device, the swan-gap device and a computer readable storage medium. The application deployment method based on the swan gap equipment comprises the steps of loading a to-be-deployed application on a container mirror image of the swan gap equipment to create a virtual server of the to-be-deployed application; when a target camera accesses the gap equipment, controlling a gap end application of the gap equipment to create a virtual client of the target camera; after the virtual server establishes communication connection with the virtual client of the target camera, controlling the virtual client of the target camera to transmit a service starting request to the virtual server; and controlling the virtual server to drive the to-be-deployed application on the container mirror image to execute the service starting request. According to the application, the development, transplantation and compilation workload of deployment and operation of the AI application can be reduced, and the deployment efficiency of the AI application is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and specifically to an application deployment method based on Hongmeng devices, Hongmeng devices, and computer-readable storage media. Background Art

[0002] AI (Artificial Intelligence) applications refer to programs or systems that use artificial intelligence technology to solve specific problems or provide certain services. With the rapid development of artificial intelligence technology, AI applications are increasingly being used on computer devices (such as Hongmeng devices). For example, AI applications such as smart transportation, smart water conservancy, smart fire detection, and smart security can be deployed on Hongmeng devices.

[0003] In the related technology, to complete the deployment and operation of AI applications on computer devices (such as Hongmeng devices), it is necessary to complete porting, compilation and operation on the computer devices (such as Hongmeng devices); therefore, the chip of the computer device needs to have corresponding dependent library code or cross-compilation tool chain to support the deployment and operation of AI applications. However, in some cases, the chip does not have the corresponding dependent library code or cross-compilation tool chain support. For example, the chip manufacturer does not provide the corresponding dependent library code during the chip selection stage. Therefore, the deployment and operation of AI applications on computer devices (such as Hongmeng devices) requires a certain amount of development, porting and compilation work, and the deployment workload of AI applications is relatively large. Summary of the invention

[0004] The present application provides an application deployment method based on Hongmeng devices, Hongmeng devices and computer-readable storage media, which can reduce the development, transplantation and compilation workload of AI application deployment and improve the deployment efficiency of AI applications.

[0005] In a first aspect, the present application provides an application deployment method based on a Hongmeng device, the method comprising:

[0006] Load the application to be deployed on the container image of the Hongmeng device to create a virtual server for the application to be deployed;

[0007] When the target camera is connected to the Hongmeng device, the Hongmeng application controlling the Hongmeng device creates a virtual client of the target camera;

[0008] After the virtual server and the virtual client of the target camera establish a communication connection, control the virtual client of the target camera to transmit a service start request to the virtual server;

[0009] The virtual server is controlled to drive the application to be deployed on the container image to execute the service startup request.

[0010] In some embodiments, the application to be deployed on the container image includes a detection module, and after the virtual server and the virtual client of the target camera establish a communication connection, controlling the virtual client of the target camera to transmit a service startup request to the virtual server includes:

[0011] After the virtual server and the virtual client of the target camera establish a communication connection, control the Hongmeng end application to call a preset service startup interface so that the virtual client of the target camera sends a service startup request to the virtual server, wherein the service startup request carries the video stream protocol address of the target camera;

[0012] The controlling the virtual server to drive the application to be deployed on the container image to execute the service startup request includes:

[0013] Controlling the virtual server to read the video stream data of the target camera according to the video stream protocol address;

[0014] The virtual server is controlled to drive a detection module to perform service processing according to the video stream data.

[0015] In some embodiments, the business processing includes inference processing, and the controlling the virtual server to drive the detection module to perform business processing according to the video stream data includes:

[0016] Controlling the virtual server to decode the video stream data to obtain decoded image frames;

[0017] The virtual server is controlled to drive the detection module to perform inference processing on the decoded image frame to obtain inference result information of the decoded image frame.

[0018] In some embodiments, the business processing further includes drawing processing, the application to be deployed on the container image includes a drawing module, and the method further includes:

[0019] Controlling the virtual server to drive the drawing module to perform drawing processing on the decoded image frame according to the inference result information to obtain a drawn image frame of the decoded image frame, wherein the inference result information includes at least one of an object detection frame and an object detection category;

[0020] Controlling the virtual server to encode the drawn image frame to obtain an encoded video stream;

[0021] Control the virtual server to push the encoded video stream to visualize the reasoning result information, and control the virtual server to report the reasoning result information to the virtual client of the target camera, so that the Hongmeng application responds and processes according to the reasoning result information.

[0022] In some embodiments, the controlling the virtual server to drive the drawing module to perform drawing processing on the decoded image frame according to the inference result information to obtain a drawing image frame of the decoded image frame includes:

[0023] Controlling the virtual server to obtain frame information of the decoded image frame;

[0024] Control the virtual server to drive the drawing module to perform drawing processing on the decoded image frame according to the inference result information and the frame information to obtain a drawn image frame of the decoded image frame, wherein the frame information includes at least one of the name of the target camera, the timestamp of the decoded image frame, and alarm information.

[0025] In some embodiments, the method further comprises:

[0026] Control the Hongmeng terminal application to call a preset model loading interface so that the virtual client of the target camera sends a model loading request to the virtual server;

[0027] When the virtual server receives the model loading request, the virtual server is controlled to load the task model of the target camera according to the model path carried in the model loading request.

[0028] In some embodiments, the method further comprises:

[0029] After the virtual client of the target camera is created, the Hongmeng application is controlled to call a preset connection interface so that the virtual client of the target camera sends a connection request to the virtual server;

[0030] When the virtual server receives the connection request, the virtual server is controlled to feed back response information of the connection request to the virtual client of the target camera, so as to establish a communication connection between the virtual server and the virtual client of the target camera.

[0031] In some embodiments, the method further comprises:

[0032] After sending a service start request to the virtual server, the Hongmeng application is controlled to call a preset stop function interface so that the virtual client of the target camera sends a stop function request to the virtual server.

[0033] On the second aspect, the present application also provides a HarmonyOS device, which includes a processor and a memory, in which a computer program is stored. When the processor calls the computer program in the memory, it executes any one of the application deployment methods based on the HarmonyOS device provided in the present application.

[0034] On the third aspect, the present application also provides a computer-readable storage medium on which a computer program is stored, and the computer program is loaded by a processor to execute the application deployment method based on Hongmeng device.

[0035] In this application, by loading the application to be deployed on the container image of the Hongmeng device, a virtual server of the application to be deployed can be created, and a virtual client of the target camera can be created through the Hongmeng end application. After the virtual server and the virtual client of the target camera establish a communication connection, the virtual client of the target camera transmits a service startup request to the virtual server, and controls the virtual server to drive the application to be deployed on the container image to execute the service startup request. On the first hand, by driving the application to be deployed on the container image loaded on the Hongmeng device, the application to be deployed can be run on the Hongmeng device; and a communication connection is established based on the virtual server and the virtual client of the target camera, so that the Hongmeng end application can drive the application to be deployed on the container image to run. On the second hand, since the container image environment loaded on the Hongmeng device can support compilation and operation at the same time, there is no need to build a cross-compilation environment to complete the compilation and operation of the application to be deployed, which reduces the workload of application transplantation and adaptation to the Hongmeng device and improves the efficiency of application deployment. Therefore, the embodiment of the present application can reduce the development, transplantation and compilation workload of AI application deployment and operation, and improve the deployment efficiency of AI applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 It is a flow chart of an application deployment method based on Hongmeng devices provided in an embodiment of the present application;

[0038] Figure 2 This is a schematic diagram of the deployment architecture of AI applications in Hongmeng devices in an embodiment of the present application;

[0039] Figure 3 is a schematic diagram of the corresponding relationship between the camera, the virtual client and the virtual server in the embodiment of the present application;

[0040] Figure 4 is a schematic diagram of the interaction between the virtual client and the virtual server in an embodiment of the present application;

[0041] Figure 5 This is a business timing diagram of the AI ​​application deployment in the embodiment of the present application;

[0042] Figure 6 It is a structural schematic diagram of the Hongmeng device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0044] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0045] In the description of the embodiments of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0046] In order to enable any person skilled in the art to implement and use the present application, the following description is provided. In the following description, details are listed for the purpose of explanation. It should be understood that those of ordinary skill in the art can recognize that the present application can also be implemented without using these specific details. In other examples, the known process will not be elaborated in detail to avoid unnecessary details that make the description of the present application embodiment obscure. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest range of principles and features disclosed in accordance with the embodiments of the present application.

[0047] The embodiments of the present application provide an application deployment method based on Hongmeng device, Hongmeng device and computer-readable storage medium.

[0048] Some embodiments of the present application are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0049] See also Figure 1 , Figure 1 A flowchart of an application deployment method based on Hongmeng devices provided in an embodiment of the present application. The application deployment method based on Hongmeng devices includes steps 101 to 104, wherein:

[0050] 101. Load the application to be deployed on the container image of the Hongmeng device to create a virtual server for the application to be deployed.

[0051] The virtual server is used to run the application to be deployed.

[0052] Among them, Hongmeng devices can be hardware devices installed with Hongmeng system, such as mobile phones, computers, etc. installed with Hongmeng system.

[0053] Among them, the applications to be deployed are AI applications that need to be deployed on the Docker image of Hongmeng devices, such as smart transportation, smart water conservancy, smart fire detection, smart security and other functional applications.

[0054] In order to better understand the embodiments of the present application, refer to Figure 2 and Figure 3 The following first introduces the software and hardware environment involved in the deployment and operation of AI applications on the Hongmeng device in the embodiment of this application, as well as some names:

[0055] 1. Hardware environment includes:

[0056] 1.1. Multiple cameras and a router: Multiple cameras and a router form a local area network.

[0057] 1.2. A Hongmeng device: The Hongmeng device is connected to a router and is in the same local area network as multiple cameras. The Hongmeng device may include a CPU and other necessary components for executing instructions.

[0058] 1.3 Graphics card (graphics card chip), which is installed on the Hongmeng device. The graphics card can include GPU and other necessary components (such as video memory, etc.).

[0059] 2. Software environment includes:

[0060] 2.1. Hongmeng devices need to meet the following software conditions: ① install Hongmeng system OpenHarmony, ② adapt Hongmeng devices to graphics card drivers, ③ complete Docker image deployment, ④ deploy Hongmeng applications. Hongmeng applications define the Init interface (i.e. connection interface), InitMode interface (i.e. model loading interface), Start interface (i.e. service startup interface) and Stop interface (i.e. stop function interface). Hongmeng applications define the Init interface, InitMode interface, Start interface and Stop interface to complete the TCP connection, model loading, service startup and stop functions of the virtual client newly connected to the camera in Hongmeng devices. Therefore, by defining a set of client interface protocols, when developing on different graphics card device platforms, Hongmeng applications can fix a set of codes to adapt to different platforms, which reflects the versatility of the current solution and improves development efficiency. Therefore, by defining a set of universal interfaces in Hongmeng applications, the versatility of multiple platform devices accessing Hongmeng devices is achieved at the interface level. For example, the pseudo code for defining the client interface protocol can be as follows:

[0061] #1. Import AI related modules

[0062] import combriconProxy from"libcombricon proxy.so

[0063] #2 Create an AI Inference Client

[0064] let proxy=await combriconProxy.createProxy()

[0065] #3. Initialize the proxy (i.e. the protocol of the Init interface)

[0066] await proxy.init()

[0067] await this.client.init(rtspUrl)

[0068] #4. Sending model (i.e. the protocol of InitMode interface)

[0069] await this.client.initMode(" / data / work / model / yolov5.mode")

[0070] #5, start reasoning (i.e. the protocol of the Start interface)

[0071] Let rtspUrl="rtsp: / / admin:water123.100.193.205:554 / Streaming / Channels / 101

[0072] this.client.startRtsp(rtspUrl)

[0073] #6, stop reasoning (i.e. the protocol of the Stop interface)

[0074] await this.client.stopRtsp()

[0075] 2.2 Graphics card demo: The graphics card demo runs on the Docker image. In this embodiment, the application to be deployed is an application that implements functions such as a detection module, a drawing module, a streaming module, and a reporting module based on an algorithm model. Most of the functions of the application to be deployed can be implemented through a virtual server; a part of the functions of the application to be deployed can be loaded into the graphics card as a graphics card demo and implemented based on the graphics card operation.

[0076] 3. Virtual server (also referred to as Docker Server or Server in this article): A virtual server (i.e., a server based on software implementation) is used to drive the application to be deployed on the Hongmeng device in the container image (i.e., Docker image) environment, thereby realizing the operation of the application to be deployed. When the application to be deployed is driven in the Docker image environment, it is equivalent to creating a virtual server. In the embodiment of the present application, on the one hand, it is necessary to develop an application (i.e., Hongmeng end application) on the Hongmeng device, and create one or more virtual clients through the Hongmeng end application, each virtual client corresponding to a camera; on the other hand, it is necessary to start the container image (i.e., Docker image) on the Hongmeng device, and the virtual server will run on the container image (i.e., Docker image), thereby creating a virtual server for the application to be deployed on the Hongmeng device. In this embodiment, by loading the application to be deployed on the Docker image to run, it has the following advantages: more systems are supported (such as supporting various types of Windows and Linux); only the operating environment required by the virtual software is virtualized, and useless configuration is minimized to the maximum extent, resource consumption is reduced, and the efficiency of AI application deployment is improved; different systems can adopt the same deployment method to improve the versatility of AI application deployment.

[0077] 4. Virtual client (also referred to as Host Client or Client in this article): A virtual client (i.e. a client implemented based on software) refers to a client created by the Hongmeng application.

[0078] Exemplarily, first, a container image (i.e., Docker image) is made through Dockerfile, and the container image (i.e., Docker image) is loaded onto the Hongmeng device. Then, a client service is developed on the Hongmeng application to create a virtual client. The detailed implementation of creating a virtual client can refer to the relevant instructions of step 102, which will not be repeated here; and a server service is developed in the container image (i.e., Docker image) to create a virtual server for the application to be deployed of the Hongmeng device. In some embodiments, the application to be deployed installed on the Hongmeng device is driven to run by the container image loaded on the Hongmeng device, and a virtual server for the application to be deployed of the Hongmeng device can be created; in addition, since compilation and running can be supported simultaneously on the container image (i.e., Docker image), there is no need to build a cross-compilation environment to complete the compilation and running of the application to be deployed, reducing the workload of transplanting AI applications to adapt to Hongmeng devices.

[0079] 102. When the target camera is connected to the Hongmeng device, the Hongmeng application that controls the Hongmeng device creates a virtual client of the target camera.

[0080] Among them, the target camera refers to the camera connected to the HarmonyOS device among multiple cameras in the same local area network as the HarmonyOS device.

[0081] In some embodiments, the Hongmeng application can create multiple virtual clients, each virtual client corresponds to a camera, and each time a new camera is connected to the Hongmeng device, the Hongmeng application will create a new virtual client. Therefore, when the target camera is connected to the Hongmeng device, the Hongmeng application will create a new virtual client as the virtual client of the target camera.

[0082] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the deployment architecture of AI applications in Hongmeng devices in this embodiment. When the target camera is connected to the Hongmeng device, the Hongmeng end application (reference Figure 2 Create a virtual client of the target camera (see "SDK App" in the Figure 2 After that, you can create a virtual client and virtual server for the target camera (see Figure 2 The communication connection between the virtual client of the target camera and the virtual server is realized through the request and response between the virtual client of the target camera and the virtual server, so as to realize the command interaction between the target camera and the AI ​​application. For example, the virtual client Client and the virtual server Server can complete the C / S request and response through the Socket communication method. The detailed request and response implementation process will be introduced later, and will not be repeated here for the sake of simplicity.

[0083] 103. After the virtual server and the virtual client of the target camera establish a communication connection, control the virtual client of the target camera to transmit a service start request to the virtual server.

[0084] 104. Control the virtual server to drive the application to be deployed on the container image to execute the service startup request.

[0085] Please refer to Figure 5 For example, after the virtual server and the virtual client of the target camera establish a communication connection, the Hongmeng application is controlled to call the preset service startup interface, so that the virtual client of the target camera sends a service startup request carrying service information to the virtual server, thereby delivering the service startup request to the virtual server, thereby utilizing the application to be deployed to perform processing tasks corresponding to the service function according to the service startup request, and completing the deployment and operation of the AI ​​application on the Hongmeng device.

[0086] There are many ways to implement step 104, illustratively including:

[0087] (1) The service start request carries the video stream protocol address of the target camera. Figure 5 At this time, step 104 may specifically include the following steps 1041A to 1043A:

[0088] 1041A. When the virtual server receives the service start request, the virtual server is controlled to parse the service start request to obtain the video stream protocol address of the target camera.

[0089] The service start request carries the video stream protocol address of the target camera.

[0090] like Figure 5 As shown in the figure, when the business manager initiates a business request in the Hongmeng application, first, the Hongmeng application calls the Init interface to establish a Socket connection between the virtual server and the virtual client of the target camera (reference Figure 5 After the Socket connection is successful, the Hongmeng application will call the InitMode interface, which is equivalent to specifying the model path corresponding to the virtual client Client, and the virtual server Server will complete the model loading according to the model path (refer to Figure 5 The virtual client Client will bind a camera (corresponding to an rtsp address, i.e., the video stream protocol address), and the Hongmeng application completes the rtsp address transmission by calling the Start interface (refer to Figure 5The application to be deployed starts the corresponding service, and the virtual server Server will pull the stream (through live555, refer to Figure 5 ), and completes video stream decoding, inference detection, OSD (On-Screen Display) processing (in this embodiment, OSD processing includes but is not limited to: superimposing text, graphics or other information in the image frame), video stream encoding and other processes (refer to Figure 5 ), and then push the encoded video stream (refer to Figure 5 At the same time, the virtual server Server sends the inference result information to the virtual client Server through the Socket connection, and then returns it to the Hongmeng application. The Hongmeng application gives the next action based on the inference result information.

[0091] 1042A. Control the virtual server to read the video stream data of the target camera according to the video stream protocol address.

[0092] 1043A. Control the virtual server to drive the detection module to perform business processing according to the video stream data.

[0093] There are multiple implementations of step 1043A, illustratively including the following method 1 and method 2:

[0094] Mode 1: In some embodiments, the business processing includes reasoning processing. In this case, step 1043A may specifically include the following steps A1 to A2:

[0095] A1. Control the virtual server to decode the video stream data to obtain decoded image frames.

[0096] A2. Control the virtual server to drive the detection module to perform inference processing on the decoded image frame to obtain inference result information of the decoded image frame.

[0097] Furthermore, after the virtual server obtains the inference result information of the decoded image frame, the inference result information can also be fed back to the virtual client. In some embodiments, the virtual server can directly feed back the inference result information (such as object detection category, number of detections, etc.) to the virtual client. At this time, after step A2, the following steps can also be included: control the virtual server to send the inference result information to the virtual client of the target camera, so that the Hongmeng end application makes the next response operation according to the inference result information, thereby meeting the AI ​​application deployment requirements of the business scenario. In some embodiments, the virtual server can first draw the inference result information (such as detection box, detection category) and other information onto the decoded image frame to obtain a drawn image frame, and then feed back the drawn image frame to the virtual client. The specific implementation process can refer to steps B1 to B4.

[0098] Mode 2: In some embodiments, the business processing includes reasoning processing and rendering processing. In this case, step 1043A may specifically include the following steps B1 to B2:

[0099] B1. Control the virtual server to decode the video stream data to obtain decoded image frames.

[0100] B2. Control the virtual server to drive the detection module to perform inference processing on the decoded image frame to obtain inference result information of the decoded image frame.

[0101] For example, in a fire detection scenario, the application to be deployed may be an intelligent fire alarm detection AI application, and the target camera may be a camera deployed at a fire detection site. Through communication between the virtual server and the virtual client, the virtual server may read the video stream data of the camera deployed at the fire detection site. The virtual server may decode the video stream data to obtain a decoded image frame. The virtual server drives the detection module to perform fire detection on the decoded image frame, and obtains the fire detection result as the inference result information of the decoded image frame.

[0102] B3. Control the virtual server to drive the drawing module to perform drawing processing on the decoded image frame according to the inference result information to obtain a drawn image frame of the decoded image frame.

[0103] The inference result information includes at least one of an object detection frame and an object detection category.

[0104] The drawn image frame is an image frame obtained after performing OSD processing such as superimposing text, graphics or other information on the decoded image frame.

[0105] There are multiple drawing processing methods for step B3. In some embodiments, the reasoning result information can be drawn onto the decoded image frame. In this case, step B3 can specifically include: controlling the virtual server to drive the drawing module to superimpose the reasoning result information onto the decoded image frame to obtain a drawn image frame of the decoded image frame. In some embodiments, the reasoning result information and the frame information of the decoded image frame can be drawn onto the decoded image frame together. In this case, step B3 can specifically include: controlling the virtual server to obtain the frame information of the decoded image frame; controlling the virtual server to drive the drawing module to draw the decoded image frame according to the reasoning result information and the frame information to obtain the drawn image frame of the decoded image frame, wherein the frame information includes at least one of the name of the target camera, the timestamp of the decoded image frame, and the alarm information.

[0106] B4. Control the virtual server to encode the drawn image frame to obtain an encoded video stream.

[0107] B5. Control the virtual server to push the encoded video stream to visualize the reasoning result information, and control the virtual server to report the reasoning result information to the virtual client of the target camera, so that the Hongmeng application responds and processes according to the reasoning result information.

[0108] (2) The service initiation request carries the image acquisition data of the target camera. In this case, step 104 may specifically include the following 1041B to 1042B:

[0109] 1041B. When the virtual server receives the service start request, the virtual server is controlled to parse the service start request to obtain the image acquisition data of the target camera.

[0110] The service initiation request carries the image acquisition data of the target camera.

[0111] 1042B. Control the virtual server to drive the detection module to perform business processing according to the image acquisition data.

[0112] For ease of understanding, the following describes the interaction between the virtual client and the virtual server. Figure 3 and Figure 4 , Figure 3 Schematic diagram of the corresponding relationship between the camera, virtual client and virtual server in this embodiment, Figure 4 Figure 1 is a schematic diagram of the interaction between the virtual client and the virtual server. Figure 3 As shown, Hongmeng applications can create multiple virtual clients (refer to Figure 3Each virtual client corresponds to a camera (refer to Figure 3 When the container image (i.e., Docker image) drives the application to be deployed on the Hongmeng device to run, a single virtual server Server is created, and a virtual server Server can interact with multiple virtual clients Client. Figure 4 As shown, the virtual client Client and the virtual server Server complete the interaction process through interface calls, including the following interface calls:

[0113] ① In some embodiments, a communication connection between a virtual client Client and a virtual server Server can be established by calling a connection interface (i.e., an Init interface), as follows: after the virtual client of the target camera is created, the Hongmeng application is controlled to call a preset connection interface so that the virtual client of the target camera sends a connection request to the virtual server; after the virtual server receives the connection request, the virtual server is controlled to feedback the response information of the connection request to the virtual client of the target camera, so as to establish a communication connection between the virtual server and the virtual client of the target camera. For example, Figure 4 As shown, first, on the virtual server Server side: the application to be deployed is run on the container image (i.e., Docker image) as the virtual server Server. The virtual server Server first creates a Socket object, where the Socket object is used to send and receive data. After binding the Socket object to a specific address and port number, the virtual server Server begins to enter the listening state and wait for the request of the virtual client Client. Then, on the virtual client Client side: when the virtual client Client of the target camera is created, the Hongmeng application calls the connection interface (i.e., Init interface). After the Hongmeng application calls the connection interface (i.e., Init interface), it will trigger the virtual client Client of the target camera to send a connection request to the virtual server Server (reference Figure 4 The virtual client of the target camera will also create a Socket object for sending and receiving data. Then, on the virtual server side: when the virtual server receives the connection request (i.e., the Init request), the virtual server accepts and establishes the connection, and reports the response information of the connection request (refer to Figure 4At this time, the communication connection between the virtual client Client of the target camera and the virtual server Server is completed.

[0114] ② In some embodiments, the model loading interface (i.e., InitMode interface) can be called to enable the virtual server Server to complete the task model loading according to the model path of the virtual client Client, as follows: control the Hongmeng end application to call the preset model loading interface so that the virtual client of the target camera sends a model loading request to the virtual server; when the virtual server receives the model loading request, control the virtual server to load the task model of the target camera according to the model path carried in the model loading request. For example, Figure 4 As shown, first, on the virtual client Client side: the Hongmeng application calls the model loading interface (i.e., the InitMode interface). After the Hongmeng application calls the model loading interface (i.e., the InitMode interface), it will trigger the virtual client Client of the target camera to send a model loading request to the virtual server Server (refer to Figure 4 Next, on the virtual server side: when the virtual server receives the model loading request (i.e., the InitMode request), the virtual server loads the task model of the target camera according to the model path carried in the model loading request, and reports the response information of the model loading request (refer to Figure 4 (as shown in "InitMode Report" in the figure).

[0115] The model path refers to the specific location where the virtual server stores and loads the machine learning or deep learning model.

[0116] ③ In some embodiments, the virtual server Server can complete the corresponding reasoning task by calling the service startup interface (i.e., Start interface), as follows: after the virtual server and the virtual client of the target camera establish a communication connection, control the Hongmeng application to call the preset service startup interface so that the virtual client of the target camera sends a service startup request to the virtual server; when the virtual server receives the service startup request, control the virtual server to drive the application to be deployed on the container image to run according to the service information carried in the service startup request, so as to perform the reasoning task according to the service information through the application to be deployed. For example, Figure 4As shown, first, on the virtual client Client side: the Hongmeng application calls the service startup interface (i.e., the Start interface). After the Hongmeng application calls the service startup interface (i.e., the Start interface), it will trigger the virtual client Client of the target camera to send a service startup request to the virtual server Server (refer to Figure 4 Next, on the virtual server side: when the virtual server receives the service start request (i.e., the Start request), the virtual server drives the application to be deployed on the container image to run according to the service information carried in the service start request, so as to execute the service processing process (such as reasoning processing tasks and drawing processing tasks) according to the service information through the application to be deployed, and reports the response information of the service start request (refer to Figure 4 (as shown in "Start Reporting" in the figure).

[0117] ④ In some embodiments, the virtual server Server can stop the corresponding reasoning task by calling the stop function interface (i.e., Stop interface), as follows: after sending a service start request to the virtual server, the Hongmeng end application is controlled to call the preset stop function interface, so that the virtual client of the target camera sends a stop function request to the virtual server; when the virtual server receives the stop function request, the virtual server is controlled to stop the corresponding reasoning task. For example, Figure 4 As shown, first, on the virtual client Client side: the Hongmeng application calls the stop function interface (i.e., the Stop interface). After the Hongmeng application calls the stop function interface (i.e., the Stop interface), it will trigger the virtual client Client of the target camera to send a stop function request to the virtual server Server (refer to Figure 4 Then, on the virtual server side: when the virtual server receives the stop function request (i.e., the Stop request), the virtual server drives the application to be deployed on the container image to stop running according to the business information carried in the stop function request, so that the application to be deployed stops executing the business processing process (such as reasoning processing tasks and drawing processing tasks), and reports the response information of the stop function request (refer to Figure 4 (as shown in "Stop Report" in the figure).

[0118] It can be seen that the virtual client Client and the virtual server Server mainly communicate through Socket. The Client sends Init, InitMode, Start, and Stop request messages, and the Server responds to the corresponding requests and completes the corresponding actions of Init, Start, and Stop. After the action is completed, the Server will give feedback to the Client through the Socket channel.

[0119] In order to better understand the embodiments of the present application, the following briefly introduces the AI ​​application development and deployment process of this embodiment from the hardware and software levels. First, in terms of hardware, prepare multiple cameras and a router to form a local area network; then use a terminal device to deploy the Hongmeng application, and the computer device is connected to the router and multiple cameras in a local area network. Secondly, OpenHarmony must be installed on the terminal device, the corresponding underlying software development must adapt the graphics card driver, and the Docker porting must be completed. Then, you can refer to the following steps to complete the AI ​​application development and deployment process:

[0120] 1. Compile and run demos and examples according to the Linux environment provided by the chip manufacturer to ensure correct results;

[0121] 2. Create a Docker image through Dockerfile and load the image on the Hongmeng device to ensure the correct result;

[0122] 3. Develop client services on the Hongmeng application side and server services on the Docker side. Perform secondary development based on the demo and add communication modules.

[0123] 4. The virtual server and virtual client communicate through Socket to make requests / responses, and complete the message parsing between the virtual client and virtual server by defining the protocol format;

[0124] 5. Each camera corresponds to a virtual client Client, which completes TCP connection, task model loading, business start and stop functions by calling Init, InitMode, Start, and Stop interfaces.

[0125] From the above content, it can be seen that in this embodiment, by loading the application to be deployed on the container image of the Hongmeng device, a virtual server of the application to be deployed can be created, and a virtual client of the target camera can be created through the Hongmeng end application. After the virtual server and the virtual client of the target camera establish a communication connection, the virtual client of the target camera transmits a service startup request to the virtual server, and controls the virtual server to drive the container image to execute the service startup request. On the first hand, by driving the application to be deployed on the container image loaded by the Hongmeng device, the application to be deployed can be run on the Hongmeng device; and a communication connection is established based on the virtual server and the virtual client of the target camera, so that the Hongmeng end application can drive the application to be deployed on the container image to run. On the second hand, since the container image environment loaded by the Hongmeng device can support compilation and operation at the same time, there is no need to build a cross-compilation environment to complete the compilation and operation of the application to be deployed, which reduces the workload of application transplantation and adaptation to the Hongmeng device and improves the efficiency of application deployment. On the third hand, by calling the compiled library of the general operating system (such as a Linux-based operating system) on the container image environment loaded by the Hongmeng device, the dependent library code corresponding to the application to be deployed does not need to rely on the chip manufacturer, thereby improving the deployment efficiency of AI applications. Fourthly, since the container image only loads the operating environment required by the virtual software (that is, only the operating environment required for the secondary development of the application to be deployed) without virtualizing the entire complete system, it can maximize the reduction of useless configurations, improve performance and save resources. Fifthly, by defining the socket communication protocol between the virtual client and the virtual server, real-time communication is achieved, so that the Hongmeng-side application can drive the application to be deployed on the container image to run. Sixthly, the container image environment loaded by the Hongmeng device can shield the software environment differences, quickly realize the deployment and / or development of AI application prototypes, improve the efficiency of chip selection, and enable developers to quickly complete the development of AI application prototypes and focus more on the development of specific business projects. Therefore, the embodiments of the present application can reduce the development, transplantation and compilation workload of AI application deployment and operation, and improve the deployment efficiency of AI applications.

[0126] In addition, in order to better implement the application deployment method based on the Hongmeng device in the embodiment of the present application, on the basis of the application deployment method based on the Hongmeng device, the embodiment of the present application also provides a Hongmeng device, such as Figure 6 As shown, the Hongmeng device 600 includes a processor 601 and a memory 602, and the processor 601 and the memory 602 are connected via a bus 603, which is, for example, an I2C (Inter-integrated Circuit) bus.

[0127] Specifically, the processor 601 is used to provide computing and control capabilities to support the operation of the entire Hongmeng device 600. The processor 601 can be a central processing unit (CPU), and the processor 601 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0128] Specifically, the memory 602 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0129] Those skilled in the art will understand that Figure 1 The structure shown in the figure is merely a block diagram of a partial structure related to the embodiment of the present application, and does not constitute a limitation on the Hongmeng device to which the embodiment of the present application is applied. The specific Hongmeng device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0130] The processor 601 is used to run the computer program stored in the memory 602, and implement any one of the Hongmeng device-based application deployment methods provided in the embodiments of the present application when executing the computer program. For example, the processor 601 is used to run the computer program stored in the memory 602, and when executing the computer program, the following steps can be implemented:

[0131] Load the application to be deployed on the container image of the Hongmeng device to create a virtual server for the application to be deployed; when the target camera is connected to the Hongmeng device, control the Hongmeng application of the Hongmeng device to create a virtual client of the target camera; after the virtual server and the virtual client of the target camera establish a communication connection, control the virtual client of the target camera to pass a service startup request to the virtual server; control the virtual server to drive the application to be deployed on the container image to execute the service startup request.

[0132] In some embodiments, the processor 601 is used to run a computer program stored in the memory 602, and when executing the computer program, the following steps may be implemented:

[0133] After the virtual server and the virtual client of the target camera establish a communication connection, the Hongmeng application is controlled to call a preset service startup interface so that the virtual client of the target camera sends a service startup request to the virtual server, wherein the service startup request carries the video stream protocol address of the target camera; the virtual server is controlled to read the video stream data of the target camera according to the video stream protocol address; and the virtual server is controlled to drive the detection module to perform service processing according to the video stream data.

[0134] In some embodiments, the processor 601 is used to run a computer program stored in the memory 602, and when executing the computer program, the following steps may be implemented:

[0135] Control the virtual server to decode the video stream data to obtain a decoded image frame; control the virtual server to drive the detection module to perform inference processing on the decoded image frame to obtain inference result information of the decoded image frame.

[0136] In some embodiments, the processor 601 is used to run a computer program stored in the memory 602, and when executing the computer program, the following steps may be implemented:

[0137] Control the virtual server to drive the drawing module to perform drawing processing on the decoded image frame according to the reasoning result information to obtain a drawn image frame of the decoded image frame, wherein the reasoning result information includes at least one of an object detection box and an object detection category; control the virtual server to encode the drawn image frame to obtain an encoded video stream; control the virtual server to push the encoded video stream to visualize the reasoning result information, and control the virtual server to report the reasoning result information to the virtual client of the target camera, so that the Hongmeng end application responds according to the reasoning result information.

[0138] In some embodiments, the processor 601 is used to run a computer program stored in the memory 602, and when executing the computer program, the following steps may be implemented:

[0139] Control the virtual server to obtain the frame information of the decoded image frame; control the virtual server to drive the drawing module to perform drawing processing on the decoded image frame according to the inference result information and the frame information, so as to obtain a drawn image frame of the decoded image frame, wherein the frame information includes at least one of the name of the target camera, the timestamp of the decoded image frame, and the alarm information.

[0140] In some embodiments, the processor 601 is used to run a computer program stored in the memory 602, and when executing the computer program, the following steps may be implemented:

[0141] Control the Hongmeng application to call a preset model loading interface so that the virtual client of the target camera sends a model loading request to the virtual server; when the virtual server receives the model loading request, control the virtual server to load the task model of the target camera according to the model path carried by the model loading request.

[0142] In some embodiments, the processor 601 is used to run a computer program stored in the memory 602, and when executing the computer program, the following steps may be implemented:

[0143] After the virtual client of the target camera is created, the Hongmeng application is controlled to call a preset connection interface so that the virtual client of the target camera sends a connection request to the virtual server; when the virtual server receives the connection request, the virtual server is controlled to feedback the response information of the connection request to the virtual client of the target camera to establish a communication connection between the virtual server and the virtual client of the target camera.

[0144] In some embodiments, the processor 601 is used to run a computer program stored in the memory 602, and when executing the computer program, the following steps may be implemented:

[0145] After sending a service start request to the virtual server, the Hongmeng application is controlled to call a preset stop function interface so that the virtual client of the target camera sends a stop function request to the virtual server.

[0146] It should be noted that technical personnel in the relevant field can clearly understand that, for the convenience and conciseness of description, the specific working process of the Harmony device described above can refer to the corresponding process in the following application deployment method implementation example based on the Harmony device, and will not be repeated here.

[0147] A person of ordinary skill in the art can understand that all or part of the steps in the above-mentioned application deployment method based on Hongmeng devices can be completed by instructions, or by controlling related hardware through instructions. The instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0148] To this end, an embodiment of the present application provides a computer-readable storage medium, in which multiple computer programs are stored. The computer programs can be loaded by a processor to execute any application deployment method based on Hongmeng devices provided in the embodiment of the present application.

[0149] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0150] In the above-mentioned computer-readable storage medium and Hongmeng device embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process and beneficial effects of the above-mentioned computer-readable storage medium, Hongmeng device and its corresponding units can refer to the description of the application deployment method based on Hongmeng device in the above embodiment, and will not be repeated here.

[0151] The above is a detailed introduction to an application deployment method based on Hongmeng devices, Hongmeng devices and computer-readable storage media provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for deploying an application based on Hongmeng devices, characterized in that: The method comprises: Load the application to be deployed on the container image of the Hongmeng device to create a virtual server for the application to be deployed; When the target camera is connected to the Hongmeng device, the Hongmeng application controlling the Hongmeng device creates a virtual client of the target camera; After the virtual server and the virtual client of the target camera establish a communication connection, control the virtual client of the target camera to transmit a service start request to the virtual server; The virtual server is controlled to drive the application to be deployed on the container image to execute the service startup request.

2. The application deployment method based on Hongmeng device according to claim 1 is characterized in that: The application to be deployed on the container image includes a detection module, and after the virtual server and the virtual client of the target camera establish a communication connection, controlling the virtual client of the target camera to transmit a service startup request to the virtual server includes: After the virtual server and the virtual client of the target camera establish a communication connection, control the Hongmeng end application to call a preset service startup interface so that the virtual client of the target camera sends a service startup request to the virtual server, wherein the service startup request carries the video stream protocol address of the target camera; The controlling the virtual server to drive the application to be deployed on the container image to execute the service startup request includes: Controlling the virtual server to read the video stream data of the target camera according to the video stream protocol address; The virtual server is controlled to drive a detection module to perform service processing according to the video stream data.

3. The application deployment method based on Hongmeng device according to claim 2 is characterized in that: The business processing includes inference processing, and the controlling the virtual server to drive the detection module to perform business processing according to the video stream data includes: Controlling the virtual server to decode the video stream data to obtain decoded image frames; The virtual server is controlled to drive the detection module to perform inference processing on the decoded image frame to obtain inference result information of the decoded image frame.

4. The application deployment method based on Hongmeng device according to claim 3 is characterized in that: The business processing further includes drawing processing, the application to be deployed on the container image includes a drawing module, and the method further includes: Controlling the virtual server to drive the drawing module to perform drawing processing on the decoded image frame according to the inference result information to obtain a drawn image frame of the decoded image frame, wherein the inference result information includes at least one of an object detection frame and an object detection category; Controlling the virtual server to encode the drawn image frame to obtain an encoded video stream; Control the virtual server to push the encoded video stream to visualize the reasoning result information, and control the virtual server to report the reasoning result information to the virtual client of the target camera, so that the Hongmeng application responds and processes according to the reasoning result information.

5. The application deployment method based on Hongmeng device according to claim 4 is characterized in that: The controlling the virtual server to drive the drawing module to perform drawing processing on the decoded image frame according to the inference result information to obtain a drawing image frame of the decoded image frame includes: Controlling the virtual server to obtain frame information of the decoded image frame; Control the virtual server to drive the drawing module to perform drawing processing on the decoded image frame according to the inference result information and the frame information to obtain a drawn image frame of the decoded image frame, wherein the frame information includes at least one of the name of the target camera, the timestamp of the decoded image frame, and alarm information.

6. The application deployment method based on Hongmeng device according to claim 1 is characterized in that: The method further comprises: Control the Hongmeng terminal application to call a preset model loading interface so that the virtual client of the target camera sends a model loading request to the virtual server; When the virtual server receives the model loading request, the virtual server is controlled to load the task model of the target camera according to the model path carried in the model loading request.

7. The application deployment method based on Hongmeng device according to claim 1 is characterized in that: The method further comprises: After the virtual client of the target camera is created, the Hongmeng application is controlled to call a preset connection interface so that the virtual client of the target camera sends a connection request to the virtual server; When the virtual server receives the connection request, the virtual server is controlled to feed back response information of the connection request to the virtual client of the target camera, so as to establish a communication connection between the virtual server and the virtual client of the target camera.

8. The application deployment method based on Hongmeng device according to claim 1 is characterized in that: The method further comprises: After sending a service start request to the virtual server, the Hongmeng application is controlled to call a preset stop function interface so that the virtual client of the target camera sends a stop function request to the virtual server.

9. A Hongmeng device, characterized in that: It includes a processor and a memory, the memory stores a computer program, and when the processor calls the computer program in the memory, it executes the application deployment method based on the Hongmeng device as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the application deployment method based on Hongmeng device as described in any one of claims 1 to 8.

Images