Unmanned aerial vehicle video stream processing method, device, equipment and medium

Abstract

The invention relates to the technical field of image processing, and discloses an unmanned aerial vehicle video stream processing method and device, equipment and a medium. The invention provides an unmanned aerial vehicle video stream processing method, the method is executed by a processor in computer equipment, a video identification model is loaded in the computer equipment, and the method comprises the following steps: obtaining video stream data of a target unmanned aerial vehicle from a video server through a first thread; storing each frame of data of the video stream into a first type of linear linked list; acquiring video frame data from the first-class linear linked list through a second thread, and processing the video frame data through a video recognition model to obtain processed video frame data; removing the processed video frame data from the first type of linear linked list, and storing the processed video frame data in a second type of linear linked list; and pushing the data in the second type of linear linked list to a video server through a third thread. According to the invention, the video stream processing and transmission efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, equipment and medium for processing video streams of unmanned aerial vehicles. Background Art

[0002] Drone inspections are regular or irregular inspections of specific areas or facilities by drones. Compared with traditional manual inspections, drone inspections have the advantages of fast speed, wide coverage, and low risk. Drones can carry high-definition cameras, infrared thermal imagers, multispectral sensors and other equipment to conduct detailed observations and data collection on the ground, buildings, infrastructure, etc. At present, the mainstream products on the market download the video after the drone inspection and copy it to a dedicated visual AI server for edge processing. If real-time video processing is required, powerful edge computing capabilities are required. If the GPU is too low and the processing time is too long, the processing will take up part of the time, and after processing, it will return to the user end. This sequential execution will have a very high delay.

[0003] In view of this, a drone video stream processing method is needed to solve the high latency problem. Summary of the invention

[0004] In view of this, the present invention provides a method, device, equipment and medium for processing drone video streams to improve the efficiency of video stream processing and transmission.

[0005] In a first aspect, the present invention provides a method for processing a drone video stream, which is executed by a processor in a computer device, and a video recognition model is also loaded in the computer device. The method includes: obtaining video stream data of a target drone from a video server through a first thread; storing each frame data of the video stream in a first type of linear linked list; obtaining video frame data from the first type of linear linked list through a second thread and processing it through a video recognition model to obtain processed video frame data; removing the processed video frame data from the first type of linear linked list and storing it in a second type of linear linked list; and pushing the data in the second type of linear linked list to the video server through a third thread.

[0006] In an optional embodiment, the video recognition model is used to identify, classify or label the video frame data; the second thread obtains the video frame data from the first type of linear linked list and processes it, including: the second thread obtains each video frame data from the first type of linear linked list, and identifies, classifies or labels each video frame data through the video recognition model to obtain processed video frame data.

[0007] In an optional implementation, the method further includes: storing the processed video frame data in a first array list; and storing the recognition processing results in the first array list in a database through a fourth thread.

[0008] In an optional implementation, the first thread, the second thread, the third thread and the fourth thread are asynchronous threads.

[0009] In an optional implementation, the first type of linear linked list is set in the memory of the computer device.

[0010] In a second aspect, the present invention provides a UAV video stream processing system, the system comprising a video server, a visual model server and a UAV; the video server is used to receive video data collected by the UAV; the visual model server is loaded with a video recognition model; the processor of the visual model server has multiple threads; the visual model processor is used to obtain the video stream data of the target UAV from the video server through a first thread; store each frame data of the video stream in a first type of linear linked list; obtain the video frame data from the first type of linear linked list through a second thread and process it through a video recognition model to obtain the processed video frame data; remove the processed video frame data from the first type of linear linked list and store it in a second type of linear linked list; push the data in the second type of linear linked list to the video server through a third thread.

[0011] In an optional implementation, the visual model server is further used to: store the processed video frame data in the first array list; and store the recognition processing results in the first array list in the database through a fourth thread.

[0012] In an optional implementation, the drone video stream processing system further includes a terminal device; the terminal device is used to send an acquisition request to the video server to acquire video data in the video server.

[0013] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the drone video stream processing method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0014] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the drone video stream processing method of the first aspect or any corresponding embodiment thereof.

[0015] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, which are used to enable a computer to execute the drone video stream processing method of the first aspect or any corresponding embodiment thereof.

[0016] The technical solution provided by this application may have the following beneficial effects:

[0017] The first thread pulls the video stream data from the video server to ensure the real-time and low latency of the data. The first type of linear linked list stores the video frame data to be processed. The second thread calls the video recognition model for processing and generates processing results. The second type of linear linked list stores the processed video frame data to ensure the order of the data. The third thread pushes the processed video stream data to the user front end to ensure that the user can view the processing results in real time. The above scheme, through multi-threading and linear linked list-like data structure, enables the drone video stream processing method to efficiently and real-time process and transmit video data, which is suitable for various inspection scenarios and improves the efficiency of video stream processing and transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 is a flowchart of a method for processing a drone video stream according to an embodiment of the present invention;

[0020] Figure 2 is a structural block diagram of a video processing system according to an embodiment of the present invention;

[0021] Figure 3 It is a flowchart of a drone AI visual model real-time recognition-business scenario according to an optional embodiment of the present invention;

[0022] Figure 4 A flowchart of a program for real-time recognition of a drone AI visual model according to an optional embodiment of the present invention;

[0023] Figure 5 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0025] In the description of the embodiments of the present application, the term "corresponding" may indicate a direct or indirect correspondence between two items, or an association relationship between the two items, or a relationship between indication and being indicated, configuration and being configured, and the like.

[0026] According to an embodiment of the present invention, an embodiment of a method for processing a drone video stream is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0027] In this embodiment, a method for processing a drone video stream is provided. The method is executed by a processor in a computer device. A video recognition model is also loaded in the computer device.

[0028] The video recognition model is a deep learning model loaded into a computer device and is used to identify time, objects, and image security-related information in video frames.

[0029] Figure 1 is a flow chart of a method for processing a drone video stream according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0030] Step S101: Obtain video stream data of a target drone from a video server through a first thread.

[0031] When the user arrives at the destination to be inspected, the drone is flown, and the drone video stream data is transmitted to the video server. The first thread pulls the video stream data from the video server. For example, the main thread A pulls the stream from the video server using the RTSP or RTMP protocol mode. The first thread is the starting point for data acquisition, providing the original video stream data for subsequent processing, ensuring that the video stream data can be transmitted from the video server to the processing device in real time and with low latency.

[0032] Step S102, storing each frame of data of the video stream into a first type of linear linked list.

[0033] Optionally, the first type of linear linked list is set in the memory of the computer device.

[0034] Each frame of data in the video stream is stored in the first type of linear linked list (such as the linear linked list LH01). The linear linked list is a dynamic data structure that supports efficient data insertion and deletion operations, can provide a buffer to store the video frame data to be processed, and uses the first-in-first-out thinking of the linear linked list to ensure video sequentiality to solve the problem of high latency.

[0035] Step S103, obtaining video frame data from the first type of linear linked list through the second thread and processing it through the video recognition model to obtain processed video frame data.

[0036] The second thread (such as thread B) reads the video frame data from the class linear linked list LH01 and calls the video recognition model for processing. The video recognition model intelligently recognizes and annotates the video frame and generates a processing result.

[0037] Step S104, removing the processed video frame data from the first type of linear linked list and storing it in the second type of linear linked list.

[0038] The processed video frame data is removed from the first type of linear linked list (such as the linear linked list LH01) and stored in the second type of linear linked list (such as the linear linked list LH02).

[0039] Step S105: Push the data in the second type of linear linked list to the video server through the third thread.

[0040] The third thread (such as thread D) reads data from the second linear linked list (such as the linear linked list LH02) and pushes it to the video server, which transmits it to the user front end. The transmission can be performed using the sliced ​​webrtc protocol. The processed video stream data is pushed to the user front end in real time to ensure that the user can view the processing results in real time.

[0041] The drone video stream processing method provided in this embodiment uses multi-threading and a linear linked list-like data structure to enable the drone video stream processing method to efficiently and real-time process and transmit video data. It is suitable for various inspection scenarios and improves the efficiency of video stream processing and transmission.

[0042] Furthermore, the first thread, the second thread, the third thread and the fourth thread are asynchronous threads.

[0043] Asynchronous threads refer to threads that can run independently of the main thread during program execution. They can be executed concurrently without blocking the execution of the main thread or other threads. Using asynchronous threads can enable multiple tasks to be performed simultaneously, reducing waiting time and improving efficiency. In the drone video stream processing method of the present invention, asynchronous threads are used to process different tasks and improve the efficiency of data processing and transmission.

[0044] Optionally, multiple asynchronous threads with the same function may be set. If there is a new use for the video frame data, asynchronous threads with more functions such as the fifth thread may be set.

[0045] In an optional implementation, the video recognition model is used to recognize, classify or label the video frame data. The process of step S103 above also includes the following steps:

[0046] The second thread obtains each video frame data from the first type of linear linked list, and identifies, classifies or labels each video frame data through a video recognition model to obtain processed video frame data.

[0047] The second thread is an asynchronous thread responsible for reading video frame data from the first linear linked list and processing it. The video frame data is obtained from the first linear linked list, and the video recognition model is called for processing to generate processed video frame data. The second thread passes the video frame data to the video recognition model and processes the output results of the model. The first linear linked list is a dynamic data structure used to store video frame data pulled from the video server. The processed video frame data refers to the video frame processed by the video recognition model, including recognition, classification or labeling results. The processed video frame data is the output result of the second thread and is stored in the second linear linked list for use by subsequent threads.

[0048] Through the combination of multi-threading and data structure, this implementation can process video data efficiently and in real time, and is suitable for various inspection scenarios.

[0049] In an optional implementation, the process of the drone video stream processing method further includes the following steps:

[0050] Step S201, storing the processed video frame data into a first array list.

[0051] The video frame data processed by the video recognition model includes recognition, classification or labeling results. The first array list (such as ArrayList AL01) is a dynamic array used to store the processed video frame data and its recognition results. The first array list is an intermediate storage structure for the processed video frame data, providing data for the fourth thread.

[0052] Step S202: storing the recognition processing results in the first array list into a database through the fourth thread.

[0053] The fourth thread (such as thread C) is an asynchronous thread responsible for storing the recognition processing results in the first array list into the database for subsequent warnings and prompts. The database is a system for storing and managing data, supporting efficient data query and analysis.

[0054] Through the combination of multi-threading and data structure, the drone video stream processing method in this embodiment can process video data efficiently and in real time, and store the processing results persistently, which is suitable for various inspection scenarios.

[0055] In this embodiment, a drone video stream processing system is also provided. The system is used to implement the above embodiments and preferred implementation modes, and those that have been explained will not be repeated here.

[0056] Figure 2 is a structural block diagram of a video processing system according to an embodiment of the present invention. This embodiment provides a video stream processing system, which includes a video server, a visual model server, and a drone;

[0057] The video server is used to receive video data collected by the drone;

[0058] The visual model server is loaded with a video recognition model; the processor of the visual model server has multiple threads; the visual model processor is used to obtain the video stream data of the target drone from the video server through the first thread; store each frame data of the video stream in a first type of linear linked list; obtain the video frame data from the first type of linear linked list through the second thread and process it through the video recognition model to obtain the processed video frame data; remove the processed video frame data from the first type of linear linked list and store it in the second type of linear linked list; push the data in the second type of linear linked list to the video server through the third thread.

[0059] In an optional implementation, the visual model server is further used for:

[0060] Store the processed video frame data into a first array list;

[0061] The recognition processing results in the first array list are stored in the database through the fourth thread.

[0062] In an optional implementation, the drone video stream processing system further includes a terminal device;

[0063] The terminal device is used to send an acquisition request to the video server to acquire video data in the video server.

[0064] Figure 3 This is a flowchart of a drone AI visual model real-time recognition-business scenario according to an optional embodiment of the present invention, and the process includes:

[0065] The pilot controls the drone to perform inspection tasks through the remote control. The camera or other sensors on the drone collect video data and transmit it to the video server. The video server receives the video data from the drone and stores or forwards it. The AI ​​vision model server obtains the video data from the video server. The AI ​​vision model in the server recognizes and annotates the video data in real time, processes each frame of the image, identifies the objects in the image, and annotates them accordingly. The data processed and annotated by AI is sent back to the video server. The video server transmits the processed data to the end user. The end user can watch the inspection after AI recognition through a computer or other device. The user can also choose to watch the video data before AI recognition, that is, the original inspection video.

[0066] Figure 4 This is a flowchart of a real-time recognition program of a drone AI visual model according to an optional embodiment of the present invention, which specifically describes the entire process of video data from acquisition to processing and then to end-user reception. The process includes:

[0067] The video data is first sent to the video server, and the main thread A pulls the video data. The main thread A is responsible for pulling the video data from the video server and storing each frame of data in the linear linked list LH01. LH01 is used to store each frame of video data pulled from the video server. The asynchronous thread B obtains each frame of data from LH01 and sends it to the AI ​​vision server for recognition, labeling and other processing. The AI ​​vision server performs intelligent analysis on the video frame, identifies the objects in the video, and performs corresponding labeling and classification. The processed data (including the recognition results) is stored in ArrayList AL01. The asynchronous thread C is responsible for parsing the data in ArrayList AL01 and storing it in the database. The database is used to store the processed data. The asynchronous thread D is responsible for pushing the processed video frame data from the linear linked list LH02 to the video server for subsequent transmission. LH02 is used to store the video frame data processed by AI. The video server pushes the processed video data to the user. The user receives the recognized video data and can view the processing results. The user can view log and warning information through the application, which may come from the data stored in the database.

[0068] The embodiment of the present invention also provides a computer device having the above Figure 2 The drone video stream processing system shown.

[0069] See also Figure 5 , Figure 5 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 5As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.

[0070] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0071] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0072] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0073] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0074] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0075] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0076] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0077] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0078] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for processing a drone video stream, the method being executed by a processor in a computer device, the computer device also being loaded with a video recognition model, characterized in that: The method comprises: Obtain the video stream data of the target drone from the video server through the first thread; storing each frame data of the video stream into a first type linear linked list; Acquire video frame data from the first type of linear linked list through a second thread and process it through a video recognition model to obtain processed video frame data; Removing the processed video frame data from the first type of linear linked list and storing it in the second type of linear linked list; The data in the second-type linear linked list is pushed to the video server through a third thread.

2. The method according to claim 1, characterized in that: The video recognition model is used to identify, classify or label video frame data; The second thread obtains and processes the video frame data from the first type of linear linked list, including: The second thread obtains each video frame data from the first type of linear linked list, and identifies, classifies or labels each video frame data through the video recognition model to obtain the processed video frame data.

3. The method according to claim 2, characterized in that The method further comprises: storing the processed video frame data in a first array list; The recognition processing results in the first array list are stored in a database through a fourth thread.

4. The method according to any one of claims 1 to 3, characterized in that: The first thread, the second thread, the third thread and the fourth thread are asynchronous threads.

5. The method according to any one of claims 1 to 3, characterized in that: The first type of linear linked list is set in the memory of the computer device.

6. A drone video stream processing system, characterized in that: The system includes a video server, a visual model server and a drone; The video server is used to receive the video data collected by the drone; The visual model server is loaded with a video recognition model; the processor of the visual model server has multiple threads; the visual model processor is used to obtain video stream data of the target drone from the video server through a first thread; store each frame data of the video stream in a first type of linear linked list; obtain video frame data from the first type of linear linked list through a second thread and process it through the video recognition model to obtain processed video frame data; remove the processed video frame data from the first type of linear linked list and store it in a second type of linear linked list; The data in the second-type linear linked list is pushed to the video server through a third thread.

7. The system according to claim 6, characterized in that The visual model server is further used for: storing the processed video frame data in a first array list; The recognition processing results in the first array list are stored in a database through a fourth thread.

8. The system according to claim 6, characterized in that The drone video stream processing system also includes a terminal device; The terminal device is used to send an acquisition request to the video server to acquire video data in the video server.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the drone video stream processing method according to any one of claims 1 to 5 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the drone video stream processing method according to any one of claims 1 to 5.

Images