Unmanned aerial vehicle single target real-time tracking method and system and edge intelligent device

By decoding and preprocessing the drone video stream, using an optimized model for object detection and coordinate conversion, determining the optimal shooting position of the drone, solving the slow identification speed, unstable tracking and data transmission in drone surveillance technology The problem of low robustness achieves efficient target object recognition and tracking.

CN119091328AActive Publication Date: 2024-12-06BEIJING HUAXIAXING TECH CO LTD
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202411185842.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-12-06
Estimated Expiration
2044-08-27

AI Technical Summary

Technical Problem

Among the existing UAV monitoring technology, the problems of slow monitoring and recognition speed, unstable tracking, and low robust data transmission lead to low target recognition and tracking efficiency.

Method used

By acquiring the drone video stream for decoding and preprocessing, the optimized YOLOv8n-obb and YOLOv8n-det models are used for object detection, the two-dimensional pixel coordinate information is obtained and the dimension is upgraded to three-dimensional world coordinate information is determined to determine the drone’s most important Excellent shooting position to achieve real-time tracking.

Benefits of technology

It improves the drone monitoring recognition speed and tracking stability, enhances the robustness of data transmission, and ensures accurate identification and tracking of target objects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091328A_ABST
    Figure CN119091328A_ABST
Patent Text Reader

Abstract

The invention discloses an unmanned aerial vehicle single target real-time tracking method and system and an edge intelligent device. The method comprises the following steps: preprocessing an unmanned aerial vehicle video stream to obtain a preprocessed image, loading a thread by using an optimized yolov8n-obb model, and detecting the preprocessed image by using the thread to obtain target information; performing dimension raising conversion on the target information to obtain three-dimensional world coordinate information of the target in reality, and determining the optimal shooting position of the unmanned aerial vehicle according to the three-dimensional world coordinate information and the orientation of the target information; sending the optimal shooting position to the unmanned aerial vehicle, moving the unmanned aerial vehicle to the optimal shooting position, obtaining a target video stream of the unmanned aerial vehicle, calculating to obtain a two-dimensional coordinate through an optimized yolov8n-det model, converting the two-dimensional coordinate into a three-dimensional coordinate, determining a second optimal shooting position according to the three-dimensional coordinate, and obtaining a target video stream of the unmanned aerial vehicle; and sending the second optimal shooting position to the unmanned aerial vehicle to control the unmanned aerial vehicle to move and track. According to the invention, the unmanned aerial vehicle monitoring identification speed and tracking stability are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle control technology, and in particular to a method, system, edge intelligent device and computer-readable storage medium for real-time tracking of a single target of an unmanned aerial vehicle. Background Art

[0002] In the field of smart cities, the technology for monitoring and tracking targets is mainly achieved through cameras, drone patrols, etc. The images and videos obtained by the lens monitoring a certain area are usually received and stored on the local server, and then the target recognition model of AI (Artificial Intelligence) deep learning is used to detect and locate the target on the server or locally, thereby achieving target tracking.

[0003] However, the existing technology has problems such as slow monitoring recognition speed, unstable tracking, and low data transmission robustness. Among them, the slow monitoring recognition speed is manifested in that after the existing monitoring technology collects data from the monitoring area, it transmits it to the background server for target detection. During this data transmission process, data packets may be lost and cannot be recognized, resulting in subsequent retransmission. After the background target is detected, a signal is transmitted to command the camera to adjust its posture. The speed of detecting the target and giving a response needs to be improved.

[0004] Tracking instability is manifested in that at a certain moment, the collected data from the surveillance camera may be detected by the background target detection model with low confidence or not detected at all, and the target is considered not to have appeared. In this case, the existing deep learning model may make fatal errors in processing the video frames shot from the camera's perspective, resulting in errors in target object recognition.

[0005] The low robustness of data transmission is manifested in: the monitoring process must be robust to different postures of the target in the lens (such as when the target just enters the monitoring area, when tracking the target), and the movement changes of the target before the lens is adjusted. The target posture with a low probability may be regarded as an abnormal situation and reduce the performance of target recognition. The sudden turn and acceleration of the target captured by the lens may make the lens unable to keep up with the movement of the target. It is necessary to improve the robustness through data enhancement and predicting the position after target detection.

[0006] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0007] The main purpose of the present invention is to provide a method, system, edge intelligent device and computer-readable storage medium for real-time tracking of a single target by a drone, aiming to solve the problems in the prior art of using drones for target monitoring and tracking, such as slow drone monitoring and recognition speed, unstable tracking and low data transmission robustness.

[0008] To achieve the above object, the present invention provides a method for real-time tracking of a single target by an unmanned aerial vehicle, the method comprising the following steps:

[0009] Obtaining a video stream sent by a drone, decoding the video stream to obtain a processable video frame, and preprocessing the processable video frame to obtain a preprocessed image;

[0010] Optimizing the first preset model and the second preset model, using the optimized first preset model to load a thread, using the thread to detect the preprocessed image, and obtaining two-dimensional pixel coordinate information of the target and the orientation of the target;

[0011] Performing a dimensional conversion on the two-dimensional pixel coordinate information to obtain three-dimensional world coordinate information of the target in reality, and determining a first optimal shooting position of the drone according to the three-dimensional world coordinate information and the orientation;

[0012] Sending the first best shooting position to the drone, and controlling the drone to move to the first best shooting position to perform low-altitude real-time tracking of the target;

[0013] Acquire a target video stream acquired by the drone at the optimal shooting position in real time, input the target video stream into an optimized second preset model for calculation to obtain target real-time coordinate data, convert the target real-time coordinate data into target three-dimensional coordinate data, determine a second optimal shooting position of the drone according to the target three-dimensional coordinate data, and send the second optimal shooting position to the drone to control the drone for real-time tracking.

[0014] Optionally, the method for real-time tracking of a single target by a drone, wherein the step of acquiring a video stream sent by the drone and decoding the video stream to obtain a processable video frame specifically includes:

[0015] When the camera of the drone captures the video stream, the video stream transmitted by the drone through the RTSP or RTP protocol is obtained;

[0016] The video stream is read and decoded by an ffmpeg converter to obtain processable video frames.

[0017] Optionally, the method for real-time tracking of a single target by a drone, wherein the preprocessing of the processable video frame to obtain a preprocessed image specifically includes:

[0018] Preloading an OpenCV framework, and using the OpenCV framework to perform geometric transformation on the processable video frame to obtain a transformed video frame;

[0019] The transformed video frame is spatially divided in equal proportion to obtain a preprocessed image, wherein the preprocessed image is a preset number of equal regions.

[0020] Optionally, in the UAV single target real-time tracking method, the first preset model is a yolov8n-obb model, and the second preset model is a yolov8n-det model;

[0021] The optimizing the first preset model and the second preset model specifically includes:

[0022] Respectively performing lightweight processing on the yolov8n-obb model and the yolov8n-det model to obtain a lightweight yolov8n-obb model and a lightweight yolov8n-det model;

[0023] The lightweight yolov8n-obb model and the lightweight yolov8n-det model are accelerated respectively by the TensorRT plug-in and the model acceleration algorithm to obtain an optimized yolov8n-obb model and an optimized yolov8n-det model.

[0024] Optionally, the method for real-time tracking of a single target of a drone, wherein the step of using the optimized first preset model to load a thread and using the thread to detect the preprocessed image to obtain two-dimensional pixel coordinate information of the target and the orientation of the target, specifically includes:

[0025] Use the optimized yolov8n-obb model to load a preset number of threads, and use the preset number of threads to perform target detection on the preset number of equal areas respectively;

[0026] When a target is found, the two-dimensional pixel coordinate information of the target and the direction of the target are obtained.

[0027] Optionally, the method for real-time tracking of a single target by a drone, wherein the step of performing up-dimensional transformation on the two-dimensional pixel coordinate information to obtain the three-dimensional world coordinate information of the target in reality, specifically includes:

[0028] Obtaining parameters of a drone camera used to shoot the video stream, wherein the drone camera parameters include factory parameters, a rotation matrix, and a translation matrix of the camera;

[0029] According to the drone camera parameters and the two-dimensional pixel coordinate information, the three-dimensional world coordinate information of the target in reality is obtained by reverse deduction through the formula from the three-dimensional world coordinate system to the two-dimensional pixel coordinate system.

[0030] Optionally, the method for real-time tracking of a single target by a drone, wherein the step of determining a first optimal shooting position of the drone according to the three-dimensional world coordinate information and the orientation specifically includes:

[0031] Calculating a better shooting position of the drone according to the three-dimensional world coordinate information;

[0032] The distance that the target may move toward the direction is predicted, and the preferred shooting position is adjusted according to the distance to obtain a first optimal shooting position of the drone.

[0033] In addition, to achieve the above-mentioned purpose, the present invention also provides a UAV single target real-time tracking system, wherein the UAV single target real-time tracking system comprises:

[0034] The video preprocessing module is used to obtain the video stream sent by the drone, decode the video stream to obtain a processable video frame, and preprocess the processable video frame to obtain a preprocessed image;

[0035] A target information acquisition module, used to optimize the first preset model and the second preset model, use the optimized first preset model to load the thread, use the thread to detect the preprocessed image, and obtain the two-dimensional pixel coordinate information of the target and the direction of the target;

[0036] An optimal shooting position determination module is used to perform up-dimensional conversion on the two-dimensional pixel coordinate information to obtain three-dimensional world coordinate information of the target in reality, and determine a first optimal shooting position of the drone according to the three-dimensional world coordinate information and the orientation;

[0037] A drone moving module, used for sending the first best shooting position to the drone, and controlling the drone to move to the first best shooting position to perform low-altitude real-time tracking of the target;

[0038] The drone tracking module is used to obtain in real time the target video stream obtained by the drone at the optimal shooting position, input the target video stream into the optimized second preset model for calculation, obtain the target real-time coordinate data, convert the target real-time coordinate data into target three-dimensional coordinate data, determine the second optimal shooting position of the drone according to the target three-dimensional coordinate data, and send the second optimal shooting position to the drone to control the drone for real-time tracking.

[0039] In addition, to achieve the above-mentioned purpose, the present invention also provides an edge intelligent device, wherein the edge intelligent device includes: a memory, a processor, and a drone single-target real-time tracking program stored in the memory and executable on the processor, and when the drone single-target real-time tracking program is executed by the processor, the steps of the drone single-target real-time tracking method as described above are implemented.

[0040] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a UAV single-target real-time tracking program, and when the UAV single-target real-time tracking program is executed by a processor, the steps of the UAV single-target real-time tracking method as described above are implemented.

[0041] In the present invention, the video stream of the drone is preprocessed to obtain a preprocessed image, the thread is loaded using the optimized yolov8n-obb model, the preprocessed image is detected using the thread, and the target information is obtained; the target information is converted to a higher dimension to obtain the three-dimensional world coordinate information of the target in reality, and the optimal shooting position of the drone is determined according to the orientation of the three-dimensional world coordinate information and the target information; the optimal shooting position is sent to the drone, the drone moves to the optimal shooting position, the target video stream of the drone is obtained, the two-dimensional coordinates are calculated by the optimized yolov8n-det model, the two-dimensional coordinates are converted into three-dimensional coordinates, the second optimal shooting position is determined according to the three-dimensional coordinates, and the second optimal shooting position is sent to the drone to control the drone to move and track. The present invention improves the speed of drone monitoring and identification and the tracking stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flow chart of the method for real-time tracking of a single target by an unmanned aerial vehicle of the present invention;

[0043] Figure 2 It is a principle diagram of the method for real-time tracking of a single target of an unmanned aerial vehicle of the present invention;

[0044] Figure 3 It is a schematic diagram of the UAV image transmission and video stream decoding in the UAV single target real-time tracking method of the present invention;

[0045] Figure 4 It is a comparison chart of the execution speed of the yolov8n model before and after acceleration in the UAV single target real-time tracking method of the present invention;

[0046] Figure 5 It is an effect diagram of parallel detection by the optimized yolov8n-obb model in the real-time tracking method of a single target of a UAV of the present invention;

[0047] Figure 6It is a schematic diagram of the transformation from the world coordinate system to the pixel coordinate system in the real-time tracking method of a single target of an unmanned aerial vehicle of the present invention;

[0048] Figure 7 It is a schematic diagram of the principle of a preferred embodiment of the UAV single target real-time tracking system of the present invention;

[0049] Figure 8 A schematic diagram of the operating environment of a preferred embodiment of the edge intelligent device of the present invention. DETAILED DESCRIPTION

[0050] The present application provides a method for real-time tracking of a single target of an unmanned aerial vehicle and related equipment. In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0052] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0053] The method for real-time tracking of a single target by an unmanned aerial vehicle according to a preferred embodiment of the present invention is as follows: Figure 1 and Figure 2 As shown, the UAV single target real-time tracking method includes the following steps:

[0054] Step S10: acquiring a video stream sent by the drone, decoding the video stream to obtain a processable video frame, and preprocessing the processable video frame to obtain a preprocessed image.

[0055] The step of obtaining the video stream sent by the drone and decoding the video stream to obtain a processable video frame specifically includes:

[0056] When the camera of the drone captures the video stream, the video stream transmitted by the drone through the RTSP or RTP protocol is obtained;

[0057] The video stream is read and decoded by an ffmpeg converter to obtain processable video frames.

[0058] In this embodiment, the drone is integrated with a camera, and the video stream captured by the camera is transmitted through the RTSP (Real-Time Streaming Protocol) or RTP (Real-Time Transport Protocol) protocol, and the received video stream is read and decoded on the edge computing device (not limited to edge computing devices, but also other small-scale intelligent computing devices, etc.) through the ffmpeg converter (ffmpeg converter is a powerful media file conversion tool, often used for transcoding, with a large number of optional commands, encoder, video length, frame rate, resolution, pixel format, sampling format, bit rate, cropping options, number of channels, etc. can be freely selected) or OpenCV framework (OpenCV is a cross-platform computer vision and machine learning software library released under the Apache2.0 license, which can run on Linux, Windows, Android and Mac OS operating systems. It is lightweight and efficient-consisting of a series of C functions and a small number of C++ classes) to obtain processable video frames.

[0059] like Figure 3 As shown, the process of decoding the video stream includes: receiving compressed data (video stream to be decoded), which is usually transmitted in the compression format of H.264 or H.265; caching the compressed data in the memory for buffering so as to perform decoding processing to ensure the smoothness of the decoding process; the decoder parses the data packet in the compressed video stream, the data packet includes a frame header and frame data, and the frame data contains coding information and compressed image data; the parsed frame data is subjected to frame separation processing to obtain independent frames (such as I frame, P frame, B frame); the decoder uses motion estimation to restore the difference between frames, interpolates based on the previous and subsequent frame data, and restores the quantized coefficients to the original DCT (discrete cosine transform) coefficients, and then converts the frequency domain data back to pixel domain data through IDCT (inverse discrete cosine transform); the pixel domain data is subjected to color space conversion, from YUV color space to RGB color space for display and processing, and finally decoding is completed to obtain a processable video frame.

[0060] Further, the preprocessing of the processable video frame to obtain a preprocessed image specifically includes:

[0061] Preloading an OpenCV framework, and using the OpenCV framework to perform geometric transformation on the processable video frame to obtain a transformed video frame;

[0062] The transformed video frame is spatially divided in equal proportion to obtain a preprocessed image, wherein the preprocessed image is a preset number of equal regions.

[0063] It is understandable that the OpenCV framework is preloaded and can be used to preprocess the processable video frame, and the result of the preprocessing is that the image is divided into a preset number of equal areas. In this embodiment, preferably, the preset number is 9, that is, the processable video frame is divided into 9 small areas in equal proportion by the OpenCV framework.

[0064] In addition, it should be noted that when using a drone for monitoring, if the drone's altitude is below a preset threshold, you can choose whether to let the drone spin slowly to prevent the target from being cut by a small area boundary (under the premise of a high-altitude perspective, the probability of being cut by a small area boundary is relatively small).

[0065] Step S20, optimizing the first preset model and the second preset model, using the optimized first preset model to load the thread, using the thread to detect the preprocessed image, and obtaining the two-dimensional pixel coordinate information of the target and the orientation of the target.

[0066] It can be understood that the first preset model is the yolov8n-obb model, and the second preset model is the yolov8n-det model; the network structure of the yolov8n-obb model (directional boundary recognition model) and the yolov8n-det model (target detection model) is mainly composed of three modules: Backbone (backbone network), Neck (neck network) and Head (detection head). Among them, the Backbone module refers to the CSPDarkNet-53 network, the Neck module uses PAN-FPN, which is an efficient and fast dual-stream FPN, and the Head module uses the Decoupled Head network structure of YOLOX.

[0067] Specifically, the yolov8n-obb model and the yolov8n-det model are optimized. First, the yolov8n-obb model and the yolov8n-det model are lightweight processed respectively to obtain a lightweight yolov8n-obb model and a lightweight yolov8n-det model. For example, the above models can be pruned. Pruning is a common model lightweight processing method, which reduces the number of parameters and computational complexity of the model by deleting some unimportant connections and neurons.

[0068] Furthermore, the lightweight yolov8n-obb model and the lightweight yolov8n-det model are accelerated respectively by the TensorRT plug-in and the model acceleration algorithm to obtain the optimized yolov8n-obb model and the optimized yolov8n-det model. Among them, TensorRT is a high-performance deep learning inference optimizer and runtime acceleration library, which can provide low-latency and high-throughput deployment inference for deep learning applications; the model acceleration algorithm mainly includes reducing the amount of model calculations, quantization operations, structural re-parameterization, using specialized hardware, data parallelism, pipeline parallel algorithms, improved sampling algorithms and other methods. Figure 4 As shown, Figure 4 This is a comparison of the inference speed of the model on the CPU and GPU before and after using TensorRT acceleration. Among them, yolov8n.pt is the original model, yolov8n.onnx is the intermediate conversion model, and yolov8n.engine is the model after TensorRT acceleration. It can be seen that lightweight acceleration of the model through TensorRT can greatly improve the recognition efficiency. In addition, the directional boundary recognition model and the target detection model can be further optimized by expanding the data set.

[0069] Furthermore, the use of the optimized first preset model to load the thread, and the use of the thread to detect the preprocessed image to obtain the two-dimensional pixel coordinate information of the target and the direction of the target specifically includes:

[0070] Use the optimized yolov8n-obb model to load a preset number of threads, and use the preset number of threads to perform target detection on the preset number of equal areas respectively;

[0071] When a target is found, the two-dimensional pixel coordinate information of the target and the direction of the target are obtained.

[0072] In this embodiment, the optimized yolov8n-obb model loads a preset number of threads, the number of threads must be the same as the number of equal regions, each thread is used to detect one region, taking 9 equal regions as an example, 9 threads are used to detect these 9 equal regions in parallel, and the target to be tracked (such as Figure 5 As shown, Figure 5 The effect diagram of parallel detection of the optimized yolov8n-obb model, in which the small vehicle in the fifth area is the discovered target, and the two-dimensional pixel coordinate information of the target and the direction of the target are obtained. It can be seen that after the detection is completed, the system instantly calculates the precise pixel position of each target in the original image, thereby ensuring efficient and accurate target positioning.

[0073] Step S30, performing up-dimensional conversion on the two-dimensional pixel coordinate information to obtain three-dimensional world coordinate information of the target in reality, and determining a first optimal shooting position of the drone according to the three-dimensional world coordinate information and the orientation.

[0074] The performing up-dimensional conversion on the two-dimensional pixel coordinate information to obtain the three-dimensional world coordinate information of the target in reality specifically includes:

[0075] Obtaining parameters of a drone camera used to shoot the video stream, wherein the drone camera parameters include factory parameters, a rotation matrix, and a translation matrix of the camera;

[0076] According to the drone camera parameters and the two-dimensional pixel coordinate information, the three-dimensional world coordinate information of the target in reality is obtained by reverse deduction through the formula from the three-dimensional world coordinate system to the two-dimensional pixel coordinate system.

[0077] In this embodiment, the acquired target information is used as the input of the coordinate dimension-upgrading conversion algorithm, and the coordinates of the target information in the real world are obtained based on the two-dimensional pixel coordinate system conversion algorithm to the three-dimensional world coordinate system.

[0078] Specifically, the camera's intrinsic parameters (camera parameters, factory-set) and extrinsic parameters (camera pose: rotation matrix, translation matrix) are obtained, and then the world coordinate system position of the target is inferred through the formula from the three-dimensional world coordinate system to the two-dimensional pixel coordinate system. When the target object is on the ground, Zw is the Z-axis value of the target world coordinate, and the corresponding height is 0, so only (Xw, Yw, 0) is required, where Xw is the X-axis value of the target world coordinate, and Yw is the Y-axis value of the target world coordinate. At this time, the coordinate dimensionality conversion can actually be regarded as a conversion between two-dimensional spaces.

[0079] like Figure 6As shown in the figure, the transformation process from the world coordinate system to the pixel coordinate system is as follows: the world coordinates (Xw, Yw, Zw) are transformed into the camera coordinates (Xc, Yc, Zc) through rigid body transformation according to the rotation moment R (3*3 matrix) and translation matrix T (3*1 matrix) in the camera extrinsic parameters, where Xc, Yc and Zc represent the X-axis value, Y-axis value and Z-axis value of the camera coordinates respectively. It should be noted that Figure 6 The camera extrinsic matrix is ​​filled into a square matrix to facilitate subsequent calculations to obtain its inverse matrix.

[0080] The camera coordinates are perspective projected through the perspective projection matrix (f represents the focal length of the camera) to obtain the two-dimensional image coordinates (x, y). Finally, the two-dimensional image coordinates are converted again according to the camera intrinsic parameters to obtain the pixel coordinates (u, v), where x and y represent the X-axis value and Y-axis value of the two-dimensional image coordinates, respectively, and u and v represent the X-axis value and Y-axis value of the pixel coordinates, respectively. It should be noted that in Figure 6 In the figure, the camera intrinsic parameter matrix is ​​also completed into a square matrix. fx and fy represent the actual physical size of each pixel in the camera in the x and y directions respectively. fx and fy are calculated by dividing the focal length f by the physical size of a single pixel on the image sensor. That is, fx and fy actually reflect the actual length unit corresponding to each pixel of the image sensor.

[0081] Through the formula from the 3D world coordinate system to the 2D pixel coordinate system, the process of inferring the world coordinate system position of the target is expressed as:

[0082]

[0083] Among them, Z c =(Zw+β / α, β=[R -1 T](row=2,colum=0), u0 and v0 represent the origin coordinates in the pixel coordinate system, Zw is the Z-axis value of the target world coordinates, and β is the matrix [R -1 T], α is the value corresponding to row 2 and column 0 of the matrix The value corresponding to row 2 and column 0, row represents the row of the matrix, and colum represents the column of the matrix.

[0084] Further, determining a first optimal shooting position of the drone according to the three-dimensional world coordinate information and the orientation specifically includes:

[0085] Calculating a better shooting position of the drone according to the three-dimensional world coordinate information;

[0086] The possible distance that the target moves toward the drone is calculated according to the orientation information, and the preferred shooting position is adjusted according to the distance to obtain a first optimal shooting position of the drone.

[0087] It can be understood that the target detection conf value (i.e., confidence threshold, which is an important parameter in target detection, determines the model's confidence score requirement for the detected object. Only when the confidence of the object is higher than the set conf value will the object be detected) is calculated according to the acquired three-dimensional world coordinate information (Xw, Yw, Zw) and the higher shooting position is used as the optimal shooting position of the drone (for changing target positions, there are their own optimal shooting positions, and the relative positions of the two remain unchanged, which has been proved by a large amount of experimental data).

[0088] Taking into account that the UAV needs a certain amount of moving time from high-altitude monitoring to low-altitude tracking, the distance that the target may move toward the UAV is calculated according to the orientation information, and then the target orientation detected by the optimized yolov8n-obb model (the vertical forward direction of the line connecting the pixel coordinates of the target object) is used to calculate the distance that the target may move toward the UAV, that is, the distance that the target is likely to move in this direction is predicted, so as to slightly adjust the optimal detection position to be transmitted to the UAV.

[0089] Step S40: sending the first best shooting position to the drone, and controlling the drone to move to the first best shooting position to perform low-altitude real-time tracking of the target.

[0090] Specifically, the best shooting position is sent to the drone, and after receiving the best shooting position, the drone calculates the movement signal of the shooting position. The drone moves to a better shooting position according to the movement signal and adjusts the corresponding camera posture.

[0091] Step S50, acquiring in real time a target video stream acquired by the drone at the optimal shooting position, inputting the target video stream into the optimized second preset model for calculation to obtain target real-time coordinate data, converting the target real-time coordinate data into target three-dimensional coordinate data, determining a second optimal shooting position of the drone according to the target three-dimensional coordinate data, and sending the second optimal shooting position to the drone to control the drone for real-time tracking.

[0092] It can be understood that when the drone moves to a low altitude, the core of the target recognition algorithm is switched to the optimized yolov8n-det model. After the drone reaches the designated optimal position, it begins to detect and track the target in real time. The optimized yolov8n-det model has an extremely fast recognition speed. The background of the edge computing device continuously performs coordinate conversion through the optimized yolov8n-det model, and sends the converted coordinates to the drone to control the drone to move and track, and the above process is repeated.

[0093] In summary, the drone of the present invention carries a high-definition camera and an edge intelligent computing device. In the edge intelligent computing device, an efficient OpenCV image preprocessing script, a lightweight accelerated optimized YOLOv8n-obb.engine and YOLOv8n.engine model, and an algorithm for converting a 2D pixel coordinate system into a 3D world coordinate system are integrated. These components ensure the efficient execution of computing tasks and the overall improvement of intelligent analysis capabilities. The drone performs a high-altitude central overlook in the monitoring area, with an angle fixed at 90°, and rotates slowly to optimize target capture. During this process, the edge intelligent computing device receives video frames transmitted by the camera in real time. Each frame of the image is first processed by the OpenCV framework, including geometric transformation and spatial division of the image, so as to divide it into 9 equal small images, thereby preparing for subsequent model detection.

[0094] On this basis, the present invention uses 9 parallel threads to detect these 9 small images at the same time. This slow rotation strategy aims to reduce false detections caused by cutting the target factor graph, thereby improving the accuracy and comprehensiveness of detection. After detecting the target, the world coordinate position and orientation of the target object are obtained through the coordinate system conversion algorithm, and then the drone obtains the best tracking and shooting position at this time and moves to it, starts the algorithm of the target detection model, and performs tracking. Before each movement, the target orientation is used to predict the position that the target may reach after the drone moves to the optimal position. The present invention effectively solves the problems of low recognition accuracy and inefficient real-time calculation of monitoring and tracking targets in a certain area of ​​the existing city. It is based on the built-in algorithm and model of the edge intelligent computing device, mounted on the drone, and uses different target detection models at different time periods for accurate recognition and real-time tracking.

[0095] Furthermore, if Figure 7 As shown, based on the above-mentioned UAV single target real-time tracking method, the present invention also provides a UAV single target real-time tracking system, wherein the UAV single target real-time tracking system includes:

[0096] The video preprocessing module 41 is used to obtain the video stream sent by the drone, decode the video stream to obtain a processable video frame, and preprocess the processable video frame to obtain a preprocessed image;

[0097] The target information acquisition module 42 is used to optimize the first preset model and the second preset model, use the optimized first preset model to load the thread, use the thread to detect the preprocessed image, and obtain the two-dimensional pixel coordinate information of the target and the direction of the target;

[0098] The best shooting position determination module 43 is used to perform up-dimensional conversion on the two-dimensional pixel coordinate information to obtain the three-dimensional world coordinate information of the target in reality, and determine the first best shooting position of the drone according to the three-dimensional world coordinate information and the orientation;

[0099] The drone moving module 44 is used to send the first best shooting position to the drone, and control the drone to move to the first best shooting position to perform low-altitude real-time tracking of the target;

[0100] The drone tracking module 45 is used to obtain in real time the target video stream obtained by the drone at the optimal shooting position, input the target video stream into the optimized second preset model for calculation, obtain the target real-time coordinate data, convert the target real-time coordinate data into target three-dimensional coordinate data, determine the second optimal shooting position of the drone according to the target three-dimensional coordinate data, and send the second optimal shooting position to the drone to control the drone for real-time tracking.

[0101] Furthermore, if Figure 8 As shown, based on the above-mentioned UAV single target real-time tracking method and system, the present invention also provides an edge intelligent device accordingly, and the edge intelligent device includes a processor 10, a memory 20 and a network receiver 30. Figure 8 Only some components of the edge smart device are shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.

[0102] In some embodiments, the memory 20 may be an internal storage unit of the edge smart device, such as a hard disk or memory of the edge smart device. In other embodiments, the memory 20 may also be an external storage device of the edge smart device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the edge smart device. Furthermore, the memory 20 may also include both an internal storage unit of the edge smart device and an external storage device. The memory 20 is used to store application software and various types of data installed on the edge smart device, such as the program code of the edge smart device. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a drone single-target real-time tracking program 40 is stored on the memory 20, and the drone single-target real-time tracking program 40 can be executed by the processor 10, thereby realizing the drone single-target real-time tracking method in the present application.

[0103] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the drone single target real-time tracking method.

[0104] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the edge smart device and to display a visual user interface. The components 10-30 of the edge smart device communicate with each other via a system bus.

[0105] In one embodiment, when the processor 10 executes the drone single target real-time tracking program 40 in the memory 20, the following steps are implemented:

[0106] Obtaining a video stream sent by a drone, decoding the video stream to obtain a processable video frame, and preprocessing the processable video frame to obtain a preprocessed image;

[0107] Optimizing the first preset model and the second preset model, using the optimized first preset model to load a thread, using the thread to detect the preprocessed image, and obtaining two-dimensional pixel coordinate information of the target and the orientation of the target;

[0108] Performing a dimensional conversion on the two-dimensional pixel coordinate information to obtain three-dimensional world coordinate information of the target in reality, and determining a first optimal shooting position of the drone according to the three-dimensional world coordinate information and the orientation;

[0109] Sending the first best shooting position to the drone, and controlling the drone to move to the first best shooting position to perform low-altitude real-time tracking of the target;

[0110] Acquire a target video stream acquired by the drone at the optimal shooting position in real time, input the target video stream into an optimized second preset model for calculation to obtain target real-time coordinate data, convert the target real-time coordinate data into target three-dimensional coordinate data, determine a second optimal shooting position of the drone according to the target three-dimensional coordinate data, and send the second optimal shooting position to the drone to control the drone for real-time tracking.

[0111] The step of acquiring the video stream sent by the drone and decoding the video stream to obtain a processable video frame specifically includes:

[0112] When the camera of the drone captures the video stream, the video stream transmitted by the drone through the RTSP or RTP protocol is obtained;

[0113] The video stream is read and decoded by an ffmpeg converter to obtain processable video frames.

[0114] The preprocessing of the processable video frame to obtain a preprocessed image specifically includes:

[0115] Preloading an OpenCV framework, and using the OpenCV framework to perform geometric transformation on the processable video frame to obtain a transformed video frame;

[0116] The transformed video frame is spatially divided in equal proportion to obtain a preprocessed image, wherein the preprocessed image is a preset number of equal regions.

[0117] Among them, the first preset model is the yolov8n-obb model, and the second preset model is the yolov8n-det model;

[0118] The optimizing the first preset model and the second preset model specifically includes:

[0119] Respectively performing lightweight processing on the yolov8n-obb model and the yolov8n-det model to obtain a lightweight yolov8n-obb model and a lightweight yolov8n-det model;

[0120] The lightweight yolov8n-obb model and the lightweight yolov8n-det model are accelerated respectively by the TensorRT plug-in and the model acceleration algorithm to obtain an optimized yolov8n-obb model and an optimized yolov8n-det model.

[0121] The step of using the optimized first preset model to load a thread and using the thread to detect the preprocessed image to obtain two-dimensional pixel coordinate information of the target and the orientation of the target specifically includes:

[0122] Use the optimized yolov8n-obb model to load a preset number of threads, and use the preset number of threads to perform target detection on the preset number of equal areas respectively;

[0123] When a target is found, the two-dimensional pixel coordinate information of the target and the direction of the target are obtained.

[0124] The step of performing up-dimensional transformation on the two-dimensional pixel coordinate information to obtain the three-dimensional world coordinate information of the target in reality specifically includes:

[0125] Obtaining parameters of a drone camera used to shoot the video stream, wherein the drone camera parameters include factory parameters, a rotation matrix, and a translation matrix of the camera;

[0126] According to the drone camera parameters and the two-dimensional pixel coordinate information, the three-dimensional world coordinate information of the target in reality is obtained by reverse deduction through the formula from the three-dimensional world coordinate system to the two-dimensional pixel coordinate system.

[0127] Wherein, determining the first optimal shooting position of the drone according to the three-dimensional world coordinate information and the orientation specifically includes

[0128] Calculating a better shooting position of the drone according to the three-dimensional world coordinate information;

[0129] The distance that the target may move toward the direction is predicted, and the preferred shooting position is adjusted according to the distance to obtain a first optimal shooting position of the drone.

[0130] In summary, the present invention provides a method for real-time tracking of a single target of a drone and related equipment, the method comprising: preprocessing the drone video stream to obtain a preprocessed image, using an optimized yolov8n-obb model to load a thread, using a thread to detect the preprocessed image, and obtaining target information; performing a dimensional conversion on the target information to obtain the three-dimensional world coordinate information of the target in reality, and determining the optimal shooting position of the drone according to the orientation of the three-dimensional world coordinate information and the target information; sending the optimal shooting position to the drone, the drone moves to the optimal shooting position, and obtains the target video stream of the drone, and calculates the two-dimensional coordinates by optimizing the yolov8n-det model, converting the two-dimensional coordinates into three-dimensional coordinates, and determining the second optimal shooting position according to the three-dimensional coordinates, and sending the second optimal shooting position to the drone to control the drone to move and track. The present invention effectively solves the problems of low recognition accuracy and inefficient real-time calculation of monitoring and tracking targets in a certain area of ​​the existing city. Based on the built-in algorithm and model of the edge intelligent computing device, it is carried on the drone, and different target detection models are used at different time periods for accurate recognition and real-time tracking.

[0131] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or edge intelligent device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or edge intelligent device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or edge intelligent device including the element.

[0132] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0133] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for real-time tracking of a single target of a UAV. It is characterized in that The UAV single target real-time tracking method comprises: Obtaining a video stream sent by a drone, decoding the video stream to obtain a processable video frame, and preprocessing the processable video frame to obtain a preprocessed image; Optimizing the first preset model and the second preset model, using the optimized first preset model to load a thread, using the thread to detect the preprocessed image, and obtaining two-dimensional pixel coordinate information of the target and the orientation of the target; Performing a dimensional conversion on the two-dimensional pixel coordinate information to obtain three-dimensional world coordinate information of the target in reality, and determining a first optimal shooting position of the drone according to the three-dimensional world coordinate information and the orientation; Sending the first best shooting position to the drone, and controlling the drone to move to the first best shooting position to perform low-altitude real-time tracking of the target; Acquire a target video stream acquired by the drone at the optimal shooting position in real time, input the target video stream into an optimized second preset model for calculation to obtain target real-time coordinate data, convert the target real-time coordinate data into target three-dimensional coordinate data, determine a second optimal shooting position of the drone according to the target three-dimensional coordinate data, and send the second optimal shooting position to the drone to control the drone for real-time tracking.

2. The method for real-time tracking of a single target by a drone according to claim 1, It is characterized in that The step of obtaining the video stream sent by the drone and decoding the video stream to obtain a processable video frame specifically includes: When the camera of the drone captures the video stream, the video stream transmitted by the drone through the RTSP or RTP protocol is obtained; The video stream is read and decoded by an ffmpeg converter to obtain processable video frames.

3. The method for real-time tracking of a single target by a drone according to claim 1, It is characterized in that The preprocessing of the processable video frame to obtain a preprocessed image specifically includes: Preloading an OpenCV framework, and using the OpenCV framework to perform geometric transformation on the processable video frame to obtain a transformed video frame; The transformed video frame is spatially divided in equal proportion to obtain a preprocessed image, wherein the preprocessed image is a preset number of equal regions.

4. The method for real-time tracking of a single target by a drone according to claim 3, It is characterized in that The first preset model is the yolov8n-obb model, and the second preset model is the yolov8n-det model; The optimizing the first preset model and the second preset model specifically includes: Respectively performing lightweight processing on the yolov8n-obb model and the yolov8n-det model to obtain a lightweight yolov8n-obb model and a lightweight yolov8n-det model; The lightweight yolov8n-obb model and the lightweight yolov8n-det model are accelerated respectively by the TensorRT plug-in and the model acceleration algorithm to obtain an optimized yolov8n-obb model and an optimized yolov8n-det model.

5. The method for real-time tracking of a single target by an unmanned aerial vehicle according to claim 4, It is characterized in that The step of using the optimized first preset model to load a thread and using the thread to detect the preprocessed image to obtain two-dimensional pixel coordinate information of the target and the direction of the target specifically includes: Use the optimized yolov8n-obb model to load a preset number of threads, and use the preset number of threads to perform target detection on the preset number of equal areas respectively; When a target is found, the two-dimensional pixel coordinate information of the target and the direction of the target are obtained.

6. The method for real-time tracking of a single target by an unmanned aerial vehicle according to claim 5, It is characterized in that The performing up-dimensional conversion on the two-dimensional pixel coordinate information to obtain the three-dimensional world coordinate information of the target in reality specifically includes: Obtaining parameters of a drone camera used to shoot the video stream, wherein the drone camera parameters include factory parameters, a rotation matrix, and a translation matrix of the camera; According to the drone camera parameters and the two-dimensional pixel coordinate information, the three-dimensional world coordinate information of the target in reality is obtained by reverse deduction through the formula from the three-dimensional world coordinate system to the two-dimensional pixel coordinate system.

7. The method for real-time tracking of a single target by an unmanned aerial vehicle according to claim 1, It is characterized in that The determining of the first optimal shooting position of the drone according to the three-dimensional world coordinate information and the orientation specifically includes: Calculating a better shooting position of the drone according to the three-dimensional world coordinate information; The distance that the target may move toward the direction is predicted, and the preferred shooting position is adjusted according to the distance to obtain a first optimal shooting position of the drone.

8. A UAV single target real-time tracking system, It is characterized in that The UAV single target real-time tracking system comprises: The video preprocessing module is used to obtain the video stream sent by the drone, decode the video stream to obtain a processable video frame, and preprocess the processable video frame to obtain a preprocessed image; A target information acquisition module, used to optimize the first preset model and the second preset model, use the optimized first preset model to load the thread, use the thread to detect the preprocessed image, and obtain the two-dimensional pixel coordinate information of the target and the direction of the target; An optimal shooting position determination module is used to perform up-dimensional conversion on the two-dimensional pixel coordinate information to obtain three-dimensional world coordinate information of the target in reality, and determine a first optimal shooting position of the drone according to the three-dimensional world coordinate information and the orientation; A drone moving module, used for sending the first best shooting position to the drone, and controlling the drone to move to the first best shooting position to perform low-altitude real-time tracking of the target; The drone tracking module is used to obtain in real time the target video stream obtained by the drone at the optimal shooting position, input the target video stream into the optimized second preset model for calculation, obtain the target real-time coordinate data, convert the target real-time coordinate data into target three-dimensional coordinate data, determine the second optimal shooting position of the drone according to the target three-dimensional coordinate data, and send the second optimal shooting position to the drone to control the drone for real-time tracking.

9. An edge intelligent device, It is characterized in that The edge intelligent device includes: a memory, a processor, and a drone single-target real-time tracking program stored in the memory and executable on the processor. When the drone single-target real-time tracking program is executed by the processor, the steps of the drone single-target real-time tracking method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a drone single-target real-time tracking program, and when the drone single-target real-time tracking program is executed by a processor, the steps of the drone single-target real-time tracking method according to any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • System and method for realizing unmanned aerial vehicle-based land and water secondary positioning

    CN110333735A

  • Target animal motion tracking method based on video sequence and related equipment

    CN115063447A

  • Target tracking and positioning method and system for unmanned aerial vehicle

    CN115861860A

  • Target positioning and recognition method and device and readable storage medium

    CN116469142A

  • Unmanned aerial vehicle monitoring and tracking method and device, electronic equipment and storage medium

    CN116486290A