Real-time target detection tracking method based on unmanned aerial vehicle

By combining YOLOv5 and SiamRPN algorithms on the drone, using ground station instructions and pre-save target images, real-time target detection and tracking of the drone is achieved, solving the stability and accuracy of target tracking in complex environments, and improving the The intelligent and autonomous navigation capabilities of drones.

CN118915785APending Publication Date: 2024-11-08THE 54TH RESEARCH INSTITUTE OF CHINA ELECTRONICS TECHNOLOGY GROUP CORPORATION
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410990367.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing drone target detection and tracking technologies are difficult to achieve efficient, stable and accurate target tracking in complex environments. Traditional methods have low detection accuracy and poor tracking stability, and deep learning-based methods are difficult to achieve real-time detection and tracking.

Method used

Using a real-time object detection and tracking method based on drones, the on-board tracking module is initialized using pre-saved target images to be tracked, combined with deep learning YOLOv5 object detection algorithm and SiamRPN tracking algorithm, fast and stable tracking is achieved through ground station instructions. Initialize the tracking module in advance to reduce the detection mode switching delay and improve tracking stability.

Benefits of technology

It improves detection accuracy and tracking stability, realizes real-time target tracking in complex environments, reduces tracking mode switching delay, and improves the overall stability of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118915785A_ABST
    Figure CN118915785A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of unmanned aerial vehicles and computer vision, in particular to a real-time target detecting and tracking method based on an unmanned aerial vehicle, which comprises the following steps of: (1) initializing an airborne tracking module of the unmanned aerial vehicle by using a pre-stored to-be-tracked target image as a template; (2) the unmanned aerial vehicle takes off to a set height; (3) the unmanned aerial vehicle starts a detection module to perform target detection on the acquired ground image according to an instruction of the ground station; (4) according to a detection image returned by the unmanned aerial vehicle, selecting a tracking target frame and sending the tracking target frame to the airborne tracking module; (5) updating the template and continuously tracking; and (6) if the tracking target is lost, switching the unmanned aerial vehicle from the tracking mode to the detection mode, and waiting for the ground station to reselect the tracking target. According to the invention, the mode of initializing the tracking module in advance is utilized, and the necessary initialization stage in the process of switching the detection mode to the tracking mode is adjusted to be completed when the whole system is started, so that the delay of the switching process is reduced, and the stability of the tracking process is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle (UAV) and computer vision technology, and in particular to a real-time target detection and tracking method based on an unmanned aerial vehicle (UAV). Background Art

[0002] As the fastest-growing civilian unmanned aerial vehicle in recent years, drones are becoming more and more popular and widely used. However, existing drone technology cannot fully realize the autonomy and intelligence of drone navigation, and there are still many problems and challenges in target detection and tracking, autonomous attack, etc.

[0003] Object detection is the process of using algorithms to determine whether a specified object exists in an image and obtain the object's location. Due to the low accuracy of traditional detection algorithms, deep learning-based detection algorithms, such as YOLOv5, have been widely used in recent years. Deep learning-based object detection algorithms require the creation of a target dataset first, then training the corresponding network weights, and then deploying them on portable mobile devices for image reasoning.

[0004] One difference between target tracking and target detection is that the target tracking algorithm requires prior information about the target position, and then infers the target position of each subsequent frame based on the target position of the initial frame. Traditional tracking algorithms such as KCF, ECO and other related filtering methods have a fast inference speed, but are easily affected by changes in target scale and occlusion, and have poor stability. Although the tracking algorithm based on deep learning has high requirements for hardware computing power, it can make better use of the initial information characteristics of the target, has strong adaptability to scale changes and occlusion, and the tracking process is more stable.

[0005] When drones use traditional methods to detect and track ground targets from the air, problems such as low detection accuracy and tracking loss are likely to occur. However, it is difficult to detect and track targets in real time using methods based on deep learning. At present, drone-based target detection and tracking technology is still in its infancy, and it is difficult to achieve efficient, stable, and accurate target tracking in complex environments. Therefore, how to realize drone intelligence, autonomous navigation, target attack, etc. is still a challenging problem. In order to solve the shortcomings of existing methods and meet the real-time requirements of detection and tracking, a drone-based target detection and tracking method is proposed. Summary of the invention

[0006] In order to avoid the problems in the background technology, the present invention provides a real-time target detection and tracking method based on a drone, which is deployed on the airborne end of the drone and can quickly and stably track the target through ground station instructions.

[0007] The technical solution adopted by the present invention is:

[0008] A real-time target detection and tracking method based on UAV is implemented based on UAV and ground station, including the following processes:

[0009] (1) Initializing the UAV's onboard tracking module using the pre-saved image of the target to be tracked as a template;

[0010] (2) The drone takes off to the set location and uses visual sensors to obtain real-time ground video images from the air;

[0011] (3) The UAV starts the detection module according to the command of the ground station, uses the target detection model to perform target detection on the acquired ground video image, extracts the detection frames of all targets and marks them in the image, and sends the detected video image to the ground station;

[0012] (4) At the ground station, the target to be tracked is selected from the detected video image. If the detection frame of the target to be tracked has been extracted by the target detection model, the corresponding detection frame is directly selected. If the detection frame has not been extracted, the bounding box of the target is drawn and the detection frame or bounding box coordinates are sent to the UAV.

[0013] (5) After receiving the detection frame or bounding box, the drone switches to tracking mode and uses the image in the frame as a template to update the onboard tracking module, and then continues to track the target;

[0014] (6) If the tracking target is lost, the UAV switches from tracking mode to detection mode, waits for the ground station to reselect the tracking target, and returns to step (3).

[0015] Furthermore, the specific process of step (1) is as follows:

[0016] (1.1) Obtain the image template of the target to be tracked in advance and store it on the onboard storage device of the UAV;

[0017] (1.2) When the system starts running, the UAV's SiamRPN tracking algorithm module is started. The SiamRPN tracking algorithm module uses the stored image template to initialize the UAV's airborne tracking module.

[0018] Furthermore, the specific process of step (2) is as follows:

[0019] (3.1) The drone takes off to the set location and uses visual sensors to obtain ground video images from the air, and transmits the ground video image stream to the ground station in real time;

[0020] (3.2) At the ground station, the detection time is selected based on the transmitted ground video image stream, and a command to start target detection is issued to the UAV.

[0021] Furthermore, the method for determining whether the tracking target is lost in step (6) is as follows:

[0022] If the tracking confidence of the current frame is lower than the threshold, the tracking target is considered lost.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] 1. The present invention proposes a deep learning detection and tracking method that can be deployed on a small airborne processing device.

[0025] 2. Compared with the previous traditional detection and tracking algorithms, the detection accuracy and tracking stability of the present invention have been greatly improved.

[0026] 3. The present invention uses the method of initializing the tracking module in advance to adjust the necessary initialization phase in the process of switching from the detection mode to the tracking mode to be completed when the entire system is started, thereby reducing the delay of the switching process and improving the stability of the tracking process. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for describing the embodiments are briefly introduced below.

[0028] Figure 1 The present invention is a flow chart of a real-time target detection and tracking method based on an unmanned aerial vehicle.

[0029] Figure 2 It is a schematic diagram of the present invention that initializes the airborne tracking module in advance to reduce the delay in the process of switching from the detection mode to the tracking mode. DETAILED DESCRIPTION

[0030] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0031] like Figure 1 As shown, the present invention provides a real-time target detection and tracking method based on an unmanned aerial vehicle, which mainly includes the following steps:

[0032] (1) First, we collected images with targets and created a dataset. We used the YOLOv5 target detection framework to train the detection network weights, and combined them with the target detection algorithm to form a target detection module that was deployed on the UAV’s onboard processing equipment.

[0033] (2) Prepare an image slice of the target to be tracked as a template in advance and deploy it together with the airborne tracking module on the airborne processing device of the UAV;

[0034] (3) According to the ground station command, the UAV begins to use the template information to initialize the UAV's onboard tracking module in advance;

[0035] (4) According to the command from the ground station, the UAV takes off to the designated location;

[0036] (5) The UAV uses visual sensors to obtain ground video images from the air and sends the ground video image stream to the ground station;

[0037] (6) The operator at the ground station waits for the right time based on the transmitted ground video images and uses input devices such as a mouse and keyboard to issue a command to the UAV to start target detection;

[0038] (7) According to the command of the ground station, the UAV switches the system to the detection mode and starts the target detection module. It uses the target detection model to detect the ground video image stream collected by the sensor, extracts the detection frames of all targets and marks them in the image, and then sends the detected video image to the ground station;

[0039] (8) The operator uses input devices such as a mouse and keyboard to select the target to be tracked from the detected video image. If the target to be tracked has a detection frame extracted by the target detection model, the corresponding detection frame can be directly selected. If the detection frame cannot be extracted, the bounding box of the target can also be manually drawn and the detection frame or bounding box coordinates can be sent to the drone.

[0040] (9) After the UAV receives the tracking command and the tracking bounding box, the system will immediately switch to the tracking mode without going through the tracking initialization process, and use the image in the box as a template to update the onboard tracking module, and then continue to track the target; Figure 2 As shown;

[0041] (10) If the tracking target is lost, the image is re-detected and the tracking target is selected again, and the process returns to step (7); wherein, if the tracking confidence of the current frame is lower than a threshold, the tracking target is considered to be lost.

[0042] The specific operation method of the mouse and keyboard in steps (6) and (8) includes the following process:

[0043] (6.1) If it is necessary to enter the detection mode, the operator presses the "q" key on the keyboard, and the system will enter the target detection mode and perform real-time detection on the incoming image;

[0044] (6.2) If the target detection module detects the target to be tracked, the operator can right-click the target detection frame to pass the detection frame as the target tracking frame to the tracking module for continuous tracking, and the system switches from detection mode to tracking mode;

[0045] (6.3) If the target detection module does not detect the target you want to track, you can manually draw a frame around the target to be tracked. That is, use the left mouse button to click and drag the cursor to draw a frame to frame the target to draw a target tracking frame. After releasing the mouse, this frame is passed to the tracking module as the target tracking frame for continuous tracking. The system switches from detection mode to tracking mode.

[0046] (6.4) If you need to switch the tracking target, press the "w" key on the keyboard to switch the tracking mode to the detection mode so that you can reselect the tracking target;

[0047] (6.5) If you want to end the detection and tracking mode, press the "e" key on the keyboard to cancel the detection or tracking mode and output a pure image stream;

[0048] How to select the tracking box based on the operator's right-click position in step (6.2):

[0049] When the operator right-clicks the target detection frame, if there are multiple target detection frames near the mouse click position, the detection frame whose center is closest to the mouse click position is passed to the onboard tracking module as the target tracking frame;

[0050] In summary, the present invention combines the YOLOv5 target detection algorithm based on deep learning and the SiamRPN target tracking algorithm, and aims at the problem that the long initialization time of the SiamRPN algorithm seriously affects the real-time and stability of tracking. A method of advancing the tracking initialization stage to the time when the entire system is initialized is proposed, thereby improving the fluency of the process from detection to tracking mode switching, enabling the entire system to track the target in real time and stably, and obtaining a detection and tracking algorithm and system with high detection accuracy, stable tracking and real-time processing.

[0051] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0052] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A real-time target detection and tracking method based on unmanned aerial vehicle, characterized in that: Based on UAV and ground station, it includes the following processes: (1) Initializing the UAV's onboard tracking module using the pre-saved image of the target to be tracked as a template; (2) The drone takes off to the set location and uses visual sensors to obtain real-time ground video images from the air; (3) The UAV starts the detection module according to the command of the ground station, uses the target detection model to perform target detection on the acquired ground video image, extracts the detection frames of all targets and marks them in the image, and sends the detected video image to the ground station; (4) At the ground station, the target to be tracked is selected from the detected video image. If the detection frame of the target to be tracked has been extracted by the target detection model, the corresponding detection frame is directly selected. If the detection frame has not been extracted, the bounding box of the target is drawn and the detection frame or bounding box coordinates are sent to the UAV. (5) After receiving the detection frame or bounding box, the drone switches to tracking mode and uses the image in the frame as a template to update the onboard tracking module, and then continues to track the target; (6) If the tracking target is lost, the UAV switches from tracking mode to detection mode, waits for the ground station to reselect the tracking target, and returns to step (3).

2. A real-time target detection and tracking method based on an unmanned aerial vehicle according to claim 1, characterized in that: The specific process of step (1) is as follows: (1.1) Obtain the image template of the target to be tracked in advance and store it on the onboard storage device of the UAV; (1.2) When the system starts running, the UAV's SiamRPN tracking algorithm module is started. The SiamRPN tracking algorithm module uses the stored image template to initialize the UAV's airborne tracking module.

3. The real-time target detection and tracking method based on an unmanned aerial vehicle according to claim 1, characterized in that: The specific process of step (2) is as follows: (3.1) The drone takes off to the set location and uses visual sensors to obtain ground video images from the air, and transmits the ground video image stream to the ground station in real time; (3.2) At the ground station, the detection time is selected based on the transmitted ground video image stream, and a command to start target detection is issued to the UAV.

4. The real-time target detection and tracking method based on an unmanned aerial vehicle according to claim 1, characterized in that: The method for determining whether the tracking target is lost in step (6) is as follows: If the tracking confidence of the current frame is lower than the threshold, the tracking target is considered lost.

Citation Information

Cited By

  • Aircraft target locking and tracking method and system for complex dynamic environment

    CN119937621A

  • A method and system for locking and tracking aircraft targets in complex dynamic environments

    CN119937621B