Unmanned aerial vehicle dynamic target tracking method based on end-to-end learning

Abstract

The invention relates to an unmanned aerial vehicle dynamic target tracking method based on end-to-end learning, and the method comprises the following steps: collecting multi-modal input data, including a visual image, IMU motion information, and environment sensor data; visual and motion features are extracted through a double-flow convolutional neural network; utilizing a graph attention network to fuse multi-modal features, and modeling an unmanned aerial vehicle-target-environment spatial relationship; generating an unmanned aerial vehicle tracking trajectory based on a deep reinforcement learning strategy, and optimizing a flight control instruction; and real-time obstacle avoidance and flight path adjustment are carried out in combination with a dynamic window method. High-precision, low-delay and real-time target tracking of the unmanned aerial vehicle in a complex dynamic environment is achieved through multi-modal feature fusion, a graph attention network and a deep reinforcement learning technology, the successful tracking rate of the method in the complex environment reaches 92.5%, real-time deployment of embedded equipment is supported, the multi-target cooperative processing capability is achieved, and the method is suitable for large-scale popularization and application. The system is suitable for scenes of security monitoring, disaster rescue, logistics distribution, film and television shooting and the like.

Description

Technical Field

[0001] The present invention relates to the technical fields of unmanned aerial vehicle (UAV) control and computer vision, and particularly to a UAV dynamic target tracking method based on end-to-end learning. This method realizes high-precision and real-time target tracking of UAVs in complex dynamic environments by integrating deep learning and reinforcement learning technologies, and is applicable to scenarios such as security monitoring, disaster relief, logistics distribution, and film shooting. Background Art

[0002] Target tracking is an important research direction in the field of computer vision and is currently widely used in sports event broadcasting, security monitoring, and fields such as UAVs, unmanned vehicles, and robots. Existing UAV dynamic target tracking technologies mainly rely on the following two types of methods:

[0003] Methods based on traditional vision algorithms:

[0004] Such methods perform target tracking by manually designing features (such as color histograms, optical flow methods, HOG, etc.) in combination with Kalman filtering or particle filtering; however, such methods have significant defects:

[0005] Poor environmental adaptability: Changes in lighting, target occlusion, or rapid movement can easily lead to tracking failure;

[0006] High computational complexity: Multi-stage processing (feature extraction → motion prediction → position correction) results in insufficient real-time performance;

[0007] Reliance on manual parameter tuning: Filter parameters need to be manually adjusted, making it difficult to adapt to dynamic scenarios.

[0008] Methods based on shallow machine learning:

[0009] Use support vector machines (SVMs) or random forests to perform short-term prediction of targets; however, their limitations include:

[0010] Limited feature expression ability: Shallow models cannot capture the complex non-linear relationships between targets and the environment;

[0011] Insufficient long-term dependence modeling: It is difficult to handle the temporal correlation in the continuous movement of targets;

[0012] Weak generalization ability: Poor robustness to untrained scenarios (such as dense occlusion, high-speed movement).

[0013] In addition, the following common problems also exist in the prior art:

[0014] Lack of end-to-end optimization: Stage-by-stage processing leads to error accumulation and limited overall performance;

[0015] Insufficient modeling of dynamic target interaction: The collaborative decision-making of UAVs with targets and the environment has not been fully modeled;

[0016] Low resource efficiency: Traditional algorithms consume excessive resources when running on embedded devices. Summary of the Invention

[0017] The purpose of the present invention is to solve the disadvantages existing in the prior art as described in the background art, and to propose a dynamic target tracking method for unmanned aerial vehicles based on end-to-end learning, aiming to achieve the following objectives:

[0018] High-precision tracking: Model the global relationship between target motion and the environment through a deep neural network.

[0019] Real-time performance and low latency: Optimize the model architecture and computational process to support deployment on embedded devices.

[0020] Strong environmental adaptability: Adapt to target occlusion, lighting changes, and complex background interference.

[0021] To achieve the above objectives, the present invention adopts the following technical solutions:

[0022] A dynamic target tracking method for unmanned aerial vehicles based on end-to-end learning, including the following steps:

[0023] S1. Collect multi-modal input data, including visual images, IMU motion information, and environmental sensor data.

[0024] S2. Extract visual and motion features through a two-stream convolutional neural network.

[0025] S3. Use a graph attention network to fuse multi-modal features and model the spatial relationship between the unmanned aerial vehicle - target - environment.

[0026] S4. Generate the tracking trajectory of the unmanned aerial vehicle based on a deep reinforcement learning strategy and optimize the flight control instructions.

[0027] S5. Combine the dynamic window method to avoid obstacles in real time and adjust the flight path.

[0028] As a further aspect of the present invention, the two-stream convolutional neural network includes an RGB stream and an infrared stream, and the output features are fused through channel concatenation.

[0029] As a further aspect of the present invention, the node update formula of the graph attention network is: where α ij is calculated by the multi-head attention mechanism, and the number of heads is 4.

[0030] As a further aspect of the present invention, the reward function of the deep reinforcement learning is defined as: where ω1 = 0.7, ω2 = 0.3, and E is the instantaneous energy consumption of the unmanned aerial vehicle.

[0031] As a further aspect of the present invention, the speed search space of the dynamic window method is: v ∈ [0, v max , ω ∈ [-ω max , ω max , where v max = 5 m / s and ω max = 1.5 rad / s.

[0032] As a further aspect of the present invention, the method supports multi-object collaborative tracking, and assigns independent decision branches to each object through a multi-head attention mechanism.

[0033] As a further aspect of the present invention, the UAV control instructions are converted into throttle, yaw angle and roll angle outputs through a PID controller.

[0034] As a further aspect of the present invention, a UAV dynamic target tracking system includes:

[0035] A data acquisition module configured to receive multi-modal sensor inputs;

[0036] A feature fusion module that realizes the multi-modal feature alignment and spatial modeling described in claims 1-7;

[0037] A decision control module that generates a flight trajectory based on reinforcement learning and avoids obstacles in real time;

[0038] A communication module that transmits control instructions to the UAV flight control system.

[0039] As a further aspect of the present invention, a UAV dynamic target tracking system is deployed on an embedded AI computing platform, supporting 60fps real-time processing and multi-object tracking.

[0040] As a further aspect of the present invention, a computer-readable storage medium stores program instructions that, when executed by a processor, implement the method described in any one of claims 1-7.

[0041] Compared with the prior art, the beneficial effects of the present invention are:

[0042] Tracking accuracy: On the OTB-100 dataset, the success rate reaches 92.5%, which is 18.2% higher than that of traditional methods (78.3%);

[0043] Real-time performance: The single-frame processing time ≤ 15 ms, supporting 60fps real-time control;

[0044] Robustness: When the target occlusion rate is 50%, the tracking error only increases by 12%. Detailed implementation

[0045] The content of the present invention can be more easily understood by referring to the following detailed description of the preferred embodiments of the present invention and the included examples. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. In case of conflict, the definitions in this specification shall prevail.

[0046] The singular forms include plural referents unless the context clearly dictates otherwise. "Optional" or "any one" means that the event or circumstance described thereafter may or may not occur, and the description includes both the case where the event occurs and the case where the event does not occur.

[0047] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments.

[0048] A method for dynamic target tracking of an unmanned aerial vehicle based on end-to-end learning includes the following steps:

[0049] Collect multi-modal input data, including visual images, IMU motion information, and environmental sensor data;

[0050] Extract visual and motion features through a two-stream convolutional neural network;

[0051] Utilize a graph attention network to fuse multi-modal features and model the spatial relationship between the unmanned aerial vehicle - target - environment;

[0052] Generate an unmanned aerial vehicle tracking trajectory based on a deep reinforcement learning strategy and optimize the flight control instructions;

[0053] Combine the dynamic window method to avoid obstacles in real time and adjust the flight path.

[0054] The two-stream convolutional neural network includes an RGB stream and an infrared stream, and the output features are fused through channel concatenation.

[0055] The node update formula of the graph attention network is: where α ij is calculated by the multi-head attention mechanism, and the number of heads is 4.

[0056] The reward function of the deep reinforcement learning is defined as: where ω1 = 0.7, ω2 = 0.3, and E is the instantaneous energy consumption of the unmanned aerial vehicle.

[0057] The speed search space of the dynamic window method is: v ∈ [0, v max , ω ∈ [-ω max , ω max , where v max = 5 m / s, ω max = 1.5 rad / s.

[0058] The method supports multi-target collaborative tracking and assigns independent decision branches to each target through the multi-head attention mechanism.

[0059] The UAV control instruction is converted into throttle, yaw angle and roll angle outputs through a PID controller.

[0060] A UAV dynamic target tracking system, comprising:

[0061] A data acquisition module configured to receive multi-modal sensor inputs;

[0062] A feature fusion module that realizes multi-modal feature alignment and spatial modeling in claims 1-7;

[0063] A decision control module that generates a flight trajectory based on reinforcement learning and avoids obstacles in real time;

[0064] A communication module that transmits control instructions to the UAV flight control system.

[0065] A UAV dynamic target tracking system is deployed on an embedded AI computing platform and supports 60fps real-time processing and multi-target tracking.

[0066] A computer-readable storage medium stores program instructions that, when executed by a processor, implement the method of any one of claims 1-7

[0067] The present invention aims to achieve the following objectives:

[0068] High-precision tracking: Modeling the global relationship between target motion and the environment through a deep neural network;

[0069] Real-time performance and low latency: Optimizing the model architecture and computational process to support deployment on embedded devices;

[0070] Strong environmental adaptability: Adapting to target occlusion, light changes, and complex background interference.

[0071] The core steps of the present invention are as follows:

[0072] ① Multi-modal data input:

[0073] Visual data: RGB images and infrared images captured by the UAV camera;

[0074] Motion data: Acceleration and angular velocity information of the UAV IMU (Inertial Measurement Unit);

[0075] Environmental data: GPS position, wind speed, and obstacle distance (obtained through LiDAR or ultrasonic sensors).

[0076] ② End-to-end feature extraction and fusion:

[0077] The two-stream convolutional neural network (TS-CNN) is used to extract visual features and motion features respectively;

[0078] The graph attention network (GAT) is used to fuse multi-modal data and model the three-dimensional spatial relationship of the drone-target-environment. The node update formula is: where α ij is calculated by the multi-head attention mechanism, and W (l) is a learnable parameter.

[0079] ③ Dynamic decision-making and trajectory generation:

[0080] Based on the deep reinforcement learning (DRL) framework, with the tracking accuracy and energy consumption as the joint reward function, the flight strategy of the drone is optimized;

[0081] The spatio-temporal Transformer is used to generate a smooth trajectory and predict the position sequence of the target within the next T frames.

[0082] ④ Real-time control and obstacle avoidance:

[0083] The predicted trajectory is converted into drone control commands (such as throttle, yaw angle);

[0084] Combined with the dynamic window approach (DWA) for real-time obstacle avoidance to ensure flight safety.

[0085] Example 1: Training and testing on standard datasets

[0086] 1. Data preparation:

[0087] Datasets: OTB-100 and VisDrone2019, covering scenarios of illumination changes, occlusion, and fast motion;

[0088] Preprocessing:

[0089] The images are uniformly scaled to a resolution of 320×240, and the frame rate is 30fps;

[0090] The IMU data and GPS information are synchronized and aligned, and the timestamp error ≤ 1ms.

[0091] 2. Model training:

[0092] Network architecture:

[0093] TS-CNN: ResNet-18 backbone network, and the two-stream output is fused into 512-dimensional features;

[0094] GAT: 4-layer graph attention network, with 4 heads in each layer;

[0095] DRL policy network: Actor-Critic architecture, with a hidden layer dimension of 256;

[0096] Training parameters:

[0097] Optimizer: AdamW (learning rate 3e-4, weight decay 1e-5);

[0098] Reward function: Tracking accuracy weight 0.7 + energy consumption weight 0.3;

[0099] Batch size: 32, number of training epochs 200.

[0100] 3. Test results:

[0101] Successful tracking rate: 92.5% (OTB-100), 89.7% (VisDrone2019);

[0102] Resource occupancy: Model size 45MB, GPU memory occupancy 1.2GB;

[0103] Extreme scenario: When the target occlusion is 50%, the tracking error is 15px (35px for traditional methods).

[0104] Example 2: Real-time deployment on an embedded platform

[0105] 1. System deployment:

[0106] Hardware platform: NVIDIA Jetson Xavier NX;

[0107] Software optimization: The model is converted into a TensorRT engine and INT8 quantization is enabled;

[0108] Sensor integration: RGB camera (30fps), IMU module (100Hz), LiDAR (10Hz).

[0109] 2. Performance testing:

[0110] End-to-end latency: 14ms (including data acquisition, processing, and control instruction generation);

[0111] Power consumption: Average power consumption 8W, peak 12W;

[0112] Obstacle avoidance test: In a dynamic obstacle environment, the obstacle avoidance success rate is 98%.

[0113] Example 3: Multi-target collaborative tracking

[0114] 1. Scene expansion:

[0115] Number of targets: Simultaneously track 3 independently moving targets;

[0116] Data annotation: Use the MOT17 dataset and expand it to multi-target annotation.

[0117] 2. Model improvement:

[0118] Introduce the multi-head attention mechanism and assign independent tracking branches to each target;

[0119] Optimize the reward function and add a conflict penalty term between targets.

[0120] 3. Results:

[0121] Multi-object tracking accuracy: 85.4% (MOT17);

[0122] Increase in resource occupancy: The model size increases to 58 MB and the latency increases to 18 ms.

[0123] The examples involved in this article are only illustrative and are used to explain some features of the method of the present invention. The appended claims are intended to claim the broadest scope conceivable, and the embodiments presented herein are only illustrative of the selected embodiments according to all possible combinations of embodiments. Therefore, the applicant's intention is that the appended claims are not limited by the selection of examples illustrating the features of the present invention. Some of the numerical ranges used in the claims also include sub-ranges within them, and variations within these ranges should also be interpreted as being covered by the appended claims where possible.

Claims

1. An end-to-end learning-based dynamic target tracking method for unmanned aerial vehicles, characterized in that, It includes the following steps: Collect multi-modal input data, including visual images, IMU motion information, and environmental sensor data; Extract visual and motion features through a two-stream convolutional neural network; Use a graph attention network to fuse multi-modal features and model the spatial relationship of the drone-target-environment; Generate a drone tracking trajectory based on a deep reinforcement learning strategy and optimize the flight control instructions; Combine the dynamic window method to avoid obstacles in real time and adjust the flight path.

2. The method for dynamically tracking a target by a drone based on end-to-end learning according to claim 1, wherein The two-stream convolutional neural network includes an RGB stream and an infrared stream, and the output features are fused through channel splicing.

3. The method for dynamically tracking an unmanned aerial vehicle target based on end-to-end learning according to claim 1, wherein, The node update formula of the graph attention network is as follows: where α ij is calculated by the multi-head attention mechanism, and the number of heads is 4.

4. A dynamic target tracking method for an unmanned aerial vehicle based on end-to-end learning according to claim 1, characterized in that, The reward function of the deep reinforcement learning is defined as: where ω1 = 0.7, ω2 = 0.3, and E is the instantaneous energy consumption of the UAV.

5. A dynamic target tracking method for an unmanned aerial vehicle based on end-to-end learning according to claim 1, characterized in that The speed search space of the dynamic window method is: v ∈ [0, v max , ω ∈ [-ω max , ω max , where v max = 5 m / s and ω max = 1.5 rad / s.

6. The method for dynamically tracking an unmanned aerial vehicle's moving target based on end-to-end learning according to claim 1, wherein, The method supports multi-target collaborative tracking, and assigns independent decision branches to each target through a multi-head attention mechanism.

7. The method for dynamically tracking an unmanned aerial vehicle's moving target based on end-to-end learning according to claim 1, wherein The drone control instructions are converted into throttle, yaw angle, and roll angle outputs through a PID controller.

8. A dynamic target tracking system for an unmanned aerial vehicle according to any one of claims 1-7, characterized in that It includes: A data acquisition module configured to receive multi-modal sensor inputs; A feature fusion module that realizes the multi-modal feature alignment and spatial modeling described in claims 1-7; A decision control module that generates a flight trajectory based on reinforcement learning and avoids obstacles in real time; A communication module that transmits control instructions to the drone flight control system.

9. A dynamic target tracking system for an unmanned aerial vehicle according to claim 8, characterized in that, Deployed on an embedded AI computing platform, supporting 60fps real-time processing and multi-target tracking.

10. A computer-readable storage medium according to any one of claims 8 to 9, characterized in that, Stores program instructions that, when executed by a processor, implement the method described in any one of claims 1-7.

Images