Unmanned aerial vehicle target tracking and obstacle avoidance fusion control method and system based on deep reinforcement learning

Abstract

This invention discloses a method and system for fusion control of UAV target tracking and obstacle avoidance based on deep reinforcement learning. The method includes the following steps: Step S1: Acquire image data collected by the UAV's onboard sensors, estimate the target state based on the acquired image data to obtain target state estimation information, and compensate for the UAV's own motion; Step S2: Construct a policy network based on deep reinforcement learning, using environmental information perceived by the UAV's onboard sensors and target state estimation information as input, and mapping it to the UAV's speed control commands through the policy network. During training, a composite reward function that fuses multiple objectives—tracking accuracy, obstacle avoidance safety, and motion stability—is used to guide target following and obstacle avoidance; Step S3: Control the UAV's flight according to the speed control commands, driving the UAV to autonomously avoid obstacles while tracking the target. This invention enables real-time coordination of UAV target tracking and obstacle avoidance.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) control technology, and in particular to a UAV target tracking and obstacle avoidance fusion control method and system based on deep reinforcement learning. Background Technology

[0002] Drones are widely used in logistics, disaster relief, power line inspection, and aerial filming to perform various complex tasks. Autonomous target tracking in complex obstacle environments requires drones to continuously track moving targets while ensuring their own flight safety and keeping the target within the limited field of view of the onboard sensors. In unstructured obstacle environments such as urban areas and jungles, the uncertainty of target movement, the obstruction of view caused by obstacles, and the drone's own dynamic constraints make this type of autonomous target tracking extremely challenging.

[0003] Currently, traditional UAV target tracking methods in obstacle-prone environments typically employ planning methods based on motion prediction and trajectory optimization. The performance of these methods heavily relies on the accuracy of front-end target prediction and path search, and usually requires precise environmental perception information as input. Furthermore, traditional UAV target tracking methods in obstacle-prone environments do not consider obstacle occlusion during the path search phase; that is, they typically design target tracking and obstacle avoidance as two independent modules to achieve target state estimation, path planning, and obstacle avoidance control. This separate design of tracking and obstacle avoidance modules leads to system response delays and control conflicts, making real-time fusion control difficult. It also results in a lack of collaborative optimization between modules; the tracking module may guide the UAV closer to an obstacle, while the obstacle avoidance module forces the UAV away from the target, creating control conflicts. This results in significant response delays in dynamic environments, making real-time, smooth fusion control difficult to achieve. Summary of the Invention

[0004] The technical problem to be solved by this invention is: In view of the above-mentioned problems existing in the prior art, this invention provides a method and system for fusion control of UAV target tracking and obstacle avoidance based on deep reinforcement learning, which can realize real-time coordination of UAV target tracking and obstacle avoidance, avoid control conflicts and response delays caused by separate control, and improve the stability of UAV control.

[0005] To solve the above-mentioned technical problems, the technical solution proposed by this invention is as follows: A method for fusion control of UAV target tracking and obstacle avoidance based on deep reinforcement learning, comprising the following steps: Step S1, Target State Estimation: Acquire image data collected by the UAV's onboard sensors, estimate the target state based on the acquired image data to obtain target state estimation information. The target state includes the target position and velocity. During the flight of the UAV, the target coordinates observed under different camera poses are uniformly transformed to the same reference viewpoint to compensate for the motion of the UAV itself. Step S2, Fusion Control Command Generation: Construct a policy network based on deep reinforcement learning, using environmental information perceived by the UAV's onboard sensors and the target state estimation information as input, and map it into the UAV's speed control command through the policy network. During the training process, the reward function of the policy network adopts a composite reward function that fuses multiple objectives such as tracking accuracy, obstacle avoidance safety, and motion stability, so as to guide the policy network to achieve target following and obstacle avoidance in the same control command. Step S3, UAV control: Control the UAV to fly according to the speed control command, and drive the UAV to achieve autonomous obstacle avoidance while tracking the target.

[0006] Furthermore, in step S1, the target state is defined as a four-dimensional vector containing the target position and velocity. ,in, Let the center coordinates of the target be in the pixel coordinate system. For the goal and The rate of pixel change in the direction; The motion characteristics of the target in the image plane are modeled using a constant velocity model. The discrete-time state transition equation of the system is expressed as: , ; in, This represents process noise and is modeled with a mean of 0 and a covariance matrix of... Gaussian process , Represents the state transition matrix. Indicates the sampling time interval.

[0007] Further, in step S1, the target state is estimated based on the extended Kalman filter to obtain target state estimation information. This step includes: Step S101, Initialization: Initialize the state vector based on the detection results of the first frame. and the corresponding error covariance matrix ; Step S102, Prediction: Predict the state at the next moment based on the motion model: , ; in, , These represent the predicted state of the target at the current moment and the state of the target at the previous moment, respectively. , These represent the prediction error covariance matrix of the target at the current time and the error covariance matrix of the target at the previous time, respectively. Step S103, Update: Change the center coordinates of the target image directly detected in the current frame. As an observation, the observation equation is modeled as follows: , ; in, The observed noise is modeled with a mean of 0 and a covariance matrix of... Gaussian process , The state vector after compensation; The Kalman gain is then calculated as follows: , in, , These represent the Kalman gain and the observation matrix, respectively. Status updated to: , Covariance updated to: , The final output of the target estimator is ,in, The detection bounding box height information is directly derived from the object detector, and the center coordinates of the target in the pixel coordinate system are... As a visual servo signal to guide the drone in tracking the target. For the goal and The rate of pixel change in the direction.

[0008] Furthermore, in step S1, the homography matrix between two consecutive frames is used to uniformly transform the target coordinates observed under different camera poses to the same reference viewpoint:

[0009]

[0010] in, The camera intrinsic parameter matrix, for and The homography matrix between time points, Indicates from Time camera coordinate system to The rotation matrix of the camera coordinate system at any given time. Indicates from Time camera coordinate system to The translation vector of the camera coordinate system at any given time. express The unit normal vector in the camera coordinate system at any given time. express The signed distance from the origin of the camera coordinate system to the plane at any given time. and for The target coordinates observed at all times, and For the goal Projected coordinates on the time-lapse image.

[0011] Furthermore, in step S2, constructing the policy network based on deep reinforcement learning includes: The target tracking task of UAVs in obstacle environments is modeled as a partially observable Markov decision. The observation space is defined to include environmental information perceived by the UAV's onboard sensors, the target state estimate, and the UAV's own state information. The action space is defined to include the UAV's own state information. x direction and y The directional speed control command and the yaw rate control command define a reward function as the composite reward function, which includes a field-of-view retention reward term representing the reward obtained for keeping the target within the field of view. Location guidance reward items used to guide drones to track targets based on the location information of drones and targets. and collision penalty items used to penalize collisions. A heading alignment penalty is used to penalize the decrease in the angle between the UAV's heading and the target's orientation. Motion smoothing penalty term used to incentivize drones to achieve stable motion. And task completion rewards , t Indicates the time.

[0012] Furthermore, the vision retention reward item Calculated based on the intersection-union ratio (IUU) between the detection box and the reference box: , in, and These represent the current target detection box and the reference box, respectively. The intersection-union ratio of the two frames; Location-guided reward items The following calculations were performed based on the difference between the actual distance between the UAV and the target and the expected distance, as well as the angle between the line connecting the UAV and the target and the UAV's heading: , in, Used to calculate the actual and expected distances between the drone and the target. The difference between them This represents the position vector of the target in the UAV's body coordinate system at the current moment. , They are respectively t time, t The angle between the line connecting the UAV and the target at time -1 and the UAV's heading. These represent the preset weighting coefficients.

[0013] Furthermore, the collision penalty item Enabled only when a collision is detected; The heading alignment penalty item The following calculations were performed based on the angle between the UAV's heading and the target's orientation: ,in, Indicates the angle between the drone's heading and the target's orientation; The motion smoothing penalty item According to drones shaft and The linear velocity and yaw rate of the shaft were calculated as follows: ,in, for t Time Drone shaft and The linear velocity and yaw rate of the shaft; The task completion reward items This takes effect only when the state of the UAV relative to the target meets preset state conditions, which include... Angle greater than the preset intersection-union ratio threshold, and the angle between the UAV's heading and the target's orientation. Less than the preset included angle threshold.

[0014] Furthermore, the policy network includes a one-dimensional convolutional encoder and an MLP fusion module. The one-dimensional convolutional encoder encodes the environmental information perceived by the UAV's recorded sensors into feature vectors. The MLP fusion module fuses the feature vectors with vectorized inputs, which include target state estimation information and UAV speed information. The one-dimensional convolutional encoder has a three-layer structure: the first layer uses multiple one-dimensional convolutional kernels in a first hidden layer to extract features from multiple consecutive frames of input data, followed by nonlinear processing using a linear rectified activation function; the second layer uses multiple one-dimensional convolutional kernels in a second hidden layer to perform convolution operations, followed by processing using a ReLU activation function; and the third layer encodes the features into a fully connected layer. The MLP fusion module adopts a single hidden layer structure. The output of the policy network maps the features into action outputs through a fully connected layer.

[0015] A control system includes a processor and a memory, the memory for storing a computer program, and the processor for executing the computer program to perform the method described above.

[0016] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention adopts a fusion control architecture based on deep reinforcement learning, which models tracking and obstacle avoidance as a unified end-to-end decision problem. By using a composite reward function that jointly optimizes tracking error and collision risk, the policy network is guided to simultaneously consider target following and obstacle avoidance in the same control command. This avoids control conflicts and response delays caused by module separation and achieves real-time coordination between tracking and obstacle avoidance.

[0018] 2. This invention adopts a composite reward structure that integrates tracking accuracy, obstacle avoidance safety, and motion stability. It can guide the agent to learn tracking accuracy, obstacle avoidance safety, and motion stability simultaneously during training. The policy network outputs continuous speed control commands, which can also meet the UAV's requirements for motion smoothness and effectively avoid the severe shaking problem of traditional methods.

[0019] 3. This invention further combines the motion-compensated extended Kalman filter method based on homography transformation to estimate the target's position and direction of motion in real time in the image plane, compressing high-dimensional image information into low-dimensional state vectors. The policy network takes the low-dimensional state vectors and LiDAR data as inputs, which can map control commands directly from sensor observations without the need for environmental maps, target absolute position truth values, or complex path search and trajectory optimization. This significantly improves the adaptability to unknown environments and uncertain target motion. Attached Figure Description

[0020] Figure 1 This is a schematic diagram illustrating the implementation process of the UAV target tracking and obstacle avoidance fusion control method based on deep reinforcement learning in this embodiment.

[0021] Figure 2 This is a schematic diagram illustrating the definition principle of some parameters involved in the reward function in this embodiment.

[0022] Figure 3 This is a schematic diagram of the overall structure of the policy network in this embodiment.

[0023] Figure 4 This is a schematic diagram of the overall system structure for achieving integrated control of UAV target tracking and obstacle avoidance in a specific application embodiment. Detailed Implementation

[0024] The present invention will be further described below with reference to the accompanying drawings and specific preferred embodiments, but this does not limit the scope of protection of the present invention.

[0025] As disclosed in this invention, unless the context clearly indicates otherwise, words such as "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. The terms "first," "second," and similar terms used in this invention disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, words such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connected" or "linked" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0026] To facilitate understanding, the relevant technical background of the present invention will first be introduced by way of example.

[0027] Traditional UAV target tracking methods in obstacle-prone environments typically employ planning approaches based on motion prediction and trajectory optimization. These methods usually decompose the problem into two stages: target state estimation and safe trajectory generation. While offering good interpretability, their performance heavily relies on the accuracy of front-end target prediction and the completeness of the environmental map. Tracking performance is critically dependent on accurate environmental map construction, target pose estimation, or complex path search and trajectory optimization, requiring precise environmental perception information as input and necessitating pre-built environmental maps or online path search. However, in real-world complex environments, target motion is uncertain, and obstacle distribution dynamically changes, making it difficult to establish accurate prediction models or global maps. This leads to trajectory generation deviating from reality, compromising tracking continuity and obstacle avoidance safety, and exhibiting poor adaptability to unknown environments and uncertain target motion.

[0028] Deep reinforcement learning-based methods learn optimal strategies through interaction between the agent and the environment. They can integrate perception and control end-to-end without requiring precise environmental models or prior target motion patterns, adaptively addressing challenges posed by target motion uncertainty, obstacle occlusion, and UAV dynamic constraints. However, deep reinforcement learning methods suffer from limitations such as simple reward function design and discretized action space, making it difficult to achieve multi-objective optimization and resulting in non-smooth motion. Furthermore, some deep learning algorithms rely on the target's true position in the global coordinate system as input, which is difficult to obtain in non-cooperative target tracking tasks (where the target does not actively send position information). Additionally, factors such as obstacle occlusion and sensor noise make it difficult to accurately estimate the target's absolute position in real time.

[0029] Furthermore, traditional methods typically design target tracking and obstacle avoidance as two independent modules, i.e., separate tracking and obstacle avoidance modules to achieve target state estimation, path planning, and obstacle avoidance control. However, this separate design approach leads to system response delays and control conflicts, making it difficult to achieve real-time fusion control. Due to the lack of collaborative optimization between modules caused by the separate design structure, the tracking module may guide the UAV closer to the obstacle, while the obstacle avoidance module may force the UAV to deviate from the target, resulting in control conflicts. Moreover, the response delay is significant in dynamic environments, making it difficult to achieve real-time, smooth fusion control.

[0030] For example, a Bézier curve-based constrained regression method is used for target motion prediction, combined with a heuristic path search front-end with kinematic and dynamic constraints and a spatiotemporally optimal trajectory optimization back-end, to achieve robust tracking of moving targets in cluttered environments. However, the path search stage fails to adequately consider obstacle occlusion, thus requiring a separate obstacle avoidance module for obstacle avoidance control. This involves designing target tracking and obstacle avoidance as two independent modules, with the tracking module generating the desired trajectory first, and the obstacle avoidance module making local corrections. This architecture inherently cannot achieve coordinated optimization of tracking accuracy and obstacle avoidance safety. Furthermore, the serial processing between modules introduces additional computational latency, making it difficult to respond in real-time to sudden changes in target movement or the sudden appearance of obstacles in dynamic environments.

[0031] This invention employs a deep reinforcement learning-based fusion control architecture, treating tracking and obstacle avoidance as a unified decision-making problem. By constructing a deep reinforcement learning-based policy network, environmental information perceived by the UAV's onboard sensors and target state estimation information are used as inputs to the policy network, mapped into speed control commands for the UAV. During training, the policy network uses a composite reward function that integrates multiple objectives, including tracking accuracy, obstacle avoidance safety, and motion stability. This allows for joint optimization of tracking error and collision risk, guiding the policy network to consider both target following and obstacle avoidance within a single control command. Furthermore, the policy network directly maps speed control commands based on sensor observations (including target state estimation information and environmental information perceived by the onboard sensors), avoiding delays caused by serial processing between modules. This enables real-time coordination between tracking and obstacle avoidance, thus avoiding control conflicts and response delays caused by the separate design of tracking and obstacle avoidance modules in traditional methods. It solves the problem of tracking failures caused by target motion uncertainty and obstacle occlusion when UAVs track moving targets in obstacle environments, ensuring stable tracking of moving targets while avoiding obstacles.

[0032] like Figure 1 As shown, the steps of the UAV target tracking and obstacle avoidance fusion control method based on deep reinforcement learning in this embodiment include: Step S1, Target State Estimation: Acquire image data collected by the UAV's onboard sensors, estimate the target state based on the acquired image data to obtain target state estimation information. The target state includes the target position and velocity. During the UAV's flight, the target coordinates observed under different camera poses are uniformly transformed to the same reference viewpoint to compensate for the UAV's own motion.

[0033] Specifically, a gimbal camera and lidar can be mounted on a rotary-wing drone platform. The camera collects image data, and the lidar senses environmental information to identify target and obstacle information, ultimately enabling the drone to continuously and stably track moving targets in obstacle environments.

[0034] Specifically, to estimate the motion of the target in the camera image plane, this embodiment defines the target's state as a four-dimensional vector containing its position and velocity. ,in, Let the center coordinates of the target be in the pixel coordinate system. For the goal and The rate of pixel change in the direction.

[0035] For the motion characteristics of the target within the image plane, this embodiment uses a constant velocity (CV) model to model the motion characteristics of the target within the image plane. This model assumes that the rate of change of the target's pixel coordinates remains constant within a short sampling period. Based on this model, the discrete-time state transition equation of the system can be expressed as: (1) (2) in, This represents process noise and is modeled with a mean of 0 and a covariance matrix of... Gaussian process , Represents the state transition matrix. Indicates the sampling time interval.

[0036] As an optional implementation, a target estimator is constructed based on the above model. The target estimator estimates the target state using an extended Kalman filter to obtain target state estimation information, thereby estimating the target's position coordinates and motion direction in the image plane online. Specifically, this includes: Step S101, Initialization: Initialize the state vector based on the detection results of the first frame. and its error covariance matrix ; Step S102, Prediction: Predict the state at the next moment based on the system model: (3) (4) in, , These represent the predicted state of the target at the current moment and the state of the target at the previous moment, respectively. , Let represent the prediction error covariance matrix of the target at the current time and the error covariance matrix of the target at the previous time, respectively.

[0037] Step S103, Update: The observation is the coordinates of the center of the target image directly detected in the current frame. The observation equation is then modeled as: (5) (6) in, The observed noise is modeled with a mean of 0 and a covariance matrix of... Gaussian process , This is the compensated state vector.

[0038] The Kalman gain is then calculated as follows: (7) in, , These represent the Kalman gain and the observation matrix, respectively.

[0039] Status Update: (8) Covariance update: (9) The final output of the target estimator is To provide target state information for subsequent tracking and control methods, wherein, The first component is the bounding box height information directly from the target detector. In scenarios where the target's physical size does not change significantly, this information indirectly reflects the relative distance between the target and the UAV. The other four components come from the state information of the target estimator. As a visual servo signal, it guides the drone to track the target. The resulting vector indirectly reflects the relationship between the UAV's heading and the target's orientation.

[0040] As an optional implementation, based on target state estimation using the extended Kalman filter method, the target coordinates observed under different camera poses are uniformly transformed to the same reference viewpoint using the homography matrix between two consecutive frames to compensate for the UAV's own motion, forming a motion-compensated extended Kalman filter method that combines homography transformation.

[0041] During flight, the camera pose of a drone continuously changes, causing the projected coordinates of the same spatial point in the world coordinate system to change on the image plane. To perform continuous filtering on the image plane, it is necessary to first decouple the coordinate changes caused by the drone's own motion. To solve this problem, this embodiment, for applications such as urban security, traffic monitoring, and field plain inspection, adopts the following assumptions: first, the drone's flight altitude remains constant; second, the tracked target moves on a flat surface. Based on these assumptions, the imaging relationship of the same ground target in two consecutive camera views can be described by a homography matrix. This embodiment utilizes the homography matrix between two consecutive frames to uniformly transform the target coordinates observed under different camera poses to the same reference viewpoint, thereby compensating for the influence of drone motion before filtering.

[0042] Specifically, assuming the drone is in and The position and attitude at each moment are respectively , , and The rotation and translation transformations of the camera relative to the body coordinate system are respectively and ,but and The pose of the time-lapse camera in the world coordinate system can be represented as follows: (10) (11) (12) (13) The relative motion of the cameras at two moments can be expressed as: (14) (15) in, Indicates from Time camera coordinate system to The rotation matrix of the camera coordinate system at any given time. Indicates from Time camera coordinate system to Translation vector of the camera coordinate system at any given time.

[0043] Based on the above two assumptions, the equation of motion of the target in the world coordinate system can be defined as: (16) in and The parameters of the plane equation represent the unit normal vector and the signed distance from the origin to the plane, respectively. For a target moving on the ground, we can take... , . These are the coordinates of a point on the plane, i.e., the coordinates of the target in the world coordinate system. Based on the UAV's... The pose information at any given time can transform the plane equations into... In the camera coordinate system at that moment: (17) (18) in, express The unit normal vector in the camera coordinate system at any given time. express The signed distance from the origin of the camera coordinate system to the plane at any given time.

[0044] but and The homography matrix between time points can be obtained as follows: (19) in This is the camera intrinsic parameter matrix. The target coordinates observed at any given time can be calculated using the homography matrix. The projection onto the time-lapse image thus unifies the target coordinates observed from different camera poses to the same reference viewpoint.

[0045] (20) in and for The target coordinates observed at all times, and For the goal Projected coordinates on the time-lapse image.

[0046] This embodiment employs the motion-compensated extended Kalman filter method combined with homography transformation, utilizing the homography matrix between two consecutive frames to uniformly transform the target coordinates observed from different camera poses to the same reference viewpoint. Before filtering, it compensates for image coordinate changes caused by the UAV's own motion. Based on this, a constant velocity model is used to estimate the target position and direction of motion in real time on the image plane, compressing high-dimensional image information into a low-dimensional state vector. This provides a compact and effective target state input for the subsequent policy network, while avoiding dependence on the true absolute position of the target. Furthermore, the policy network uses this low-dimensional state vector and LiDAR data as input, eliminating the need for environmental maps, true absolute target positions, and complex path searches and trajectory optimizations. It can directly map control commands from sensor observations, significantly improving adaptability to unknown environments and uncertain target motion.

[0047] Step S2, Fusion Control Command Generation: Construct a policy network based on deep reinforcement learning. The environmental information perceived by the UAV's onboard sensors and the target state estimation information are used as inputs. The policy network maps these information into the speed control commands of the UAV. During the training process, the reward function of the policy network adopts a composite reward function that integrates multiple objectives such as tracking accuracy, obstacle avoidance safety, and motion stability, so as to guide the policy network to achieve target following and obstacle avoidance in the same control command.

[0048] This embodiment constructs a policy network based on deep reinforcement learning, using target state estimation information and environmental information perceived by lidar as inputs. The end-to-end policy network directly maps these into control commands for the UAV, thereby driving the UAV to autonomously avoid obstacles while tracking the target.

[0049] Traditional deep learning networks output discrete actions, which limits the performance of drones and leads to frequent jitter and abrupt acceleration / deceleration during flight, affecting system stability. In this embodiment, the policy network outputs continuous speed control commands, which can meet the drone's requirement for smooth motion and effectively avoid the problems of severe jitter and target loss in traditional methods.

[0050] As an optional implementation method, constructing a policy network based on deep reinforcement learning includes: The target tracking task of UAVs in obstacle environments can be modeled as a partially observable Markov decision. A partially observable Markov decision can be represented by a six-tuple. It means that among them Representing the state space, Represents the action space, Let be the state transition probability. For the reward function, Represents the set of all possible observations. Indicates when an action has been performed Afterwards, and the state transitions At this time, the observation was obtained. The probability ( ).

[0051] The observation space is defined to include environmental information perceived by the UAV's onboard sensors (data from lidar), target state estimates, and the UAV's own state information. Specifically, the observation space is composed of external perception information from the two-dimensional lidar and target detector, as well as internal perception information from the inertial measurement unit. The observation at time can be represented as ,in This represents three consecutive frames of data from a two-dimensional lidar system. The target state estimate comes from the target estimator. This provides the speed information for the UAV. To reduce computational complexity and unify the dimensions of the observation space, this embodiment downsamples the lidar input data to 512 dimensions.

[0052] Define the action space including drones x direction and y Directional speed control commands and yaw rate control commands. Specifically, the action space is controlled by the UAV. x direction and yThe system consists of directional speed control commands and yaw rate control commands, with maximum output values ​​of 2.5 m / s, 2.0 m / s, and 0.5 rad / s, respectively. Since UAV speed control is a high-level control command, executed by the low-level closed-loop controller within the embedded flight controller, this control method effectively reduces model errors during the simulation-to-physical transfer process and provides inherent robustness of the speed commands across different environments.

[0053] The reward function is defined as a composite reward function, which includes a field-of-view retention reward term representing the reward obtained for keeping the target within the field of view. Location guidance reward items used to guide drones to track targets based on the location information of drones and targets. and collision penalty items used to penalize collisions. A heading alignment penalty is used to penalize the decrease in the angle between the UAV's heading and the target's orientation. Motion smoothing penalty term used to incentivize drones to achieve stable motion. And task completion rewards , t The time point is indicated. Specifically, the composite reward function includes multiple reward functions, each corresponding to a different optimization objective, such as maintaining the target's field of view, maintaining the relative distance, avoiding collisions, aligning the course, smoothing motion, and achieving the task. By reasonably setting the weights of each reward item, the agent can simultaneously learn tracking accuracy, obstacle avoidance safety, and motion stability during training, thus guiding the agent to learn these three aspects simultaneously during training.

[0054] As an optional implementation method, t The composite reward function at time step 1 can be expressed as: (twenty one) in, , , , and These represent the non-negative weights of each reward item.

[0055] Maintaining Vision Rewards This represents the reward for keeping the target within the field of view, calculated based on the intersection-union ratio (IUU) between the detection box and the reference box. For example, it can be calculated as follows: (twenty two) In the formula and These represent the current target detection box and the reference box, respectively. This represents the Intersection over Union (IoU) ratio between the two bounding boxes. The agent receives a higher reward when the detection box and the reference box are highly aligned; the reward decreases as the deviation increases. If there is no overlap between the two boxes, the reward is zero.

[0056] As an optional implementation, location-guided reward items This is used to guide the drone to track the target based on the location information of both the drone and the target. The distance is calculated based on the difference between the actual distance and the desired distance between the drone and the target, as well as the angle between the line connecting the drone and the target and the drone's heading. A position guidance reward is then given based on the relative distance change and heading alignment. For example, the following calculation method can be used: (twenty three) in, Used to calculate the actual and expected distances between the drone and the target. The difference between them This represents the position vector of the target in the UAV's body coordinate system at the current moment. , They are respectively t time, t The angle between the line connecting the UAV and the target at time -1 and the UAV's heading. These represent preset weighting coefficients. If the distance between the drone and the target is close to the preset reference distance, and the drone's flight path is aligned with the target, a reward is given; otherwise, a penalty is imposed.

[0057] As an optional implementation method, collision penalty item It is a sparse collision penalty that is only activated when a collision is detected. Its value is set with reference to the maximum reward within the round, aiming to create a significant value cliff. This ensures that the final reward of a failed round is much lower than that of a normal tracking round, thereby forcing the policy network to prioritize avoiding collision penalties through a large value gap. For example, it can be configured as follows: (twenty four) As an optional implementation, a heading alignment penalty item To prevent the agent from getting trapped in a local optimum while flying around the target, a penalty is calculated based on the angle between the UAV's heading and the target's orientation. For example, it can be calculated as follows: ,in, Indicates the angle between the drone's heading and the target's orientation, such as Figure 2 As shown.

[0058] As an optional implementation, a motion smoothing penalty term Used to incentivize drones to achieve stable movement, based on the drone shaft and The linear velocity and yaw rate of the shaft were calculated as follows: (25) in, For drones shaft and The linear velocity and yaw rate of the shaft.

[0059] As an optional implementation method, task completion rewards This is a sparse reward term that only takes effect when the drone's state relative to the target meets preset state conditions. For example, it can be configured as follows: (26) in, .

[0060] Traditional deep learning methods often employ simple reward function designs, focusing solely on tracking error or obstacle avoidance success rate. They lack comprehensive consideration of multiple objectives, such as motion smoothness, field-of-view maintenance, and energy consumption. The lack of collaborative optimization of tracking accuracy, obstacle avoidance safety, and motion stability in the reward function design makes it difficult for the agent to learn a globally optimal strategy, resulting in low tracking success rate and uneven motion under complex conditions. This embodiment utilizes the aforementioned composite reward function, forming a composite reward structure with six reward functions: a field-of-view maintenance reward based on the cross-union ratio of detection boxes; a position guidance reward based on relative distance change and heading alignment; a sparse collision penalty term; a heading alignment penalty term based on the angle between the UAV's heading and the target's orientation; a motion smoothing penalty term based on the rate of change of velocity; and a sparse task achievement reward based on multiple conditions. This integrates multiple optimization objectives such as target field-of-view maintenance, relative distance maintenance, collision avoidance, heading alignment, motion smoothness, and task achievement, guiding the policy network to simultaneously consider target following and obstacle avoidance within a single control command. By setting appropriate weights for each reward item, the optimization objectives can be kept in balance in terms of magnitude, guiding the agent to learn tracking accuracy, obstacle avoidance safety, and motion stability simultaneously during training, thus avoiding getting trapped in local optima.

[0061] In this embodiment, the policy network includes a one-dimensional convolutional encoder and an MLP fusion module. The one-dimensional convolutional encoder is used to encode the environmental information perceived by the UAV's recorded sensors into a feature vector. The MLP fusion module is used to combine the feature vector with other vectorized inputs (for example, specifically target state estimation information output by a target estimator). and drone speed information Drone speed information Specifically, the fusion of the drone's linear velocity and yaw rate in the x and y directions is performed. The one-dimensional convolutional encoder consists of three layers. The first layer uses multiple one-dimensional convolutional kernels through the first hidden layer to extract features from multiple consecutive frames of input data, and then performs non-linear processing using a linear rectified activation function. The second layer uses multiple one-dimensional convolutional kernels through the second hidden layer to perform convolution operations and processes them using the ReLU activation function. The third layer is used to encode the features into a fully connected layer. The MLP fusion module adopts a single hidden layer structure. The output of the policy network maps the features into action outputs through a fully connected layer.

[0062] Specifically, such as Figure 3 As shown, the one-dimensional convolutional encoder encodes the two-dimensional LiDAR input into a 512-dimensional feature vector, while the MLP fusion module fuses this feature vector with other vectorized inputs. The one-dimensional convolutional encoder consists of three layers: the first hidden layer uses 32 one-dimensional convolutional kernels of size 5 and stride 2 to extract features from three consecutive frames of input data, followed by non-linear processing using the rectified ReLU activation function; the second hidden layer further uses 32 one-dimensional convolutional kernels of size 3 and stride 2 for convolution operations, also using the ReLU activation function; the third layer is a fully connected layer that encodes the features into a 512-dimensional vector. The MLP fusion module uses a single hidden layer structure with a hidden layer feature dimension of 256. Finally, a fully connected layer maps the features to a 3-dimensional action output. The overall network structure is as follows. Figure 3 As shown in Table 1, the input N of the lidar is 512.

[0063] Table 1: Network Framework

[0064] Furthermore, the policy network was trained using the Proximal Policy Optimization (PPO) algorithm. Some key parameters during training are shown in Table 2.

[0065] Table 2 Training Algorithm Parameters

[0066] Step S3: Drone control: Control the drone to fly according to the speed control command, and drive the drone to achieve autonomous obstacle avoidance while tracking the target.

[0067] This embodiment further provides a control system, including a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the method as described above.

[0068] In specific application embodiments, such as Figure 4As shown, firstly, a target state estimator based on Extended Kalman Filter (EKF) is designed to estimate the target's position coordinates and motion direction in the image plane online. Subsequently, a motion controller based on deep reinforcement learning is constructed. This controller takes the target state estimation information and the environmental information perceived by the LiDAR as inputs and directly maps them into control commands for the UAV through an end-to-end policy network, thereby driving the UAV to achieve autonomous obstacle avoidance while tracking the target.

[0069] This embodiment further provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0070] Those skilled in the art will understand that the above embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce implementations of the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0071] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention should fall within the protection scope of the present invention.

Claims

1. A method for fusion control of target tracking and obstacle avoidance in unmanned aerial vehicles based on deep reinforcement learning, characterized by the following steps: include: Step S1, Target State Estimation: Acquire image data collected by the UAV's onboard sensors, estimate the target state based on the acquired image data to obtain target state estimation information. The target state includes the target position and velocity. During the flight of the UAV, the target coordinates observed under different camera poses are uniformly transformed to the same reference viewpoint to compensate for the motion of the UAV itself. Step S2, Fusion Control Command Generation: Construct a policy network based on deep reinforcement learning, using environmental information perceived by the UAV's onboard sensors and the target state estimation information as input, and map it into the UAV's speed control command through the policy network. During the training process, the reward function of the policy network adopts a composite reward function that fuses multiple objectives such as tracking accuracy, obstacle avoidance safety, and motion stability, so as to guide the policy network to achieve target following and obstacle avoidance in the same control command. Step S3, UAV control: Control the UAV to fly according to the speed control command, and drive the UAV to achieve autonomous obstacle avoidance while tracking the target.

2. The UAV target tracking and obstacle avoidance fusion control method based on deep reinforcement learning according to claim 1, characterized in that, In step S1, the target state is defined as a four-dimensional vector containing the target position and velocity. ,in, Let the center coordinates of the target be in the pixel coordinate system. For the goal and The rate of pixel change in the direction; The motion characteristics of the target in the image plane are modeled using a constant velocity model. The discrete-time state transition equation of the system is expressed as: ， ； in, This represents process noise and is modeled with a mean of 0 and a covariance matrix of... Gaussian process , Represents the state transition matrix. Indicates the sampling time interval.

3. The UAV target tracking and obstacle avoidance fusion control method based on deep reinforcement learning according to claim 2, characterized in that, In step S1, the target state is estimated based on the extended Kalman filter to obtain target state estimation information. The steps include: Step S101, Initialization: Initialize the state vector based on the detection results of the first frame. and the corresponding error covariance matrix ; Step S102, Prediction: Predict the state at the next moment based on the motion model: ， ； in, , These represent the predicted state of the target at the current moment and the state of the target at the previous moment, respectively. , These represent the prediction error covariance matrix of the target at the current time and the error covariance matrix of the target at the previous time, respectively. Step S103, Update: Change the center coordinates of the target image directly detected in the current frame. As an observation, the observation equation is modeled as follows: ， ； in, The observed noise is modeled with a mean of 0 and a covariance matrix of... Gaussian process , The state vector after compensation; The Kalman gain is then calculated as follows: ， in, , These represent the Kalman gain and the observation matrix, respectively. Status updated to: ， Covariance updated to: ， The final output of the target estimator is ,in, The detection bounding box height information is directly derived from the object detector, and the center coordinates of the target in the pixel coordinate system are... As a visual servo signal to guide the drone in tracking the target. For the goal and The rate of pixel change in the direction.

4. The UAV target tracking and obstacle avoidance fusion control method based on deep reinforcement learning according to claim 1, characterized in that, In step S1, the homography matrix between two consecutive frames is used to uniformly transform the target coordinates observed under different camera poses to the same reference viewpoint: in, The camera intrinsic parameter matrix, for and The homography matrix between time points, Indicates from Time camera coordinate system to The rotation matrix of the camera coordinate system at any given time. Indicates from Time camera coordinate system to The translation vector of the camera coordinate system at any given time. express The unit normal vector in the camera coordinate system at any given time. express The signed distance from the origin of the camera coordinate system to the plane at any given time. and for The target coordinates observed at all times, and For the goal Projected coordinates on the time-lapse image.

5. The UAV target tracking and obstacle avoidance fusion control method based on deep reinforcement learning according to claim 1, characterized in that, In step S2, constructing a policy network based on deep reinforcement learning includes: The target tracking task of UAVs in obstacle environments is modeled as a partially observable Markov decision. The observation space is defined to include environmental information perceived by the UAV's onboard sensors, the target state estimate, and the UAV's own state information. The action space is defined to include the UAV's own state information. x direction and y The directional speed control command and the yaw rate control command define a reward function as the composite reward function, which includes a field-of-view retention reward term representing the reward obtained for keeping the target within the field of view. Location guidance reward items used to guide drones to track targets based on the location information of drones and targets. and collision penalty items used to penalize collisions. A heading alignment penalty is used to penalize the decrease in the angle between the UAV's heading and the target's orientation. Motion smoothing penalty term used to incentivize drones to achieve stable motion. And task completion rewards , t Indicates the time.

6. The UAV target tracking and obstacle avoidance fusion control method based on deep reinforcement learning according to claim 5, characterized in that, The vision retention reward item Calculated based on the intersection-union ratio (IUU) between the detection box and the reference box: ， in, and These represent the current target detection box and the reference box, respectively. The intersection-union ratio of the two frames; Location-guided reward items The following calculations were performed based on the difference between the actual distance between the UAV and the target and the expected distance, as well as the angle between the line connecting the UAV and the target and the UAV's heading: ， in, Used to calculate the actual and expected distances between the drone and the target. The difference between them This represents the position vector of the target in the UAV's body coordinate system at the current moment. , They are respectively t time, t The angle between the line connecting the UAV and the target at time -1 and the UAV's heading. These represent the preset weighting coefficients.

7. The UAV target tracking and obstacle avoidance fusion control method based on deep reinforcement learning according to claim 5, characterized in that, The collision penalty item Enabled only when a collision is detected; The heading alignment penalty item The following calculations were performed based on the angle between the UAV's heading and the target's orientation: ,in, Indicates the angle between the drone's heading and the target's orientation; The motion smoothing penalty item According to drones shaft and The linear velocity and yaw rate of the shaft were calculated as follows: ,in, for t Time Drone shaft and The linear velocity and yaw rate of the shaft; The task completion reward items This takes effect only when the state of the UAV relative to the target meets preset state conditions, which include... Angle greater than the preset intersection-union ratio threshold, and the angle between the UAV's heading and the target's orientation. Less than the preset included angle threshold.

8. The UAV target tracking and obstacle avoidance fusion control method based on deep reinforcement learning according to any one of claims 1 to 7, characterized in that, The policy network includes a one-dimensional convolutional encoder and an MLP fusion module. The one-dimensional convolutional encoder encodes environmental information perceived by the UAV's sensors into feature vectors. The MLP fusion module fuses the feature vectors with vectorized inputs, which include target state estimation information and UAV velocity information. The one-dimensional convolutional encoder has a three-layer structure: the first layer uses multiple one-dimensional convolutional kernels in a first hidden layer to extract features from multiple consecutive frames of input data, followed by nonlinear processing using a linear rectified activation function; the second layer uses multiple one-dimensional convolutional kernels in a second hidden layer to perform convolution operations, followed by processing using a ReLU activation function; and the third layer encodes the features into a fully connected layer. The MLP fusion module uses a single hidden layer structure. The output of the policy network maps the features into action outputs through a fully connected layer.

9. A control system comprising a processor and a memory, the memory being used to store a computer program, characterized in that, The processor is used to execute the computer program to perform the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

Images