An unmanned aerial vehicle target tracking method based on reinforcement learning for realizing self-obstacle avoidance
By employing a policy distillation method based on task attention mechanisms, and combining attention mechanisms with D3QN to train UAV networks, the challenges of target tracking and obstacle avoidance for UAVs in complex environments are solved, achieving efficient task execution and collision reduction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTHEAST UNIV
- Filing Date
- 2023-10-31
- Publication Date
- 2026-05-01
AI Technical Summary
Existing UAV target tracking systems lack effective obstacle avoidance capabilities when performing tasks, especially in complex environments where it is difficult to achieve efficient target tracking and obstacle avoidance simultaneously. Furthermore, existing multi-task learning methods are prone to training instability and negative transfer problems.
A policy distillation method based on task attention mechanism is adopted. The UAV target tracking and obstacle avoidance tasks are trained through three networks. End-to-end training is carried out using Duel Dual Deep Q Network (D3QN). The task weights are dynamically adjusted by combining the attention mechanism, and an adaptive loss function is designed to optimize the student network policy.
It achieves high-precision target tracking by UAVs in complex and dynamic environments while reducing collision events, and has strong generalization ability, which is superior to existing methods.
Smart Images
Figure CN117369502B_ABST
Abstract
Description
A Reinforcement Learning-Based Method for Unmanned Aerial Vehicle (UAV) Target Tracking to Achieve Obstacle Avoidance Technical Field
[0001] This invention relates to a target tracking method for unmanned aerial vehicles (UAVs) that enables self-obstacle avoidance based on reinforcement learning, belonging to the field of UAV target tracking and autonomous obstacle avoidance technology. Background Technology
[0002] Unmanned aerial vehicles (UAVs) are characterized by their light weight, small size, high maneuverability, and low cost. In recent years, the UAV field has made significant progress in military applications, aerial photography, surveillance, and formation flying. Among these advancements, UAV target tracking and obstacle avoidance are crucial issues requiring attention in engineering applications and are increasingly attracting research from experts and scholars. However, a common limitation of many existing UAV-based tracking systems is the lack of obstacle avoidance considerations when performing tracking tasks. In the real world, the operating environment for UAVs may include various obstacles such as buildings, trees, power lines, and other aircraft. Therefore, when performing target tracking tasks, UAVs must possess the ability to avoid aerial obstacles to ensure mission safety and efficiency.
[0003] Current research largely focuses on single navigation and single target tracking tasks for UAVs, limiting their applicability in certain scenarios. With the development of deep reinforcement learning, its application in solving multi-task problems for UAVs has been extensively studied. Some experts use joint training methods to achieve multi-task implementation, but integrating the objectives of multiple tasks into a common reward function can be very complex, and gradient signals between different tasks may cancel each other out, making training unstable and difficult to converge. Therefore, some scholars have proposed transfer learning, shared representation learning, and progressive neural networks. These methods can promote knowledge sharing between different tasks. However, these methods perform better between strongly related tasks. In UAV tracking and obstacle avoidance tasks, negative transfer problems may arise when there are significant differences between tasks. To address these challenges, we propose a policy distillation framework to extract policies learned from two single-task networks of the UAV. These two single-task networks are called the teacher network, and the network that extracts the training policy is called the student network. Here, the policy is the probability distribution of the operation to be taken in the current state. We extract the separately trained obstacle avoidance and tracking policies into a memory buffer, ensuring that the memory buffer contains information from both tasks. We extract multi-task strategies by designing the loss in the student network. However, designing the student network's strategy solely through a weighted sum of the two teacher networks fails to ensure the student network rationally acquires appropriate actions based on the drone's current state. We aim to make the drone focus more on tracking when there are no obstacles and more on obstacle avoidance when there are obstacles. Introducing an attention mechanism, a key technique, helps the model flexibly select and adjust between different tasks. By adaptively focusing on specific tasks, the model can better handle interference and target conflicts between different tasks. We perceive collision risk based on the drone's current environmental state and obtain parameters from the attention mechanism to adaptively adjust the weights of the two tasks. Summary of the Invention
[0004] This invention addresses the technical problems existing in the prior art by providing a UAV target tracking method based on reinforcement learning to achieve self-obstacle avoidance. This application is a novel target tracking and obstacle avoidance solution based on policy distillation of task attention mechanism, which is applied to the task of UAV target tracking and self-obstacle avoidance.
[0005] To achieve the above objectives, the technical solution of the present invention is as follows: a method for tracking unmanned aerial vehicle (UAV) targets to achieve obstacle avoidance based on reinforcement learning, comprising the following steps:
[0006] S1: This method comprises three networks: a drone target tracking network and a drone obstacle avoidance network, referred to as the teacher network; and a multi-task network that simultaneously performs drone target tracking and obstacle avoidance, referred to as the student network. The three networks are trained in three scenarios. A scenario for drone target tracking is constructed in a virtual robotics experimental platform, including walls, tables, chairs, green plants, a sink, stairs, etc., as well as randomly moving pedestrians (who can autonomously avoid obstacles). The constructed scenario may contain occlusions, increasing the complexity and dynamism of the scenario.
[0007] S2: The robot in the scenario can autonomously avoid obstacles, randomly turning left or right when encountering one. The robot's speed is 0.8 m / s. To enable the drone to learn a target tracking strategy instead of the robot's inherent path, we set the robot's movement to random. Every 150 counts, we randomly assign a number, 'a'. If 'a' is less than or equal to 0.2, the robot's left wheel speed increases by 0.5 m / s, meaning the robot turns right; if 'a' is greater than or equal to 0.8, the robot's right wheel speed increases by 0.5 m / s, meaning the robot turns left; if 'a' is between 0.2 and 0.8, the robot continues to move. This setting ensures that the robot's path is random in each round, allowing the drone to better learn a reasonable strategy based on the images captured by the camera.
[0008] S3: Image Acquisition. A camera is installed on the underside of the drone to acquire images in the downward field of view. The images are 64×64 pixel color (RGB) images.
[0009] S4: A three-layer convolutional neural network and a spatial softmax layer are then used to process the input image in the UAV target tracking network to extract useful features. Simultaneously, a Long Short-Term Memory (LSTM) network is used to encode historical UAV actions, enabling the network to learn historical information. Tracking is performed using visual information within the visible range when the target is not occluded, while tracking is performed using the encoded historical UAV actions when the target is occluded. Furthermore, an attention mechanism module is added to better focus on either the visual or motion channels.
[0010] S5: The entire tracking network is trained using a Duel Dual Deep Q-Network (D3QN), continuously interacting with the environment and outputting learned actions. The underlying UAV controller then drives the UAV in a simulation environment to obtain new states and receive rewards for taking actions to reach those states. The network is then trained for a total of 2000 rounds, resulting in Model 1.
[0011] S6: Build a scenario in the virtual robot experimental platform for drone obstacle avoidance, which mainly includes objects such as walls and stakes that may pose obstacles to the drone.
[0012] S7: Image acquisition. Four cameras are installed around the drone's body, in the front, back, left, and right directions. Each direction has a 90-degree field of view and is used to acquire images around the drone. The images are RGB images with a pixel size of 64×64.
[0013] S8: Similarly, a three-layer convolutional neural network and a spatial softmax layer are used to process the input image in the drone obstacle avoidance network to extract useful features.
[0014] S9: The entire tracking network is trained using D3QN, continuously interacting with the environment and outputting the learned actions. Then, the underlying UAV controller drives the UAV to move in the simulation environment, obtaining new states and receiving rewards for taking actions to reach those states. The network is then trained for a total of 2000 rounds, resulting in Model 2.
[0015] S10: The learned Model 1 and Model 2 from the two environments are stored in a buffer. Using policy distillation, the first two learned network models are used as the teacher network, and the multi-task network is used as the student network. A scenario is built in the virtual robot experimental platform for UAV target tracking while simultaneously achieving obstacle avoidance. This scenario mainly includes objects such as walls and stakes that may pose obstacles to the UAV.
[0016] S11: The robot in the scene can autonomously avoid obstacles. When it encounters an obstacle, it will randomly turn left or right. The robot runs at a speed of 0.8 meters per second. The robot's running path is random, so that the drone can better learn reasonable strategies based on the images collected by the camera.
[0017] S12: Image Acquisition. Four cameras are installed around the drone's fuselage, facing forward, backward, left, and right, each with a 90-degree field of view, to acquire images from all around the drone. A camera is installed on the underside of the drone's fuselage to acquire images from the downward field of view. The images are 64×64 pixel RGB images. A total of five images are used as input.
[0018] S13: Use a three-layer convolutional neural network and a spatial softmax layer to process the image input to the multi-task network and extract useful features.
[0019] S14: The entire tracking network is trained using D3QN, continuously interacting with the environment and outputting learned actions. The underlying UAV controller then drives the UAV in a simulated environment to obtain new states and receive rewards for taking actions to reach those states. This process continues training the network for 2000 rounds. The design of the loss function is a key focus of our research. After storing the models of the two teacher networks in a buffer, we extract policies using policy distillation and design a weight using an attention mechanism. This weight allows the UAV to learn which policy based on its current state in the environment. By designing the loss function, the student network adaptively learns two policies, distilling a multi-task policy. Testing the trained model shows that the UAV can simultaneously track targets and avoid obstacles in complex and dynamic environments.
[0020] The robot's speed in step S2 is shown in the following formula:
[0021]
[0022] Where v0 refers to the robot's speed, which is 0.8 m / s. Here, m / s refers to meters per second. L v represents the speed of the revolver. R This represents the speed of the right wheel. 'a' represents a number randomly selected from 0 to 1.
[0023] In step S5, the drone's motion space is discrete, consisting of five actions: forward, backward, left, right, and stop. While drone movements in the real world are generally continuous, we use a discrete motion space to enhance the system's portability to other platforms. After outputting the discrete actions, we employ a relatively low-level Proportional-Integral-Derivative (PID) control to drive the motors and control the drone's movement. Simultaneously, we use D3QN for end-to-end training of the entire network, with the reward design formula as follows: r = e -Δpos Where, Δpos=||pos u -pos t ||2 represents the L2 norm of the relative distance between the drone and the target robot, r represents the reward, e is the symbol for the natural base, and pos u and pos t These represent the positions of the drone and the robot, respectively.
[0024] In step S9, we also use D3QN to train the entire network end-to-end, and the reward design formula is as follows:
[0025]
[0026]
[0027] r = r1 + r2.
[0028] The first reward represents the obstacle avoidance reward, d min This represents the minimum distance between the drone and an obstacle. If the sensor detects a minimum distance of less than 0.3 meters, a penalty is applied; a distance between 0.3 and 0.6 meters also incurs a penalty. Otherwise, the drone is considered safe, with neither a reward nor a penalty. If the drone takes one of four actions (forward, backward, left, or right), a reward of 0.1 is given; otherwise, no reward is given. This setting encourages the drone to explore its environment rather than simply stopping to avoid collisions. The final reward is the sum of these two rewards.
[0029] In step S13, the network structure diagram of our proposed policy distillation method based on the attention mechanism is shown in Figure 1, which includes two teacher networks and one student network. We extract the weight parameter V corresponding to the attention mechanism from the obstacle avoidance network. B The loss function for the student network is designed as follows:
[0030]
[0031] Task A is a tracking task, and Task B is an obstacle avoidance task. B This refers to the normalized value of the value function in the obstacle avoidance network, which can reflect the degree of collision risk to a certain extent. If the collision risk in the environment where the drone is currently located is low, that is, V... B A high V value indicates a relatively safe environment, meaning the student network focuses more on the tracking task, or task A. Conversely, a high collision rate in the current environment (V) indicates a safer environment. B If the value is low, the environment is relatively dangerous, and collisions are likely to occur, then the student network will pay more attention to the obstacle avoidance task, which is the B task. and π S These represent the strategies for Task A, Task B, and the student network, respectively. Additionally:
[0032]
[0033] Where n represents the number of samples, i varies from 1 to n, and Y and These represent the true value and the predicted value, respectively.
[0034] In step S14, the reward design formula is as follows:
[0035] r1=e -Δpos ,
[0036]
[0037]
[0038] r = r1 + r2 + r3.
[0039] The first reward represents minimizing the distance between the drone and the robot, corresponding to the tracking task. The second and third rewards represent the drone's obstacle avoidance task. Finally, these three rewards are added together to obtain the final reward.
[0040] Compared with existing technologies, the advantages of this invention are as follows: This invention proposes a novel target tracking and obstacle avoidance solution based on a task attention mechanism and policy distillation. First, two tasks, UAV target tracking and UAV obstacle avoidance, are trained separately. Both networks are trained using a Duel Dual Deep Q Network (D3QN) to learn the corresponding policies in an end-to-end manner. Then, the two trained policies are retrieved from a memory buffer. At the same time, the collision risk can be perceived by the current state of the UAV environment, thereby allocating the weights of the two tasks in the attention mechanism. Therefore, the method of this invention can adaptively focus on the corresponding task according to the current state. Simulation experiments were conducted using a virtual robot experimental platform. The study presents convincing experimental results: (1) it outperforms state-of-the-art methods by achieving superior tracking accuracy and extended tracking duration in different environments while reducing collision events; (2) the developed distillation policy exhibits strong generalization ability when applied to previously unencountered environments. Attached Figure Description
[0041] Figure 1 shows the overall framework of the present invention, which includes two teacher networks and one student network. Detailed Implementation
[0042] To enhance understanding of the present invention, the embodiments will be described in detail below with reference to the accompanying drawings.
[0043] Example 1: Referring to Figure 1, a method for tracking a drone target to achieve obstacle avoidance based on reinforcement learning includes the following steps:
[0044] S1: This method comprises three networks: a drone target tracking network and a drone obstacle avoidance network, referred to as the teacher network; and a multi-task network that simultaneously performs drone target tracking and obstacle avoidance, referred to as the student network. The three networks are trained in three scenarios. A scenario for drone target tracking is constructed in a virtual robotics experimental platform, including walls, tables, chairs, green plants, a sink, stairs, etc., as well as randomly moving pedestrians (who can autonomously avoid obstacles). The constructed scenario may contain occlusions, increasing the complexity and dynamism of the scenario.
[0045] S2: The robot in the scenario can autonomously avoid obstacles, randomly turning left or right when encountering one. The robot's speed is 0.8 m / s. To enable the drone to learn a target tracking strategy instead of the robot's inherent path, we set the robot's movement to random. Every 150 counts, we randomly assign a number, 'a'. If 'a' is less than or equal to 0.2, the robot's left wheel speed increases by 0.5 m / s, meaning the robot turns right; if 'a' is greater than or equal to 0.8, the robot's right wheel speed increases by 0.5 m / s, meaning the robot turns left; if 'a' is between 0.2 and 0.8, the robot continues to move. This setting ensures that the robot's path is random in each round, allowing the drone to better learn a reasonable strategy based on the images captured by the camera.
[0046] S3: Image Acquisition. A camera is installed on the underside of the drone to acquire images in the downward field of view. The images are 64×64 pixel color (RGB) images.
[0047] S4: A three-layer convolutional neural network and a spatial softmax layer are then used to process the input image in the UAV target tracking network to extract useful features. Simultaneously, a Long Short-Term Memory (LSTM) network is used to encode historical UAV actions, enabling the network to learn historical information. Tracking is performed using visual information within the visible range when the target is not occluded, while tracking is performed using the encoded historical UAV actions when the target is occluded. Furthermore, an attention mechanism module is added to better focus on either the visual or motion channels.
[0048] S5: The entire tracking network is trained using a Duel Dual Deep Q-Network (D3QN), continuously interacting with the environment and outputting learned actions. The underlying UAV controller then drives the UAV in a simulation environment to obtain new states and receive rewards for taking actions to reach those states. The network is then trained for a total of 2000 rounds, resulting in Model 1.
[0049] S6: Build a scenario in the virtual robot experimental platform for drone obstacle avoidance, which mainly includes objects such as walls and stakes that may pose obstacles to the drone.
[0050] S7: Image acquisition. Four cameras are installed around the drone's body, in the front, back, left, and right directions. Each direction has a 90-degree field of view and is used to acquire images around the drone. The images are RGB images with a pixel size of 64×64.
[0051] S8: Similarly, a three-layer convolutional neural network and a spatial softmax layer are used to process the input image in the drone obstacle avoidance network to extract useful features.
[0052] S9: The entire tracking network is trained using D3QN, continuously interacting with the environment and outputting the learned actions. Then, the underlying UAV controller drives the UAV to move in the simulation environment, obtaining new states and receiving rewards for taking actions to reach those states. The network is then trained for a total of 2000 rounds, resulting in Model 2.
[0053] S10: The learned Model 1 and Model 2 from the two environments are stored in a buffer. Using policy distillation, the first two learned network models are used as the teacher network, and the multi-task network is used as the student network. A scenario is built in the virtual robot experimental platform for UAV target tracking while simultaneously achieving obstacle avoidance. This scenario mainly includes objects such as walls and stakes that may pose obstacles to the UAV.
[0054] S11: The robot in the scene can autonomously avoid obstacles. When it encounters an obstacle, it will randomly turn left or right. The robot runs at a speed of 0.8 meters per second. The robot's running path is random, so that the drone can better learn reasonable strategies based on the images collected by the camera.
[0055] S12: Image Acquisition. Four cameras are installed around the drone's fuselage, facing forward, backward, left, and right, each with a 90-degree field of view, to acquire images from all around the drone. A camera is installed on the underside of the drone's fuselage to acquire images from the downward field of view. The images are 64×64 pixel RGB images. A total of five images are used as input.
[0056] S13: Use a three-layer convolutional neural network and a spatial softmax layer to process the image input to the multi-task network and extract useful features.
[0057] S14: The entire tracking network is trained using D3QN, continuously interacting with the environment and outputting learned actions. The underlying UAV controller then drives the UAV in a simulated environment to obtain new states and receive rewards for taking actions to reach those states. This process continues training the network for 2000 rounds. The design of the loss function is a key focus of our research. After storing the models of the two teacher networks in a buffer, we extract policies using policy distillation and design a weight using an attention mechanism. This weight allows the UAV to learn which policy based on its current state in the environment. By designing the loss function, the student network adaptively learns two policies, distilling a multi-task policy. Testing the trained model shows that the UAV can simultaneously track targets and avoid obstacles in complex and dynamic environments.
[0058] The robot's speed in step S2 is shown in the following formula:
[0059]
[0060] Where v0 refers to the robot's speed, which is 0.8 m / s. Here, m / s refers to meters per second. L v represents the speed of the revolver. R This represents the speed of the right wheel. 'a' represents a number randomly selected from 0 to 1.
[0061] In step S5, the drone's motion space is discrete, consisting of five actions: forward, backward, left, right, and stop. While drone movements in the real world are generally continuous, we use a discrete motion space to enhance the system's portability to other platforms. After outputting the discrete actions, we employ a relatively low-level Proportional-Integral-Derivative (PID) control to drive the motors and control the drone's movement. Simultaneously, we use D3QN for end-to-end training of the entire network, with the reward design formula as follows: r = e -Δpos Where, Δpos=||pos u -pos t ||2 represents the L2 norm of the relative distance between the drone and the target robot, r represents the reward, e is the symbol for the natural base, and pos u and pos t These represent the positions of the drone and the robot, respectively.
[0062] In step S9, we also use D3QN to train the entire network end-to-end, and the reward design formula is as follows:
[0063]
[0064]
[0065] r = r1 + r2.
[0066] The first reward represents the obstacle avoidance reward, d min This represents the minimum distance between the drone and an obstacle. If the sensor detects a minimum distance of less than 0.3 meters, a penalty is applied; a distance between 0.3 and 0.6 meters also incurs a penalty. Otherwise, the drone is considered safe, with neither a reward nor a penalty. If the drone takes one of four actions (forward, backward, left, or right), a reward of 0.1 is given; otherwise, no reward is given. This setting encourages the drone to explore its environment rather than simply stopping to avoid collisions. The final reward is the sum of these two rewards.
[0067] In step S13, the network structure diagram of our proposed policy distillation method based on the attention mechanism is shown in Figure 1, which includes two teacher networks and one student network. We extract the weight parameter V corresponding to the attention mechanism from the obstacle avoidance network. B The loss function for the student network is designed as follows:
[0068]
[0069] Task A is a tracking task, and Task B is an obstacle avoidance task. B This refers to the normalized value of the value function in the obstacle avoidance network, which can reflect the degree of collision risk to a certain extent. If the collision risk in the environment where the drone is currently located is low, that is, V... B A high V value indicates a relatively safe environment, meaning the student network focuses more on the tracking task, or task A. Conversely, a high collision rate in the current environment (V) indicates a safer environment. B If the value is low, the environment is relatively dangerous, and collisions are likely to occur, then the student network will pay more attention to the obstacle avoidance task, which is the B task. and π S These represent the strategies for Task A, Task B, and the student network, respectively. Additionally:
[0070]
[0071] Where n represents the number of samples, i varies from 1 to n, and Y and These represent the true value and the predicted value, respectively.
[0072] In step S14, the reward design formula is as follows:
[0073] r1=e -Δpos ,
[0074]
[0075]
[0076] r = r1 + r2 + r3.
[0077] The first reward represents minimizing the distance between the drone and the robot, corresponding to the tracking task. The second and third rewards represent the drone's obstacle avoidance task. Finally, these three rewards are added together to obtain the final reward.
[0078] It should be noted that the above embodiments are not intended to limit the scope of protection of the present invention. Equivalent transformations or substitutions made based on the above technical solutions all fall within the scope of protection of the claims of the present invention.
Claims
1. A method for tracking unmanned aerial vehicle (UAV) targets to achieve obstacle avoidance based on reinforcement learning, characterized in that, The method includes the following steps: S1: The method comprises three networks: a drone target tracking network and a drone obstacle avoidance network, referred to as the teacher network; and a multi-task network that simultaneously achieves drone target tracking and obstacle avoidance, referred to as the student network. The three networks are trained in three scenarios. A scenario for drone target tracking is built in a virtual robot experimental platform. S2: The robot in the scenario can autonomously avoid obstacles, randomly turning left or right upon encountering an obstacle. The robot's operating speed is 0.8 meters per second. S3: Image acquisition. A camera is installed under the drone's fuselage to acquire images from the downward field of view. The images are 64×64 pixel color (RGB) images. S4: A three-layer convolutional neural network and a spatial soft maximization algorithm are then used. The softmax layer processes the input image in the UAV target tracking network, extracting useful features. Simultaneously, a Long Short-Term Memory (LSTM) network is used to encode historical UAV actions, enabling the network to learn from historical information. S5: The entire tracking network is trained using a Duel Dual Deep Q-Network (D3QN), continuously interacting with the environment and outputting learned actions. The underlying UAV controller then drives the UAV in a simulation environment, obtaining new states and receiving rewards for taking actions to reach those states. The network is trained for 2000 rounds to obtain Model 1. S6: A scenario for UAV obstacle avoidance is built in a virtual robot experimental platform. S7: Images are acquired by installing four cameras around the UAV's fuselage in four directions (front, back, left, and right), each with a 90-degree field of view, to capture images of the UAV's surroundings. These images are 64×64 pixel RGB images. S8: A three-layer convolutional neural network and a spatial soft max layer are then used to... The softmax layer processes the input image in the UAV obstacle avoidance network and extracts useful features. S9: The entire tracking network is trained using D3QN, continuously interacting with the environment and outputting learned actions. The underlying UAV controller then drives the UAV to move in the simulation environment, obtains a new state, and receives the reward corresponding to the action taken to reach the state. The network continues to be trained for a total of 2000 rounds to obtain Model 2. S10: Model 1 and Model 2 learned in the two environments are stored in the memory bank (buffer). Through policy distillation, the first two learned network models are used as the teacher network, and the multi-task network is used as the student network. A scenario for UAV target tracking and obstacle avoidance is built in the virtual robot experimental platform, mainly including walls, stakes, and other objects that may pose obstacles to the UAV. S11: The robot in the scenario can achieve autonomous obstacle avoidance. When it encounters an obstacle, it randomly turns left or right. The robot's running speed is 0.At 8 meters per second, the robot's movement path is random, allowing the drone to better learn reasonable strategies based on the images captured by the cameras. S12: Image acquisition. Four cameras are installed around the drone's fuselage, in the front, back, left, and right directions, each with a 90-degree field of view, to acquire images from all around the drone. A camera is installed below the drone's fuselage to acquire images from the downward field of view. The images are 64×64 pixel RGB images, for a total of five images as input. S13: A three-layer convolutional neural network and a spatial softmax layer are used to process the input images in the multi-task network to extract useful features. S14: The entire tracking network is trained using D3QN, continuously interacting with the environment and outputting learned actions. The underlying drone controller then drives the drone's movement in a simulation environment, obtaining new states and receiving rewards for taking actions to reach those states. The network continues training for 2000 rounds. The trained model, after testing, demonstrates that the drone can track targets and avoid obstacles simultaneously in complex and dynamic environments.
2. The UAV target tracking method based on reinforcement learning for obstacle avoidance as described in claim 1, characterized in that: In step S1, the environment includes walls, tables, chairs, green plants, sinks, stairs, and randomly moving pedestrians. In step S6, the environment mainly includes walls, stakes, and objects that create obstacles for the drone. In step S10, the environment mainly includes walls, stakes, objects that create obstacles for the drone, and the target being tracked, namely the robot.
3. The UAV target tracking method for achieving obstacle avoidance based on reinforcement learning according to claim 1, characterized in that: The robot's speed in step S2 is shown in the following formula: Where v0 refers to the robot's speed, which is 0.8 m / s, and m / s refers to meters per second. L v represents the speed of the revolver. R 'a' represents the speed of the right wheel, and 'a' represents a number randomly selected from 0 to 1.
4. The UAV target tracking method for achieving obstacle avoidance based on reinforcement learning according to claim 1, characterized in that: In step S5, the drone's motion space is discrete, consisting of five actions: forward, backward, left, right, and stop. This enhances the system's portability to other platforms. After outputting the discrete actions, a relatively low-level proportional-integral-derivative (PID) control is used to drive the motors and control the drone's movement. Simultaneously, D3QN is used for end-to-end training of the entire network. The reward design formula is as follows: r = e -Δpos Where, Δpos=||pos u -pos t ||2 represents the L2 norm of the relative distance between the drone and the target robot, r represents the reward, e is the symbol for the natural base, and pos u and pos t These represent the positions of the drone and the robot, respectively.
5. The UAV target tracking method based on reinforcement learning for obstacle avoidance according to claim 1, characterized in that: In step S9, D3QN is used to train the entire network end-to-end, and the reward design formula is as follows: r = r1 + r2. The first reward represents the obstacle avoidance reward, d min This represents the minimum distance between the drone and an obstacle. If the sensor detects that the minimum distance to the obstacle is less than 0.3 meters, a penalty is imposed. There is also a penalty if the distance is between 0.3 and 0.6 meters. Otherwise, it is considered safe, and there is neither a reward nor a penalty. If the drone takes one of the four actions (forward, backward, left, or right), a reward of 0.1 is given. Otherwise, there is no reward. This setting better encourages the drone to explore the environment, rather than choosing to stay in place without taking any other actions to avoid collisions. The final reward is the sum of these two rewards.
6. The UAV target tracking method for achieving obstacle avoidance based on reinforcement learning according to claim 1, characterized in that: In step S13, the network structure of the policy distillation method based on the attention mechanism includes two teacher networks and one student network. The weight parameter V corresponding to the attention mechanism is extracted from the obstacle avoidance network. B The loss function for the student network is designed as follows: Task A is a tracking task, and Task B is an obstacle avoidance task. B This refers to the normalized value of the value function in the obstacle avoidance network, which reflects the degree of collision risk. If the collision risk in the environment where the drone is located is low in the current state, that is, V... B A high value indicates a relatively safe environment, meaning the student network focuses more on the tracking task, i.e., task A. Conversely, a high collision rate in the current environment (i.e., task V) indicates a safer environment. B If the value is low, the environment is relatively dangerous, and collisions are likely to occur, then the student network will focus more on the obstacle avoidance task, which is task B. and π S These represent the strategies for Task A, Task B, and the student network, respectively. Additionally: Where n represents the number of samples, i varies from 1 to n, and Y and These represent the true value and the predicted value, respectively.
7. The UAV target tracking method for achieving obstacle avoidance based on reinforcement learning according to claim 1, characterized in that: In step S14, the reward design formula is as follows: r1 = e -Δpos , r = r1 + r2 + r3. Where, The first reward represents minimizing the distance between the drone and the robot, which is the reward for the tracking task. The second and third rewards represent the drone's obstacle avoidance task. Finally, these three rewards are added together to obtain the final reward.