Multi-quadrotor unmanned aerial vehicle rendezvous task path planning method based on reinforcement learning

Abstract

The application discloses a multi-four-rotor unmanned aerial vehicle gathering type task path planning method based on reinforcement learning, first constructs a multi-four-rotor unmanned aerial vehicle Gym environment based on PyBullet, abstracts a state space and an action space of the four-rotor unmanned aerial vehicle, sets a reward function mechanism, then uses an improved deep reinforcement learning algorithm to make path planning decisions, finally trains the improved deep reinforcement learning network, controls the four-rotor unmanned aerial vehicle through output action information, and makes each four-rotor unmanned aerial vehicle successfully reach a specified target task in the shortest time. The method uses N-step returns in the TD3 algorithm, obtains more accurate return estimates, faster learning speed and better generalization performance, uses priority experience replay to reduce sampling bias, reduces model bias caused by uneven sampling, improves algorithm stability, makes the TD3 algorithm more suitable for continuous multi-dimensional decision problems, and can plan an optimal route in the shortest time under the condition that real-time performance and accuracy are reached.

Description

Technical Field

[0001] This invention belongs to the field of path planning technology for multi-quadrotor unmanned aerial vehicles (UAVs), specifically relating to a path planning method for swarm-type tasks of multi-quadrotor UAVs based on reinforcement learning. Background Technology

[0002] Quadrotor drones use the lift generated by multiple rotors to balance the aircraft's gravity, enabling hovering, vertical takeoff and landing, and low requirements for takeoff sites, although their flight speed is relatively slow. Therefore, multirotor drones are suitable for complex environments and limited-range applications, such as aerial photography, surveillance, and architectural modeling. With the continuous development of drone technology, they have been widely used in the civilian sector, and the complexity of the tasks they perform is constantly increasing. Since the payload and flight capabilities of a single drone are limited, cooperation among multiple drones is necessary to improve mission execution capabilities and scope.

[0003] Since almost all UAV missions involve shortest path planning, this problem has become a key focus and a challenging area of ​​research in UAV path planning in recent years. Shortest path planning can be further categorized into swarm-based and allocation-based tasks based on their specific characteristics. Swarm-based tasks aim to plan the optimal path for each UAV to reach the same destination from its starting point. The goal of these tasks is typically to ensure all UAVs arrive at the destination simultaneously and complete the task as quickly as possible. In this case, the objective is generally to minimize the total mission time or the total path length. Compared to allocation-based tasks, swarm-based tasks have greater versatility.

[0004] Compared to existing algorithms based on rules or heuristic search, reinforcement learning-based path planning methods offer better adaptability and scalability. Existing methods require manual design and adjustment of rules according to the environment, while reinforcement learning methods allow agents to autonomously learn and adapt to the environment. Because the agents in reinforcement learning possess autonomous decision-making capabilities, they can learn optimal behavior through interaction with the environment. Furthermore, deep learning algorithms have powerful perceptual capabilities, and deep reinforcement learning algorithms, which combine deep learning and reinforcement learning, can handle higher-dimensional inputs, making them more suitable for the multi-UAV topic discussed in this paper. Therefore, compared to existing methods, deep reinforcement learning algorithms can better handle unknown situations and changes, enabling agents to perform continuous decision-making tasks in complex environments.

[0005] Current technologies for solving swarm-based tasks involving multiple UAVs still face many challenges, including environment modeling, low learning efficiency, and complex action and state spaces. First, for agent projects based on deep reinforcement learning algorithms, the construction of a simulated environment is fundamental to the entire experiment, and the design of the UAV system must rely on simulation tools. Therefore, establishing appropriate UAV simulators is crucial for academic research and the development of safety-critical applications. However, many current simulation environments based on deep reinforcement learning algorithms lack real-world portability, with many sacrificing realism for high sample throughput. Furthermore, the training efficiency of multi-UAV path planning using deep reinforcement learning algorithms is generally low. In most simulated environments, path planning rewards are sparse, with agents only receiving reward signals after the task is completed. Moreover, the difficulty of effective exploration in complex environments makes it difficult to begin training in the early stages. Finally, such multi-UAV path planning problems typically involve multiple agents and multiple obstacles, resulting in high-dimensional and complex state spaces, action spaces, and reward functions, increasing the difficulty of modeling and solving the problem. Since the action space of multi-UAV path planning is often very large, effective search strategies are needed to address the challenges of high-dimensional action spaces. In conclusion, clustered path planning is of great significance for the execution of missions involving multiple UAVs. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention proposes a multi-quadrotor UAV swarming task path planning method based on reinforcement learning, focusing on solving swarming tasks in multi-UAV path planning.

[0007] The technical solution of this invention is: a path planning method for swarm-type missions of multi-quadrotor UAVs based on reinforcement learning, the specific steps of which are as follows:

[0008] S1. Construct a PyBullet-based multi-quadrotor UAV Gym environment;

[0009] S2. Abstract the state space and action space of the quadcopter drone, and set up a reward function mechanism to enable the drone to interact with the environment;

[0010] S3. Use an improved deep reinforcement learning algorithm for path planning decisions, and perform path planning for each quadcopter drone in a clustered task.

[0011] S4. The improved deep reinforcement learning network controls the angular and linear velocities of the quadcopter drones by outputting motion information, enabling each quadcopter drone to successfully reach the specified target task in the shortest possible time.

[0012] Furthermore, step S1 specifically includes the following:

[0013] S11. Construct a dynamic simulation model of a multi-quadrotor UAV;

[0014] The dynamic equations of a multi-quadrotor UAV are derived from the equations of motion and aerodynamic effects of the quadrotor UAV, thus completing the construction of a dynamic simulation model of the multi-quadrotor UAV, as detailed below:

[0015] Use PyBullet to build force and torque models for each quadcopter in Gym, and use the physics engine to calculate and update the dynamic equations of all quadcopters.

[0016] Set the arm length of each quadcopter drone to be Quality is Inertial properties are The physical constants and convex collision shape are described by separate URDF files for the configuration of the "x" type quadcopter UAV.

[0017] First, set the gravitational acceleration in PyBullet. And the physical stepping frequency, the force applied to the four motors and torque around the drone's Z-axis With motor speed It is proportional to the square of . and The expression is as follows:

[0018] (1)

[0019] (2)

[0020] in, and This represents a pre-defined constant.

[0021] If the model is to be controlled in real time, the dynamic equations of the quadcopter UAV are expressed as follows:

[0022] (3)

[0023] in, Represents the Jacobian matrix. Represents the inertia matrix. Represents generalized acceleration. Indicates Coriolis and the effect of gravity, superscript This indicates the transpose operation.

[0024] In practice, flying close to the ground or near other drones generates additional aerodynamic effects. PyBullet models these effects separately and uses them in combination, including: propeller drag. Ground effect acting on a single motor Downwash effect acting on the center of mass .

[0025] The rotating propellers of a quadcopter drone generate drag. ,resistance linear velocity of quadcopter drones The angular velocity of the propeller and the constant drag coefficient matrix. They are directly proportional, as expressed below:

[0026] (4)

[0027] in, This represents the angular velocity of the propeller, 60 represents 60s; constant drag coefficient torque. The specific expression is as follows:

[0028] (5)

[0029] in, Indicates the vertical drag coefficient. Represents the parallel drag coefficient, and the matrix The least squares method is used to fit the data.

[0030] When hovering at a very low height, the ground effect exists, and the impact of the ground effect on each motor is considered. With propeller radius ,speed ,high and constant The proportional relationship is as follows:

[0031] (6)

[0032] When two quadcopter drones pass through the same path at different altitudes, a downwash effect occurs. The effect of the downwash effect can be simplified to a single force acting on the drone's center of mass; its magnitude... Depending on the coordinate system of the two drones , , Distance in and constants determined experimentally , , , The expression is as follows:

[0033] (7)

[0034] S12. Construct observation and maneuvering space for multiple quadrotor UAVs;

[0035] In the constructed Gym environment, each action performed by a quadcopter drone outputs an observation vector. The observation space expression for multiple quadcopter drones is as follows:

[0036] (8)

[0037] in, Indicates the number of quadcopter drones; Indicates the location of the quadcopter drone; Represents a quaternion, used for attitude control of quadrotor drones; , , These represent the roll angle, pitch angle, and yaw angle, respectively, which are three angles used for attitude estimation. for Indicates the first The linear velocity of a quadcopter drone for Indicates the first The angular velocity of a quadcopter drone; for This indicates the motor speed of all drones.

[0038] In this invention, the quadcopter drone uses a lidar to detect obstacles. The model is configured to have a lidar system. one, and use this A lidar unit is used to observe the environment.

[0039] Among them, this The scanning angle range of each lidar is: The angle between the two lasers is ; Represents the horizontal plane The ray length of each radar; The ray length of the i-th radar is expressed as follows:

[0040] (9)

[0041] Environmental information The definition is as follows:

[0042] (10)

[0043] in, The one-hot code of the i-th radar is expressed as follows:

[0044] (11)

[0045] For any quadcopter UAV, the action space expression is as follows:

[0046] (12)

[0047] in, This indicates the speed input to the quadcopter drone. , , Represents the components of a unit vector. This indicates the required speed; and the motion space can also be represented by the rotational speeds of the four motors, as shown in the following expression:

[0048] (13)

[0049] Finally, the input is converted into pulse width modulation (PWM) and the motor speed is delegated to the controller, which consists of position and attitude control subroutines.

[0050] Furthermore, step S2 is specifically as follows:

[0051] S21. State space and motion space of an abstract quadcopter drone;

[0052] The status of a quadcopter drone includes: the quadcopter drone's position and quaternions. Side roll angle Pitch Yaw angle linear velocity angular velocity Motor speeds of all drones The angle between the drone's first-person view and the line connecting the target and the Global coordinates of the drone Global coordinates of the target Differences between .

[0053] Replace the drone's global position with its relative position to the target in the state. That is Then the state of the drone as follows:

[0054] (14)

[0055] The status of the drone and the environmental conditions detected by lidar The state space of the quadcopter drone can be obtained. The expression is as follows:

[0056] (15)

[0057] The action space of a multi-quadcopter UAV environment consists of the velocities input to the quadcopter UAV. Referring to equation (12), the action space expression for any quadcopter UAV is as follows:

[0058] (16)

[0059] S22. Set up a reward function mechanism to enable the quadcopter drone to interact with the environment;

[0060] reward function Indicates the state Take action below The obtained environmental feedback; a reward function consisting of three parts is set to enable the quadcopter drone to reach the assembly-type mission objective point as quickly as possible, as follows:

[0061] First, set a distance reward between the quadcopter drone and the target point. To enable the quadcopter drone to reach its target, The expression is as follows:

[0062] (17)

[0063] in, This indicates the distance between the quadcopter drone and the target. Indicates the first The distance between the quadcopter drone and the target, Indicates the first The distance between the quadcopter drone and the target at the next moment.

[0064] Secondly, a distance reward is set for the quadcopter drone and the obstacle. To encourage drones to stay away from obstacle placement, The expression is as follows:

[0065] (18)

[0066] in, This represents the ray length of the i-th radar, i.e., the detection distance of the quadcopter drone to obstacles or other quadcopter drones. Indicates the first The distance between the quadcopter drone and the target, and the setting of a safe distance between the drone and the obstacle. .

[0067] Finally, an angular reward is set between the quadcopter drone and the target point. If the drone is prompted to move closer to the target, The larger the penalty, the greater the punishment. The expression is as follows:

[0068] (19)

[0069] Furthermore, step S3 is specifically as follows:

[0070] The ITD3 algorithm is derived by improving the TD3 algorithm with N-step reward and priority experience replay. The ITD3 algorithm consists of four sub-networks, namely two commentator networks and two actor networks. The improved deep reinforcement learning algorithm is implemented by the ITD3 algorithm.

[0071] First, the N-step reward is introduced into the TD3 algorithm. The N-step reward adds up the rewards of the next n time steps, providing more comprehensive information than the single-step reward.

[0072] In the case of sparse rewards, most state transitions If there is no reward information, then a one-step reward will be ineffective; an N-step reward mitigates the problem of reward sparsity by sampling N transitions.

[0073] By incorporating N-step rewards, the equations of the TD3 algorithm's critic network are modified, resulting in the equation at the N-th step. In round sampling, the modified time difference error function expression as follows:

[0074] (20)

[0075] in, and This represents the parameters of the dual-critic network. Indicates the first Step-by-step return, Indicates the first The reward of taking a step, and Indicates the current state and action. and Indicates the target state and action. The value function representing the critic network, The value function represents the target critic network. This represents the discount factor.

[0076] Secondly, the original TD3 algorithm uses priority empirical replay, at the beginning of the sample, the first The sampling probability of each transformation is defined as The expression is as follows:

[0077] (twenty one)

[0078] in, Indicates the first Prioritizing experience; This represents a constant used to adjust the sampling weights. This determines the priority level to use, when When the value is 0, uniform random sampling will be used.

[0079] Then, the sampling weights are used to update each transition of the network. It is calculated by the following formula, which represents the importance of each transferred data point. Indicates the size of the small batch. This represents maximizing the sampling weights, used for normalization:

[0080] (twenty two)

[0081] Finally, using proportional priority, the transfer priority is updated based on the time difference error, as shown in the following formula:

[0082] (twenty three)

[0083] in, Indicates time difference error. This indicates a pre-set small value to avoid having a priority of 0.

[0084] Furthermore, step S4 is specifically as follows:

[0085] The ITD3 network training network is implemented using two neural networks: an actor network consisting of three fully connected layers to perform the mapping from state to action, and a critic network using four fully connected layers to estimate the Q-value.

[0086] In the ITD3 network, for two actor networks, the input is the state, and the output is the action. The critic network takes the state-action pairs as its input and produces the state-action value function (Q-value). The specific training process of the ITD3 algorithm is as follows:

[0087] First, a small sample is preferentially drawn from the experience replay buffer. ,Will Input into the actor target network. Then, in the next iteration... and state-action pairs Input the target network of the commentator.

[0088] After obtaining two target Q-values ​​( and After that, choose the smaller one to calculate the objective function. The objective function expression is as follows:

[0089] (twenty four)

[0090] in, Indicates return, discount factor The values ​​are the same as those in equation (20). For the random parameters of the critic network.

[0091] On the other hand, Inputting the critic network yields two Q-values ​​( and Then, use them to calculate. The mean squared error is calculated, and the sum of the mean squared errors is backpropagated to update the parameters of the two critic networks, with N-step replay added to the time difference error update.

[0092] Next, the Q-values ​​obtained from the first critic network are input into the actor model network, and the parameters of the actor network are updated in the direction of increasing Q-values ​​(updated once every two iterations).

[0093] Finally, a soft update method is used to update all target networks.

[0094] After training, the angular velocity and linear velocity of the quadcopter drones are controlled by the output motion information, so that each quadcopter drone can successfully reach the specified target mission in the shortest possible time and complete the path planning for the mobilization mission.

[0095] The beneficial effects of this invention are as follows: The method first constructs a PyBullet-based multi-quadcopter UAV Gym environment. By abstracting the state space and action space of the quadcopter UAVs and setting a reward function mechanism, it then uses an improved deep reinforcement learning algorithm for path planning and decision-making. Finally, it trains an improved deep reinforcement learning network and controls the quadcopter UAVs using the output action information, enabling each quadcopter UAV to successfully reach the specified target task in the shortest possible time. The method utilizes N-step rewards in the TD3 algorithm to obtain more accurate reward estimation, faster learning speed, and better generalization performance. It uses priority experience replay to reduce sampling bias, minimize model bias caused by imbalanced sampling, and improve algorithm stability. This makes the TD3 algorithm more suitable for continuous multi-dimensional decision-making problems, enabling it to plan the optimal route in the shortest time while achieving the specified real-time performance and accuracy. Attached Figure Description

[0096] Figure 1This is a flowchart of a multi-quadrotor UAV swarm path planning method based on reinforcement learning according to the present invention.

[0097] Figure 2 This is a model diagram of an "x"-shaped quadcopter drone in an embodiment of the present invention.

[0098] Figure 3 This is a schematic diagram illustrating the principle of lidar detection of horizontal environmental information in an embodiment of the present invention.

[0099] Figure 4 This is a state diagram of a quadcopter drone in an embodiment of the present invention.

[0100] Figure 5 This is a schematic diagram of the ITD3 algorithm in an embodiment of the present invention.

[0101] Figure 6 This is a diagram of the neural network structure in ITD3 in this embodiment of the invention. Detailed Implementation

[0102] The method of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0103] like Figure 1 The flowchart shown is a method for swarm-type mission path planning of multi-quadrotor UAVs based on reinforcement learning according to the present invention. The specific steps are as follows:

[0104] S1. Construct a PyBullet-based multi-quadrotor UAV Gym environment;

[0105] S2. Abstract the state space and action space of the quadcopter drone, and set up a reward function mechanism to enable the drone to interact with the environment;

[0106] S3. Use an improved deep reinforcement learning algorithm for path planning decisions, and perform path planning for each quadcopter drone in a clustered task.

[0107] S4. The improved deep reinforcement learning network controls the angular and linear velocities of the quadcopter drones by outputting motion information, enabling each quadcopter drone to successfully reach the specified target task in the shortest possible time.

[0108] In this embodiment, step S1 is specifically as follows:

[0109] S11. Construct a dynamic simulation model of a multi-quadrotor UAV;

[0110] Use PyBullet to build force and torque models for each quadcopter drone in Gym, and use the physics engine to calculate and update the dynamic equations of all drones.

[0111] like Figure 2As shown, the simplified "x"-shaped quadcopter drone power model constructed in this embodiment sets the arm length of each drone to be... Quality is Inertial properties are The physical constants and convex collision shape are described by separate URDF files for the configuration of the "x" type quadcopter UAV.

[0112] First, set the gravitational acceleration in PyBullet. And the physical stepping frequency (more precise control frequency than Gym stepping); in addition to physical properties and constants, URDF information can also be used in PyBullet to load CAD models of quadcopters; the forces applied to the four motors. and torque around the drone's Z-axis With motor speed It is proportional to the square of . and The expression is as follows:

[0113] (1)

[0114] (2)

[0115] in, and This represents a pre-defined constant.

[0116] and Assuming the pulse width modulation (PWM) input is linearly related and the control of the model is real-time, the motion equations of the quadcopter UAV are as follows:

[0117] (3)

[0118] in, Represents the Jacobian matrix. Represents the inertia matrix. Represents generalized acceleration. Indicates Coriolis and the effect of gravity, superscript This indicates the transpose operation.

[0119] In practice, flying close to the ground or near other drones may produce additional aerodynamic effects, such as Figure 2 middle , and The related forces are represented. In PyBullet, they can be modeled separately and used in combination, including: propeller drag. Ground effect acting on a single motor and the downwash effect acting on the center of mass. .

[0120] The rotating propellers of a quadcopter drone generate drag. This is a force that acts in the opposite direction to the direction of motion. Resistance. linear velocity of quadcopter drones Angular velocity of the propeller and coefficient matrix They are directly proportional, as expressed below:

[0121] (4)

[0122] in, This represents the angular velocity of the propeller, where 60 represents 60 seconds. To simulate cross-coupling, a matrix containing nine coefficients needs to be fitted. The fitting requires a certain degree of symmetry: the drag coefficient and the cross-coupling between the x and y axes should be the same. Due to symmetry, wind speed in the z-direction produces the same force in both the x and y directions. Furthermore, the drag in the z-direction caused by the velocity in the x-direction should be the same as the drag in the z-direction caused by the velocity in the y-direction. Therefore, the drag coefficient matrix... The specific expression is as follows:

[0123] (5)

[0124] in, Indicates the vertical drag coefficient. Represents the parallel drag coefficient, and the matrix The least squares method is used to fit the data.

[0125] When hovering at very low altitudes, the ground effect occurs, meaning the thrust generated by the interaction between the propeller airflow and the ground on the quadcopter drone increases, thus affecting the impact of the ground effect on each motor. With propeller radius ,speed ,high and constant The proportional relationship is as follows:

[0126] (6)

[0127] When two quadcopter drones pass through the same path at different altitudes, a downwash effect occurs. This downwash effect reduces the lift of the lower part of the drone. The effect of the downwash effect can be simplified to a single force acting on the center of mass of the drone, the magnitude of which is... Depending on the coordinate system of the two drones , , Distance in and constants determined experimentally , , , The expression is as follows:

[0128] (7)

[0129] S12. Construct observation and maneuvering space for multiple quadrotor UAVs;

[0130] In the constructed Gym environment, each action performed by a quadcopter drone outputs an observation vector. The observation space expression for multiple quadcopter drones is as follows:

[0131] (8)

[0132] in, Indicates the number of quadcopter drones; Indicates the location of the quadcopter drone; Represents a quaternion, used for attitude control of quadrotor drones; , , These represent the roll angle, pitch angle, and yaw angle, respectively, which are three angles used for attitude estimation. for Indicates the first The linear velocity of a quadcopter drone for Indicates the first The angular velocity of a quadcopter drone; for This indicates the motor speed of all drones.

[0133] like Figure 3 As shown, in this embodiment, the quadcopter drone uses a lidar to detect obstacles. The model is set to include a lidar system for the quadcopter drone. one, and use this A lidar unit is used to observe the environment.

[0134] Among them, this The scanning angle range of each lidar is: (In this embodiment) The angle between the two lasers is ; Represents the horizontal plane The length of the radar ray; if a sensor does not detect any object within a finite distance, then the length of the ray is the maximum detectable distance. Otherwise, the length is the distance between the UAV and the point detected by the radar; The ray length of the i-th radar is expressed as follows:

[0135] (9)

[0136] Environmental information The definition is as follows:

[0137] (10)

[0138] in, Let represent the one-hot code of the i-th radar, if the radar detects a detectable object within a finite distance. The value is 1 if it is true and 0 otherwise, as shown in the expression below:

[0139] (11)

[0140] For any quadcopter UAV, the action space expression is as follows:

[0141] (12)

[0142] in, This indicates the speed input to the quadcopter drone. , , Represents the components of a unit vector. This indicates the required speed; and the motion space can also be represented by the rotational speeds of the four motors, as shown in the following expression:

[0143] (13)

[0144] Finally, the input is converted to pulse width modulation (PWM) and the motor speed is delegated to the controller, which consists of position and attitude control subroutines.

[0145] In this embodiment, step S2 is specifically as follows:

[0146] S21. State space and motion space of an abstract quadcopter drone;

[0147] quadcopter drone The states include: the position of the quadcopter drone and quaternions (used for attitude control of the quadcopter drone). Roll angle Pitch angle Yaw angle linear velocity angular velocity Motor speeds of all drones ;like Figure 4 The drone shown Angle between the first-person perspective direction and the line connecting the target and the Global coordinates of the drone Global coordinates of the target Differences between .

[0148] To enable the drone to reach the target faster and improve convergence speed, the drone's global position is replaced with its relative position to the target in the state. That is Then the state of the drone as follows:

[0149] (14)

[0150] The status of the drone and the environmental conditions detected by lidar The state space of the quadcopter drone can be obtained. The expression is as follows:

[0151] (15)

[0152] The action space of a multi-quadcopter UAV environment consists of the velocities input to the quadcopter UAV. Referring to equation (12), the action space expression for any quadcopter UAV is as follows:

[0153] (16)

[0154] S22. Set up a reward function mechanism to enable the quadcopter drone to interact with the environment;

[0155] The reward function setting has a significant impact on the performance of deep reinforcement learning models and determines the drone's strategy. Reward Function Indicates the state Take action below The environmental feedback obtained is used to evaluate the quality of actions taken in the current state. If... Large, indicating the current state Take action below To facilitate achieving the goal, in the next strategy update, in the state Take action below The probability will increase; otherwise, the probability will decrease.

[0156] To enable the quadcopter drone to reach the assembly mission objective point as quickly as possible, this embodiment sets up a reward function consisting of three parts, as follows:

[0157] First, set a distance reward between the quadcopter drone and the target point. To enable the quadcopter drone to reach its target, The settings are as follows: If the drone approaches the target, the reward is positive, with the maximum reward upon reaching the target point; if the drone moves away from the target, the reward is negative, and if it fails to reach the target within a preset time, the maximum penalty is [missing information]. . The expression is as follows:

[0158] (17)

[0159] in, This indicates the distance between the quadcopter drone and the target. Indicates the first The distance between the quadcopter drone and the target, Indicates the first The distance between the quadcopter drone and the target at the next moment.

[0160] Secondly, a distance reward is set for the quadcopter drone and the obstacle. Setting up to keep drones away from obstacles As follows: If the distance between the drone and the nearest obstacle is less than... If the drone collides with an obstacle, it will be penalized; if the drone collides with an obstacle, the penalty will be... If the distance between the drone and the nearest obstacle is less than This indicates that the drone is safe and will not be punished. The expression is as follows:

[0161] (18)

[0162] in, This represents the ray length of the i-th radar, i.e., the detection distance of the quadcopter drone to obstacles or other quadcopter drones. Indicates the first The distance between the quadcopter drone and the target, and the setting of a safe distance between the drone and the obstacle. .

[0163] Finally, an angular reward is set between the quadcopter drone and the target point. If the drone is prompted to move closer to the target, (like Figure 4 The larger the value (as shown), the greater the penalty. The expression is as follows:

[0164] (19)

[0165] In this embodiment, step S3 is specifically as follows:

[0166] In this embodiment, the ITD3 algorithm is used to implement path planning for multi-UAV swarming missions in unknown environments. The TD3 algorithm solves the overestimation bias problem of Deep Deterministic Policy Gradient (DDPG) and is a deterministic policy reinforcement learning algorithm suitable for high-dimensional continuous action spaces.

[0167] The TD3 algorithm addresses the issues of Q-value estimation error and excessive variance by employing a dual Q-network and a delayed update strategy. However, when the environment has delayed rewards, the TD3 algorithm may require more experience to learn how to make correct decisions. To address this issue, this embodiment introduces an N-step reward mechanism into the TD3 algorithm.

[0168] The ITD3 algorithm is derived by improving the TD3 algorithm with N-step reward and priority experience replay. The ITD3 algorithm consists of four sub-networks, namely two commentator networks and two actor networks. The improved deep reinforcement learning algorithm is implemented by the ITD3 algorithm.

[0169] First, the N-step reward is introduced into the TD3 algorithm. The N-step reward adds up the rewards of the next n time steps, providing more comprehensive information than the single-step reward. Therefore, the algorithm can make better use of delayed rewards and improve learning efficiency.

[0170] In cases of sparse rewards, most state transitions... If there is no reward information, then a one-step reward will be ineffective; an N-step reward is achieved by sampling N transitions to make the environment reward sparse (here, instance values ​​are set). ).

[0171] This embodiment adds an N-step reward to TD3, increasing the chances of finding rewarded transfers and learning from them, thus improving learning efficiency. By adding an N-step reward, the equations of the TD3 algorithm's critic network are modified, and in the... In round sampling, the modified time difference function expression as follows:

[0172] (20)

[0173] in, and This represents the parameters of the dual-critic network. Indicates the first Step-by-step return, Indicates the first The reward of taking a step, and Indicates the current state and action. and Indicates the target state and action. The value function representing the critic network, The value function represents the target critic network. This represents the discount factor.

[0174] Secondly, in the original TD3 algorithm, each experience is sampled uniformly. However, without prioritization, the learning efficiency is low. Adding priority can solve this problem. Prioritized experience replay is a technique to enhance DRL performance. Based on experience replay, it prioritizes samples according to the importance of the experience, allowing important samples to be sampled more frequently, thereby improving the model's learning efficiency and performance. At the beginning of the sample, the first... The sampling probability of each transformation is defined as The expression is as follows:

[0175] (twenty one)

[0176] in, Indicates the first Prioritizing experience; This represents a constant used to adjust the sampling weights. This determines the priority level to use, when When the value is 0, uniform random sampling will be used.

[0177] Then, the sampling weights are used to update each transition of the network. It is calculated by the following formula, which represents the importance of each transferred data point. This indicates the size of the mini-batch. This represents maximizing the sampling weights, used for normalization:

[0178] (twenty two)

[0179] Finally, using proportional priority, the transfer priority is updated based on the time difference error, as shown in the following formula:

[0180] (twenty three)

[0181] in, Indicates time difference error. This indicates a pre-set small value to avoid having a priority of 0.

[0182] In this embodiment, step S4 is specifically as follows:

[0183] The ITD3 network training network is implemented using two neural networks: an actor network consisting of three fully connected layers to perform the mapping from state to action, and a critic network using four fully connected layers to estimate the Q-value.

[0184] In the ITD3 network, for two actor networks, the input is the state, and the output is the action. The critic network takes the state-action pairs as its input and produces the state-action value function (Q-value). Then, as follows... Figure 5 As shown, the training process of the ITD3 algorithm is as follows:

[0185] First, a small sample is preferentially drawn from the experience replay buffer. ,Will Input into the actor target network. Then, in the next iteration... and state-action pairs Input the target network of the commentator.

[0186] After obtaining two target Q-values ​​( and After that, choose the smaller one to calculate the objective function. The objective function expression is as follows:

[0187] (twenty four)

[0188] in, Indicates return, discount factor The values ​​are the same as those in equation (20). For the random parameters of the critic network.

[0189] On the other hand, Inputting the critic network yields two Q-values ​​( and Then, use them to calculate. The mean squared error is calculated, and the sum of the mean squared errors is backpropagated to update the parameters of the two critic networks, with N-step replay added to the time difference error update.

[0190] Next, the Q-values ​​obtained from the first critic network are input into the actor model network, and the parameters of the actor network are updated in the direction of increasing Q-values ​​(updated once every two iterations).

[0191] Finally, a soft update method is used to update all target networks.

[0192] Based on empirical risk minimization, the complexity of a neural network is related to the number of samples. Therefore, based on the state space and action space (size of the observations) obtained in the above steps of this embodiment, the network structure of ITD3 is designed as follows: Figure 6 .

[0193] In this embodiment, the ITD3 actor network consists of three FC neural network layers with 512, 128, and 4 nodes respectively. The first and second layers are activated by rectified linear units (ReLU), and the third layer is activated by hyperbolic tangent (tanh) to ensure that the output of the actor network is within the range of FC layers. Within the range. After three FCs, the actor will input... Mapping motion commands to drones For the commentator network, the Q-value is estimated using four fully connected (FC) cells. Input First, input an FC (1024 nodes) with ReLU activation, then input the vector and action. The vectors are merged into a single 1028-dimensional vector. After passing through two ReLU activation functions and one tanh activation function, the critic network transforms this vector into a Q-value.

[0194] After training, the angular velocity and linear velocity of the quadcopter drones are controlled by the output motion information, so that each quadcopter drone can successfully reach the specified target mission in the shortest possible time and complete the path planning for the mobilization mission.

[0195] In summary, the method of this invention first performs Gym environment modeling for multiple quadcopter UAVs based on PyBullet, improving the portability of the invention. Secondly, it utilizes N-step rewards in the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm to obtain more accurate reward estimation, faster learning speed, and better generalization performance. Simultaneously, it uses priority experience replay to reduce sampling bias, minimizing model bias caused by imbalanced sampling and improving algorithm stability, thus making the TD3 algorithm more suitable for continuous multidimensional decision problems. Finally, it trains an improved TD3 (ITD3) network, controlling the angular and linear velocities of the quadcopter UAV to enable it to successfully reach the assembly-type task objective point in the shortest time under unknown environmental conditions.

[0196] The embodiments described above are provided to help readers understand the principles of the present invention and should be understood as not limiting the scope of protection of the present invention to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of the present invention.

Claims

1. A path planning method for swarm-type missions using multi-quadrotor UAVs based on reinforcement learning, the specific steps of which are as follows: S1. Construct a PyBullet-based multi-quadrotor UAV Gym environment; S2. Abstract the state space and action space of the quadcopter drone, and set up a reward function mechanism to enable the drone to interact with the environment; S3. Use an improved deep reinforcement learning algorithm for path planning decisions, and perform path planning for each quadcopter drone in a clustered task. The ITD3 algorithm is obtained by improving the TD3 algorithm with N-step reward and priority experience replay. The ITD3 algorithm consists of four sub-networks, namely two commentator networks and two actor networks. The improved deep reinforcement learning algorithm is implemented by the ITD3 algorithm. First, the N-step reward is introduced into the TD3 algorithm. The N-step reward adds up the rewards of the next n time steps, providing more comprehensive information than the single-step reward. In the case of sparse rewards, most state transitions If there is no reward information, then a one-step reward will be ineffective; an N-step reward mitigates the problem of reward sparsity by sampling N transitions. By incorporating N-step rewards, the equations of the TD3 algorithm's critic network are modified, resulting in the equation at the N-th step. In round sampling, the modified time difference error function expression as follows: （20）； in, and This represents the parameters of the dual-critic network. Indicates the first Step-by-step return Indicates the first The reward of taking a step, and Indicates the current state and action. and Indicates the target state and action. The value function representing the critic network, The value function represents the target critic network. Indicates the discount factor; Secondly, the original TD3 algorithm uses priority empirical replay, at the beginning of the sample, the first The sampling probability of each transformation is defined as The expression is as follows: （21）； in, Indicates the first Prioritizing experience; This represents a constant used to adjust the sampling weights. This determines the priority level to use, when When the value is 0, uniform random sampling will be used; Then, the sampling weights are used to update each transition of the network. It is calculated by the following formula, which represents the importance of each transferred data point. Indicates the size of the small batch. This represents maximizing the sampling weights, used for normalization: （22）； Finally, using proportional priority, the transfer priority is updated based on the time difference error, as shown in the following formula: （23）； in, Indicates time difference error. This indicates a pre-set small value to avoid having a priority of 0; S4. The improved deep reinforcement learning network controls the angular and linear velocities of the quadcopter drones by outputting motion information, enabling each quadcopter drone to successfully reach the specified target task in the shortest possible time.

2. The method for planning the swarm-type mission path of a multi-quadrotor UAV based on reinforcement learning according to claim 1, characterized in that, In step S1, the specific details are as follows: S11. Construct a dynamic simulation model of a multi-quadrotor UAV; The dynamic equations of a multi-quadrotor UAV are derived from the equations of motion and aerodynamic effects of the quadrotor UAV, thus completing the construction of a dynamic simulation model of the multi-quadrotor UAV, as detailed below: Use PyBullet to build force and torque models for each quadcopter in Gym, and use the physics engine to calculate and update the dynamic equations of all quadcopters. Set the arm length of each quadcopter drone to be Quality is Inertial properties are Physical constants and convex collision shapes are described in separate URDF files for the configuration of the "x" type quadcopter UAV; First, set the gravitational acceleration in PyBullet. And the physical stepping frequency, the force applied to the four motors and torque around the drone's Z-axis With motor speed It is proportional to the square of . and The expression is as follows: （1）； （2）； in, and This represents a pre-defined constant; If the model is to be controlled in real time, the dynamic equations of the quadcopter UAV are expressed as follows: （3）； in, Represents the Jacobian matrix. Represents the inertia matrix. Represents generalized acceleration. Indicates Coriolis and the effect of gravity, superscript Indicates the transpose operation; In practice, flying close to the ground or near other drones generates additional aerodynamic effects. PyBullet models these effects separately and uses them in combination, including: propeller drag. Ground effect acting on a single motor Downwash effect acting on the center of mass ; The rotating propellers of a quadcopter drone generate drag. ,resistance linear velocity of quadcopter drones The angular velocity of the propeller and the constant drag coefficient matrix. They are directly proportional, as expressed below: （4）； in, This represents the angular velocity of the propeller, 60 represents 60s; constant drag coefficient torque. The specific expression is as follows: （5）； in, Indicates the vertical drag coefficient. Represents the parallel drag coefficient, and the matrix The data is fitted using the least squares method; When hovering at a very low height, the ground effect exists, and the impact of the ground effect on each motor is considered. With propeller radius ,speed ,high and constant The proportional relationship is as follows: （6）； When two quadcopter drones pass through the same path at different altitudes, a downwash effect occurs. The effect of the downwash effect can be simplified to a single force acting on the drone's center of mass, the magnitude of which is... Depending on the coordinate system of the two drones , , Distance in and constants determined experimentally , , , The expression is as follows: （7）； S12. Construct observation and maneuvering space for multiple quadrotor UAVs; In the constructed Gym environment, each action performed by a quadcopter drone outputs an observation vector. The observation space expression for multiple quadcopter drones is as follows: （8）； in, Indicates the number of quadcopter drones; Indicates the location of the quadcopter drone; Represents a quaternion, used for attitude control of quadrotor drones; , , These represent the roll angle, pitch angle, and yaw angle, respectively, which are three angles used for attitude estimation. for Indicates the first The linear velocity of a quadcopter drone for Indicates the first The angular velocity of a quadcopter drone; for This indicates the motor speed of all drones; Quadcopter drones use lidar to detect obstacles; the model is configured to have lidar. one, and use this A lidar unit is used to observe the environment; Among them, this The scanning angle range of each lidar is: The angle between the two lasers is ; Represents the horizontal plane The ray length of each radar; The ray length of the i-th radar is expressed as follows: （9）； Environmental information The definition is as follows: （10）； in, The one-hot code of the i-th radar is expressed as follows: （11）； For any quadcopter UAV, the action space expression is as follows: （12）； in, This indicates the speed input to the quadcopter drone. , , Represents the components of a unit vector. This indicates the required speed; and the motion space is represented by the rotational speeds of the four motors, as shown in the following expression: （13）； Finally, the input is converted into pulse width modulation (PWM) and the motor speed is delegated to the controller, which consists of position and attitude control subroutines.

3. The method for planning the swarm-type mission path of a multi-quadrotor UAV based on reinforcement learning according to claim 2, characterized in that, Step S2 is as follows: S21. State space and motion space of an abstract quadcopter drone; The state of a quadcopter drone includes: the quadcopter drone's position, quaternions, etc. Side roll angle Pitch Yaw angle linear velocity angular velocity Motor speeds of all drones The angle between the drone's first-person view and the line connecting the target and the Global coordinates of the drone Global coordinates of the target Differences between ; Replace the drone's global position with its relative position to the target in the state. ,for Then the state of the drone as follows: （14）； The status of drones and the environmental conditions detected by lidar The state space of the quadcopter drone can be obtained. The expression is as follows: （15）； The action space of a multi-quadcopter UAV environment consists of the velocities input to the quadcopter UAV. Referring to equation (12), the action space expression for any quadcopter UAV is as follows: （16）； S22. Set up a reward function mechanism to enable the quadcopter drone to interact with the environment; reward function Indicates the state Take action below The obtained environmental feedback; a reward function consisting of three parts is set to enable the quadcopter drone to reach the assembly-type mission objective point as quickly as possible, as follows: First, set a distance reward between the quadcopter drone and the target point. To enable the quadcopter drone to reach its target, The expression is as follows: （17）； in, This indicates the distance between the quadcopter drone and the target. Indicates the first The distance between the quadcopter drone and the target, Indicates the first The distance between the quadcopter drone and the target at the next moment; Secondly, a distance reward is set for the quadcopter drone and the obstacle. To encourage drones to stay away from obstacle placement, The expression is as follows: （18）； in, This represents the ray length of the i-th radar, i.e., the detection distance of the quadcopter drone to obstacles or other quadcopter drones. Indicates the first The distance between the quadcopter drone and the target, and the setting of a safe distance between the drone and the obstacle. ; Finally, an angular reward is set between the quadcopter drone and the target point. If the drone is prompted to move closer to the target, The larger the penalty, the greater the punishment. The expression is as follows: （19）。 4. The method for planning the swarm-type mission path of a multi-quadrotor UAV based on reinforcement learning according to claim 3, characterized in that, Step S4 is as follows: The ITD3 network training network is implemented by two neural networks: an actor network consisting of three fully connected layers to perform the mapping from state to action, and a critic network using four fully connected layers to estimate the Q-value; In the ITD3 network, for the two actor networks, the input is the state, and the output is the action; the critic network takes the state-action pair as its input and produces the state-action value function Q-value; the specific training process of the ITD3 algorithm is as follows: First, a small sample is preferentially drawn from the experience replay buffer. ,Will Input into the actor target network; then, in the next iteration... and state-action pairs Input the critic's target network; After obtaining two target Q-values , Then, the smaller one is selected to calculate the objective function. The objective function expression is as follows: （24）； in, Indicates return, discount factor The values ​​are the same as those in equation (20). For random parameters of the critic network; On the other hand, Inputting the critic network yields two Q-values. , Then, use them to calculate The mean squared error is calculated, and the sum of the mean squared errors is backpropagated to update the parameters of the two critic networks, with N-step replay added to the time difference error update; Next, the Q-value obtained from the first critic network is input into the actor model network, and the parameters of the actor network are updated in the direction of increasing Q-value, with an update every two iterations; Finally, a soft update method is used to update all target networks; After training, the angular velocity and linear velocity of the quadcopter drones are controlled by the output motion information, so that each quadcopter drone can successfully reach the specified target mission in the shortest possible time and complete the path planning for the mobilization mission.

Images