Urban multi-unmanned aerial vehicle cooperative route planning method based on deep reinforcement learning

Abstract

The invention discloses an urban multi-unmanned aerial vehicle cooperative route planning method based on deep reinforcement learning, and the method comprises the steps: building a flight environment model based on urban three-dimensional point cloud data, and achieving the efficient obstacle query through an R-Tree index; for multiple unmanned aerial vehicles, acquiring distance information of surrounding obstacles by adopting a ray-based environment sensing mode, and constructing a state space containing a current unmanned aerial vehicle state, a target relative position, relative positions of other unmanned aerial vehicles and the distance information of the surrounding obstacles; outputting continuous flight control actions through a deep reinforcement learning strategy of parameter sharing; a multi-target reward function is designed, and the multiple unmanned aerial vehicles are guided to achieve safe and efficient collaborative route planning; in the execution stage, each unmanned aerial vehicle makes an independent decision only by depending on the shared strategy network and local observation information of the unmanned aerial vehicle, and distributed cooperative flight is realized. According to the method, the complexity of a traditional method for environment perception and decision calculation is effectively reduced, and the method is suitable for a multi-unmanned aerial vehicle planning task in a complex urban airspace.

Description

Technical Field

[0001] This invention belongs to the field of low-altitude airspace management, multi-UAV collaborative control and path planning technology, specifically involving a method for urban multi-UAV collaborative route planning based on deep reinforcement learning. Background Technology

[0002] With the rapid development of the low-altitude economy, unmanned aerial vehicles (UAVs) are being deployed more and more widely in applications such as urban inspection, logistics delivery, emergency response, and urban sensing. In real urban environments, multiple UAVs often need to perform tasks simultaneously in complex three-dimensional spaces with dense buildings and limited airspace. How to achieve efficient collaborative flight and path planning of multiple UAVs while ensuring flight safety has become a key technical problem that urgently needs to be solved in the application of low-altitude unmanned systems.

[0003] Existing research on UAV path planning typically relies on certain idealized assumptions, such as simplifying the environment into regular geometry or pre-constructing an airway network suitable for UAVs. However, in high-density urban environments, UAVs inevitably need to traverse narrow spaces between buildings or height-restricted areas, placing higher demands on path planning algorithms in terms of 3D spatial perception, trajectory rationality, and multi-UAV collaborative mechanisms. Traditional path planning methods mainly include search-based and sampling-based methods. These methods can achieve relatively reliable results in single-UAV scenarios, but when extended to multi-UAV collaborative tasks, their computational complexity increases rapidly, and the generated flight trajectories often lack smoothness, making it difficult to quickly converge to a high-quality solution in complex or narrow environments.

[0004] Furthermore, while some numerical optimization-based path planning methods can handle complex constraints without relying on explicit environment modeling, their iterative solution process is typically accompanied by high computational overhead and long solution latency, making it difficult to meet the requirements of real-time collaborative flight of multiple UAVs. In recent years, thanks to the advantages of deep reinforcement learning in experience-based learning and environment adaptation, it has gradually been introduced into the field of UAV path planning. Early research focused primarily on single-agent scenarios, subsequently developing multi-agent reinforcement learning frameworks to address collaborative tasks. Although multi-agent reinforcement learning methods have shown great potential in collaborative decision-making problems, their training process is usually accompanied by significant computational costs, and with the increase in the number of agents, training stability and algorithm scalability face severe challenges. Therefore, how to construct a multi-UAV path planning method that combines safety, collaboration, scalability, and sample efficiency in complex urban 3D environments remains an important research issue. Summary of the Invention

[0005] The purpose of this invention is to address the problems existing in the prior art and provide a method for collaborative flight path planning of multiple UAVs in urban environments based on deep reinforcement learning. This method acquires environmental information around the UAVs using a ray-sensing model. Under the independent policy reinforcement learning paradigm, it enables multiple homogeneous UAVs to share a unified policy network through a parameter sharing mechanism. Furthermore, it combines a near-end policy optimization algorithm to achieve flight control in a continuous action space, thereby improving the collaborative efficiency and system scalability of multiple UAVs in complex environments while ensuring flight safety.

[0006] The technical solution adopted in this invention is as follows:

[0007] A method for collaborative flight path planning of multiple UAVs in urban areas based on deep reinforcement learning includes the following steps:

[0008] Step 1: 3D Flight Environment Modeling: First, the city's 3D point cloud data is cleaned and noise is removed. Then, a 3D flight environment model is constructed based on R-Tree spatial indexing technology for real-time querying of the spatial relationship between the UAV and obstacles. R-Tree is used to organize buildings and obstacles in the environment, forming a 3D environment model for UAV flight simulation. In this 3D environment model, the following is defined: Each drone is assigned a takeoff and target position, and each drone is modeled as an independent intelligent agent.

[0009] Step 2: Construct a state space based on ray sensing: launch with the UAV as the center. Each ray is used to obtain the cutoff distance of each ray. Combined with the current motion state information of the UAV, the relative position information of the target point and the relative position information of other UAVs, a state vector is constructed.

[0010] Environmental perception model construction: for each drone location Starting from the current position, launch into three-dimensional space. A ray is used to acquire obstacle distance information in the environment surrounding the drone. Each ray has a preset maximum detection range. When a ray does not collide with an obstacle within the maximum detection range, its return distance is taken as the maximum detection range. The emission directions of the rays include: rays pointing from the drone's current position to the target position, rays pointing in the opposite direction from the drone to the target position, rays in the vertically upward and vertically downward directions along the world coordinate system, and other rays distributed at preset angles in the plane formed by the drone's current position and the target position, wherein the ray density in the area in front of the target position is higher than that in the area behind it.

[0011] The state space is an observation vector based on local perception, which includes at least: the current flight speed of the UAV, the relative position information of the UAV's current position relative to the target position, the relative position information of other UAVs relative to the current UAV, and the obstacle distance information corresponding to each ray.

[0012] Step 3: Construct the action space: Define the UAV state space as normalized velocity control vectors along the forward, lateral, and vertical directions;

[0013] For each UAV agent, an observation vector is constructed based on its local perception information. The observation vector includes at least the following at each time step: The speed of the drone itself The relative position information of the drone's current position with respect to the target position. ,in Represents a predefined normalized constant corresponding to the environmental range, and the relative position information of other drones relative to this drone. ,in Representing the current state of drones and the first The three-dimensional relative distance between the drones and the obstacle distance information obtained based on the ray sensing model. In order to enhance the sensitivity to nearby obstacles, the distance returned by each ray is mapped to the corresponding proximity value through an inverse function. ,in It is a predefined constant.

[0014] The action space of a UAV is defined as its normalized velocity control vectors in the forward, lateral, and vertical directions within the body coordinate system. For the first... A drone at all times Its actions are as follows:

[0015]

[0016] in, , , These represent the normalized speed commands for the UAV in the forward, lateral, and vertical directions, respectively. This is based on the UAV's maximum horizontal flight speed. and maximum climb (descent) speed Normalized motion can be mapped to actual velocity vectors in the body coordinate system:

[0017]

[0018] Since the drone's motion control is performed in the body coordinate system, the velocity vector in the body coordinate system needs to be transformed to the world coordinate system to achieve state updates. The drone's heading angle in the horizontal plane is determined by the direction from its current position to the target position, and is calculated as follows:

[0019]

[0020] Based on the aforementioned heading angle, the velocity in the body coordinate system can be mapped to the world coordinate system through rotational transformation, and the transformation relationship is as follows:

[0021]

[0022] At each discrete time step The drone's location status is updated as follows:

[0023]

[0024] Step 4: Set up a multi-objective reward function: Design a comprehensive reward function that includes objective guidance reward, collision penalty, time consumption penalty, and motion smoothness penalty;

[0025] Based on the requirements of UAV cooperative flight missions, a comprehensive reward function is constructed. This reward function includes at least the following reward or penalty items: reward for reaching the target location. Penalties for collisions Rewards for drones approaching the target location Penalties for drones straying from their target location Penalty when the drone's altitude gradually moves away from the target altitude Penalties when drones approach environmental boundaries Penalties for drones approaching obstacles Penalties for drones approaching other drones Time step punishment Smoothness penalty when movement changes are too large .

[0026] The comprehensive reward function is a weighted combination of multiple reward sub-items, with different reward sub-items corresponding to different weights. The weight values ​​of the reward when the UAV reaches the target and the collision penalty are greater than the weights of the other sub-items. The weight parameters of each reward sub-item are configured according to different flight scenarios or mission requirements to optimize the collaborative route planning effect while ensuring flight safety.

[0027] Step 5: Construct a parameter-sharing deep reinforcement learning policy network: Using the Proximal Policy Optimization (PPO) algorithm, a deep neural network model containing an Actor network and a Critic network is constructed. All homogeneous UAVs share the same set of network parameters and output continuous flight control actions based on their respective state spaces. The parameters are updated centrally during the training phase.

[0028] During training and execution, each UAV inputs its local observations into the shared policy network and outputs corresponding continuous control actions, thereby enabling collaborative behavior learning among multiple UAVs under independent decision-making conditions.

[0029] Step 6: Deploy the converged policy network to the UAV control system, and output control commands to drive the UAV to fly based on the real-time collected ray data and status information.

[0030] Step 7: Distributed Execution: After training is completed, each UAV relies solely on the shared policy network and its own local observation information to make independent decisions during the actual operation phase, without the need for a centralized control unit, thereby realizing distributed cooperative flight of multiple UAVs.

[0031] This invention employs a sparse ray perception model to replace high-dimensional visual images as environmental input. While maintaining environmental perception capabilities, it significantly reduces the computational load on the neural network, improving the real-time performance and stability of the algorithm in complex 3D urban environments. By introducing a parameter-sharing reinforcement learning mechanism, multiple homogeneous UAVs can share learning experiences, significantly improving the sample efficiency and training stability of multi-UAV collaborative learning. Collaborative decision-making is achieved under an independent policy execution paradigm, avoiding the computational bottleneck caused by centralized control. It possesses good system scalability and distributed execution capabilities, making it suitable for low-altitude flight scenarios in densely built-up, airspace-restricted urban environments. Attached Figure Description

[0032] Figure 1 This is a flowchart of the present invention;

[0033] Figure 2 This is a top-view schematic diagram of the ray sensing model;

[0034] Figure 3 This is a side view schematic diagram of the ray sensing model;

[0035] Figure 4 This is a schematic diagram of a neural network structure;

[0036] Figure 5 This is an example diagram of the path planning of the present invention. Detailed Implementation

[0037] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0038] See Figure 1 The present invention includes the following specific steps:

[0039] Step 1: 3D Flight Environment Modeling: First, the city's 3D point cloud data is cleaned to remove noise, and then a 3D flight environment model is constructed based on R-Tree spatial indexing technology for real-time querying of the spatial relationship between the UAV and obstacles. In the 3D environment model, the following is defined: Each drone is assigned a takeoff and target position, and each drone is modeled as an independent intelligent agent.

[0040] Step 2: Construct a state space based on ray sensing: launch with the UAV as the center. A ray is used to obtain the cutoff distance of each ray. Combined with the UAV's current motion state information, the target point's relative position information, and the relative position information of other UAVs, a state vector is constructed. For each UAV position... Each ray returns the distance between itself and the nearest obstacle it encounters. The maximum detection range is The emission directions of the rays include: rays emitted from the current position of the UAV towards the target position, rays emitted in the opposite direction from the current position of the UAV towards the target position, and rays emitted vertically upward and downward in the world coordinate system, respectively. The remaining rays are distributed at preset angles within a reference plane determined by the current position of the UAV and the target position, with the ray density in the forward region along the target direction being higher than the ray density in the backward region.

[0041] For each UAV agent, an observation vector is constructed based on its local perception information. The observation vector includes at least the following at each time step: The speed of the drone itself The relative position information of the drone's current position with respect to the target position. ,in Represents a predefined normalized constant corresponding to the environmental range, and the relative position information of other drones relative to this drone. ,in Representing the current state of drones and the first The three-dimensional relative distance between the drones and the obstacle distance information obtained based on the ray sensing model. In order to enhance the sensitivity to nearby obstacles, the distance returned by each ray is mapped to the corresponding proximity value through an inverse function. ,in It is a predefined constant.

[0042] Step 3: Construct the action space: Define the UAV's state space as normalized velocity control vectors along the forward, lateral, and vertical directions. For the first... A drone at all times Its actions are as follows:

[0043]

[0044] in, , , These represent the normalized speed commands for the UAV in the forward, lateral, and vertical directions, respectively. This is based on the UAV's maximum horizontal flight speed. and maximum climb (descent) speed Normalized motion can be mapped to actual velocity vectors in the body coordinate system:

[0045]

[0046] Since the drone's motion control is performed in the body coordinate system, the velocity vector in the body coordinate system needs to be transformed to the world coordinate system to achieve state updates. The drone's heading angle in the horizontal plane is determined by the direction from its current position to the target position, and is calculated as follows:

[0047]

[0048] Based on the aforementioned heading angle, the velocity in the body coordinate system can be mapped to the world coordinate system through rotational transformation, and the transformation relationship is as follows:

[0049]

[0050] At each discrete time step The drone's location status is updated as follows:

[0051]

[0052] Step 4: Set up a multi-objective reward function: Design a comprehensive reward function that includes target guidance rewards, collision penalties, time consumption penalties, and motion smoothness penalties. The reward function should include at least the following reward or penalty items: reward for reaching the target position. Penalties for collisions Rewards for drones approaching the target location Penalties for drones straying from their target location Penalty when the drone's altitude gradually moves away from the target altitude Penalties when drones approach environmental boundaries Penalties for drones approaching obstacles Penalties for drones approaching other drones Time step punishment Smoothness penalty when movement changes are too large .

[0053] Step 5: Construct a parameter-sharing deep reinforcement learning policy network: Using the Proximal Policy Optimization (PPO) algorithm, a deep neural network model containing an Actor network and a Critic network is constructed. All homogeneous UAVs share the same set of network parameters and output continuous flight control actions based on their respective state spaces, thereby realizing collaborative behavior learning of multiple UAVs under independent decision-making conditions.

[0054] Step 6: Deploy the converged policy network to the UAV control system. Based on the real-time collected ray data and status information, output control commands to drive the UAV to fly. This enables multi-UAV cooperative flight without a centralized control unit.

[0055] Example

[0056] This embodiment uses a city street scene as the research area to verify the collaborative path planning task of multiple UAVs. The specific process of this embodiment includes three steps:

[0057] Step 1: Collect 3D point cloud data (.ply format) of the urban area, with a point count on the order of 10. 6 After denoising and downsampling, the point cloud data is organized using an R-Tree spatial index structure to support rapid collision detection and ray sensing queries during UAV flight. The environment is defined as follows: A continuous space. Set the number of drones. .

[0058] Step 2: Refer to Figure 2 , Figure 3 As shown, in this embodiment, the number of rays is set. The maximum sensing distance is set to The rays are distributed in a fan shape around the UAV. Besides the four designated directions, the remaining rays follow... , , , They are distributed around the drones.

[0059] Step 3: Set predefined constants , drones At time step state space vector .

[0060] Step 4: In this embodiment, the maximum horizontal flight speed of the drone is set to... m / s, maximum climb (descent) speed set to m / s, time step set to s.

[0061] Step 5: In this embodiment, the reward function weights are set as follows: , , , , , , , , , .

[0062] Step 6: In this embodiment, a multi-UAV cooperative decision-making model is constructed using a parameter-sharing deep reinforcement learning framework. The PPO algorithm is employed to build a unified policy network and value network, with all homogeneous UAVs sharing the same set of network parameters. (See [link to documentation]). Figure 4 As shown, during training, each UAV, based on its own local observation vector, inputs its observations into the shared policy network and independently outputs corresponding continuous control actions. The value network is used to evaluate the expected reward in the current state and guides the policy network update. Through the parameter sharing mechanism, the sample efficiency and training stability of multi-UAV collaborative learning are improved, while the model parameter size is reduced.

[0063] After training, during flight, each UAV relies solely on its own local perception information to make independent decisions, achieving collaborative obstacle avoidance and collaborative route planning without explicit communication with other UAVs. The path planning results for each UAV in this embodiment are shown in the attached diagram. Figure 5 As shown.

[0064] The above implementation is one embodiment of the present invention and is not intended to limit the present invention. Any modifications and improvements made without departing from the concept of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for collaborative flight path planning of multiple unmanned aerial vehicles (UAVs) in urban areas based on deep reinforcement learning, characterized in that: Includes the following steps: Step 1: 3D flight environment modeling: First, clean the 3D point cloud data of the city to remove noise, and then build a 3D flight environment model based on R-Tree spatial indexing technology for real-time querying of the spatial relationship between the UAV and obstacles; Step 2: Construct a state space based on ray sensing: launch with the UAV as the center. Each ray is used to obtain the cutoff distance of each ray. Combined with the current motion state information of the UAV, the relative position information of the target point and the relative position information of other UAVs, a state vector is constructed. Step 3: Construct the action space: Define the UAV state space as normalized velocity control vectors along the forward, lateral, and vertical directions; Step 4: Set up a multi-objective reward function: Design a comprehensive reward function that includes objective guidance reward, collision penalty, time consumption penalty, and motion smoothness penalty; Step 5: Construct a parameter-sharing deep reinforcement learning policy network: The PPO algorithm is optimized using a proximal policy, and a deep neural network model containing an Actor network and a Critic network is constructed. All homogeneous UAVs share the same set of network parameters and output continuous flight control actions based on their respective state spaces. Step 6: Deploy the converged policy network to the UAV control system, and output control commands to drive the UAV to fly based on the real-time collected ray data and status information.

2. The method according to claim 1, characterized in that: The launch described in step 2 Each ray has a preset maximum detection range. When the ray does not collide with an obstacle within the maximum detection distance, its return distance is taken as the maximum detection distance.

3. The method according to claim 1, characterized in that: The emission directions of the rays include: rays pointing from the current position of the UAV to the target position, rays pointing in the opposite direction from the UAV to the target position, rays pointing vertically upward and downward along the world coordinate system, and other rays distributed at preset angles in the plane formed by the current position of the UAV and the target position, wherein the ray density in the area in front of the target position is higher than that in the area behind it.

4. The method according to claim 1, characterized in that: The state space described in step 2 is an observation vector based on local perception, which includes at least: the current flight speed of the UAV, the relative position information of the current position of the UAV with respect to the target position, the relative position information of other UAVs with respect to the current UAV, and the obstacle distance information corresponding to each ray.

5. The method according to claim 1, characterized in that: The flight control actions described in step 5 are represented in a continuous action space, and the actions include the forward speed command, lateral speed command, and vertical speed command of the UAV.

6. The method according to claim 1, characterized in that: The target guidance rewards include: rewards when the drone approaches the target position, penalties when the drone deviates from the target position, penalties when the drone's altitude gradually deviates from the target altitude, penalties when the drone approaches the environmental boundary, penalties when the drone approaches an obstacle, penalties when the drone approaches other drones, and rewards when the drone reaches the target.

7. The method according to claim 1, characterized in that: The comprehensive reward function is a weighted combination of multiple reward sub-items, with different reward sub-items corresponding to different weights. The weight values ​​of the reward when the drone reaches the target and the collision penalty are greater than the weights of the other sub-items. The weight parameters of each reward sub-item are configured according to different flight scenarios or mission requirements.

8. The method according to claim 1, characterized in that: The parameter-sharing deep reinforcement learning policy network updates parameters centrally during the training phase and is independently invoked by each UAV during the execution phase, achieving distributed cooperative flight without explicit communication.

Images