A three-dimensional dynamic coverage method based on deep deterministic policy gradient

Abstract

The application discloses a three-dimensional dynamic coverage method based on a deep deterministic policy gradient. The application belongs to the field of unmanned aerial vehicle control and mainly solves the problem of dynamic area coverage of multiple unmanned aerial vehicles in a three-dimensional environment. The method introduces a region surface area calculation method based on a digital elevation model and performs effective distribution of points of interest (PoIs) on the basis of the method. Meanwhile, a multi-agent cooperative trajectory control strategy based on a deep deterministic policy gradient method is developed to guide the flight of the unmanned aerial vehicle. Compared with existing work, the method has greater coverage, higher connectivity and lower energy consumption.

Description

Technical Field

[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) control and is particularly suitable for using multiple UAVs to dynamically cover irregular three-dimensional areas. Background Technology

[0002] Regional coverage has long been a research hotspot, and current research has been widely applied to communication coverage, disaster monitoring, data collection, and public safety. Recently, the use of drones for dynamic coverage missions has become increasingly common in both military and civilian fields. For example, during the 2019 Amazon rainforest fires, firefighters were unable to enter the fire zone to assess the situation; drones, with their high-altitude operation and high flexibility, were used to detect the specific conditions in the fire-affected areas.

[0003] Over the past decade, numerous region coverage algorithms have been proposed. Coverage algorithms based on radial basis function representations of the ergodic problem and the steady-state thermal equations of the potential field are used to achieve centralized feedback control in multi-agent systems. In deep reinforcement learning control frameworks, agents learn control policies from their interactions with the environment to cover a specified target area. However, these methods all perform coverage tasks in two-dimensional environments. In reality, the operating environments of drones often resemble hills with varying elevations, forests with fires, or cities experiencing earthquakes—all areas with highly uneven terrain surfaces.

[0004] Solving the dynamic coverage problem of multiple UAVs in a 3D environment presents several challenges. First, the surface area of ​​a 3D region is highly unevenly distributed within the environment, and the size of the surface area ultimately affects the overall coverage quality. Second, for larger 3D regions, multiple UAVs are often required to perform coverage tasks, necessitating the maintenance of connectivity between them; UAVs disconnected from the system for extended periods will permanently lose connection. Furthermore, UAVs have limited energy resources, therefore coverage tasks must be performed efficiently to extend the lifespan of the entire system.

[0005] This invention considers a scenario where multiple drones perform dynamic coverage tasks in an irregular three-dimensional region. To more efficiently control the drones to dynamically cover a specified three-dimensional surface, a control strategy is developed that maximizes the overall coverage while minimizing the energy consumption of all drones, and ensures the connectivity of the drone swarm during the coverage task. Summary of the Invention

[0006] The purpose of this invention is to propose a depth-deterministic policy gradient-based dynamic coverage (MADDPG-DSC) method for scenarios where multiple UAVs perform dynamic coverage tasks in an irregular 3D region. Consider using N UAVs to perform the coverage task. Each UAV knows its own position information via GPS and maintains a fixed altitude difference Δh with the region surface throughout its flight. Each UAV has a finite coverage area, and the projection of its coverage area onto the region surface is a circle with radius R centered on the UAV's projection point on the region. Due to the finite coverage area, the UAVs must constantly move back and forth to cover the entire 3D region. During this movement, the UAVs should maintain connectivity as much as possible, and due to limited energy resources, the UAVs must minimize their flight distance to complete the coverage task efficiently. Detailed Implementation

[0007] To achieve this objective, the present invention employs the following steps:

[0008] Step 1: An irregular 3D scene Q is constructed using a Digital Elevation Model (DEM). The DEM data primarily employs a regular grid method to sample the 3D region and stores the height information of the sampling points. Based on the regular grid, the projection of Q onto the XY plane is divided into 22500 (150*150) regularly arranged grids of length Δx and width Δy. Each grid's four vertices correspond to four sampling points on the terrain Q. These sampling points divide Q into several regularly arranged patches with the same projection size on the XY plane. The area of ​​each patch is calculated based on the height value corresponding to its center point, and the height data corresponding to the center points of the eight surrounding patches are also needed. The center point O of the patch whose area is to be calculated is connected to the center points A1 to A8 of the eight surrounding patches, and the midpoints of these eight patches are connected sequentially to obtain eight triangles in the 3D coordinate system. Next, we calculate the area of ​​each of the eight triangles. For example, for triangle OA1A2, we first calculate the Euclidean distances a, b, and c of the three sides of the triangle. Then, the area of ​​the triangle is calculated as follows:

[0009]

[0010] Where s = (a + b + c) / 2, the area of ​​these 8 triangles and S are finally calculated. Since these triangles exceed the area of ​​the central facet, they need to be trimmed. Define O and A. i The center point of the line is B. i Connect B1 to B8 sequentially. According to the principle of similar triangles, O and B... i The sum of the areas S of the triangles formed m =S / 4, S mThis is the approximate three-dimensional surface area of ​​the central facet. The smaller the values ​​of Δx and Δy, the higher the accuracy of this approximate area.

[0011] Step 2: Divide the 3D irregular terrain Q into 25 (5*5) identical, regularly arranged cells based on a regular grid. Each cell contains 900 (30*30) regular grid cells. The cell is defined as the smallest unit for UAV area coverage. The 3D surface area of ​​a cell is calculated as the sum of the 3D surface areas of the corresponding patches of these 900 regular grid cells. The 3D surface area of ​​each cell is divided into four levels, each assigned 1 to 4 Points of Interest (PoIs). Through the allocation of PoIs, the 3D irregular terrain Q is transformed into a PoI model, converting the UAV's 3D surface coverage problem into a PoI coverage problem.

[0012] Step 3: Define three performance metrics to be optimized: coverage, energy consumption, and connectivity. Coverage is defined as the ratio of the number of covered points of interest to the total number of points of interest, i.e. The definition stipulates that when a point of interest falls within the coverage area of ​​a drone, it is considered covered. The coverage area of ​​the drone on the 3D region surface is defined as a circle with radius r centered on the drone's projection onto the region surface. The value of r can be determined based on the drone's hardware and flight altitude. During flight, all drones maintain a fixed altitude difference Δh with the region surface. Only the energy consumption of the drone during flight is considered, and the energy consumption value is defined to have a simple linear relationship with the drone's flight distance. Energy consumption increases linearly with flight distance; using This represents a drone swarm network, where This refers to a collection of drones, i.e. This indicates the total number of drones. This indicates the connectivity between drones, specifying when two drones L i L j When the distance between them is less than the communication range R of the drone, the drones can communicate with each other, denoted as E. ij =1, when any two drones in the system can establish a communication link, the entire system is fully connected, denoted as f. c =1, otherwise f c =0.

[0013] Step 4: Propose a MADDPG-based control strategy, MADDPG-DSC. The UAV's control strategy can be modeled as a distributed partially observable Markov decision process (DPOMDP). Generally, a DPOMDP is represented as... in It is the joint state space of all agents; It is the action space; It is a joint observation space of agents; It is the conditional probability of state transition after the action is performed. γ is the discount factor; It is the reward function. The joint state of all agents consists of four parts: agent location, coverage information of points of interest, connectivity between drones, and energy consumption of each drone. At each time step, each drone can only observe its own and its neighbors' state information; the agent's action consists of two components, namely... Let x and y represent the motion components of the UAV in the x and y directions, respectively. Therefore, the motion space is represented as follows: Where T represents the total number of time slots. A mechanics model is used to control the drone's movement, converting the action values ​​output by the agent into forces. Assuming the initial velocity of the drone is v0, then after one round of actions, the drone's velocity is v = v0 + d. t *F i t / mass, where mass represents the mass model of the drone, d t This represents the length of a time step. To reduce computational complexity, the UAV's position in the XY plane is updated as follows: The reward function is defined in the following form:

[0014]

[0015] in and These are the coverage and connectivity metrics for the current time step; m c and m f These are parameters that can be adjusted manually; This represents the energy consumption increment at the current time step compared to the previous time step. The numerator of the reward function represents the total reward, including coverage gain and connectivity penalty, while the denominator represents the total energy consumption. MADDPG-DSC employs a centralized training and distributed execution framework. Policy information from other agents is used as additional information for training the critic network, but this information is not used during the agent execution phase. In the experiments, a four-layer fully connected network is used as the network model for both the critic and the actor, with each layer using 64, 64, 128, and 256 neurons respectively, and ReLU as the activation function.

[0016] The policy of each agent is represented as u i Based on the idea of ​​deterministic strategies, the agent's expected return J(u) i The gradient of ) is expressed as:

[0017]

[0018] in It is a lumped-action value function, i.e., a model of a critic network. This function takes as input the action information of all agents and some state information x, and outputs the Q-value of the i-th agent. In the simplest case, x consists of the observation information of all agents, i.e. Experience replay buffer Contains tuples Update using the following loss function:

[0019]

[0020] in This indicates a delay parameter θ′ i The target policy set. All drone commentator networks are trained in a centralized manner; however, each drone's actor network is updated only using its own observations. At each time step, each drone reacts to the environment based on its own observations and policy model, outputting a corresponding action value and executing the action, then updating its state information. This process is repeated at the next time step until all time steps are completed or the target area is fully covered.

[0021] The proposed 3D dynamic overlay method based on depth-deterministic policy gradient has demonstrated its superiority through simulation experiments. The simulation constructed an irregular 3D region projected onto the ground as a square with a side length of 4 units, where one unit corresponds to 1125 meters in reality. A regular grid with a length and width of 30 meters was used to process the 3D region, dividing it into 25 (5*5) patches. Each patch projected onto the ground as a square with a side length of 0.8 units. A corresponding number of Points of Interest (PoIs) were assigned to each patch based on its area, resulting in a total of 54 PoIs. The UAV's communication range was defined as 0.47 units, and the UAV maintained a 0.18-unit altitude difference with the region's surface during flight. (Appendix) Figure 1 This demonstrates the 3D region constructed in the simulation experiment, and a schematic diagram illustrating the principle of calculating surface area using a digital elevation model (DEM). (Attached) Figure 2 The distribution map of interest points in the interest point model is given. (Attached) Figure 3 The network architecture of MADDPG-DSC is demonstrated, employing a framework of centralized training and distributed execution. (Appendix) Figure 4 The document provides flight path diagrams for three drones performing a coverage mission when the drone's coverage range is 0.3 units. (Attached) Figure 5The graph shows the relationship between coverage rate and training time slots when MADDPG-DSC and DDQN use 3 and 5 drones respectively to perform coverage tasks, assuming the drone coverage range is 0.3 units. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the surface area calculation of the present invention;

[0023] Figure 2 This is a schematic diagram of the distribution of points of interest in this invention;

[0024] Figure 3 This is a diagram of the MADDPG-DSC network structure of the present invention;

[0025] Figure 4 This is a schematic diagram of the trajectory of the three drones covering the mission of this invention;

[0026] Figure 5 This is a schematic diagram of the model convergence of the present invention.

Claims

1. A 3D dynamic coverage method based on depth-deterministic policy gradient, comprising the following steps: Step 1: Establish an irregular 3D scene using a digital elevation model. Q DEM data primarily employs a regular grid method to sample the three-dimensional region and stores the height information of the sampling points; based on the regular grid, Q exist XY The projection of the plane is divided into 22500 (150*150) regularly arranged segments of length [missing information]. Width A grid, where the four vertices of each grid correspond to the terrain. Q The four sampling points on the surface will... Q Divided into several regular arrangements and in XY The projections on the plane are identical in size; the area of ​​each surface is calculated based on the height value corresponding to its center point, and the height data corresponding to the center points of the eight surrounding surfaces is also needed; the center points of the surfaces whose areas are to be calculated are then connected. O Center point of the surrounding 8 facets A 1 ~A 8. Connect the midpoints of these 8 faces in sequence to obtain 8 triangles in a 3D coordinate system; next, calculate the area of ​​each of these 8 triangles. For any one of these 8 triangles, first calculate the Euclidean distance between the three sides of the triangle. a , b , c The area of ​​the triangle is then calculated as follows: (1) in Finally, the sum of the areas of these 8 triangles was calculated. S Since these triangles exceed the boundaries of the central facet, they need to be trimmed; definition O and A i The center point of the line is B i , connected in sequence B 1 ~B 8. According to the principle of similar triangles, by O and B i The sum of the areas of the triangles formed , S m That is, the approximate three-dimensional surface area of ​​the central facet. and The smaller the value, the higher the accuracy of the approximate three-dimensional surface area; Step 2: Identify the 3D irregular terrain based on the regular grid. Q The area is divided into 25 (5*5) identical, regularly arranged cells, each containing 900 (30*30) regular grids. The cell is defined as the smallest unit for UAV area coverage. The approximate 3D surface area of ​​a cell is calculated as the sum of the 3D surface areas of the corresponding patches within these 900 regular grids. The approximate 3D surface areas of the cells are divided into different levels, and a corresponding number of points of interest (POIs) are assigned to each level. Through the allocation of POIs, the 3D irregular terrain is... Q The problem of UAV coverage of a 3D region surface is transformed into a problem of coverage of points of interest. Step 3: Model the UAV control strategy as a distributed partially observable Markov decision process. A distributed partially observable Markov decision process is represented as follows: ,in It is the joint state space of all agents; It is the action space; It is a joint observation space of agents; It is the conditional probability of state transition after the action is performed. ; It is a discount factor; It is a reward function; the joint state of all agents consists of four parts: agent location, coverage information of points of interest, connectivity between drones, and energy consumption of each drone; at each time step, each drone can only observe its own and its neighbors' state information; each agent's action consists of two components, namely... , respectively representing the drone in x and y The directional motion component, therefore the motion space is represented as ,in This represents the total number of time slots; a mechanical model is used to control the movement of the drone, converting the action values ​​output by the agent into forces, i.e. Assuming the initial velocity of the drone is After one round of action output, the drone's speed value is... ,in Indicates the mass parameters of the drone. This represents the length of a time step; to reduce computational complexity, the drone... XY The position in the plane is updated to The reward function is defined in the following form: ， (2) in and These are the coverage and connectivity metrics for the current time step, respectively. and These are parameters that can be adjusted manually; This represents the energy consumption increment of the current time step compared to the previous time step; the numerator of the reward function represents the total reward, including coverage gain and connectivity penalty, while the denominator represents the total energy consumption; the 3D dynamic coverage method based on deep deterministic policy gradient adopts a framework of centralized training and decentralized execution, where the policy information of the remaining agents is used as additional information for training the critic network, but this information is not used in the execution phase of the agents; a four-layer fully connected network is used as the network model for critics and actors, where the number of neurons used in each layer is 64, 64, 128, and 256, respectively, and the ReLU function is used as the activation function; The strategy of each agent is represented as Based on the idea of ​​deterministic strategies, the agent's expected return... The gradient is expressed as: , (3) in It is a lumped-action-value function, i.e., a model of a commentator network; the function takes into account the action information of all agents and some state information. x Output the first i Agent Q value; x It consists of the observation information of all agents, that is Experience replay buffer Contains tuples ; Update using the following loss function: , (4) , (5) in Indicates a delay parameter The target policy set; all drone commentator networks are trained in a centralized manner, while each drone actor network is updated using only its own observation information; at each time step, each drone reacts to the environment based on its own observation information and policy model, outputs the corresponding action value and executes the action, then updates the state information, and repeats the previous steps in the next time step until all time steps are completed or the target area is covered.

2. The method according to claim 1, characterized in that, The performance metric to be optimized in the 3D dynamic coverage method based on depth deterministic policy gradient is defined as follows: Three performance metrics to be optimized are defined: coverage, energy consumption, and connectivity. Coverage is defined as the ratio of the number of covered points of interest to the total number of points of interest. The definition stipulates that when a point of interest falls within the coverage area of ​​a drone, it is considered covered. The coverage area of ​​the drone on the surface of the three-dimensional region is defined as having a radius centered on the projection of the drone onto the region's surface. r The circle, r The value can be determined based on the drone's hardware and flight altitude. During flight, all drones maintain a fixed altitude difference with the area surface. ; Considering only the energy consumption during the drone's flight, a simple linear relationship is established between the energy consumption and the drone's flight distance, i.e. Energy consumption increases linearly with flight distance; using This represents a drone swarm network, where This refers to a collection of drones, i.e. , This indicates the total number of drones. This indicates the connectivity between drones, specifying when two drones... , The distance between them is less than the communication range of the drone. R At that time, drones can communicate with each other, as shown in the following diagram. When any two drones in the system can establish a communication link, the entire system is fully connected, represented as... Otherwise ; Its goal is to maximize coverage while minimizing the total energy consumption of the drones, and to ensure the connectivity of the entire system as much as possible.

Images