Multi-unmanned aerial vehicle navigation collaborative environment monitoring implementation method based on deep reinforcement learning
By combining deep reinforcement learning and dynamic programming, the path planning of UAVs is optimized, which solves the problem of data collection efficiency of UAVs under limited energy and random node distribution, and realizes efficient and stable data collection and energy management.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANJING UNIV OF POSTS & TELECOMM
- Filing Date
- 2026-01-28
- Publication Date
- 2026-05-08
AI Technical Summary
Existing technologies struggle to efficiently collect data in UAV path planning under conditions of limited energy and random node distribution. Furthermore, traditional algorithms suffer from high computational complexity or insufficient generalization, making them ill-suited for flexible and ever-changing environments.
A two-stage optimization method combining deep reinforcement learning and dynamic programming is adopted. By generating training data, planning paths, and allocating energy constraints, the path of the UAV is optimized to maximize data collection efficiency and minimize energy consumption. The path planning and task allocation are performed by combining deep reinforcement learning models and dynamic programming.
Under energy constraints, the data collection rate and information freshness of UAVs are improved, the generalization of the algorithm and the stability of path planning are enhanced, and it can adapt to flexible and ever-changing environments.
Smart Images
Figure CN121995935A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of unmanned aerial vehicle (UAV) navigation application technology, and more specifically, to a method for multi-UAV navigation collaborative environment monitoring based on deep reinforcement learning. Background Technology
[0002] With the rapid development of IoT technology, sensor nodes have been widely used in environmental monitoring, agricultural management, urban planning, and smart homes. These sensor nodes collect environmental data and provide real-time information to support various smart applications. Drones, due to their flexibility and maneuverability, have become an important tool for IoT node data collection. Drones can cover wide areas and quickly access and collect data from dispersed sensor nodes. However, due to the randomness of node data generation and the limited energy of drones, how to efficiently complete assigned tasks under various constraints is of significant research importance. Specifically, optimizing drone path planning to maximize data collection efficiency and minimize energy consumption remains a pressing issue.
[0003] Traditional path planning algorithms mainly include exhaustive search, dynamic programming, nearest neighbor algorithm, greedy algorithm, and quadratic optimization method. These algorithms have achieved some success in solving the Traveling Salesman Problem, but they have many shortcomings in practical applications. While exhaustive search and dynamic programming can find the optimal solution, their high computational complexity makes them unsuitable for large-scale problems. Approximate and heuristic algorithms, such as nearest neighbor and greedy algorithms, are computationally efficient, but they often fail to guarantee the optimality of the solution and are prone to getting trapped in local optima.
[0004] To improve solution efficiency and quality, advanced heuristic algorithms such as genetic algorithms, simulated annealing, and ant colony optimization have been proposed. Genetic algorithms generate new solutions by simulating natural selection and genetic mechanisms, are suitable for large-scale problems, and possess global search capabilities. Simulated annealing algorithms gradually reduce randomness by simulating the physical annealing process, escaping local optima, and have a wide search space. Ant colony optimization simulates ant foraging behavior, finding the optimal path through pheromone transmission and updates, and is suitable for dynamic optimization problems. These advanced heuristic algorithms improve path planning performance to some extent, but they have long computation times, complex parameter settings, and are difficult to apply in real-time for multi-UAV path planning.
[0005] In recent years, with the development of deep learning and reinforcement learning technologies, path planning methods based on deep reinforcement learning have gradually become a research hotspot. Centralized multi-UAV deep reinforcement learning, centralized training and distributed execution multi-agent reinforcement learning, and graph neural networks have been applied to path planning problems, demonstrating good performance by adjusting path planning strategies in real time in dynamic and complex environments. However, most of these methods directly learn policies in the joint action space or rely on complex multi-agent cooperative mechanisms, resulting in insufficient generalization ability in flexible and varied environments. Summary of the Invention
[0006] To address the aforementioned problems in existing technologies, the purpose of this invention is to provide a method for solving the path planning problem for multi-UAV data collection. This method can provide an effective and stable solution based on the random node distribution, maximizing the data collection rate of UAVs under energy constraints while ensuring information freshness. The purpose of this invention is to provide such a method.
[0007] To address the aforementioned problems, the technical solution adopted in this invention is as follows: A method for multi-UAV navigation collaborative environment monitoring based on deep reinforcement learning, comprising the following steps: Step 1, acquiring training data: uniformly distributing UAV network nodes, and generating the required amount of training data within a custom-sized interval using a random generation method; Step 2, training deep reinforcement learning based on the training data: inputting the training data into a deep reinforcement learning model in the form of a spatial set, the deep reinforcement learning model training itself based on the training data, the neural network being used to approximate the policy function, obtaining the UAV's path planning policy and probability distribution, wherein the training data is represented as a state space, action space, and corresponding reward function space, the state space being represented as S = (node weight, AOI of IoT node, UAV position, node to be visited, node round), the action space being represented as a = (UAV movement direction, next visited node), and the reward function being represented as r = total path length; Step 3, inputting the data of each round into the deep reinforcement learning model, the model outputting the planned route. As shown in formula (1):
[0008] Formula (1) In the above formula, The representative model M is given by the input dataset as The parameters are At that time, the obtained strategy and its probability distribution The path length is the path length under the corresponding strategy; Step 4, calculate the loss function value of the total path, and formula (2) represents the total path: Formula (2); Step 5: X(t) is the original uniformly distributed dataset. After each round of data training, the training data is updated by gradient descent to obtain a dataset with a more complex distribution. This dataset is then mixed with the original dataset and used as the input data in Step 2.
[0009] In the above formula, M represents the model being trained. As the baseline model, The model M outputs the policy probability distribution, and T is the training level control parameter, determined by the training progress. Step 6: IoT nodes randomly generate data using a Poisson process. The drone adds nodes with data to the list of nodes to be visited and inputs this data into the deep reinforcement learning model to obtain the path planning scheme.
[0010] Step 7, after obtaining the high-quality global flight path of the UAV. Subsequently, a multi-UAV task allocation method based on dynamic programming was introduced, which intelligently divides the path among multiple UAVs according to the energy constraints of the UAVs.
[0011] Further, in step 2: the data is input into the deep reinforcement learning model, and the model outputs the path planning strategy and probability distribution of the drone; Step 2.1: The data is input into the neural network, which is used to approximate the policy function and outputs the probability of choosing action a in state S; Step 2.2: The reinforcement learning model includes a state space, an action space, and a reward function; the state space is S = (node weight, AOI of the IoT node, drone position, node to be visited, node round), the action space is a = (drone movement direction, next node to be visited), and the reward function is r = total path length, outputting the reward for action a.
[0012] Further, step 7 includes the following steps: Step 7.1, each node contains its location information, current cached data volume, and corresponding AOI status. Taking into account the flight energy consumption, data acquisition and transmission energy of the UAV when performing the corresponding task segment, a continuous sub-path is calculated. Energy consumption is ; Step 7.2, during the state transition process, state DP(l,k) represents the state DP(l,k) that uses the first k UAVs to complete the previous task. The minimum cumulative cost when a node accesses a task, and the corresponding state transition relationship can be expressed as:
[0013] Step 7.3: After all node access tasks have been considered, the system selects the appropriate node to access, provided that the maximum number of drones is not exceeded. The minimum state is taken as the final solution, and the optimal node access sub-path for each drone is determined by backtracking the dynamic programming process.
[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) The present invention optimizes the path planning of UAVs to maximize the data collection efficiency of UAVs and minimize the energy consumption of UAVs; (2) Under energy constraints, the present invention maximizes the data collection rate of the UAV while ensuring the freshness of the information; (3) The path planning problem solution method of the present invention for multi-UAV data collection can provide an effective and stable solution based on the random node distribution. (4) Based on the complex multi-agent cooperative mechanism of the present invention, the algorithm has strong generalization ability in flexible and ever-changing environments. Attached Figure Description
[0015] Figure 1 This is a schematic diagram of the algorithm flowchart; Figure 2 This is a schematic diagram of the algorithm's convergence. Figure 3 This is a path diagram; Figure 4 This is a comparison chart of average AOI. Figure 5 This is a comparison chart of effective data collection rates. Detailed Implementation
[0016] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.
[0017] Example 1 like Figure 1 As shown, the multi-UAV navigation cooperative environment monitoring method proposed in this invention based on deep reinforcement learning includes the following steps: Step 1, Obtain Training Data: Uniformly distribute drone network nodes and generate the required amount of training data within a custom-sized interval using a random generation method; Step 2, Train Deep Reinforcement Learning Based on Training Data: Input the training data into the deep reinforcement learning model in the form of a spatial set. The deep reinforcement learning model trains itself based on the training data. The neural network is used to approximate the policy function to obtain the drone's path planning policy and probability distribution. The training data is represented as a state space, action space, and corresponding reward function space. The state space is represented as S = (node weight, AOI of IoT node, drone position, node to be visited, node round), the action space is represented as a = (drone movement direction, next node to be visited), and the reward function is represented as r = total path length. The output is the probability of choosing action a in state S. The action is determined based on the probability value, and the model outputs the drone's path planning policy and probability distribution. Step 3: Input the data from each round into the deep reinforcement learning model, and the model outputs the planned route. As shown in formula (1): Formula (1) In the above formula, The representative model M is given by the input dataset as The parameters are At that time, the obtained strategy and its probability distribution This represents the path length under the corresponding strategy. ; Step 4: Calculate the loss function value of the total path. After backpropagation, optimize the deep reinforcement learning model using stochastic gradient descent. Equation (2) represents the total path: Formula (2) Step 5: X(t) is the original uniformly distributed dataset. After each round of training, the training data is updated using gradient descent to obtain a more complex dataset. This dataset is then mixed with the original dataset and used as the input data in Step 2.
[0018] In the above formula, M represents the model being trained. As the baseline model, It is the policy probability distribution output by model M, and T is the training level control parameter, which is determined by the training progress; Step 6: The IoT nodes randomly generate data using a Poisson process. The drone adds nodes with data to the list of nodes to be visited and inputs this data into a deep reinforcement learning model to obtain a path planning scheme.
[0019] Step 7, after obtaining the high-quality global flight path of the UAV. Subsequently, a multi-UAV task allocation method based on dynamic programming was introduced, which intelligently divides the path among multiple UAVs according to the energy constraints of the UAVs.
[0020] Step 7.1: Each node contains its location information, current cached data volume, and corresponding AOI status. Taking into account the UAV's flight energy consumption, data acquisition and transmission energy during the execution of the corresponding task segment, a continuous sub-path is calculated. Energy consumption is .
[0021] Step 7.2: During the state transition process, state DP(l,k) represents the completion of the previous task using the first k UAVs. The minimum cumulative cost when a node accesses a task, and the corresponding state transition relationship can be expressed as:
[0022] Step 7.3: After all node access tasks have been considered, the system selects the appropriate node to access, provided that the maximum number of drones is not exceeded. The minimum state is taken as the final solution, and the optimal node access sub-path for each drone is determined by backtracking the dynamic programming process.
[0023] Example 2 Traditional UAV path planning schemes mostly consider the case where IoT node locations are fixed. However, in environmental monitoring systems, each IoT node does not generate data regularly. Therefore, when collecting data, UAVs only need to consider IoT nodes that have data. In this case, the distribution of IoT nodes is randomly changing. Traditional methods do not consider the robustness of the algorithm, and cannot derive a stable path solution when encountering situations where the node distribution is constantly changing. The main objective of this invention is to provide a path planning method for multi-UAV environmental monitoring data collection, which helps to increase the amount of IoT node data collected by UAVs and reduce the information AOI (Area of Interest).
[0024] This invention proposes a two-stage joint optimization method combining deep reinforcement learning and dynamic programming, aiming to achieve efficient and scalable data acquisition scheduling under multiple practical conditions, including random node data generation, limited information freshness, and UAV energy constraints. Considering the randomness of IoT node data generation, an adaptive path planning algorithm is first used to address the challenge of constantly changing node distribution. Then, dynamic programming is used to partition the access sequence and allocate tasks among multiple UAVs, comprehensively optimizing energy consumption and information freshness performance while strictly satisfying single-UAV energy constraints. Compared to schemes that rely solely on deep reinforcement learning for end-to-end decision-making, the proposed joint optimization framework decouples "path structure learning" from "task executability assurance," significantly improving the system's effective energy utilization and multi-UAV load balancing performance while ensuring AOI constraints. This provides a feasible and engineering-valued solution for practical deployment in dynamic low-altitude IoT scenarios.
[0025] Reference Figure 2 and Figure 3 Assuming the maximum drone speed is For each IoT node, data acquisition and upload times are assumed to be the same. N=100 nodes are distributed within a 1km × 1km area, with one communication base station deployed and a communication radius R = 100. The node data generation process follows a Poisson distribution, with a packet arrival rate of λ = 0.019 packets / s. The generated data is first stored in the node's local cache queue, awaiting access and collection by the drone. (Refer to...) Figure 4 and Figure 5 This indicates that the method of the present invention has a relatively high average AOI and effective data collection rate.
[0026] Step 1: Generate the location coordinates of 100 nodes in the interval [0,1000] using a random method, and generate the corresponding node data arrival sequence according to the Poisson arrival model.
[0027] Step 2: Input the current system state into the deep reinforcement learning model, and the model outputs the UAV's path planning strategy Π and probability P.
[0028] Step 2.1: The system state S is input into the neural network, which is used to approximate the policy function. It outputs the action to be selected in the current state. The probability distribution. Status information includes the drone's current location, remaining energy, node locations, node cache status, and their AOI values.
[0029] Step 2.2: Based on the state Choose the action The state space, action space, and reward function of the reinforcement learning model are updated. The node AOI starts from 0 and increases linearly over time. The drone starts from the charging station location by default and moves in the directions of east, south, west, north, and their diagonals.
[0030] Step 3: Based on the generated strategy, for nodes that have successfully collected data, their cache queues are cleared, the corresponding AOIs are reset, and the AOIs of unaccessed nodes continue to accumulate over time.
[0031] Step 4: Calculate the loss function value of the total path. After backpropagation, use the policy gradient method to update the parameters of the deep reinforcement learning model.
[0032] Step 5: After each round of data training, update the training data using gradient descent to obtain a more complex dataset. Mix this dataset with the original dataset and use it as the input data in Step 2. Use w as the weight for each dataset; the higher the weight value of the complex dataset, the more difficult the training data and the more complex its distribution.
[0033] Step 6: After completing model training, a distributed dataset of 100 nodes is randomly generated. Each node generates data packets randomly according to a Poisson process. Before each round of node visits, the drone adds nodes with data to the list to be visited, and uses the amount of data packets generated by each node as a weight value, inputting it into the deep reinforcement learning model to obtain a path planning scheme.
[0034] Step 7: Under the premise of satisfying the energy constraints of the drones, reasonably allocate the path to multiple drones for execution.
[0035] Step 7.1: Represent the path output by the deep reinforcement learning model as an ordered sequence of node visits. Each node in the sequence contains its location information, current data cache state, and corresponding AOI value. Calculate the flight energy and data collection energy required to visit adjacent nodes.
[0036] Step 7.2: Taking into account factors such as UAV energy consumption and node data timeliness, compare all feasible task partitioning schemes and select the scheme with the lowest overall cost as the optimal result.
[0037] Step 7.3: Based on the results of dynamic programming, assign a corresponding node access sub-path to each UAV. Each UAV then executes its own task in parallel, completing the node data collection and uploading operations sequentially according to the assigned access order.
Claims
1. A method for multi-UAV navigation cooperative environment monitoring based on deep reinforcement learning, characterized in that, The method includes the following steps: Step 1, acquiring training data: Uniformly distribute UAV network nodes and generate the required amount of training data within a custom-sized interval using a random generation method; Step 2, deep reinforcement learning training based on the training data: Input the training data into the deep reinforcement learning model in the form of a spatial set. The deep reinforcement learning model is trained based on the training data. The neural network is used to approximate the policy function to obtain the UAV's path planning policy and probability distribution. The training data is represented as a state space, action space, and corresponding reward function space. The state space is represented as S = (node weight, AOI of the IoT node, UAV position, node to be visited, node round), the action space is represented as a = (UAV movement direction, next visited node), and the reward function is represented as r = total path length; Step 3, input the data of each round into the deep reinforcement learning model, and the model outputs the planned route. As shown in formula (1): Official (1) In the above formula, The representative model M is given by the input dataset as The parameters are At that time, the obtained strategy and its probability distribution The path length is the path length under the corresponding strategy; Step 4, calculate the loss function value of the total path, and formula (2) represents the total path: Formula (2); Step 5: X(t) is the original uniformly distributed dataset. After each round of data training, the training data is updated by gradient descent to obtain a dataset with a more complex distribution. This dataset is then mixed with the original dataset and used as the input data in Step 2. In the above formula, M represents the model being trained. As the baseline model, The model M outputs the policy probability distribution, and T is the training level control parameter, determined by the training progress. Step 6: IoT nodes randomly generate data using a Poisson process. The drone adds nodes with data to the list of nodes to be visited and inputs this data into the deep reinforcement learning model to obtain the path planning scheme. Step 7, after obtaining the high-quality global flight path of the UAV. Subsequently, a multi-UAV task allocation method based on dynamic programming was introduced, which intelligently divides the path among multiple UAVs according to the energy constraints of the UAVs.
2. The method for multi-UAV navigation cooperative environment monitoring based on deep reinforcement learning according to claim 1, characterized in that, Step 2: Input the data into the deep reinforcement learning model, and the model outputs the drone's path planning strategy and probability distribution; Step 2.1: The data is input into the neural network, which is used to approximate the policy function and outputs the probability of choosing action a in state S; Step 2.2: The reinforcement learning model includes a state space, an action space, and a reward function; the state space is S = (node weight, AOI of the IoT node, drone position, node to be visited, node round), the action space is a = (drone movement direction, next node to be visited), and the reward function is r = total path length, outputting the reward for action a.
3. The method for multi-UAV navigation cooperative environment monitoring based on deep reinforcement learning according to claim 1, characterized in that, Step 7 includes the following steps: Step 7.1, each node contains its location information, current cached data volume, and corresponding AOI status. Taking into account the flight energy consumption, data acquisition and transmission energy of the UAV when performing the corresponding task segment, a continuous sub-path is calculated. Energy consumption is ; Step 7.2, during the state transition process, state DP(l,k) represents the state DP(l,k) that uses the first k UAVs to complete the previous task. The minimum cumulative cost when a node accesses a task, and the corresponding state transition relationship can be expressed as: Step 7.3: After all node access tasks have been considered, the system selects the appropriate node to access, provided that the maximum number of drones is not exceeded. The minimum state is taken as the final solution, and the optimal node access sub-path for each drone is determined by backtracking the dynamic programming process.