A Trajectory Planning Method for Multi-UAV Sensor Data Collection Based on Deep Reinforcement Learning

Abstract

This invention provides a multi-UAV sensor data acquisition trajectory planning method based on deep reinforcement learning, comprising the following steps: S1, establishing the basic framework of a UAV-assisted sensor data acquisition model in a land scenario; S2, constructing a persistent trajectory planning strategy based on an improved deep reinforcement learning algorithm according to the basic framework, and generating a UAV flight trajectory map that satisfies energy balance constraints and maximizes data acquisition value based on velocity vectors. This invention introduces a feature extraction unit module containing self-attention weight calculation to extract spatiotemporal differential features of sensor observations based on traditional multi-agent deep reinforcement learning algorithms, and combines a packet retransmission mechanism and an energy balance coefficient to optimize UAV flight energy consumption while improving the value and reliability of data acquisition. This system improves the processing efficiency of UAVs for dynamic monitoring tasks.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) communication and trajectory planning technology, and more particularly to a multi-UAV sensor data collection trajectory planning method based on deep reinforcement learning. Background Technology

[0002] In recent years, the rapid development of drone technology and the rise of IoT sensing and monitoring tasks have brought tremendous opportunities and challenges to various industries. As an aircraft with autonomous flight capabilities, drones possess advantages such as flexibility, efficiency, and deployability, making them a research hotspot in the field of IoT data acquisition. By using drones as mobile base stations or data collectors, efficient coverage of sensor nodes in remote areas and complex land environments can be achieved. The rise of this technology provides powerful data backhaul capabilities for large-scale sensor networks, enabling them to operate in more complex and intelligent environments. Therefore, utilizing drones to assist in sensor data acquisition has become an important means to improve the timeliness and coverage of monitoring systems. This is of great significance for the application of drones in precision agriculture, ecological monitoring, infrastructure inspection, disaster early warning, and other fields.

[0003] However, traditional methods of periodic manual inspections or vehicle-based data collection suffer from drawbacks such as long task cycles, low response efficiency, and high labor costs when dealing with large-scale monitoring areas. These methods are ill-suited for scenarios with extremely high real-time requirements, such as wildfire warnings and sudden environmental pollution events. Because the physical quantities collected by sensor nodes exhibit dynamic and non-linear changes over time, the value of the data is significantly time-sensitive. If the data is not collected and processed within its prime timeframe, it loses its scientific value in supporting decision-making. Therefore, constructing a highly mobile, robust, and intelligently value-identifying mobile data acquisition system has become a key bottleneck restricting the further development of precision monitoring.

[0004] Traditional path planning methods, such as A Algorithms such as Dijkstra's algorithm or Myopic's algorithm primarily find the shortest path by searching graph nodes. However, in large-scale sensor networks and high-dimensional state spaces, these algorithms suffer from problems such as an exponential increase in search space, high computational complexity, and difficulty in perceiving the time-varying value of data. This causes UAVs to frequently wander ineffectively in areas of stable data, making it impossible to efficiently collect data in regions of abrupt change.

[0005] Compared to traditional methods, Deep Reinforcement Learning (DRL) optimizes strategies through environmental interaction and reward / penalty mechanisms, demonstrating stronger adaptability and path improvement efficiency. However, basic DRL algorithms have significant drawbacks in complex multi-UAV scenarios: traditional network architectures struggle to capture the spatiotemporal correlation features between distant sensors, resulting in weak global perception capabilities; and uniform random sampling mechanisms cannot effectively extract sparse, high-value samples, leading to slow convergence and difficulty in achieving accurate data collection within limited energy consumption. Summary of the Invention

[0006] In view of this, the purpose of this invention is to propose a multi-UAV sensor data acquisition trajectory planning method based on deep reinforcement learning, so as to solve the technical problem that existing traditional network architectures are unable to capture the spatiotemporal correlation features between long-distance sensors.

[0007] The technical means employed in this invention are as follows:

[0008] A multi-UAV sensor data acquisition trajectory planning method based on deep reinforcement learning includes the following steps: S1. Establish the basic framework of the UAV data acquisition system model in a land scenario; S2. Based on the basic framework, a continuous trajectory planning strategy based on an improved deep reinforcement learning algorithm is constructed. The continuous trajectory planning strategy includes a feature extraction unit that calculates self-attention weights and a multi-agent deep reinforcement learning decision network based on a weighted sampling mechanism for sample prediction errors. The continuous trajectory planning strategy aims to maximize the sum of the cumulative information differences collected. It takes the real-time environmental state as input, trains under energy balance constraints, and outputs the velocity vectors of each UAV in each time slot. The flight decisions of the UAVs are adjusted according to the energy consumption of each UAV during the data collection process until the remaining energy of the UAVs reaches the safe threshold for returning to home. Based on the velocity vectors, a UAV flight trajectory map that satisfies the energy balance constraints and maximizes the value of data collection is generated.

[0009] Furthermore, S1 specifically includes the following steps: S11. Constructing an information difference model for sensor observations: Using temporal differential logic, the current observations of sensor nodes are acquired in real time and compared with historical observations from the previous moment. The difference between the two is calculated, and an information difference function is constructed based on this difference to characterize the acquisition value of the node's current data. The formula for calculating the information difference function is as follows:

[0010] in, For sensors exist The observation value at a given moment, the information difference degree is used to characterize the value of the data collected at that node, and is used as a state input or reward feedback parameter in the UAV trajectory planning process to guide the UAV's flight decision; S12. Considering terrain occlusion characteristics, the probability of line-of-sight and non-line-of-sight links occurring. Angle of elevation of the drone relative to the sensor Related; Total path loss The expected value of path loss under different link states is used to characterize the loss, and the calculation formula is as follows:

[0011] in, For line-of-sight communication path loss, This represents the path loss for non-line-of-sight communication. Introducing a packet-segmented transmission protocol and an automatic retransmission request mechanism: defining the probability of successful transmission of a single data packet. Here is a nonlinear mapping function with respect to the signal-to-interference-plus-noise ratio (SINR), used to simulate the random packet loss and retransmission logic in UAV-sensor communication. The formula is as follows:

[0012] in, The threshold center point for the probability of success For reliability sensitivity; S13, UAV flight power Based on the aerodynamic characteristics of UAVs, the specific formula is as follows:

[0013] in, For the drone's flight speed, and The power coefficient is a constant. By monitoring the remaining battery power in real time and combining the straight-line distance from the current location to the starting point with the return-to-home power requirement, the forced return-to-home power threshold is calculated in real time. When the remaining power is less than or equal to the forced return-to-home power threshold, the drone executes the shortest path return-to-home action.

[0014] Furthermore, S2 specifically includes the following steps: S21. Obtain the coordinate information of the current position of the UAV, the position of each sensor node and the channel state information as input, restrict the activity area of ​​the UAV, use the self-attention mechanism of the feature extraction unit module including self-attention weight calculation to extract the correlation weight between state sequences, identify the time-varying pattern of data value, and output a high-dimensional feature vector representing the scene features and the UAV flight speed vector. S22. Define a multidimensional constraint reward function to guide the UAV to move towards a region with high information difference, and introduce a priority experience replay mechanism to perform weighted sampling of high-value training samples based on the sample prediction error. Output the UAV velocity vector through an improved multi-agent deep reinforcement learning decision network algorithm. S23. The UAV performs data acquisition actions and updates sensor observations. It calculates the estimated energy consumption of the UAV returning to the starting point in real time. If the remaining energy is higher than the return threshold, it continues to perform repeated acquisition tasks. Otherwise, it returns to the starting point according to the shortest path. After the UAV enters the return state, it is removed from the subsequent trajectory planning and data acquisition decision. Finally, a complete trajectory planning scheme containing the coordinate sequence of each UAV is output.

[0015] Furthermore, S21 specifically includes the following steps: S211. Construct a feature extraction network based on Transformer unit modules. Input the temporal state sequences of the UAV and sensors into the feature extraction module with a self-attention mechanism, and calculate the correlation weights between state features. The calculation formula is as follows:

[0016] in, For querying the matrix, The key matrix, For value matrices, This is a similarity matrix. As a scaling factor, through Normalized weights are used, and the feature extraction network serves as the input layer of the improved multi-agent deep reinforcement learning decision network algorithm, outputting a high-dimensional feature vector representing scene features to achieve multi-agent collaborative decision-making. S212. Superimpose the velocity vector of the next time slot onto the initial coordinates of the UAV to obtain the temporary coordinates of the UAV. It is stipulated that the UAV shall not fly beyond the boundary. Iterate through the actions of the UAV and check whether the UAV has overstepped the boundary. If overstepping the boundary is found, correct the trajectory and impose a penalty, and re-input the action until the UAV learns to fly in the designated area. If there is no overstepping the boundary, output the temporary coordinates of the UAV at this time as the current coordinates of the UAV.

[0017] Furthermore, S22 specifically includes the following steps: S221. Define a comprehensive reward function with multidimensional constraints. The reward function consists of a UAV collision penalty, an out-of-bounds penalty, and a positive feedback reward consisting of the product of the energy balance coefficient and the difference in the collected information. S222. The energy balance coefficient is used to measure the degree of energy consumption balance among multiple sensors during data acquisition. It measures the degree of data acquisition balance by calculating the ratio of the square of the sum of energy consumed by each sensor node to the sum of the squares of all energy. This guides the drone swarm to make the coefficient approach 1, thereby suppressing the drones from wandering in local areas. The formula for the energy balance coefficient is as follows:

[0018] in, For sensors The energy consumed from the start of the system; S223. Establish a probabilistic channel model for land scenarios, comprehensively considering line-of-sight and non-line-of-sight links, and calculate the path loss formula for different link states as follows:

[0019] in, For line-of-sight communication path loss, The probability of a line-of-sight link occurring. For non-line-of-sight communication path loss, Given the probability of non-line-of-sight links occurring, the reward value for the current state is calculated using the reward function and the channel model, and is used to update the policy network.

[0020] Furthermore, S23 specifically includes the following steps: S231. The UAV obtains a positive reward after each successful transmission of a data packet using the packet retransmission mechanism, and resets the cumulative information difference of the sensor after all data packets have been successfully transmitted. The retransmission probability is calculated as follows:

[0021] in, The center point of the success probability threshold. For signal-to-interference-to-noise ratio, To improve reliability sensitivity, the transmission success probability mapped by the signal-to-interference-plus-noise ratio is used in conjunction with a random sampling discrimination mechanism to simulate the packet retransmission process, thereby providing differentiated reward signals based on the collected results. S232. If the drone's current remaining battery power and estimated energy consumption for returning to the starting point are monitored in real time, the drone will be forced to terminate the data collection task and return to the starting point, generating the shortest path return trajectory from the current position to the starting point.

[0022] Furthermore, the UAV data acquisition system model described in S1 is a network composed of multiple UAV nodes, each of which can receive and store data; all UAVs start from the same point and repeatedly collect data from sensors distributed in the land scene.

[0023] Furthermore, the improved multi-agent deep reinforcement learning decision network algorithm architecture includes: Using MADDPG as the basic algorithm framework, the feature extraction unit module with self-attention weight calculation described in S211 is integrated to process the observation sequence. The comprehensive reward function described in S221 is used to guide the policy update. The policy and value function of the agent are approximated by a deep neural network. The system is trained by combining priority experience replay and target network to achieve multi-agent collaborative decision-making and obtain the velocity vector of the UAV in the next time slot.

[0024] Furthermore, the improved MADDPG algorithm process includes the following: Define the state space and action space of the drone in the terrestrial environment. The action space is the velocity vector of the drone's movement, and the state space includes the drone's real-time coordinates, the position of each sensor node, and the drone's current remaining battery power. The reward function for the drone is determined, which consists of four parts: negative feedback reward for drone collisions, negative feedback reward for exceeding the boundary, positive feedback reward for the product of energy balance coefficient and the difference in collected information, with the boundary penalty assigned the highest weight coefficient. The agent is trained using deep reinforcement learning algorithms. During training, the agent continuously learns and optimizes its path planning strategy by interacting with a simulated land environment in order to collect the most valuable information before the battery runs out.

[0025] Furthermore, the improved MADDPG algorithm includes a priority experience replay mechanism, the specific steps of which are as follows: The priority of each sample is calculated using the TD error. Samples with a higher prediction error than the current experience pool average are assigned a sampling weight higher than the experience pool average to accelerate the network's learning of high-value scenarios such as drastic changes in information variability. The priority probability distribution formula is as follows:

[0026] in, For TD error, To control the degree of priority, To prevent extremely small positive numbers with an error of 0; The update step size is adjusted by using importance sampling weights to ensure the stability and convergence of the training process.

[0027] Compared with the prior art, the present invention has the following advantages: The TP-MADDPG algorithm provided by this invention extracts the spatiotemporal differential features of sensor observations by introducing a Transformer module, enabling UAVs to accurately perceive information value and effectively solving the problem of poor data acquisition timeliness in traditional methods. Simultaneously, by combining a probabilistic channel model and a packet retransmission mechanism, it significantly improves communication reliability in complex land environments. Furthermore, by introducing an energy balance coefficient into the reward function and setting a return-to-home safety threshold, it not only achieves load balancing under multi-UAV collaboration but also ensures optimal system energy consumption and continuous task execution. Finally, the priority experience replay mechanism accelerates the algorithm's convergence process, giving the system higher processing efficiency and robustness in dynamic monitoring tasks. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a framework diagram of the method of the present invention.

[0030] Figure 2 This is a flowchart illustrating the logic of the method of the present invention.

[0031] Figure 3 This is a diagram of the algorithm framework of the present invention.

[0032] Figure 4 The image shows a two-dimensional trajectory of a drone when there are 2 drones and 30 sensors.

[0033] Figure 5 Comparison chart of system reward value algorithms when the number of drones is 2 and the number of sensors is 30.

[0034] Figure 6 This is a comparison chart of algorithm results for information difference under different numbers of drones when the number of sensors is 30.

[0035] Figure 7 This is a comparison chart of the algorithm results for the energy balance coefficient under different numbers of drones when the number of sensors is 30.

[0036] Figure 8 This is a comparison chart of the algorithm results for transmission success rate under different numbers of drones when the number of sensors is 30. Detailed Implementation

[0037] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0038] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0039] like Figure 1 and 2 As shown, this invention provides a multi-UAV intelligent path planning method based on deep reinforcement learning, comprising the following steps: S1. Establish the basic framework of the UAV data acquisition system model in the land scenario. The data collection model consists of multiple UAVs, each of which can transmit data. All UAVs start from the same point to collect sensor data, and the positions of multiple sensors are fixed.

[0040] S11. Constructing an information difference model for sensor observations: Using time-series differential logic, the current observations of sensor nodes are acquired in real time and compared with historical observations from the previous moment. The difference between the two is calculated, and an information difference function is constructed based on this difference to characterize the acquisition value of the node's current data. The calculation formula is as follows:

[0041] in, For sensors exist The observation value at a given moment, the information difference degree is used to characterize the value of the data collected at that node, and is used as a state input or reward feedback parameter in the UAV trajectory planning process to guide the UAV's flight decision; S12. Considering terrain occlusion characteristics and taking into account both line-of-sight and non-line-of-sight links, its occurrence probability... Angle of elevation of the drone relative to the sensor Related. Total path loss The expected value of path loss under different link states is used to characterize the loss, and the calculation formula is as follows:

[0042] in, For line-of-sight communication path loss, This addresses path loss in non-line-of-sight communication. A packet-segmentation protocol and an automatic retransmission request mechanism are introduced. The probability of successful transmission of a single data packet is defined. Here is a nonlinear mapping function with respect to the signal-to-interference-plus-noise ratio (SINR), used to simulate the random packet loss and retransmission logic in UAV-sensor communication. The formula is as follows:

[0043] in, The threshold center point for the probability of success For reliability sensitivity; S13, UAV flight power Based on the aerodynamic characteristics of UAVs, the specific formula is as follows:

[0044] The above formula comprehensively considers the induced power, form drag power, and parasitic power of the rotorcraft UAV, where For the drone's flight speed, and The power coefficient is a constant. The system monitors the remaining battery power in real time and calculates the forced return power threshold in real time by combining the straight-line distance from the current position to the starting point and the return power requirement. When the remaining power is less than or equal to the forced return power threshold, the drone executes the shortest path return maneuver.

[0045] S2. Generate a trajectory planning strategy based on the basic framework, and construct a continuous trajectory planning strategy based on an improved deep reinforcement learning algorithm. The optimization strategy utilizes a deep reinforcement learning framework that introduces a Transformer module and priority experience replay to maximize the sum of the cumulative information differences collected. Under the constraint of energy balance, the flight trajectory of the UAV is planned. The flight decision of the UAV is adjusted according to the energy consumption of each UAV during the data collection process until the remaining energy of the UAV reaches the safe threshold for returning to home.

[0046] S21. Obtain the coordinate information of the current position of the UAV, the position of each sensor node and the channel state information as input, restrict the activity area of ​​the UAV, use the self-attention mechanism of the Transformer module to extract the correlation weight between state sequences, identify the time-varying pattern of data value, and output a high-dimensional feature vector representing the scene features and the UAV flight speed vector. S211. Construct a feature extraction network based on the Transformer module. Input the temporal state sequences of the UAV and sensors into the feature extraction module with a self-attention mechanism, and calculate the correlation weights between state features. The calculation formula is as follows:

[0047] in, For querying the matrix, The key matrix, For value matrices, This is a similarity matrix. As a scaling factor, through Normalized weights are used, and the feature extraction network serves as the input layer of the improved multi-agent deep reinforcement learning decision network algorithm, outputting a high-dimensional feature vector representing scene features to achieve multi-agent collaborative decision-making. The improved architecture of the multi-agent deep reinforcement learning algorithm includes: using MADDPG as the basic algorithm framework, integrating a Transformer feature extraction module to process the observation sequence, and using a comprehensive reward function to guide policy updates; approximating the agent's policy and value function through a deep neural network, and combining priority experience replay and target network for training to achieve multi-agent collaborative decision-making and obtain the velocity vector of the UAV's next time slot flight.

[0048] The improved MADDPG algorithm process includes the following steps: Define the state space and action space of the drone in the terrestrial environment. The action space is the velocity vector of the drone's movement, and the state space includes the drone's real-time coordinates, the position of each sensor node, and the drone's current remaining battery power. The reward function for the drone is determined, which consists of four parts: negative feedback reward for drone collisions, negative feedback reward for exceeding the boundary, positive feedback reward for the product of energy balance coefficient and the difference in collected information, with the boundary penalty assigned the highest weight coefficient. The agent is trained using deep reinforcement learning algorithms. During training, the agent continuously learns and optimizes its path planning strategy by interacting with a simulated land environment in order to collect the most valuable information before the battery runs out.

[0049] The priority of each sample is calculated using the TD error. Samples with larger prediction errors are assigned higher sampling weights to accelerate the network's learning of high-value scenarios such as drastic changes in information variability. The priority probability distribution formula is as follows:

[0050] in For TD error, To control the degree of priority, To prevent extremely small positive numbers with an error of 0.

[0051] The update step size is adjusted by using importance sampling weights to ensure the stability and convergence of the training process.

[0052] S212. Superimpose the velocity vector of the next time slot onto the initial coordinates of the UAV to obtain the temporary coordinates of the UAV. It is stipulated that the UAV shall not fly beyond the boundary. Iterate through the actions of the UAV and check whether the UAV has overstepped the boundary. If overstepping the boundary is found, correct the trajectory and impose a penalty, and re-input the action until the UAV learns to fly in the designated area. If there is no overstepping the boundary, output the temporary coordinates of the UAV at this time as the current coordinates of the UAV.

[0053] S22. Define a multidimensional constraint reward function to guide the UAV to move towards a region with high information difference, and introduce a priority experience replay mechanism to perform weighted sampling of high-value training samples based on TD error, and output the UAV velocity vector through the improved MADDPG algorithm. S221. Define a comprehensive reward function with multidimensional constraints. The reward function consists of a UAV collision penalty, an out-of-bounds penalty, and a positive feedback reward consisting of the product of the energy balance coefficient and the difference in the collected information. S222. The energy balance coefficient is used to measure the degree of energy consumption balance among multiple sensors during data acquisition. It measures the degree of data acquisition balance by calculating the ratio of the square of the sum of energy consumed by each sensor node to the sum of the squares of all energy. This guides the drone swarm to make the coefficient approach 1, thereby suppressing the drones from wandering in local areas. The formula for the energy balance coefficient is as follows:

[0054] in, For sensors The energy consumed from the start of the system; S223. Establish a probabilistic channel model for land scenarios, comprehensively considering line-of-sight and non-line-of-sight links, and calculate the path loss formula for different link states as follows:

[0055] in, For line-of-sight communication path loss, The probability of a line-of-sight link occurring. For non-line-of-sight communication path loss, Given the probability of non-line-of-sight links occurring, the reward value for the current state is calculated using the reward function and the channel model, and is used to update the policy network.

[0056] S23. The UAV performs data acquisition actions and updates sensor observations. It calculates the estimated energy consumption of the UAV returning to the starting point in real time. If the remaining energy is higher than the return threshold, it continues to perform repeated acquisition tasks. Otherwise, it returns to the starting point according to the shortest path. After the UAV enters the return state, it is removed from the subsequent trajectory planning and data acquisition decision. Finally, a complete trajectory planning scheme containing the coordinate sequence of each UAV is output.

[0057] S231. The UAV obtains a positive reward after each successful transmission of a data packet using the packet retransmission mechanism, and resets the cumulative information difference of the sensor after all data packets have been successfully transmitted. The retransmission probability is calculated as follows:

[0058] in, The center point of the success probability threshold. For signal-to-interference-to-noise ratio, To improve reliability sensitivity, the transmission success probability mapped by the signal-to-interference-plus-noise ratio is used in conjunction with a random sampling discrimination mechanism to simulate the packet retransmission process, thereby providing differentiated reward signals based on the collected results. S232. If the drone's current remaining battery power and estimated energy consumption for returning to the starting point are monitored in real time, the drone will be forced to terminate the data collection task and return to the starting point, generating the shortest path return trajectory from the current position to the starting point.

[0059] This embodiment proposes a multi-UAV sensor data collection trajectory planning method based on deep reinforcement learning. It uses an improved multi-agent deep deterministic policy gradient algorithm (TP-MADDPG) that incorporates Transformer feature extraction and priority experience replay mechanism to maximize the cumulative information difference of the collected data while taking into account the energy efficiency fairness of UAVs as the optimization objective.

[0060] This embodiment conducts experiments in real-world task scenarios, testing different scales of drone numbers and sensor counts. The comparison algorithms used in this paper include the Multi-Agent Deep Deterministic Policy Gradient Algorithm (MADDPG), the Multi-Agent Attention Evaluation Algorithm (MAAC), the Multi-Agent Proximal Policy Optimization Algorithm (MAPPO), and the Greed Algorithm.

[0061] like Figure 4The image shows the trajectory of two drones on a two-dimensional plane with 30 sensors. The drones are represented by the number of uav and the sensors by the number of sensors.

[0062] like Figure 5 The image shows a comparison of the algorithms for system reward values ​​when there are 2 drones and 30 sensors.

[0063] like Figure 6 The image shows a comparison of algorithm results for the difference in information collected by drones during a flight mission, with 30 sensors and different numbers of drones.

[0064] like Figure 7 The image shows a comparison of the algorithm results for the energy balance coefficient of a drone completing a flight mission with different numbers of drones and 30 sensors.

[0065] like Figure 8 The image shows a comparison of algorithm results for the success rate of data transmission when a drone completes a flight mission with different numbers of drones and 30 sensors.

[0066] Depend on Figures 4-8 It can be seen that the TP-MADDPG algorithm outperforms other algorithms in all aspects. This is because the algorithm can help UAVs plan their flight paths better.

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-UAV sensor data acquisition trajectory planning method based on deep reinforcement learning, characterized in that, Includes the following steps: S1. Establish the basic framework of the UAV data acquisition system model in a land scenario; S2. Based on the basic framework, a continuous trajectory planning strategy based on an improved deep reinforcement learning algorithm is constructed. The continuous trajectory planning strategy includes a feature extraction unit that calculates self-attention weights and a multi-agent deep reinforcement learning decision network based on a weighted sampling mechanism for sample prediction errors. The continuous trajectory planning strategy aims to maximize the sum of the cumulative information differences collected. It takes the real-time environmental state as input, trains under energy balance constraints, and outputs the velocity vectors of each UAV in each time slot. The flight decisions of the UAVs are adjusted according to the energy consumption of each UAV during the data collection process until the remaining energy of the UAVs reaches the safe threshold for returning to home. Based on the velocity vectors, a UAV flight trajectory map that satisfies the energy balance constraints and maximizes the value of data collection is generated.

2. The multi-UAV sensor data acquisition trajectory planning method based on deep reinforcement learning according to claim 1, characterized in that, S1 specifically includes the following steps: S11. Constructing an information difference model for sensor observations: Using temporal differential logic, the current observations of sensor nodes are acquired in real time and compared with historical observations from the previous moment. The difference between the two is calculated, and an information difference function is constructed based on this difference to characterize the acquisition value of the node's current data. The formula for calculating the information difference function is as follows: in, For sensors exist The observation value at a given moment, the information difference degree is used to characterize the value of the data collected at that node, and is used as a state input or reward feedback parameter in the UAV trajectory planning process to guide the UAV's flight decision; S12. Considering terrain occlusion characteristics, the probability of line-of-sight and non-line-of-sight links occurring. Angle of elevation of the drone relative to the sensor Related; Total path loss The expected value of path loss under different link states is used to characterize the loss, and the calculation formula is as follows: in, For line-of-sight communication path loss, This represents the path loss for non-line-of-sight communication. Introducing a packet-segmented transmission protocol and an automatic retransmission request mechanism: defining the probability of successful transmission of a single data packet. Here is a nonlinear mapping function with respect to the signal-to-interference-plus-noise ratio (SINR), used to simulate the random packet loss and retransmission logic in UAV-sensor communication. The formula is as follows: in, The threshold center point for the probability of success For reliability sensitivity; S13, UAV flight power Based on the aerodynamic characteristics of UAVs, the specific formula is as follows: in, For the drone's flight speed, and The power coefficient is a constant. By monitoring the remaining battery power in real time and combining the straight-line distance from the current location to the starting point with the return-to-home power requirement, the forced return-to-home power threshold is calculated in real time. When the remaining power is less than or equal to the forced return-to-home power threshold, the drone executes the shortest path return-to-home action.

3. The multi-UAV sensor data acquisition trajectory planning method based on deep reinforcement learning according to claim 1, characterized in that, S2 specifically includes the following steps: S21. Obtain the coordinate information of the current position of the UAV, the position of each sensor node and the channel state information as input, restrict the activity area of ​​the UAV, use the self-attention mechanism of the feature extraction unit module including self-attention weight calculation to extract the correlation weight between state sequences, identify the time-varying pattern of data value, and output a high-dimensional feature vector representing the scene features and the UAV flight speed vector. S22. Define a multidimensional constraint reward function to guide the UAV to move towards a region with high information difference, and introduce a priority experience replay mechanism to perform weighted sampling of high-value training samples based on the sample prediction error. Output the UAV velocity vector through an improved multi-agent deep reinforcement learning decision network algorithm. S23. The UAV performs data acquisition actions and updates sensor observations. It calculates the estimated energy consumption of the UAV returning to the starting point in real time. If the remaining energy is higher than the return threshold, it continues to perform repeated acquisition tasks. Otherwise, it returns to the starting point according to the shortest path. After the UAV enters the return state, it is removed from the subsequent trajectory planning and data acquisition decision. Finally, a complete trajectory planning scheme containing the coordinate sequence of each UAV is output.

4. The multi-UAV sensor data acquisition trajectory planning method based on deep reinforcement learning according to claim 3, characterized in that, S21 specifically includes the following steps: S211. Construct a feature extraction network based on Transformer unit modules. Input the temporal state sequences of the UAV and sensors into the feature extraction module with a self-attention mechanism, and calculate the correlation weights between state features. The calculation formula is as follows: in, For querying the matrix, The key matrix, For value matrices, This is a similarity matrix. As a scaling factor, through Normalized weights are used, and the feature extraction network serves as the input layer of the improved multi-agent deep reinforcement learning decision network algorithm, outputting a high-dimensional feature vector representing scene features to achieve multi-agent collaborative decision-making. S212. Superimpose the velocity vector of the next time slot onto the initial coordinates of the UAV to obtain the temporary coordinates of the UAV. It is stipulated that the UAV shall not fly beyond the boundary. Iterate through the actions of the UAV and check whether the UAV has overstepped the boundary. If overstepping the boundary is found, correct the trajectory and impose a penalty, and re-input the action until the UAV learns to fly in the designated area. If there is no overstepping the boundary, output the temporary coordinates of the UAV at this time as the current coordinates of the UAV.

5. The multi-UAV sensor data acquisition trajectory planning method based on deep reinforcement learning according to claim 4, characterized in that, S22 specifically includes the following steps: S221. Define a comprehensive reward function with multidimensional constraints. The reward function consists of a UAV collision penalty, an out-of-bounds penalty, and a positive feedback reward consisting of the product of the energy balance coefficient and the difference in the collected information. S222. The energy balance coefficient is used to measure the degree of energy consumption balance among multiple sensors during data acquisition. It measures the degree of data acquisition balance by calculating the ratio of the square of the sum of energy consumed by each sensor node to the sum of the squares of all energy. This guides the drone swarm to make the coefficient approach 1, thereby suppressing the drones from wandering in local areas. The formula for the energy balance coefficient is as follows: in, For sensors The energy consumed from the start of the system; S223. Establish a probabilistic channel model for land scenarios, comprehensively considering line-of-sight and non-line-of-sight links, and calculate the path loss formula for different link states as follows: in, For line-of-sight communication path loss, The probability of a line-of-sight link occurring. For non-line-of-sight communication path loss, Given the probability of non-line-of-sight links occurring, the reward value for the current state is calculated using the reward function and the channel model, and is used to update the policy network.

6. The multi-UAV sensor data acquisition trajectory planning method based on deep reinforcement learning according to claim 3, characterized in that, S23 specifically includes the following steps: S231. The UAV obtains a positive reward after each successful transmission of a data packet using the packet retransmission mechanism, and resets the cumulative information difference of the sensor after all data packets have been successfully transmitted. The retransmission probability is calculated as follows: in, The center point of the success probability threshold. For signal-to-interference-to-noise ratio, To improve reliability sensitivity, the transmission success probability mapped by the signal-to-interference-plus-noise ratio is used in conjunction with a random sampling discrimination mechanism to simulate the packet retransmission process, thereby providing differentiated reward signals based on the collected results. S232. If the drone's current remaining battery power and estimated energy consumption for returning to the starting point are monitored in real time, the drone will be forced to terminate the data collection task and return to the starting point, generating the shortest path return trajectory from the current position to the starting point.

7. The multi-UAV sensor data acquisition trajectory planning method based on deep reinforcement learning according to claim 1, characterized in that, The UAV data acquisition system model described in S1 is a network composed of multiple UAV nodes, each of which can receive and store data; all UAVs start from the same point and repeatedly collect data from sensors distributed in the land scene.

8. The multi-UAV sensor data acquisition trajectory planning method based on deep reinforcement learning according to claim 5, characterized in that, The improved multi-agent deep reinforcement learning decision network algorithm architecture includes: Using MADDPG as the basic algorithm framework, the feature extraction unit module with self-attention weight calculation described in S211 is integrated to process the observation sequence. The comprehensive reward function described in S221 is used to guide the policy update. The policy and value function of the agent are approximated by a deep neural network. The system is trained by combining priority experience replay and target network to achieve multi-agent collaborative decision-making and obtain the velocity vector of the UAV in the next time slot.

9. The multi-UAV sensor data acquisition trajectory planning method based on deep reinforcement learning according to claim 8, characterized in that, The improved MADDPG algorithm process includes the following: Define the state space and action space of the drone in the terrestrial environment. The action space is the velocity vector of the drone's movement, and the state space includes the drone's real-time coordinates, the position of each sensor node, and the drone's current remaining battery power. The reward function for the drone is determined, which consists of four parts: negative feedback reward for drone collisions, negative feedback reward for exceeding the boundary, positive feedback reward for the product of energy balance coefficient and the difference in collected information, with the boundary penalty assigned the highest weight coefficient. The agent is trained using deep reinforcement learning algorithms. During training, the agent continuously learns and optimizes its path planning strategy by interacting with a simulated land environment in order to collect the most valuable information before the battery runs out.

10. The multi-UAV sensor data acquisition trajectory planning method based on deep reinforcement learning according to claim 8, characterized in that, The improved MADDPG algorithm includes a priority experience replay mechanism, and the specific steps are as follows: The priority of each sample is calculated using the TD error. Samples with a higher prediction error than the current experience pool average are assigned a sampling weight higher than the experience pool average to accelerate the network's learning of high-value scenarios such as drastic changes in information variability. The priority probability distribution formula is as follows: in, For TD error, To control the degree of priority, To prevent extremely small positive numbers with an error of 0; The update step size is adjusted by using importance sampling weights to ensure the stability and convergence of the training process.

Images