Unmanned aerial vehicle data acquisition path planning method based on improved TD3 algorithm

Abstract

The invention discloses an unmanned aerial vehicle data acquisition path planning method based on an improved TD3 algorithm. The method comprises the following steps: 1) constructing a system model including an environment model, an unmanned aerial vehicle kinematics model and a communication model; 2) designing a reward function based on the system model; 3) constructing a priority experience playback buffer area by adopting a PER mechanism, and managing the priority of the sample experience in the buffer area by using a binary tree structure; and 4) constructing an improved TD3 algorithm model based on an Actor-Critic network, optimizing the model by adopting a self-attention mechanism and a dropout technology, training the improved model based on a reward function and a priority experience playback buffer area, and completing a data acquisition path planning task of the unmanned aerial vehicle by using the trained model. According to the method, the data acquisition time of the unmanned aerial vehicle in a complex environment and the energy consumption overhead of the unmanned aerial vehicle can be reduced, so that a data acquisition path with the maximum data acquisition rate and the minimum energy consumption rate is found.

Description

Technical Field

[0001] This invention relates to unmanned aerial vehicle (UAV) path planning and intelligent optimization algorithm technology, and more particularly to path planning for UAVs to collect data from IoT devices in urban scenarios. Specifically, it is a UAV data collection path planning method based on an improved TD3 algorithm. Background Technology

[0002] With the rapid development and widespread adoption of IoT technology, drones have emerged as a unique advantage over traditional ground equipment: compared to fixed ground base stations or sensors, drones do not require pre-deployment and can flexibly overcome geographical limitations, quickly establishing temporary communication links in remote areas, complex terrains, or emergency scenarios, increasing response speed several times over; their dynamic three-dimensional layout capabilities allow them to adjust flight altitude and position as needed, optimizing line-of-sight communication links, expanding coverage, avoiding local interference, and significantly improving network coverage quality and data transmission stability.

[0003] As a core prerequisite for drones to complete various tasks, path planning directly determines data acquisition efficiency, network coverage quality, mission execution reliability, and energy utilization efficiency, playing a crucial role in improving the overall performance of drones. However, traditional optimization algorithms such as graph search, spatial sampling, and potential field methods, while achieving good trajectory planning results in simple environments, struggle to efficiently solve autonomous UAV trajectory planning problems in complex, dynamic, and highly uncertain environments. Heuristic algorithms, while addressing combinatorial optimization problems like autonomous UAV trajectory planning by simulating natural phenomena or the self-organizing behavior of biological groups, are easy to implement, heuristic, and parallelizable, but suffer from slow search speeds and high memory consumption. Even reinforcement learning algorithms, widely used in recent years, while adaptable to continuous state and action spaces and capable of handling decision-making in dynamic environments, still face several scenario-specific problems: First, insufficient state representation capabilities make it difficult to model the dependencies between user features in multi-user UAV data acquisition scenarios, leading to incomplete action decisions using global user information. Second, the problem of reward sparsity is prominent; the long task cycle for UAVs to complete full data acquisition, coupled with a lack of effective feedback signals in intermediate steps, easily leads to slow training convergence and policy oscillations. Third, the risk of overfitting in value estimation; traditional reinforcement learning algorithms like TD3 and DDPG are prone to this in complex environments. The value is overestimated, and the Actor network may rely too much on specific user features or environmental patterns, reducing the generalization ability of the policy; fourth, the sample efficiency is low. Reinforcement learning needs to obtain experience samples through a large number of environmental interactions, but the cost of sample collection in actual drone flight is high and the difficulty of scene reproduction is great, resulting in long training cycles and insufficient adaptability to actual deployment. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a UAV data acquisition path planning method based on an improved TD3 algorithm. This method can reduce the time and energy consumption of UAVs in complex environments, thereby finding a data acquisition path that maximizes the data collection rate and minimizes energy consumption.

[0005] The technical solution to achieve the objective of this invention is: A method for UAV data acquisition path planning based on an improved TD3 algorithm includes the following steps: 1) Constructing the system model: The system model includes an environment model, a UAV kinematics model, and a communication model, specifically: 1-1) Constructing an environment model: Generate an environment model containing multiple buildings and with a size of [missing information]. A 3D scene, in which randomly distributed... The first fixed-location IoT node, the second The positions of the nodes and the UAV are represented by formulas (1) and (2), respectively, where Indicates the first The horizontal coordinates of each IoT node It is the drone at all times Position on the horizontal plane, It is the drone's flight altitude. This refers to the duration of task execution. (1), (2), To simulate a real environment, the specific location and height of buildings are generated using a statistical model recommended by the International Telecommunication Union (ITU). Building data is generated based on the Rayleigh distribution mean of building coverage ratio, number of buildings per unit area, and building height distribution. 1-2) Constructing the UAV kinematic model: Discretizing the continuous time domain into... There are time steps, each with a length of . ,and They are not necessarily equal; each time step consists of a fixed flight time. and the hover time used for data It consists of two parts. If there are no nodes at the current location where data can be collected, the hovering time is 0. The drone adjusts its speed at each time step. and horizontal direction angle Update its horizontal position At the same time, maintain a high altitude Furthermore, the drone must always meet the constraints shown in formula (3) during flight: (3); 1-2-1) The position update formula for the UAV is expressed by formula (4), where Indicates the distance it traveled: (4); 1-2-2) The distance traveled by the drone is equal to the fixed flight time. Multiply by the average flight speed at the current time step , represented as ; 1-2-3) The objective of minimizing the total task completion time is... It means that among them ; 1-3) Constructing the communication model: A practical "ground-to-air" G2A channel model is adopted. The communication model is represented by large-scale fading and small-scale fading. These fading values ​​are calculated based on a simulated 3D map that considers buildings as propagation scatterers, as detailed below: 1-3-1) Calculating large-scale fading: Based on the inspection of drones and the first To accurately determine whether the direct communication link between IoT nodes is blocked by buildings, the following information is needed: Does a line-of-sight link exist between the IoT nodes? And what about the drone and the...? The large-scale fading of the G2A channel associated with an IoT node is expressed by formula (5): (5), in Indicates drones and the first Free space path loss between IoT nodes Indicates drones and the first The distance between IoT nodes Indicates the carrier frequency. Represents the speed of light. Indicates a line-of-sight link. and These represent the propagation losses for line-of-sight links and non-line-of-sight links, respectively. 1-3-2) Calculation of Small-Scale Fading and Information Gain: For non-line-of-sight links, the small-scale fading coefficient... Assuming Rayleigh fading, for line-of-sight links, then... Assuming Rice fading with a Rice factor of 15 dB, the drone reaches the... The channel gain of each IoT node is shown in formula (6): (6), 1-3-3) Calculation of Signal-to-Noise Ratio (SNR) and Transmission Rate: The IoT node can only transmit at a constant power when the drone wakes it up. Start uploading data; otherwise, they will remain silent until the [date / time]. In the first time step, the first The uplink signal-to-noise ratio between an IoT node and a drone is expressed by formula (7): (7), in It is the first Channel gain during each time step hovering phase This represents the additive white Gaussian noise power at the UAV receiver, for the... Uplink data collection services related to each IoT node, setting a predefined signal-to-noise ratio threshold. Only when At that time, the first Only when a certain number of IoT nodes can be woken up and provided with services by the drone, is defined to indicate the first IoT node. Can the first IoT node be in the first...? The indicator function that satisfies the signal-to-noise ratio requirement at each time step is shown in formula (8): (8), Assuming that each IoT node is served at most once in a single implementation, then formula (9) represents the first... The indication function for each IoT node, where It is a binary variable used to indicate the first... Are the IoT nodes already being served by drones? (9), if Then the first The first IoT node has been served during the task; otherwise, the second... One IoT node was not serviced, and then the drone and the first The transmission rate between IoT nodes is expressed as shown in formula (10): (10); 1-3-4) Calculation of drone hovering time: Only in the first... One IoT node Only when the first time step is served can the drone upload the first time step. The data comes from the IoT nodes, therefore, the drone's hovering time is the [number]th [node]. The maximum duration of data upload from the served IoT nodes in each time step is represented by formula (11): (11), in Indicates the first The size of the information file for each IoT node This is to prevent values ​​with a denominator of 0; 2) Reward Function Design: Guiding the drone to complete the data collection task of all IoT nodes in the shortest possible time is a typical "sparse reward" problem, meaning there is only a clear feedback when the task is completed. A reward shaping mechanism is used to transform the original sparse reward into a dense reward. The reward function consists of three components: time consumption penalty, data collection reward, and out-of-bounds / violation penalty. A pheromone mechanism is integrated into the reward function, as detailed below: 2-1) Definition of UAV pheromones: Pheromones are virtual "odor" distributions that reflect the state of the environment and are used to guide UAVs to unserved nodes. The definition is shown in formula (12): (12) in It is in the Remaining pheromones in each time step It is the drone in the The number of IoT nodes served in each time step It is a positive constant representing the pheromone captured by each IoT node. It is a positive constant representing the loss of pheromones. This refers to the penalty when an action results in a drone crossing boundaries or violating regulations; 2-2) Definition of reward function: By integrating the reward shaping mechanism and pheromones into the reward function, the final definition of the reward function is shown in formula (13): (13) in It is caused by pheromones The resulting reward function, approximate The function is used, but its gradient is smoother than the latter, and the drone receives a reward for the remaining time when the mission is completed. ,in It is the maximum time step. It is the time step required to complete the task; 3) Constructing a Priority Experience Replay Buffer using the PER mechanism: The PER mechanism samples from the experience replay buffer according to priority. The priority is proportional to the absolute value of the time difference (TD) error. Samples with large TD errors indicate that the Critic network's Q-value estimation for these samples is inaccurate, while samples with small TD errors indicate that the Critic network is overfitting. The core objective of PER is to prioritize sampling based on the "importance" of the samples, specifically implemented as follows: 3-1) Experience Sample Storage and Initialization Calculation: The experience samples generated by the interaction between the UAV and the environment are stored in the form of quadruples. Stored in the buffer, where Indicates the first The state at each time step Indicates the first Actions at each time step Indicates the first Rewards for each time step Indicates the execution of an action After transitioning to the next state, when new samples are stored, the priority experience replay buffer will calculate the initial TD error for each sample, as shown in formula (14): (14) in It is the first TD error for each sample The goal value, These represent two independent Critic network pairs in the TD3 algorithm. of Value prediction, then sample prioritization based on TD3 error. To ensure that new samples have an initial sampling probability, the sample priority is calculated as follows. ,in It is a tiny positive number, taking the value of ,avoid The sample priority is set to 0 to ensure that all samples have a chance to be sampled. 3-2) Sampling mini-batch samples: Probability-based sampling is implemented using a Sum-Tree data structure. A Sum-Tree is a binary tree in which each leaf node stores the priority of the sample. Non-leaf nodes store the sum of the priorities of their child nodes. During sampling, a randomly generated value is used to find the corresponding leaf node in the Sum-Tree to achieve sampling according to priority ratio. 3-3) Calculate the importance sampling IS weights and correct the sampling bias: The loss function is corrected using the IS weights, as shown in formula (15): (15) in, It is the first The IS weights of each sample, It is the first The sampling probability of a sample. It is the minimum sampling probability in the mini-batch. It is a deviation correction factor; 3-4) Update network parameters and sample priorities simultaneously by combining IS weights: After each mini-batch training, the sample priorities are... The value prediction will be updated, the TD error of the sample will be recalculated, and its priority will be updated. At the same time, the node values ​​of the Sum-Tree will be updated synchronously to ensure that the priority of the next round of sampling is up-to-date. The network parameters are updated in combination with IS weights as shown in formula (16): (16) in This is the loss function value of the Critic network. This is the number of samples in the mini-batch; 4) TD3 Algorithm Model Improvement and Training: Based on the original TD3 algorithm, and addressing the complexity of path planning for UAVs in 3D environments, the TD3-DPATT model was designed. The TD3-DPATT model improves the performance of the original Actor-Critic network structure by integrating a self-attention mechanism and introducing dropout technology into the Actor network architecture. The TD3-DPATT model is trained based on a reward function and a priority experience replay buffer, including: 4-1) Building the TD3-DPATT model: The TD3-DPATT model includes one Actor network and two Critic networks. Each Actor network or Critic network consists of two sub-networks: an online network and a target network. The online Actor network is composed of... Parameterization, the two online Critic networks are respectively composed of and Parameterization; using parameters as The target Actor network and parameters are: and The two target Critic networks, TD3-DPATT, deeply integrate self-attention mechanism and dropout technology into the network structure and training process of TD3, forming an improved solution adapted to multi-user scenarios; 4-1-1) Attention Mechanism Embedded in State Representation: The self-attention mechanism is used as the core feature encoding layer of the Actor or Critic network. After the Actor and Critic networks receive the original state input, the global features are first expanded to user-level dimensions to complete feature preprocessing. Then, the expanded global features are concatenated with the local features of each user to construct user-level sequence features. Finally, a multi-head attention mechanism is used to encode the sequence features in high dimension. The feature transfer efficiency is optimized by combining residual connections and layer normalization, capturing the dependencies between multiple users and generating more representative state features. 4-1-2) Integrating Dropout Technology into the Actor Action Generation Link: Dropout technology is incorporated as controlled noise into the action generation process of the online Actor network. In the link from the hidden layer to the action output layer of the Actor network, during each training iteration, some neurons are dropped out with probability. It was randomly and temporarily shut down; 4-1-3) Parameter Update of TD3-DPATT Model: First, TD3-DPATT uses two independent... and right The value estimates were calculated by two online Critic networks and two target Critic networks, respectively. The minimum value among the values ​​is used to calculate the optimal value. The value is shown in formula (17): (17) in It is a discount factor that smooths out immediate and future rewards; The target Actor network outputs the optimal action for the next time step, and then two online Critic networks... and The update is performed by minimizing the loss function using formula (18): (18) Meanwhile, the online Actor network is updated using policy gradients, while the target Critic and target actor networks are updated using soft updates. The online Actor network update is shown in formula (19), and the target Critic and target actor network updates are shown in formula (20). Indicates the soft update rate: (19) (20); 4-2) Training the TD3-DPATT model: The UAV first preprocesses the current environmental state, using information augmentation and dimensional expansion to enhance key information; then it generates flight actions based on the Actor network, incorporating exploration noise to balance exploring new areas and utilizing known strategies. After the action is executed, the reward signal is calculated based on environmental feedback to generate a new environmental state. Finally, the interaction experience of "current state - executed action - feedback reward - new state - mission termination flag" is stored in the priority experience replay buffer until the data collection task is completed or the maximum number of steps in a single training session is reached. When the interaction data stored in the priority experience buffer reaches the preset amount, the network update process is initiated. Then, the dual Critic network is updated first to optimize the accuracy of action value judgment, and then the Actor network is updated with a delay to improve trajectory decision performance. At the same time, a soft update mechanism is used to synchronize the target network parameters.

[0006] This technical solution addresses the challenges of communication congestion caused by buildings and the high complexity of scenarios with multiple IoT nodes in drone data collection. It proposes a drone path planning approach based on an improved TD3 algorithm. The core objective is to minimize data collection time and drone energy consumption. This approach is grounded in real-world environmental constraints, employing a path loss channel model applicable to actual environments to accurately characterize the impact of buildings on communication links. Through relevant path planning, the drone can flexibly adjust its altitude, optimizing the signal-to-noise ratio and data transmission rate. At the model level, a priority experience replay strategy is integrated into the TD3 algorithm, sampling high-value samples according to TD error priority to improve learning efficiency. Simultaneously, a self-attention mechanism is used in the Actor-Critic network, enabling the model to focus on high-feature state information, improving the model's efficiency in utilizing key information and the rationality of decision-making. At the algorithm level, a feature extraction layer and a dropout layer are introduced on top of the TD3 algorithm. The feature extraction layer expands the dimensionality of low-dimensional features, allowing the model to analyze various features more comprehensively. The dropout layer suppresses overfitting, enhancing training stability and generalization ability. The state, action, and reward functions are clearly defined according to the actual scenario, and soft update and delayed update mechanisms ensure model convergence.

[0007] This technical solution has the following advantages: 1. Real-world scenario modeling to ensure the practicality of the solution: To address the problem that traditional simplified LoS or probabilistic channel models cannot accurately depict environmental building occlusion, a practical urban LoS channel model based on the ITU statistical model is adopted. By generating real building distribution, the LoS / NLoS link status between the UAV and IoT nodes is accurately determined. The channel gain is calculated by combining large-scale and small-scale fading. It does not rely on perfect channel state information, making the trajectory design more adaptable to the actual environment. 2. The TD3 algorithm is effectively improved to enhance training efficiency and stability: To address the problems of inefficient sampling and overfitting in traditional deep reinforcement learning algorithms, the PER strategy and dropout technique are integrated into the TD3 algorithm: The priority experience replay buffer samples high-value samples based on the TD error priority, and the sampling bias is corrected by IS weights to accelerate the learning of key experiences; the Actor network is embedded with a dropout layer to randomly shield some neurons, reduce dependence on local features, and suppress overfitting. The synergy of the two enables the algorithm to quickly converge to the near-optimal strategy in complex scenarios with 35 sensors. At the same time, in order to improve the TD3 algorithm's ability to focus on key features, a self-attention mechanism is introduced.

[0008] By adopting this technical solution, UAVs can quickly learn near-optimal path strategies in complex scenarios with multiple IoT nodes. This not only solves the communication limitations of traditional path planning but also overcomes the instability of existing deep reinforcement learning algorithms, achieving synergistic optimization of data collection efficiency, energy consumption control, and training robustness.

[0009] This method can reduce the time and energy consumption of drones in complex environments, thereby finding a data collection path that maximizes the data collection rate and minimizes the energy consumption rate. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating the method used in the embodiment. Figure 2 This is a schematic diagram of the initialization system model process in the embodiment; Figure 3 This is a schematic diagram of the reward function construction process in the embodiment; Figure 4 This is a schematic diagram illustrating the process of constructing a priority experience replay buffer in the embodiment; Figure 5 This is a schematic diagram of the TD3-DPATT training process in the embodiment; Figure 6 This is a schematic diagram illustrating the simulation results of task completion time under different numbers of IoT nodes in the embodiment; Figure 7 This is a schematic diagram illustrating the simulation results of drone energy consumption under different numbers of IoT nodes in the embodiment. Detailed Implementation

[0011] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this is not intended to limit the scope of the invention.

[0012] Example:

[0013] Reference Figure 1A method for UAV data acquisition path planning based on an improved TD3 algorithm includes the following steps: 1) Constructing the system model: The system model includes an environment model, a UAV kinematics model, and a communication model, specifically: 1-1) Constructing an environment model: Generate an environment model containing multiple buildings and with a size of [missing information]. A 3D scene, in which randomly distributed... The first fixed-location IoT node, the second The positions of the nodes and the UAV are represented by formulas (1) and (2), respectively, where Indicates the first The horizontal coordinates of each IoT node It is the drone at all times Position on the horizontal plane, It is the drone's flight altitude. This refers to the duration of task execution. (1), (2), To simulate a real environment, the specific location and height of buildings are generated using a statistical model recommended by the International Telecommunication Union (ITU). Building data is generated based on the Rayleigh distribution mean of building coverage ratio, number of buildings per unit area, and building height distribution. 1-2) Constructing the UAV kinematic model: Discretizing the continuous time domain into... There are time steps, each with a length of . ,and They are not necessarily equal; each time step consists of a fixed flight time. and the hover time used for data It consists of two parts. If there are no nodes at the current location where data can be collected, the hovering time is 0. The drone adjusts its speed at each time step. and horizontal direction angle Update its horizontal position At the same time, maintain a high altitude Furthermore, the drone must always meet the constraints shown in formula (3) during flight: (3); 1-2-1) The position update formula for the UAV is expressed by formula (4), where Indicates the distance it traveled: (4); 1-2-2) The distance traveled by the drone is equal to the fixed flight time. Multiply by the average flight speed at the current time step , represented as ; 1-2-3) The objective of minimizing the total task completion time is... It means that among them ; 1-3) Constructing the communication model: A practical "ground-to-air" G2A channel model is adopted. The communication model is represented by large-scale fading and small-scale fading. These fading values ​​are calculated based on a simulated 3D map that considers buildings as propagation scatterers, as detailed below: 1-3-1) Calculating large-scale fading: Based on the inspection of drones and the first To accurately determine whether the direct communication link between IoT nodes is blocked by buildings, the following information is needed: Does a line-of-sight link exist between the IoT nodes? And what about the drone and the...? The large-scale fading of the G2A channel associated with an IoT node is expressed by formula (5): (5), in Indicates drones and the first Free space path loss between IoT nodes Indicates drones and the first The distance between IoT nodes Indicates the carrier frequency. Represents the speed of light. Indicates a line-of-sight link. and These represent the propagation losses for line-of-sight links and non-line-of-sight links, respectively. 1-3-2) Calculation of Small-Scale Fading and Information Gain: For non-line-of-sight links, the small-scale fading coefficient... Assuming Rayleigh fading, for line-of-sight links, then... Assuming Rice fading with a Rice factor of 15 dB, the drone reaches the... The channel gain of each IoT node is shown in formula (6): (6), 1-3-3) Calculation of Signal-to-Noise Ratio (SNR) and Transmission Rate: The IoT node can only transmit at a constant power when the drone wakes it up. Start uploading data; otherwise, they will remain silent until the [date / time]. In the first time step, the first The uplink signal-to-noise ratio between an IoT node and a drone is expressed by formula (7): (7), in It is the first Channel gain during each time step hovering phase This represents the additive white Gaussian noise power at the UAV receiver, for the... Uplink data collection services related to each IoT node, setting a predefined signal-to-noise ratio threshold. Only when At that time, the first Only when a certain number of IoT nodes can be woken up and provided with services by the drone, is defined to indicate the first IoT node. Can the first IoT node be in the first...? The indicator function that satisfies the signal-to-noise ratio requirement at each time step is shown in formula (8): (8), Assuming that each IoT node is served at most once in a single implementation, then formula (9) represents the first... The indication function for each IoT node, where It is a binary variable used to indicate the first... Are the IoT nodes already being served by drones? (9), if Then the first The first IoT node has been served during the task; otherwise, the second... One IoT node was not serviced, and then the drone and the first The transmission rate between IoT nodes is expressed as shown in formula (10): (10); 1-3-4) Calculation of drone hovering time: Only in the first... One IoT node Only when the first time step is served can the drone upload the first time step. The data comes from the IoT nodes, therefore, the drone's hovering time is the [number]th [node]. The maximum duration of data upload from the served IoT nodes in each time step is represented by formula (11): (11), in Indicates the first The size of the information file for each IoT node This is to prevent values ​​with a denominator of 0; 2) Reward Function Design: Guiding the drone to complete the data collection task of all IoT nodes in the shortest possible time is a typical "sparse reward" problem, meaning there is only a clear feedback when the task is completed. A reward shaping mechanism is used to transform the original sparse reward into a dense reward. The reward function consists of three components: time consumption penalty, data collection reward, and out-of-bounds / violation penalty. A pheromone mechanism is integrated into the reward function, as detailed below: 2-1) Definition of UAV pheromones: Pheromones are virtual "odor" distributions that reflect the state of the environment and are used to guide UAVs to unserved nodes. The definition is shown in formula (12): (12) in It is in the Remaining pheromones in each time step It is the drone in the The number of IoT nodes served in each time step It is a positive constant representing the pheromone captured by each IoT node. It is a positive constant representing the loss of pheromones. This refers to the penalty when an action results in a drone crossing boundaries or violating regulations; 2-2) Definition of reward function: By integrating the reward shaping mechanism and pheromones into the reward function, the final definition of the reward function is shown in formula (13): (13) in It is caused by pheromones The resulting reward function, approximate The function is used, but its gradient is smoother than the latter, and the drone receives a reward for the remaining time when the mission is completed. ,in It is the maximum time step. It is the time step required to complete the task; 3) Constructing a Priority Experience Replay Buffer using the PER mechanism: The PER mechanism samples from the experience replay buffer according to priority. The priority is proportional to the absolute value of the time difference (TD) error. Samples with large TD errors indicate that the Critic network's Q-value estimation for these samples is inaccurate, while samples with small TD errors indicate that the Critic network is overfitting. The core objective of PER is to prioritize sampling based on the "importance" of the samples, specifically implemented as follows: 3-1) Experience Sample Storage and Initialization Calculation: The experience samples generated by the interaction between the UAV and the environment are stored in the form of quadruples. Stored in the buffer, where Indicates the first The state at each time step Indicates the first Actions at each time step Indicates the first Rewards for each time step Indicates the execution of an action After transitioning to the next state, when new samples are stored, the priority experience replay buffer will calculate the initial TD error for each sample, as shown in formula (14): (14) in It is the first TD error for each sample The goal value, These represent two independent Critic network pairs in the TD3 algorithm. of Value prediction, then sample prioritization based on TD3 error. To ensure that new samples have an initial sampling probability, the sample priority is calculated as follows. ,in It is a tiny positive number, taking the value of ,avoid The sample priority is set to 0 to ensure that all samples have a chance to be sampled. 3-2) Sampling mini-batch samples: Probability-based sampling is implemented using a Sum-Tree data structure. A Sum-Tree is a binary tree in which each leaf node stores the priority of the sample. Non-leaf nodes store the sum of the priorities of their child nodes. During sampling, a randomly generated value is used to find the corresponding leaf node in the Sum-Tree to achieve sampling according to priority ratio. 3-3) Calculate the importance sampling IS weights and correct the sampling bias: The loss function is corrected using the IS weights, as shown in formula (15): (15) in, It is the first The IS weights of each sample, It is the first The sampling probability of a sample. It is the minimum sampling probability in the mini-batch. It is a deviation correction factor; 3-4) Update network parameters and sample priorities simultaneously by combining IS weights: After each mini-batch training, the sample priorities are... The value prediction will be updated, the TD error of the sample will be recalculated, and its priority will be updated. At the same time, the node values ​​of the Sum-Tree will be updated synchronously to ensure that the priority of the next round of sampling is up-to-date. The network parameters are updated in combination with IS weights as shown in formula (16): (16) in This is the loss function value of the Critic network. This is the number of samples in the mini-batch; 4) TD3 Algorithm Model Improvement and Training: Based on the original TD3 algorithm, and addressing the complexity of path planning for UAVs in 3D environments, the TD3-DPATT model was designed. The TD3-DPATT model improves the performance of the original Actor-Critic network structure by integrating a self-attention mechanism and introducing dropout technology into the Actor network architecture. The TD3-DPATT model is trained based on a reward function and a priority experience replay buffer, including: 4-1) Building the TD3-DPATT model: The TD3-DPATT model includes one Actor network and two Critic networks. Each Actor network or Critic network consists of two sub-networks: an online network and a target network. The online Actor network is composed of... Parameterization, the two online Critic networks are respectively composed of and Parameterization; using parameters as The target Actor network and parameters are: and The two target Critic networks, TD3-DPATT, deeply integrate self-attention mechanism and dropout technology into the network structure and training process of TD3, forming an improved solution adapted to multi-user scenarios; 4-1-1) Attention Mechanism Embedded in State Representation: The self-attention mechanism is used as the core feature encoding layer of the Actor or Critic network. After the Actor and Critic networks receive the original state input, the global features are first expanded to user-level dimensions to complete feature preprocessing. Then, the expanded global features are concatenated with the local features of each user to construct user-level sequence features. Finally, a multi-head attention mechanism is used to encode the sequence features in high dimension. The feature transfer efficiency is optimized by combining residual connections and layer normalization, capturing the dependencies between multiple users and generating more representative state features. 4-1-2) Integrating Dropout Technology into the Actor Action Generation Link: Dropout technology is incorporated as controlled noise into the action generation process of the online Actor network. In the link from the hidden layer to the action output layer of the Actor network, during each training iteration, some neurons are dropped out with probability. It was randomly and temporarily shut down; 4-1-3) Parameter Update of TD3-DPATT Model: First, TD3-DPATT uses two independent... and right The value estimates were calculated by two online Critic networks and two target Critic networks, respectively. The minimum value among the values ​​is used to calculate the optimal value. The value is shown in formula (17): (17) in It is a discount factor that smooths out immediate and future rewards; The target Actor network outputs the optimal action for the next time step, and then two online Critic networks... and The update is performed by minimizing the loss function using formula (18): (18) Meanwhile, the online Actor network is updated using policy gradients, while the target Critic and target actor networks are updated using soft updates. The online Actor network update is shown in formula (19), and the target Critic and target actor network updates are shown in formula (20). Indicates the soft update rate: (19) (20); 4-2) Training the TD3-DPATT model: The UAV first preprocesses the current environmental state, using information augmentation and dimensional expansion to enhance key information; then it generates flight actions based on the Actor network, incorporating exploration noise to balance exploring new areas and utilizing known strategies. After the action is executed, the reward signal is calculated based on environmental feedback to generate a new environmental state. Finally, the interaction experience of "current state - executed action - feedback reward - new state - mission termination flag" is stored in the priority experience replay buffer until the data collection task is completed or the maximum number of steps in a single training session is reached. When the interaction data stored in the priority experience buffer reaches the preset amount, the network update process is initiated. Then, the dual Critic network is updated first to optimize the accuracy of action value judgment, and then the Actor network is updated with a delay to improve trajectory decision performance. At the same time, a soft update mechanism is used to synchronize the target network parameters.

[0014] Specifically, in this example: The 3D environment required for the experiment was generated based on a statistical model recommended by the International Telecommunication Union. The location, density, and height of buildings were randomly distributed to simulate real urban occlusion scenarios. The experimental site was set in a 1000×1000m² Cartesian coordinate system. The location and communication link status of the drone and IoT nodes were dynamically calculated based on this environment. This example was implemented on a Linux server equipped with an NVIDIA GeForce RTX 3060 and a development environment using Python 3.8 and PyTorch 1.4. By simulating drone flight, IoT node data upload, and communication link fading, the actual data collection scenario was reproduced. The randomness of experimental data such as IoT node distribution and building height improved the generalization of algorithm verification.

[0015] In the simulation experiment, the core parameters were configured as follows: UAV flight speed The range is Fixed flight altitude Optimized to Flight time per step for Maximum number of training rounds for Maximum time step for a single training round for Number of IoT nodes Values Transmit power of each node for Data file size for SNR threshold for Regarding algorithm hyperparameters, noise was explored. attenuation rate for Experience replay buffer capacity for Soft update rate for Discount factor for mini-batch size for Actor-Critic network learning rate All Furthermore, the Actor uses tanh activation, and the Critic uses ReLU activation, specifically: 1) Initialize the system model, such as Figure 2 As shown, it includes: Step 1: Initialize basic environment parameters and set scene size. The number of drones is set according to different mission requirements. and number of users At the same time, set the maximum number of time steps. for Drone flight altitude User location file name ; Step 2: Construct the environment model, initialize the Urban_world class, and set building generation parameters: maximum building height. Minimum height Road width Based on the parameters, the number, size, location, and height of buildings are generated, and a building height matrix is ​​constructed. Building location and height information matrix If the user location file does not exist, in and Randomly generate user locations within the range and save them to Specify a file; if the file exists, read the user's position from the file. Step 3: Initialize the UAV kinematic model parameters and set the maximum single flight distance for the UAV. When resetting the environment, and Randomly generate the initial drone within the range , Coordinates, fixed flight altitude ,initialization List stores drone instances; Step 4: Initialize communication model parameters: Set signal-to-noise ratio threshold UAV launch power Noise power ; and define The function determines whether it is line-of-sight communication by comparing the pathpoint height with the building height matrix using the waypoints between the drone and the user. Step 5: Initialize state vector parameters: The state vector dimension is... This includes the user's current coverage status, historical coverage status, drone location, and pheromones; when resetting the environment, the state vector is initialized, and the initial coverage status and line-of-sight status are calculated. Set and fill vectors with initial values; Step 6: Initialize task termination parameters: cumulative number of users covered Initialize to 0, initialize Setting it to False indicates whether all user data collection has been completed. Indicate whether the training round has ended; Step 7 Initialize environment interaction and training parameters: Called at the beginning of each training round. Function resets the environment; definition The function serves as the main function for environmental interaction and initializes experience-related interaction metrics, which are used for state, action, and reward interactions in the subsequent training process. 2) Construct a reward function based on the system model, such as... Figure 3 As shown, it includes: Step 8: Determine whether the drone violates boundary constraints after performing the action; if the drone exceeds the map boundary, apply boundary penalty and revert the drone's position to the position before performing the action, then re-enter Step 8; if no constraints are violated, the drone successfully performs the selected action and proceeds to Step 9. Step 9: Use the formula Update pheromone parameters And calculate the number of newly collected users; Step 10: Calculate the reward value for the current step after the reward shaping mechanism. The calculation formula is as follows: ; Step 11: Determine if the data collection task for all users has been completed; if so, proceed to Step 12; if not, proceed to Step 13. Step 12 Calculate the additional remaining time reward for completing all user data collection, using the following formula: ,in ; Step 13: If not all user data is collected, the drone will only receive the reward value for the current step after the reward shaping mechanism, and will not receive the remaining time reward. Step 14: Determine the training rounds Is the task complete? (If so, indicate task completion.) or Then set End of Otherwise, proceed to Step 8 to continue the action execution and reward calculation process for the next time step. 3) Employ the PER mechanism to construct a priority experience replay buffer, such as... Figure 4 As shown, it includes: Step 15 initializes the PER buffer and Sum-Tree data structure, and initializes the binary tree structure. If the initialization is successful, proceed to Step 16 and terminate if an error occurs. Step 16: Perform environment interaction and store experience in the buffer. If the buffer is not full, update the pointer and proceed to Step 17. If the buffer is full, overwrite the oldest experience and proceed to Step 17. Step 17: Determine whether to start network training: Check If the buffer has enough experience, proceed to Step 18; if the experience is insufficient, continue collecting experience and return to Step 16. Step 18: Priority-based tiered sampling. If 256 experiences are successfully sampled, proceed to Step 19; otherwise, resample. Step 19 Calculate importance sampling weights for bias correction: If the weights are calculated successfully, return the empirical data and proceed to Step 20; otherwise, recalculate the weights. Step 20: Calculate the TD error for priority update: If the TD error is successfully calculated, proceed to Step 21; otherwise, re-propagate forward. Step 21: Update network parameters by applying importance weights. If the network update is successful, proceed to Step 22; otherwise, reduce the learning rate and update again. Step 22 Update experience priority to complete the PER loop. If the priority update is successful, proceed to Step 23; otherwise, proceed to Step 20 to recalculate the TD error. Step 23: Perform delayed Actor updates and soft updates to the target network: Determine... Is it equal to If the delayed update condition is met, then update the Actor network and soft-update the target network, and proceed to Step 24; if the delayed update condition is not met, then... Proceed directly to Step 24; Step 24: Determine whether training should continue: Check the current... Whether to end or reach the maximum. If training is not completed, return to Step 16 for environment interaction and experience collection; if training is completed, end the PER training process directly. 4) Train the improved TD3 algorithm model, such as... Figure 5As shown, it includes: Step 25 Initialize the training environment and model parameters: This includes creating the environment, initializing the Actor-Critic architecture and self-attention mechanism of the TD3 network, creating the PER buffer, and setting the exploration noise. and training rounds If initialization is successful, proceed to Step 26; otherwise, check the parameter configuration and re-initialize. Step 26: Begin a new training round And reset the environment state: reset the drone and user positions, obtain the initial state vector. If the environment reset is successful, proceed to Step 27; otherwise, recreate the environment instance. Step 27 Determine the current round Is it finished?: Check The sign, if express Continue, and proceed to Step 28 to select the action; if express If finished, proceed to Step 36. Results recording and preparation for the next round; Step 28: The Actor network selects actions and adds exploration noise: After processing user features through a self-attention layer, the MLP outputs actions, Gaussian noise is added for exploration, and the final action is obtained. If the action is successfully generated, proceed to Step 29; otherwise, retry using the action from the previous step. Step 29 Execute actions and interact with the environment to obtain feedback: Execute drone movement, calculate the new user coverage status, update pheromones, perform boundary checks and reward calculations, and return to the next state. Instant rewards Termination mark Update time step If the environment interaction is successful, then Accumulate rewards and proceed to Step 30; otherwise, return to the drone's position and re-execute the action. Step 30: Store the experience in the priority experience replay buffer: Store the experience quadruple Store the new experience in the PER buffer, assign it the highest priority, and update the leaf nodes of the Sum-Tree. If the experience is successfully stored, proceed to Step 31. If the buffer is full, overwrite the oldest experience and continue storing. Step 31 Determine if network training conditions are met: Check if... If the buffer has sufficient experience, proceed to Step 32 to begin network updates; if the experience is insufficient, proceed to Step 35 to update the status and continue collecting experience. Step 32: Sample training batches from the PER buffer: Perform stratified sampling based on priority to obtain experience batches. If sampling is successful, proceed to Step 33; otherwise, wait for more experience. Step 33 Update Critic parameters: Calculate the target Value, get the current Calculate the TD error. If the Critic update is successful, proceed to Step 34; otherwise, reduce the learning rate and update again. Step 34: Determine whether to perform Actor update: Check the delay counter. Is it equal to If the delay condition is met, update the Actor and reset it. Then proceed to Step 35; if the delay condition is not met, then... Proceed directly to Step 35; Step 35 Update PER priority and record drone trajectory: Update the empirical priority based on TD error, record the drone position at the current time step into the trajectory array. If the priority and trajectory update are successful, return to Step 27 to continue. Loop; otherwise, recalculate the TD error. Step 36: Determine whether to save the model and continue training: Check if... If training is not completed, return to Step 26; begin a new session. If the training is completed, the training process ends.

[0016] To evaluate the effectiveness of the UAV path planning method in this example, UAV data collection time and energy consumption were used as evaluation metrics for the model. During the experiment, the data collection time and energy consumption of the UAV in each training round were recorded in a log file. At the same time, the improved TD3 algorithm stored a quadruple consisting of the UAV's previous time step state, current state, reward value, and executed action in the experience pool at each time step in each training round. Subsequently, priority experience replay was used to extract small batches of samples from the experience pool for parameter training of the neural network. If the parameter value used to evaluate the performance of the neural network at the current time step is higher than the parameter value at the previous time step, the current parameters of the neural network were saved to a file.

[0017] Simulation Verification: Currently, the main performance indicators for evaluating UAV data acquisition path planning methods are task completion time, flight energy consumption, and algorithm convergence stability. Task completion time represents the average time it takes for the UAV to complete data collection from all IoT nodes. Flight energy consumption reflects the energy consumption efficiency of the UAV in performing the task. Convergence stability reflects the fluctuation of reward value and convergence speed during algorithm training. In scenarios with different numbers of IoT nodes, the comprehensive performance of the method is evaluated by comparing these indicators.

[0018] Following the method in this example, we obtain: Figure 6 The simulation results for task completion time and flight energy consumption of the example method TD3-DPATT, control method 1, control method 2, and control method 3 under different numbers of IoT nodes are shown in the figure. Among them, the example method is based on the TD3 algorithm and introduces four core techniques: information enhancement (integrating "pheromone" to represent the interaction state between the environment and the UAV), reward shaping (converting sparse rewards into dense rewards), PER buffer (sampling based on sample priority to improve reinforcement learning efficiency), and self-attention mechanism (making the algorithm focus on high-value core features). Control method 1 (TD3-TDCTM) designs a new reward function based on the TD3 algorithm to guide the UAV in trajectory optimization, and also optimizes the original structure of the TD3 algorithm. Control method 2 optimizes the shortest access path based on the ant colony algorithm (RRT), and control method 3 explores the node access sequence based on the random tree algorithm (RRT).

[0019] In the experiment, all methods were tested in a 3D urban environment of 1000×1000m², with the number of IoT nodes ranging from 5 to 35. The UAV's fixed flight altitude was optimized to 95 m to ensure communication quality. Control method 1 used a common experience buffer, and the network lacked the ability to focus on key features, resulting in low algorithm decision-making efficiency and slow algorithm convergence speed. Control method 2 did not utilize the UAV's environmental perception capabilities, and path redundancy was still significant. Control method 3's exploration mechanism was random, and the optimization effect was limited. Therefore, in a 3D urban environment and scenarios with different numbers of IoT nodes, the method in this example, through the synergistic effect of four core techniques, shortened the task completion time compared to the control method, reduced the UAV's flight energy consumption, and maintained stable convergence performance, resulting in superior overall performance.

Claims

1. A method for UAV data acquisition path planning based on an improved TD3 algorithm, characterized in that, Includes the following steps: 1) Constructing the system model: The system model includes an environment model, a UAV kinematics model, and a communication model, specifically: 1-1) Constructing an environment model: Generate an environment model containing multiple buildings and with a size of [missing information]. A 3D scene, in which randomly distributed... The first fixed-location IoT node, the second The positions of the nodes and the UAV are represented by formulas (1) and (2), respectively, where Indicates the first The horizontal coordinates of each IoT node It is the drone at all times Position on the horizontal plane, It is the drone's flight altitude. This refers to the duration of task execution. （1）， （2）， The specific location and height of buildings are generated using a statistical model recommended by the International Telecommunication Union (ITU). Building data is generated based on the Rayleigh distribution mean of building coverage ratio, number of buildings per unit area, and building height distribution. 1-2) Constructing the UAV kinematic model: Discretizing the continuous time domain into... There are time steps, each with a length of . ,and They are not necessarily equal; each time step consists of a fixed flight time. and the hover time used for data It consists of two parts. If there are no nodes at the current location where data can be collected, the hovering time is 0. The drone adjusts its speed at each time step. and horizontal direction angle Update its horizontal position At the same time, maintain a high altitude Furthermore, the drone must always meet the constraints shown in formula (3) during flight: （3）； 1-2-1) The position update formula for the UAV is expressed by formula (4), where Indicates the distance it traveled: （4）； 1-2-2) The distance traveled by the drone is equal to the fixed flight time. Multiply by the average flight speed at the current time step , represented as ; 1-2-3) The objective of minimizing the total task completion time is... It means that, among them ; 1-3) Constructing the communication model: A practical "ground-to-air" G2A channel model is adopted. The communication model is represented by large-scale fading and small-scale fading. These fading values ​​are calculated based on a simulated 3D map that considers buildings as propagation scatterers, as detailed below: 1-3-1) Calculating large-scale fading: Based on the inspection of drones and the first To accurately determine whether the direct communication link between IoT nodes is blocked by buildings, the following information is needed: Does a line-of-sight link exist between the IoT nodes? And what about the drone and the...? The large-scale fading of the G2A channel associated with an IoT node is expressed by formula (5): （5）， in Indicates drones and the first Free space path loss between IoT nodes Indicates drones and the first The distance between IoT nodes Indicates the carrier frequency. Represents the speed of light. Indicates a line-of-sight link. and These represent the propagation losses for line-of-sight links and non-line-of-sight links, respectively. 1-3-2) Calculation of Small-Scale Fading and Information Gain: For non-line-of-sight links, the small-scale fading coefficient... Assuming Rayleigh fading, for line-of-sight links, then... Assuming Rice fading with a Rice factor of 15 dB, the drone reaches the... The channel gain of each IoT node is shown in formula (6): （6）； 1-3-3) Calculation of Signal-to-Noise Ratio (SNR) and Transmission Rate: The IoT node can only transmit at a constant power when the drone wakes it up. Start uploading data; otherwise, they will remain silent until the [date / time]. In the first time step, the first The uplink signal-to-noise ratio between an IoT node and a drone is expressed by formula (7): （7）， in It is the first Channel gain during each time step hovering phase This represents the additive white Gaussian noise power at the UAV receiver, for the... Uplink data collection services related to each IoT node, setting a predefined signal-to-noise ratio threshold. Only when At that time, the first Only when a certain number of IoT nodes can be woken up and provided with services by the drone, is defined to indicate the first IoT node. Can the first IoT node be in the first...? The indicator function that satisfies the signal-to-noise ratio requirement at each time step is shown in formula (8): （8）， Assuming that each IoT node is served at most once in a single implementation, then formula (9) represents the first... The indication function for each IoT node, where It is a binary variable used to indicate the first... Are the IoT nodes already being served by drones? （9）， if Then the first The first IoT node has been served during the task; otherwise, the second... One IoT node was not serviced, and then the drone and the first The transmission rate between IoT nodes is expressed as shown in formula (10): （10）； 1-3-4) Calculation of drone hovering time: Only in the first... One IoT node Only when the first time step is served can the drone upload the first time step. The data comes from the IoT nodes, therefore, the drone's hovering time is the [number]th [node]. The maximum duration of data upload from the served IoT nodes in each time step is represented by formula (11): （11）， in Indicates the first The size of the information file for each IoT node This is to prevent values ​​with a denominator of 0; 2) Reward Function Design: Guiding the drone to complete the data collection task of all IoT nodes in the shortest time is a typical "sparse reward" problem, meaning there is only a clear feedback when the task is completed. A reward shaping mechanism is used to transform the original sparse reward into a dense reward. The reward function consists of three parts: time consumption penalty, data collection reward, and out-of-bounds / violation penalty. A pheromone mechanism is integrated into the reward function, as detailed below: 2-1) Definition of UAV pheromones: Pheromones are virtual "odor" distributions that reflect environmental conditions and are used to guide UAVs to unserved nodes. The definition is shown in formula (12): （12）， in It is in the Remaining pheromones in each time step It is the drone in the The number of IoT nodes served in each time step It is a positive constant representing the pheromone captured by each IoT node. It is a positive constant representing the loss of pheromones. This refers to the penalty when an action results in a drone crossing boundaries or violating regulations; 2-2) Definition of reward function: By integrating the reward shaping mechanism and pheromones into the reward function, the final definition of the reward function is shown in formula (13): （13）， in It is caused by pheromones The resulting reward function, approximate The function is a function, but its gradient is smoother than the latter, and the drone receives a reward for the remaining time when the mission is completed. ,in It is the maximum time step. It is the time step required to complete the task; 3) Constructing a Priority Experience Replay Buffer using the PER mechanism: The PER mechanism samples from the experience replay buffer according to priority. The priority is proportional to the absolute value of the time difference (TD) error. Samples with large TD errors indicate that the Critic network's Q-value estimation for these samples is inaccurate, while samples with small TD errors indicate that the Critic network is overfitting. The core objective of PER is to prioritize sampling based on the "importance" of the samples, specifically implemented as follows: 3-1) Experience Sample Storage and Initialization Calculation: The experience samples generated by the interaction between the UAV and the environment are stored in the form of quadruples. Stored in the buffer, where Indicates the first The state at each time step Indicates the first The action at each time step Indicates the first Rewards for each time step Indicates the execution of an action After transitioning to the next state, when new samples are stored, the priority experience replay buffer will calculate the initial TD error for each sample, as shown in formula (14): （14）， in It is the first TD error for each sample The goal value, These represent two independent Critic network pairs in the TD3 algorithm. of Value prediction, then sample prioritization based on TD3 error. To ensure that new samples have an initial sampling probability, the sample priority is calculated as follows. ,in It is a tiny positive number, taking the value of ,avoid The sample priority is set to 0 to ensure that all samples have a chance to be sampled. 3-2) Sampling mini-batch samples: Probability-based sampling is implemented using a Sum-Tree data structure. A Sum-Tree is a binary tree in which each leaf node stores the priority of the sample. Non-leaf nodes store the sum of the priorities of their child nodes. During sampling, a randomly generated value is used to find the corresponding leaf node in the Sum-Tree to achieve sampling according to priority ratio. 3-3) Calculate the importance sampling IS weights and correct the sampling bias: The loss function is corrected using the IS weights, as shown in formula (15): （15）， in, It is the first The IS weights of each sample, It is the first The sampling probability of a sample. It is the minimum sampling probability in the mini-batch. It is a deviation correction factor; 3-4) Update network parameters and sample priorities simultaneously by combining IS weights: After each mini-batch training, the sample priorities are... The value prediction will be updated, the TD error of the sample will be recalculated, and its priority will be updated. At the same time, the node values ​​of the Sum-Tree will be updated synchronously to ensure that the priority of the next round of sampling is up-to-date. The network parameters are updated in combination with IS weights as shown in formula (16): （16）， in This is the loss function value of the Critic network. This is the number of samples in the mini-batch; 4) TD3 Algorithm Model Improvement and Training: Based on the original TD3 algorithm, and addressing the complexity of path planning for UAVs in 3D environments, the TD3-DPATT model was designed. The TD3-DPATT model improves the performance of the original Actor-Critic network structure by integrating a self-attention mechanism and introducing dropout technology into the Actor network architecture. The TD3-DPATT model is trained based on a reward function and a priority experience replay buffer, including: 4-1) Building the TD3-DPATT model: The TD3-DPATT model includes one Actor network and two Critic networks. Each Actor network or Critic network consists of two sub-networks: an online network and a target network. The online Actor network is composed of... Parameterization, the two online Critic networks are respectively composed of and Parameterization; using parameters as The target Actor network and parameters are: and The two target Critic networks, TD3-DPATT, deeply integrate self-attention mechanism and dropout technology into the network structure and training process of TD3, forming an improved solution adapted to multi-user scenarios; 4-1-1) Attention Mechanism Embedded in State Representation: The self-attention mechanism is used as the core feature encoding layer of the Actor or Critic network. After the Actor and Critic networks receive the original state input, the global features are first expanded to user-level dimensions to complete feature preprocessing. Then, the expanded global features are concatenated with the local features of each user to construct user-level sequence features. Finally, a multi-head attention mechanism is used to encode the sequence features in high dimension. The feature transfer efficiency is optimized by combining residual connections and layer normalization, capturing the dependencies between multiple users and generating more representative state features. 4-1-2) Integrating Dropout Technology into the Actor Action Generation Link: Dropout technology is incorporated as controlled noise into the action generation process of the online Actor network. In the link from the hidden layer to the action output layer of the Actor network, during each training iteration, some neurons are dropped out with probability. It was randomly and temporarily shut down; 4-1-3) Parameter Update of TD3-DPATT Model: First, TD3-DPATT uses two independent... and right The value estimates were calculated by two online Critic networks and two target Critic networks, respectively. The minimum value among the values ​​is used to calculate the optimal value. The value is shown in formula (17): （17）， in It is a discount factor that smooths out immediate and future rewards; The target Actor network outputs the optimal action for the next time step, and then two online Critic networks... and The update is performed by minimizing the loss function using formula (18): （18）， Meanwhile, the online Actor network is updated using policy gradients, while the target Critic and target actor networks are updated using soft updates. The online Actor network update is shown in formula (19), and the target Critic and target actor network updates are shown in formula (20). Indicates the soft update rate: （19）， （20）； 4-2) Training the TD3-DPATT model: The UAV first preprocesses the current environmental state, using information augmentation and dimensional expansion to enhance key information; then it generates flight actions based on the Actor network, incorporating exploration noise to balance exploring new areas and utilizing known strategies. After the action is executed, the reward signal is calculated based on environmental feedback to generate a new environmental state. Finally, the interaction experience of "current state - executed action - feedback reward - new state - mission termination flag" is stored in the priority experience replay buffer until the data collection task is completed or the maximum number of steps in a single training session is reached. When the interaction data stored in the priority experience buffer reaches a preset amount, the network update process is initiated. Then, the dual Critic network is updated first to optimize the accuracy of action value judgment, and then the Actor network is updated with a delay to improve trajectory decision performance. At the same time, a soft update mechanism is used to synchronize the target network parameters.

Images