A heterogeneous unmanned aerial vehicle data ferry path planning method based on reinforcement learning
By using the HRMA-MATD3 algorithm, leveraging the role-aware multi-stream attention mechanism and decentralized decision-making model, the problem of low efficiency in UAV role differentiation and policy learning in heterogeneous UAV data transfer tasks is solved. This achieves efficient data acquisition and caching forwarding, improving the system's adaptability and collaborative efficiency.
Patent Information
- Application Number
- CN202611149239.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-30
- Publication Date
- 2026-11-10
AI Technical Summary
Existing UAV data acquisition and relay methods struggle to effectively distinguish different UAV roles in heterogeneous UAV collaborative tasks, resulting in low policy learning efficiency, insufficient generalization ability, and difficulty in adapting to dynamic ground user needs and complex communication environments.
The HRMA-MATD3 algorithm based on reinforcement learning is adopted. Through role-aware multi-stream attention mechanism and distributed decision-making model, the task responsibilities of data collection drones and relay drones are distinguished. A multi-agent collaborative path planning method is constructed. Combined with data flow, link connection, priority guidance and safety penalty reward function, the distributed autonomous decision-making of drones is realized.
It improves data acquisition and buffering/forwarding efficiency in integrated air-space-ground communication scenarios, reduces end-to-end latency, enhances system adaptability and collaborative efficiency, and avoids the problem of symmetrical UAV actions.
Smart Images

Figure CN123065461A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of collaborative control of unmanned aerial vehicle (UAV) swarms, and specifically relates to a method for heterogeneous UAV data transfer path planning based on reinforcement learning. Background Technology
[0002] In recent years, with the development of technologies such as unmanned aerial vehicle (UAV) communication, low-Earth orbit (LEO) satellite communication, and deep reinforcement learning (DRL), integrated air-space-ground communication networks have received widespread attention in scenarios such as emergency communication, communication in remote areas, and data transmission in areas with scarce infrastructure. UAVs offer advantages such as flexible deployment, high mobility, and low cost, enabling them to provide temporary data collection services to ground users in areas where ground communication facilities are difficult to cover or are damaged. LEO satellite networks, with their wide coverage and strong backhaul capabilities, can provide long-distance transmission channels for data collected by UAVs. Therefore, combining UAVs with LEO satellites to build a data transfer communication system for ground users has significant application value.
[0003] In unmanned aerial vehicle (UAV) data transfer systems, different UAVs typically perform different tasks. Data acquisition UAVs are primarily responsible for collecting latency-sensitive data from ground users and temporarily storing the data in an onboard cache; relay UAVs are mainly responsible for approaching the data acquisition UAV carrying the cached data and forwarding its data to the satellite backhaul network. Because data acquisition UAVs and relay UAVs differ in their communication targets, flight altitudes, mission objectives, and decision-making requirements, traditional homogeneous UAV path planning methods are insufficient to effectively describe this heterogeneous collaborative relationship.
[0004] Existing UAV data acquisition and relay methods mostly employ heuristic optimization, clustering, trajectory planning, or traditional reinforcement learning. While these methods can achieve some success in specific scenarios, they typically assume consistent UAV roles, static task requirements, or sufficient communication resources. This makes them ill-suited for complex scenarios such as dynamically updated ground user needs, limited UAV cache capacity, varying link quality based on location, and collaborative relaying by multiple UAV types. Furthermore, conventional deep reinforcement learning methods often directly concatenate observation information from different types of entities into a unified vector, making it difficult to distinguish the different semantic relationships between ground users, acquisition UAVs, and relay UAVs. This results in low policy learning efficiency and insufficient generalization ability.
[0005] Multi-Agent Deep Reinforcement Learning (MADRL) enables multiple agents to learn collaborative decision-making strategies through interaction with the environment, making it suitable for drone swarm control in dynamic and uncertain environments. However, in heterogeneous drone data transfer tasks, if all drones share the same policy network, action symmetry problems can easily arise, causing multiple drones to perform similar behaviors, resulting in duplicated services, local clustering, or uneven utilization of relay resources. While attention mechanisms can improve the representation of multi-entity environments, existing attention methods typically do not explicitly distinguish between different entity types, making it difficult to fully characterize the role differences between data collection drones and relay drones in data collection, cache unloading, and relay rendezvous processes. Therefore, designing an intelligent path planning method that can perceive drone roles, distinguish the semantics of multiple entity types, and balance data collection efficiency, information timeliness, and safe flight constraints remains an urgent problem to be solved. Summary of the Invention
[0006] The purpose of this invention is to address the problem of heterogeneous UAV data ferry path planning and relay forwarding decision-making, proposing a reinforcement learning-based heterogeneous UAV data ferry path planning method to improve the efficiency of ground user data acquisition, buffer forwarding efficiency, and service timeliness in integrated air-space-ground communication scenarios. To achieve this objective, the steps employed in this invention are as follows:
[0007] Step 1: Build a heterogeneous UAV data transfer system architecture. This system architecture consists of five parts: ground users, data acquisition UAVs, relay UAVs, satellite backhaul network, and intelligent decision-making model. Ground users generate latency-sensitive data, data acquisition UAVs collect and cache ground user data, relay UAVs receive cached data from data acquisition UAVs and forward it to the satellite backhaul network, and intelligent decision-making model generates path planning and collaborative control decisions for each UAV.
[0008] Step 2: Establish a system model for the heterogeneous UAV data transfer mission; divide the mission area into a two-dimensional horizontal region, and discretize the mission execution process into multiple time steps; let the mission area be... Discrete-time set is Then we have:
[0009] A = {(x, y) | 0 ≤ x ≤ X} max , 0≤y≤Y max T = 0, 1, ..., T max -1} (1)
[0010] Among them, X max and Y max These represent the length and width of the task area, respectively, and T represents the task time steps. maxThis represents the maximum number of task time steps, with each time step having a length of Δt. x and y represent the planar coordinates in the two-dimensional horizontal task region.
[0011] Let the collection of drones be... Relay drones are All drones are grouped together For drone i with role κ, its set is: Where κ∈C,R, and its three-dimensional position q i (t) and horizontal position p i (t) are respectively represented as:
[0012]
[0013] Where C represents the data collection drone role, R represents the relay drone role, and h κ h represents the flight altitude of the drone with the role of κ. C Indicates the flight altitude of the drone, h R Indicates the flight altitude of the relay drone, x i (t) and y i (t) represent the horizontal coordinates of drone i at time step t; by setting h C ≠h R This allows the data acquisition drone and the relay drone to be located at different altitudes to reflect the differences in their roles in data acquisition and relay tasks.
[0014] At each time step, the UAV generates a horizontal speed command based on the continuous actions output by the intelligent decision-making model; let the speed command of UAV i at time step t be... The current horizontal velocity is v i (t), maximum acceleration is Its acceleration is then updated as follows:
[0015]
[0016] Where Δt represents the duration of a single time step, a represents the unconstrained expected acceleration. i (t) represents the actual acceleration after acceleration constraints.
[0017] Furthermore, the drone's speed and position are updated based on the constrained acceleration:
[0018]
[0019] in, This represents the maximum horizontal speed of drone i; the above motion model can prevent instantaneous changes in drone speed that do not conform to physical constraints.
[0020] Step 3: Establish the communication link model and data buffer model; assuming the ground user set is... The position of ground user m at time step t is s. m (t)=[x m (t), y m (t), 0] T The data acquisition link distance between data acquisition drone i and ground user m, and the data relay link distance between relay drone j and data acquisition drone i are respectively:
[0021]
[0022] in, This indicates the three-dimensional distance from the drone to the ground user. This represents the three-dimensional distance between the relay drone and the data acquisition drone; if Then the data acquisition distance between the drone i and the ground user m meets the data acquisition distance condition; if Then the data relay distance condition is satisfied between relay UAV j and data acquisition UAV i; where R CG and R RC These represent the maximum communication distances of the data acquisition link and the data relay link, respectively.
[0023] For any distance The wireless link employs a distance-dependent line-of-sight channel gain model, with link gain g(t) and link transmission rate r... χ (t) is:
[0024]
[0025] Where β0 represents the channel gain at the reference distance, α represents the path loss exponent, and B χ N represents the total bandwidth allocated to link type χ. χ (t) represents the number of links simultaneously served by the same launched UAV at time step t, P tx χ represents the transmit power, N0 represents the noise power spectral density, χ = CG represents the link from the data acquisition UAV to the ground user, and χ = RC represents the link from the relay UAV to the data acquisition UAV. Indicates the distance of a general wireless link.
[0026] set up This indicates the data acquisition correlation indicator variable between the data acquisition drone i and the ground user m. This indicates the data relay association variable between relay UAV j and data acquisition UAV i; the remaining data volume of ground user m at time step t is b. m (t), the amount of cached data collected by drone i is B. i(t), whose maximum cache capacity is The amount of data collected by drone i from ground user m is:
[0027]
[0028] Among them, c i,m (t) represents the amount of data transmitted from ground user m to data acquisition drone i within one time step. This represents the data acquisition rate between the drone i and the ground user m.
[0029] The amount of data collected by drone i at time step t and unloaded by relay drone is represented as follows:
[0030]
[0031] in, This indicates the amount of cached data that drone i completed unloading at this time step. This represents the data relay rate between relay drone j and data acquisition drone i.
[0032] Furthermore, the remaining data volume for ground users and the cache size of the data collection drones are updated according to the following formula:
[0033]
[0034] Among them, when b m When (t+1)=0, the current data demand of ground user m is considered to be completed and enters the cooling or refresh state; when the cooling ends, the ground user generates a new data demand.
[0035] Step 4: Establish the optimization objective for heterogeneous UAV data transfer path planning. The objective of this task is to enable the data collection UAV to collect more ground user data, the relay UAV to forward more cached data, and reduce the age of ground user data, while satisfying UAV motion constraints, communication distance constraints, link association constraints, and cache capacity constraints. η represents the set of ground users who are active at time step t. m (t) represents the data age of ground user m, c i,m (t) represents the amount of data collected by UAV i from ground user m within time step t. Let represent the amount of cached data collected from drone i to the relay drone within time step t. Then, the optimization objective is expressed as:
[0036]
[0037] Where Π represents the set of trajectory control decisions for all UAVs, and λ c λ represents the data collection weight.r λ represents the cache forwarding weight. a λ represents the average data age penalty weight. m T represents the maximum data age penalty weight. max This represents the maximum number of discrete-time steps in the task execution, where t represents the discrete-time step index. Let represent the set of data collection drones, M represent the set of ground users, i represent the index of the data collection drones, and m represent the index of the ground users. This objective function simultaneously considers throughput, cache unloading capacity, average information timeliness, and worst-case user waiting time.
[0038] Step 5: Model the heterogeneous UAV data transfer path planning problem as a distributed partially observable Markov decision process, which is represented as follows: in Represents the global state space. Represents the joint action space, Represents the state transition function. Represents the reward function, The observation function is represented by γ, which represents the discount factor. The global environmental state includes the position and velocity of all UAVs, the remaining data volume of ground users, the age of ground user data, the activity status of ground users, and the buffer status of the UAVs collecting data. Each UAV independently outputs continuous two-dimensional actions based solely on its own local observations.
[0039]
[0040] Among them, a i (t) represents the normalized action output by the intelligent decision-making model. This represents the speed command obtained from the motion mapping, a i,x (t) and a i,y (t) represent the components of the normalized action in the x and y directions, respectively. This represents the maximum horizontal speed of drone i. The link association is not directly output as an agent action, but is automatically determined based on communication distance, channel gain, buffer status, and bandwidth sharing rules.
[0041] Step 6: Based on the MATD3 (Multi-Agent Twin Delayed Deep Deterministic Policy Gradient) algorithm, a reinforcement learning-based heterogeneous UAV data transfer path planning method, HRMA-MATD3 (Heterogeneous Relational Multi-Stream Attention enhanced MATD3), is proposed to improve the overall data transfer rate and reduce end-to-end latency. This algorithm first constructs a policy network based on role-aware multi-stream attention. For each UAV, its observation information is divided into ground user stream, data acquisition UAV stream, and relay UAV stream according to entity type. For any semantic stream, the entity features in that stream are... With learnable type embedding u ρ The data is then stitched together and embedded using an independent encoder.
[0042]
[0043] in, This represents the encoded features of entity e in the semantic flow ρ. This represents the feature encoding network corresponding to the semantic stream, ρ represents the semantic stream type index, satisfying ρ∈GU,C,R, where GU,C and R represent the ground user stream, the data collection UAV stream, and the relay UAV stream, respectively; e represents the entity index in the semantic stream. u represents the input feature of entity e in the semantic flow ρ. ρ This represents the learnable type embedding corresponding to the semantic flow ρ. This represents the concatenation result of entity features and type embedding.
[0044] To differentiate the mission responsibilities of data collection drones and relay drones, the state x of each drone is defined. i Later, role coding was introduced. i And generate attention query vectors:
[0045]
[0046] Among them, e i f is used to characterize the role type of drone i. q (·) represents the query vector generation network, x i The state characteristics of UAV i are represented by role coding, which enables the parameter sharing policy network to form differentiated decision-making behaviors for data acquisition UAVs and relay UAVs.
[0047] Furthermore, a multi-head scaling dot product attention mechanism is applied to each type of semantic stream. For the h-th attention head of semantic stream ρ, its attention weights and context vectors are as follows:
[0048]
[0049] in, This represents the attention weight of drone i on entity e. Let h represent the context vector output by the h-th attention head in the semantic flow ρ. and These represent the query, key, and value vectors, respectively, d h The feature dimension of a single attention head. Let represent the set of entities observable by drone i under semantic flow ρ, and let e′ represent the entity index in the attention weight normalization summation process.
[0050] The outputs of each attention head are concatenated to obtain the semantic flow context features, and the drone actions are output by the shared policy network:
[0051]
[0052] Where H represents the number of attention heads. f represents the output projection matrix. π (·) represents the policy network. The context features corresponding to the semantic flow ρ are represented by tanh(·), which is used to constrain the action within the range of [-1, 1]. Through this structure, the data collection drone can focus on ground users with high data age, high amount of remaining data, or close proximity, while also perceiving relay drones that can be used for cache unloading; the relay drone can focus on data collection drones with high cache amount, close proximity, or greater need for relay services.
[0053] Step 7: Construct a hybrid reward function for the data transfer task. This function consists of data flow reward, connectivity reward, priority guidance reward, security penalty, and artificial potential field reward. It guides the data collection drone to switch between "collecting ground user data" and "finding a relay drone to unload cache," and guides the relay drone to actively approach the data collection drone carrying cached data. For drone i, the total reward function obtained at time step t is expressed as:
[0054] r i (t)=ω f r i flow (t)+ω l r i link (t)+ω p r i pri(t)-ω s r i safe (t)+ω Φ (Φ i (t)-Φ i (t-1))(16)
[0055] Where, r i flow (t) represents the data stream reward, used to incentivize data collection by drones or data relay by drones; r i link (t) represents the connection reward, used to encourage the establishment of effective communication links; r i pri (t) represents priority-guided rewards, which determine priority service targets based on ground user data age, target distance, and data collection drone buffer occupancy rate; r i safe (t) represents a safety penalty used to restrict drones from crossing boundaries and other drones from approaching too closely; Φ i (t)-Φ i (t-1) represents the change in the artificial potential field, used to encourage drones to approach effective targets and avoid clustering of similar drones; ω f ω l ω p ω s and ω Φ These represent the weights of the corresponding reward items.
[0056] Step 8: Train the intelligent decision-making model using a multi-agent reinforcement learning framework with centralized training and distributed execution. During the training phase, evaluate the network's ability to assess policy quality using joint observations and joint actions. During the execution phase, each UAV independently outputs motion actions based solely on its own local observations, semantic flow attention features, and role encoding. Let u... θ Represents a shared policy network. and This represents the two centralized evaluation networks corresponding to UAV i, and the transfer samples (o) obtained from the experience replay pool. t u t r t o t+1 The target action and target value are calculated as follows:
[0057]
[0058] Where θ′ represents the target policy network parameters, The network parameters represent the target evaluation parameters, ∈ represents truncated Gaussian noise, γ represents the discount factor, and o t u represents the joint observation at time step t. tr represents the joint action at time step t. i,t o represents the reward obtained by drone i at time step t. t+1 This indicates the joint observation at the next time step. This indicates the target motion after adding truncated Gaussian noise. This represents the target evaluation value obtained by taking the smaller value from the outputs of the two target evaluation networks, y. i Let represent the target Q value corresponding to UAV i, and l represent the evaluation network number. l = 1 or l = 2. By taking the smaller value from the two evaluation networks, the risk of overestimation of the evaluation network can be reduced.
[0059] The network loss function is expressed as:
[0060]
[0061] The policy network is optimized using a delayed update method, and its loss function is expressed as:
[0062]
[0063] Where N represents the number of drone agents, u -i,t o represents the set of actions of all drones except drone i. i,t The local observation of UAV i at time step t is represented by the method of combining centralized training and distributed execution. This invention improves the stability of policy learning by utilizing global information during the training phase and maintains the distributed autonomous decision-making capability of UAV during the execution phase. Attached Figure Description
[0064] Figure 1 This is a schematic diagram of the overall scenario of a heterogeneous drone data transfer system;
[0065] Figure 2 This is a structural diagram of a heterogeneous unmanned aerial vehicle (UAV) collaborative decision-making model;
[0066] Figure 3 This is a comparison chart of the training results of our method and the baseline method. Detailed Implementation
[0067] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.
[0068] In the following description, this specification will refer to the reinforcement learning-based heterogeneous UAV data ferry path planning method proposed in this invention as HRMA-MATD3 (Heterogeneous Relational Multi-Stream Attentionenhanced Multi-Agent Twin Delayed Deep Deterministic Policy Gradient).
[0069] HRMA-MATD3 first sets the following operating conditions:
[0070] Taking the heterogeneous UAV data transfer path planning task as an example, (see attached) Figure 1 This is a schematic diagram of the overall scenario of the heterogeneous UAV data transfer system proposed in this invention, consisting of ground users, data acquisition UAVs, relay UAVs, and a satellite backhaul network. During mission execution, the data acquisition UAVs plan their paths based on the data needs of the ground users and their own cache status, and collect latency-sensitive data generated by the ground users within their communication range. The relay UAVs rendezvous with the data acquisition UAVs based on their location and cache status, and forward the cached data from the data acquisition UAVs to the satellite backhaul network. This system enables data transfer services in areas with scarce infrastructure through the coordinated movement and data forwarding of heterogeneous UAVs.
[0071] Taking the role-aware multi-stream attention reinforcement learning network structure as an example, (see attached) Figure 2 The proposed heterogeneous UAV collaborative decision-making model structure is presented. During training, each UAV first obtains its own state, the ground user's state, the states of other UAVs, and role information, and forms the input to the policy network through the observation encoding module. The policy network divides different types of entities into multiple semantic streams, extracts attention features for each, and then outputs the continuous motion actions of each UAV through the shared policy network. The environment calculates rewards based on UAV motion, communication links, cache changes, and data refresh status, and stores sample data in the experience replay unit. The evaluation network updates network parameters based on joint observations and joint actions, thereby completing the training of the intelligent decision-making model.
[0072] Based on the above conditions, the HRMA-MATD3 proposed in this invention has been implemented in a Python simulation environment, and simulation results demonstrate the effectiveness of the method. The specific implementation steps of HRMA-MATD3 are as follows:
[0073] Step 1: Establish a heterogeneous UAV data transfer system architecture. This architecture consists of five parts: ground users, data acquisition UAVs, relay UAVs, satellite backhaul network, and intelligent decision-making model. Ground users generate latency-sensitive data; data acquisition UAVs collect and cache ground user data; relay UAVs receive cached data from data acquisition UAVs and forward it to the satellite backhaul network; and the intelligent decision-making model generates path planning and collaborative control decisions for each UAV.
[0074] Step 2: Establish a system model for the heterogeneous UAV data transfer mission. Divide the mission area into a two-dimensional horizontal region and discretize the mission execution process into multiple time steps. Let the mission area be... Discrete-time set is Then we have:
[0075] A = {(x, y) | 0 ≤ x ≤ X} max , 0≤y≤Y max T = 0, 1, ..., T max -1} (20)
[0076] Among them, X max and Y max These represent the length and width of the task area, respectively, and T represents the task time steps. max This represents the maximum number of task time steps, with each time step having a length of Δt. x and y represent the planar coordinates in the two-dimensional horizontal task region.
[0077] Let the collection of drones be... Relay drones are All drones are grouped together For drone i with role κ, its set is: Where κ∈C,R, and its three-dimensional position q i (t) and horizontal position p i (t) are respectively represented as:
[0078]
[0079] Where C represents the data collection drone role, R represents the relay drone role, and h κ h represents the flight altitude of the drone with the role of κ. C Indicates the flight altitude of the drone, h R Indicates the flight altitude of the relay drone, x i (t) and y i (t) represent the horizontal coordinates of drone i at time step t; by setting h C ≠h RThis allows the data acquisition drone and the relay drone to be located at different altitudes to reflect the differences in their roles in data acquisition and relay tasks.
[0080] At each time step, the UAV generates a horizontal speed command based on the continuous actions output by the intelligent decision-making model. Let the speed command of UAV i at time step t be... The current horizontal velocity is v i (t), maximum acceleration is Its acceleration is then updated as follows:
[0081]
[0082] Where Δt represents the duration of a single time step, a represents the unconstrained expected acceleration. i (t) represents the actual acceleration after acceleration constraints.
[0083] Furthermore, the drone's speed and position are updated based on the constrained acceleration:
[0084]
[0085] in, This represents the maximum horizontal speed of drone i. The above motion model helps avoid sudden, abrupt changes in drone speed that violate physical constraints.
[0086] Step 3: Establish the communication link model and data buffer model. Let the set of ground users be... The position of ground user m at time step t is s. m (t)=[x m (t), y m (t), 0] T The data acquisition link distance between data acquisition drone i and ground user m, and the data relay link distance between relay drone j and data acquisition drone i are respectively:
[0087]
[0088] in, This indicates the three-dimensional distance from the drone to the ground user. This represents the three-dimensional distance between the relay drone and the data collection drone. If... Then the data acquisition distance between the drone i and the ground user m meets the data acquisition distance condition; if Then the data relay distance condition is satisfied between relay drone j and data collection drone i. Where R CG and R RC These represent the maximum communication distances of the data acquisition link and the data relay link, respectively.
[0089] For any distance The wireless link employs a distance-dependent line-of-sight channel gain model, with link gain g(t) and link transmission rate r... χ (t) is:
[0090]
[0091] Where β0 represents the channel gain at the reference distance, α represents the path loss exponent, and B χ N represents the total bandwidth allocated to link type χ. χ (t) represents the number of links simultaneously served by the same launched UAV at time step t, P tx χ represents the transmit power, N0 represents the noise power spectral density, χ = CG represents the link from the data acquisition UAV to the ground user, and χ = RC represents the link from the relay UAV to the data acquisition UAV. Indicates the general wireless link distance.
[0092] set up This indicates the data acquisition correlation indicator variable between the data acquisition drone i and the ground user m. This variable represents the data relay association indicator between relay UAV j and data acquisition UAV i. The remaining data volume of ground user m at time step t is b. m (t), the amount of cached data collected by drone i is B. i (t), whose maximum cache capacity is The amount of data collected by drone i from ground user m is:
[0093]
[0094] Among them, c i,m (t) represents the amount of data transmitted from ground user m to data acquisition drone i within one time step. This represents the data acquisition rate between the drone i and the ground user m.
[0095] The amount of data collected by drone i at time step t and unloaded by relay drone is represented as:
[0096]
[0097] in, This indicates the amount of cached data that drone i completed unloading at this time step. This represents the data relay rate between relay drone j and data acquisition drone i.
[0098] Furthermore, the remaining data volume for ground users and the cache size of the data collection drones are updated according to the following formula:
[0099]
[0100] Among them, when b m When (t+1)=0, the current data demand of ground user m is considered to be completed and enters the cooling or refresh state; when the cooling ends, the ground user generates a new data demand.
[0101] Step 4: Establish the optimization objective for heterogeneous UAV data transfer path planning. The objective of this task is to enable data collection UAVs to collect more ground user data, relay UAVs to forward more cached data, and reduce the age of ground user data, all while satisfying UAV motion constraints, communication distance constraints, link association constraints, and cache capacity constraints. Let... η represents the set of ground users who are active at time step t. m If (t) represents the data age of ground user m, then the optimization objective is expressed as:
[0102]
[0103] Where Π represents the set of trajectory control decisions for all UAVs, and λ c λ represents the data collection weight. r λ represents the cache forwarding weight. a λ represents the average data age penalty weight. m This represents the maximum data age penalty weight. The objective function simultaneously considers throughput, cache unloading capacity, average information timeliness, and worst-case user wait time.
[0104] Step 5: Model the heterogeneous UAV data transfer path planning problem as a distributed partially observable Markov decision process. This process is represented as follows: in Represents the global state space. Represents the joint action space, Represents the state transition function. Represents the reward function, Let represent the observation function, and γ represent the discount factor. The global environmental state includes the position and velocity of all UAVs, the amount of remaining ground user data, the age of ground user data, the activity status of ground users, and the buffer status of the UAVs collecting data. Each UAV independently outputs continuous two-dimensional actions based solely on its own local observations:
[0105]
[0106] Among them, a i (t) represents the normalized action output by the intelligent decision-making model. This indicates the speed command obtained from the motion mapping. i,x (t) and a i,y(t) represent the components of the normalized action in the x and y directions, respectively. This represents the maximum horizontal speed of drone i. Link association is not directly output as an agent action, but is automatically determined based on communication distance, channel gain, buffer status, and bandwidth sharing rules.
[0107] Step 6: Based on the MATD3 algorithm, a reinforcement learning-based heterogeneous UAV data ferry path planning method, HRMA-MATD3, is proposed to improve the overall data ferry transmission rate and reduce end-to-end latency. This algorithm first constructs a policy network based on role-aware multi-stream attention. For each UAV, its observation information is divided into ground user stream, acquisition UAV stream, and relay UAV stream according to entity type. For any semantic stream ρ, the entity features in that stream are... With learnable type embedding u ρ The data is then stitched together and embedded using an independent encoder.
[0108]
[0109] in, This represents the encoded features of entity e in the semantic flow ρ. This represents the feature encoding network corresponding to the semantic stream. ρ represents the semantic stream type index, satisfying ρ∈GU,C,R, where GU,C and R represent the ground user stream, the data collection UAV stream, and the relay UAV stream, respectively; e represents the entity index in the semantic stream. u represents the input feature of entity e in the semantic flow ρ. ρ This represents the learnable type embedding corresponding to the semantic flow ρ. This represents the concatenation result of entity features and type embedding.
[0110] To differentiate the mission responsibilities of data collection drones and relay drones, the state x of each drone is defined. i Later, role coding was introduced. i And generate attention query vectors:
[0111]
[0112] Among them, e i f is used to characterize the role type of drone i. q (·) represents the query vector generation network. x i This represents the state characteristics of drone i. Through role encoding, the parameter sharing policy network can formulate differentiated decision-making behaviors for data acquisition drones and relay drones.
[0113] Furthermore, a multi-head scaled dot product attention mechanism is applied to each type of semantic stream. For the h-th attention head of the semantic stream ρ, its attention weights and context vectors are as follows:
[0114]
[0115] in, This represents the attention weight of drone i on entity e. and These represent the query, key, and value vectors, respectively, d h The feature dimension of a single attention head. Let i represent the set of entities observable by drone i under semantic flow ρ.
[0116] The outputs of each attention head are concatenated to obtain the semantic flow context features, and the drone actions are output by the shared policy network:
[0117]
[0118] Where H represents the number of attention heads. f represents the output projection matrix. π (·) represents the policy network, and tanh(·) is used to constrain actions within the range [-1, 1]. Through this structure, data collection drones can focus on ground users with high data age, high remaining data volume, or close proximity, while simultaneously sensing relay drones that can be used for cache offloading; relay drones can focus on data collection drones with high cache volume, close proximity, or greater need for relay services.
[0119] Step 7: Construct a hybrid reward function for data transfer tasks. This reward function consists of data flow reward, connectivity reward, priority guidance reward, security penalty, and artificial potential field reward. It guides the data collection drone to switch between "collecting ground user data" and "finding a relay drone to unload cache," and guides the relay drone to actively approach the data collection drone carrying cached data. For drone i, its reward function is expressed as:
[0120] r i (t)=ω f r i flow (t)+ω l r i link (t)+ω p r i pri (t)-ω s r i safe (t)+ω Φ (Φ i (t)-Φ i(t-1))(35)
[0121] Where, r i flow (t) represents the data stream reward, used to incentivize data collection by drones or data relay by drones; r i link (t) represents the connection reward, used to encourage the establishment of effective communication links; r i pri (t) represents priority-guided rewards, which determine priority service targets based on ground user data age, target distance, and data collection drone buffer occupancy rate; r i safe (t) represents a safety penalty used to restrict drones from crossing boundaries and other drones from approaching too closely; Φ i (t)-Φ i (t-1) represents the change in the artificial potential field, used to encourage drones to approach effective targets and avoid clustering of similar drones; ω f ω l ω p ω s and ω Φ These represent the weights of the corresponding reward items.
[0122] Step 8: Train the intelligent decision-making model using a multi-agent reinforcement learning framework with centralized training and distributed execution. During the training phase, the evaluation network assesses policy quality using joint observations and joint actions; during the execution phase, each UAV independently outputs motion actions based solely on its own local observations, semantic flow attention features, and role encoding. Let μ... θ Represents a shared policy network. and Let i represent the two centralized evaluation networks corresponding to UAV i. The transfer samples (o) obtained from the experience replay pool are... t u t r t o t+1 The target action and target value are calculated as follows:
[0123]
[0124] Where θ′ represents the target policy network parameters, denoted as the target evaluation network parameter, ∈ represents truncated Gaussian noise, and γ represents the discount factor. t u represents the joint observation at time step t. t r represents the joint action at time step t. i,t o represents the reward obtained by drone i at time step t. t+1 This indicates the joint observation at the next time step. This indicates the target motion after adding truncated Gaussian noise. This represents the target evaluation value obtained by taking the smaller value from the outputs of the two target evaluation networks, y. i Let represent the target Q value corresponding to UAV i, and l represent the evaluation network number, where l = 1 or l = 2. By taking the smaller value from the two evaluation networks, the risk of overestimation of the evaluation network can be reduced.
[0125] The network loss function is expressed as:
[0126]
[0127] The policy network is optimized using a delayed update method, and its loss function is expressed as:
[0128]
[0129] Where N represents the number of drone agents, u -i,t o represents the set of actions of all drones except drone i. i,t This represents the local observation of UAV i at time step t. By combining centralized training and distributed execution, this invention utilizes global information to improve the stability of policy learning during the training phase and maintains the distributed autonomous decision-making capability of the UAV during the execution phase.
[0130] Using the above methods, this invention can achieve the following beneficial effects: First, by modeling the data collection drone and the relay drone as heterogeneous agents with different altitudes, communication objects, cache states, and task objectives, it can more accurately describe the integrated air-ground-space data transfer task; Second, through role encoding and multi-stream attention mechanisms, it can explicitly distinguish the different semantic relationships between ground users, data collection drones, and relay drones, alleviating the action symmetry problem under parameter-sharing policy networks; Third, by fusing a hybrid reward function that integrates data flow, link connection, priority guidance, security constraints, and artificial potential fields, it can simultaneously consider data throughput, cache unloading efficiency, information timeliness, and safe flight constraints; Fourth, through a multi-agent reinforcement learning framework with centralized training and distributed execution, it can improve policy convergence stability during the training phase and achieve distributed path planning and collaborative data transfer for heterogeneous drones during the execution phase.
[0131] In the simulation experiment, to verify the effectiveness and stability of the present invention in heterogeneous UAV data transfer tasks, an integrated air-ground data transfer scenario was constructed within a rectangular task area. The simulation area was set to 1500m × 1500m, each task round lasted 300 time steps, and each time step was 1s long. The default scenario included 50 ground users, 6 data collection UAVs, and 3 relay UAVs. Ground users and UAVs were randomly initialized at the start of each task round, and the initial positions of the UAVs were adjusted with a safe distance. Each ground user generated 10Mbit of data to be collected, entered a cooling state after its data was fully collected, and generated new data requests after the cooling period. At the algorithm training level, all learning methods were trained under the same environmental settings, with a maximum training round count of 1000 and an experience replay pool capacity of 3 × 10⁻⁶. 5 The training batch size was set to 64. The method of this invention adopts a centralized training and distributed execution framework. The entity attention feature dimension in the policy network is set to 64, the number of attention heads is set to 4, and a one-dimensional role encoding is introduced to distinguish the task responsibilities of data collection drones and relay drones. Simulation experiments compared the performance of the method of this invention with the nearest-to-nearest method, the bipolar artificial potential field method, and the MAPPO (Multi-Agent Proximal Policy Optimization) method under different ground user scales. The number of ground users was set to 20, 30, 40, and 50, respectively, and the tests were repeated under different random initialization scenarios.
[0132] Comparison of relevant experimental performance is attached. Figure 3 As shown, the performance of different methods is evaluated from four aspects: end-to-end throughput, average network information age, service demand satisfaction rate, and end-to-end latency. Experimental results show that the method of this invention can achieve higher end-to-end throughput and service demand satisfaction rate under different ground user scales, while reducing average network information age and end-to-end latency. Compared with the nearest-first method, the method of this invention can not only avoid the problem of UAVs chasing only local nearest targets and ignoring global service demands; compared with the bipolar artificial potential field method, the method of this invention can learn more flexible acquisition-relay cooperative behavior; compared with the MAPPO method that does not introduce multi-stream attention and role encoding, the method of this invention can more effectively distinguish the different semantic relationships between ground users, acquisition UAVs, and relay UAVs, avoiding similar or inefficient actions of heterogeneous UAVs under the shared strategy. It can be seen that this invention can maintain good data acquisition, buffer forwarding, and cooperative scheduling capabilities under the conditions of increased ground user numbers and increased task load, and improve the task adaptability, service timeliness, and overall cooperative efficiency of heterogeneous UAV data transfer systems.
[0133] The contents not described in detail in this application are existing technologies known to those skilled in the art.
Claims
1. A heterogeneous UAV data transfer path planning method based on reinforcement learning, comprising the following steps: Step 1: Establish a heterogeneous UAV data transfer system architecture. This architecture consists of five parts: ground users, data collection UAVs, relay UAVs, satellite backhaul network, and intelligent decision-making model. Ground users generate latency-sensitive data. Data collection drones are used to collect ground user data and cache it. Relay drones are used to receive the cached data from the data collection drones and forward it to the satellite backhaul network. Intelligent decision-making models are used to generate path planning and collaborative control decisions for each drone. Step 2: Establish a system model for the heterogeneous UAV data transfer mission; divide the mission area into a two-dimensional horizontal region, and discretize the mission execution process into multiple time steps; let the mission area be... Discrete-time set is Then we have: A={(x,y)|0≤x≤X max ,0≤y≤Y max ,T=0,1,...,T max -1} (1) Among them, X max and Y max These represent the length and width of the task area, respectively, and T represents the task time steps. max This represents the maximum number of task time steps, with each time step having a length of Δt. x and y represent the planar coordinates within the two-dimensional horizontal task region. Let the collection of drones be... Relay drones are All drones are grouped together For drone i with role κ, its set is: Where κ∈C,R, and its three-dimensional position q i (t) and horizontal position p i (t) are respectively represented as: q i (t)=[x i (t),y i (t),h κ ] T ,p i (t)=[x i (t),y i (t)] T , κ∈C,R,h c ≠h R (2) Where C represents the data collection drone role, R represents the relay drone role, and h κ h represents the flight altitude of the drone with the role of κ. C Indicates the flight altitude of the drone, h R Indicates the flight altitude of the relay drone, x i (t) and y i (t) represent the horizontal coordinates of drone i at time step t; by setting h C ≠h R This allows the data acquisition drone and the relay drone to be located at different altitude levels to reflect the differences in their roles in data acquisition and relay tasks; At each time step, the UAV generates a horizontal speed command based on the continuous actions output by the intelligent decision-making model; let the speed command of UAV i at time step t be... The current horizontal velocity is v i (t), maximum acceleration is Its acceleration is then updated as follows: Where Δt represents the duration of a single time step, a represents the unconstrained expected acceleration. i (t) represents the actual acceleration after acceleration constraints; Furthermore, the drone's speed and position are updated based on the constrained acceleration: in, This represents the maximum horizontal speed of drone i; the above motion model can prevent instantaneous changes in drone speed that do not conform to physical constraints. Step 3: Establish the communication link model and data buffer model; assuming the ground user set is... The position of ground user m at time step t is s. m( t)=[x m (t), y m (t), 0] T The data acquisition link distance between data acquisition drone i and ground user m, and the data relay link distance between relay drone j and data acquisition drone i are respectively: in, This indicates the three-dimensional distance from the drone to the ground user. This represents the three-dimensional distance between the relay drone and the data acquisition drone; if Then the data acquisition distance between the drone i and the ground user m meets the data acquisition distance condition; if Then the data relay distance condition is satisfied between relay UAV j and data acquisition UAV i; where R CG and R RC These represent the maximum communication distances of the data acquisition link and the data relay link, respectively. For any distance The wireless link employs a distance-dependent line-of-sight channel gain model, with link gain g(t) and link transmission rate r... χ (t) is: Where β0 represents the channel gain at the reference distance, α represents the path loss exponent, and B χ N represents the total bandwidth allocated to link type X. χ (t) represents the number of links simultaneously served by the same launched UAV at time step t, P tx χ represents the transmit power, N0 represents the noise power spectral density, X = CG represents the link from the data acquisition UAV to the ground user, and χ = RC represents the link from the relay UAV to the data acquisition UAV. Indicates the general wireless link distance; set up This indicates the data acquisition correlation indicator variable between the data acquisition drone i and the ground user m. This indicates the data relay association variable between relay UAV j and data acquisition UAV i; the remaining data volume of ground user m at time step t is b. m (t), the amount of cached data collected by drone i is B. i (t), whose maximum cache capacity is The amount of data collected by drone i from ground user m is: Among them, c i,m( t) represents the amount of data transmitted from ground user m to data collection drone i within one time step. This represents the data acquisition rate between the data acquisition drone i and the ground user m; The amount of data collected by drone i at time step t and unloaded by relay drone is represented as: in, This represents the amount of cached data that drone i completed unloading at this time step. This represents the data relay rate between relay drone j and data acquisition drone i; Furthermore, the remaining data volume for ground users and the cache size of the data collection drones are updated according to the following formula: Among them, when b m When (t+1) = 0, the current data request of ground user m is considered to be completed and enters the cooling or refresh state; when the cooling is over, the ground user generates a new data request. Step 4: Establish the optimization objective for heterogeneous UAV data transfer path planning. The objective of this task is to enable the data collection UAV to collect more ground user data, the relay UAV to forward more cached data, and reduce the age of ground user data, while satisfying UAV motion constraints, communication distance constraints, link association constraints, and cache capacity constraints. η represents the set of ground users who are active at time step t. m (t) represents the data age of ground user m, c i,m( t) represents the amount of data collected by UAV i from ground user m within time step t. Let represent the amount of cached data collected from drone i to the relay drone within time step t. Then, the optimization objective is expressed as: Where Π represents the set of trajectory control decisions for all UAVs, and λ c λ represents the data collection weight. r λ represents the cache forwarding weight. a λ represents the average data age penalty weight. m T represents the maximum data age penalty weight. max This represents the maximum number of discrete-time steps in the task execution, where t represents the discrete-time step index. Let represent the set of data collection drones, M represent the set of ground users, i represent the index of the data collection drones, and m represent the index of the ground users. This objective function considers throughput, cache unloading capacity, average information timeliness, and worst-case user waiting time simultaneously. Step 5: Model the heterogeneous UAV data transfer path planning problem as a distributed partially observable Markov decision process, which is represented as follows: in Represents the global state space. Represents the joint action space, Represents the state transition function. Represents the reward function, The observation function is represented by γ, which represents the discount factor. The global environmental state includes the position and velocity of all UAVs, the remaining data volume of ground users, the age of ground user data, the activity status of ground users, and the buffer status of the UAVs collecting data. Each UAV independently outputs continuous two-dimensional actions based solely on its own local observations. Where, α i (t) represents the normalized action output by the intelligent decision-making model. This represents the speed command obtained from the motion mapping, a i,x (t) and α i,y( t) represent the components of the normalized action in the x and y directions, respectively. This represents the maximum horizontal speed of drone i. The link association is not directly output as an agent action, but is automatically determined based on communication distance, channel gain, buffer status, and bandwidth sharing rules. Step 6: Based on the MATD3 algorithm, a reinforcement learning-based heterogeneous UAV data transfer path planning method, HRMA-MATD3, is proposed to improve the overall data transfer rate and reduce end-to-end latency. This algorithm first constructs a policy network based on role-aware multi-stream attention. For each UAV, its observation information is divided into ground user stream, data acquisition UAV stream, and relay UAV stream according to entity type. For any semantic stream, the entity features in that stream are... With learnable type embedding u ρ The data is then stitched together and embedded using an independent encoder. in, This represents the encoded features of entity e in the semantic flow ρ. This represents the feature encoding network corresponding to the semantic stream, ρ represents the semantic stream type index, satisfying ρ∈GU,C,R, where GU,C and R represent the ground user stream, the data collection UAV stream, and the relay UAV stream, respectively; e represents the entity index in the semantic stream. u represents the input feature of entity e in the semantic flow ρ. ρ This represents the learnable type embedding corresponding to the semantic flow ρ. This represents the concatenation result of entity features and type embedding; To differentiate the mission responsibilities of data collection drones and relay drones, the state x of each drone is defined. i Later, role coding was introduced. i And generate attention query vectors: Among them, e i f is used to characterize the role type of drone i. q (·) represents the query vector generation network, x i The self-state characteristics of UAV i are represented by role coding, which enables the parameter sharing policy network to form differentiated decision-making behaviors for data acquisition UAVs and relay UAVs. Furthermore, a multi-head scaling dot product attention mechanism is applied to each type of semantic stream. For the h-th attention head of semantic stream ρ, its attention weights and context vectors are as follows: in, This represents the attention weight of drone i on entity e. Let h represent the context vector output by the h-th attention head in the semantic flow ρ. These represent the query, key, and value vectors, respectively, d h The feature dimension of a single attention head. Let represent the set of entities observable by drone i under semantic flow ρ, and let e′ represent the entity index in the attention weight normalization summation process; The outputs of each attention head are concatenated to obtain the semantic flow context features, and the drone actions are output by the shared policy network: Where H represents the number of attention heads. f represents the output projection matrix. π (·) represents the policy network. The context features corresponding to the semantic flow ρ are represented by tanh(·), which is used to constrain the action within the range of [-1, 1]. Through this structure, the data collection drone can focus on ground users with high data age, high amount of remaining data, or close proximity, while also perceiving relay drones that can be used for cache unloading; the relay drone can focus on data collection drones with high cache amount, close proximity, or greater need for relay services. Step 7: Construct a hybrid reward function for the data transfer task. This function consists of data flow reward, connectivity reward, priority guidance reward, security penalty, and artificial potential field reward. It guides the data collection drone to switch between "collecting ground user data" and "finding a relay drone to unload cache," and guides the relay drone to actively approach the data collection drone carrying cached data. For drone i, the total reward function obtained at time step t is expressed as: in, This refers to a data stream reward, used to incentivize drones to collect data or relay drones to forward data. This represents a connection reward, used to encourage the establishment of effective communication links; This indicates a priority-based reward system, which determines priority service targets based on ground user data age, target distance, and data collection drone cache occupancy rate. This indicates a safety penalty, used to restrict drones from crossing boundaries and other drones from getting too close; Φ i (t)-Φ i (t-1) represents the change in the artificial potential field, used to encourage drones to approach effective targets and avoid clustering of similar drones; ω f ω l ω p ω s and ω Φ These represent the weights of the corresponding reward items; Step 8: Train the intelligent decision-making model using a multi-agent reinforcement learning framework with centralized training and distributed execution. During the training phase, evaluate the network's ability to assess policy quality using joint observations and joint actions. During the execution phase, each UAV independently outputs motion actions based solely on its own local observations, semantic flow attention features, and role encoding. Let μ be the value of this action. θ Represents a shared policy network. and This represents the two centralized evaluation networks corresponding to UAV i, and the transfer samples (o) obtained from the experience replay pool. t u t r t o t+1 The target action and target value are calculated as follows: Where θ′ represents the target policy network parameters, The parameters of the target evaluation network are represented by ε, γ, and o. t u represents the joint observation at time step t. t r represents the joint action at time step t. i,t o represents the reward obtained by drone i at time step t. t+1 This indicates the joint observation at the next time step. This indicates the target motion after adding truncated Gaussian noise. This represents the target evaluation value obtained by taking the smaller value from the outputs of the two target evaluation networks, y. i Let represent the target Q value corresponding to UAV i, and l represent the evaluation network number. l = 1 or l = 2. By taking the smaller value from the two evaluation networks, the risk of overestimation of the evaluation network can be reduced. The network loss function is expressed as: The policy network is optimized using a delayed update method, and its loss function is expressed as: Where N represents the number of drone agents, u -i,t o represents the set of actions of all drones except drone i. i,t The local observation of UAV i at time step t is represented by the method of combining centralized training and distributed execution. This invention improves the stability of policy learning by utilizing global information during the training phase and maintains the distributed autonomous decision-making capability of UAV during the execution phase.
