An unmanned aerial vehicle edge computing scheduling method based on multi-agent reinforcement learning

Abstract

The application discloses a kind of unmanned aerial vehicle edge computing scheduling method based on multi-agent reinforcement learning.The method first constructs the multi-layer unmanned aerial vehicle MEC network including ground user group, low-altitude unmanned aerial vehicle, high-altitude unmanned aerial vehicle and restricted area in simulation environment, establishes ground user movement, sudden task arrival, air-to-air communication link, task cache and energy consumption model.Then, trajectory planning, task offloading and bandwidth coordination are modeled as a distributed partially observable Markov decision process, a heterogeneous hybrid action space, a feasible offloading action mask, and a multi-objective reward function are designed, and HHA-MAPPO is used to achieve online decision-making with centralized training and distributed execution.Simulation results demonstrate the effectiveness of the method.

Description

Technical Field

[0001] This invention belongs to the field of mobile edge computing for unmanned aerial vehicles (UAVs) and multi-agent reinforcement learning, and specifically relates to a UAV edge computing scheduling method based on multi-agent reinforcement learning. Background Technology

[0002] Unmanned Aerial Vehicles (UAVs)-assisted Mobile Edge Computing (MEC) can provide flexible computing services in scenarios where ground infrastructure is sparse, overloaded, damaged, or temporarily unavailable. It is suitable for tasks requiring rapid deployment and low-latency processing, such as emergency rescue, support for temporary public events, smart city sensing, and services in remote areas. UAVs can act as aerial edge nodes, providing ground users or IoT devices with capabilities for task data collection, forwarding, computation offloading, and result transmission.

[0003] In a multi-layered UAV MEC network, Ground User Groups (GUGs), Low-Altitude UAVs (L-UAVs), and High-Altitude UAVs (H-UAVs) constitute a hierarchical service system. GUGs continuously generate bursty computing tasks; L-UAVs need to track GUGs and perform local aggregation, partial computation, and task offloading; H-UAVs, acting as aerial edge servers, provide strong computing power and cross-regional service capabilities. The flight trajectory of L-UAVs affects their coverage area and the number of tasks they can serve, while the location and bandwidth allocation of H-UAVs affect the offloading feasibility and transmission rate of L-UAVs. Therefore, there is a strong coupling relationship between trajectory planning, task offloading, bandwidth coordination, queue backlog, task latency, and energy consumption.

[0004] In existing research on unmanned aerial vehicle (UAV) MEC (Multi-access Edge Computing), some methods employ convex optimization, stepwise planning, or heuristic algorithms to solve the trajectory planning and task unloading problem. These methods typically rely on a relatively complete global state, accurate channel models, and centralized iterative solutions. However, they struggle to guarantee real-time online decision-making capabilities under conditions of rapidly changing wireless links, sudden arrival of ground tasks, limited UAV perception, or communication congestion. Other methods utilize reinforcement learning (RL), deep reinforcement learning (DRL), or multi-agent reinforcement learning (MARL) to improve online adaptability. However, these methods often assume identical UAV roles or discretize continuous motion control, making it difficult to simultaneously adapt to the hybrid motion control requirements of multi-layered heterogeneous UAV networks.

[0005] Furthermore, in real-world wireless environments, low-altitude UAVs cannot offload tasks to any high-altitude UAV at any given time. When a candidate high-altitude UAV is out of communication range, the link rate is insufficient, there are currently no service backlog tasks, or the candidate offloading ratio exceeds the current time slot's transmission capacity, directly sampling the corresponding offloading action will result in an inactive action, reducing training sample efficiency and affecting policy stability. Therefore, how to jointly achieve continuous trajectory control, discrete task offloading ratio selection, bandwidth allocation mode selection, and inactive action shielding for the heterogeneous roles of low-altitude and high-altitude UAVs under partially observable conditions is a crucial technical problem that multi-layer UAV MEC networks urgently need to solve. Summary of the Invention

[0006] The purpose of this invention is to address the problem of highly coupled trajectory planning, task offloading, and bandwidth coordination in multi-layer UAV MEC networks, and the difficulty of stable online decision-making under partially observable, bursty traffic, and mixed action conditions using existing methods. This invention proposes a UAV edge computing scheduling method based on multi-agent reinforcement learning, thereby improving system throughput and energy efficiency while reducing task latency and task overflow risk. To achieve this objective, the steps employed in this invention are as follows:

[0007] Step 1: Model the joint trajectory planning, task offloading, and bandwidth coordination problems in the multi-layer UAV MEC network as a distributed partially observable Markov Decision Process (Dec-POMDP), determine the global state space for evaluating the network during the centralized training phase, and construct role-based partial observation spaces for L-UAV and H-UAV respectively for online decision-making by UAVs during the distributed execution phase.

[0008] Step 2: Construct the hybrid action space and multi-objective reward function for L-UAV and H-UAV respectively. L-UAV actions include continuous horizontal velocity actions and discrete unloading target and unloading ratio actions. H-UAV actions include continuous horizontal velocity actions and discrete bandwidth allocation mode actions. L-UAV rewards include mission success reward, energy consumption penalty, mission backlog penalty, following GUG reward, and successful unloading reward. H-UAV rewards include assistance service reward, energy consumption penalty, mission backlog penalty within service range, load balancing reward, and service communication link reward. The final reward for both types of UAVs is a linear coupling of the above components.

[0009] Step 3: Determine the set of L-UAVs associated with each H-UAV and perform effective offloading based on the offloading actions of L-UAVs, calculate the corresponding A2A link bandwidth according to the bandwidth allocation mode selected by the H-UAVs, and construct feasible offloading action masks for the discrete offloading action branches of L-UAVs. Determine whether the candidate offloading actions are feasible based on the amount of backlog tasks that can be served, the communication distance between L-UAVs and H-UAVs, the reference A2A uplink rate, and the maximum allowable offloading ratio. Before action sampling, infeasible offloading actions are masked out.

[0010] Step 4: Construct and train a heterogeneous hybrid action multi-agent proximal policy optimization algorithm model, including a role-based policy network for L-UAV and H-UAV, a role-based evaluation network, an observation encoder, a gated recurrent unit (GRU), a multi-head attention module, a continuous action output head, and a discrete action output head. Acquire the global state and role-related local observations of each UAV in the simulation environment. Input the local observations into the policy network to generate hybrid actions for L-UAV and H-UAV, and execute the hybrid actions in the simulation environment. Obtain role-based rewards based on environmental feedback. Write the local observations, actions, rewards, value estimates, old action probabilities, recurrent hidden states, and action masks into the trajectory cache. Update the policy network and evaluation network using generalized advantage estimation and proximal policy optimization until the reward function converges or reaches the preset number of training rounds to obtain the trained decision model.

[0011] Step 5: Deploy the trained L-UAV model and H-UAV model into the multi-layer UAV MEC simulation system, so that each UAV can output trajectory control, task offloading and bandwidth allocation decisions online based on its own local observations, thereby realizing joint trajectory planning and task offloading of the multi-layer UAV MEC network.

[0012] Furthermore, the heterogeneous hybrid action space includes the L-UAV action space and the H-UAV action space. For the i-th L-UAV, its actions consist of continuous horizontal speed control actions and discrete task unloading actions; the j-th H-UAV's actions consist of continuous horizontal speed control actions and discrete bandwidth mode actions, specifically expressed as follows:

[0013]

[0014] in, For L-UAV sets; For H-UAV sets; For the i-th L-UAV, the mixed action in time slot t; v i (t) represents the continuous horizontal velocity motion of the i-th L-UAV, v x，i (t) and v y，i (t) represents the velocity components in the x and y directions, respectively; bi (t) is the link-level offloading ratio action vector of the i-th L-UAV towards the candidate H-UAV; a i，j (t) represents the unloading ratio action of the i-th L-UAV towards the j-th H-UAV; For the j-th H-UAV, the hybrid operation in time slot t; u j (t) represents the continuous horizontal velocity motion of the j-th H-UAV, u x，j (t) and u y，j (t) represents the velocity components in the x and y directions, respectively; Let j be the discrete bandwidth mode component of the j-th H-UAV. In the H-UAV discrete action branch, These correspond to average allocation, channel priority, backlog priority, and conservative allocation, respectively.

[0015] Furthermore, the specific expression of the action constraints satisfied by the heterogeneous hybrid action space is as follows:

[0016]

[0017] in, For discrete unloading ratio sets; This is an indicator function that takes the value 1 if the condition within the parentheses is true, and 0 otherwise. and These are the maximum horizontal speeds of the L-UAV and H-UAV, respectively. The above single-objective constraint causes each L-UAV to use local services within a time slot or to select at most one H-UAV for offloading.

[0018] Furthermore, the processing volume of an L-UAV task is determined by both local processing volume and remote offloading processing volume. The total offloading ratio of the i-th L-UAV in time slot t is defined as:

[0019]

[0020] The index of the H-UAV selected for remote assisted offloading in time slot t by the i-th L-UAV is defined as:

[0021]

[0022] Where, when j i When (t) = 0, it indicates that the i-th L-UAV did not select an H-UAV and performed pure local processing. In this case, the A2A uplink rate used for unified representation is taken as 0.

[0023] Based on the above definition, the effective processing volume is represented as:

[0024]

[0025] in, and Γ i (t) represents the local processing task volume, remote offloading task volume, and total processing task volume of the i-th L-UAV in time slot t, respectively; F represents the current backlog of tasks that the i-th L-UAV can serve; L For L-UAV local computing capabilities; Δt is the time slot length; C cyc The number of CPU cycles required to process a unit of data; Let be the A2A uplink rate from the i-th L-UAV to its selected H-UAV.

[0026] Based on the offloading actions of the L-UAVs, the set of L-UAVs associated with the j-th H-UAV and having offloading actions is determined, and the link bandwidth is calculated based on the bandwidth allocation mode selected by the j-th H-UAV. The associated set of L-UAVs is:

[0027]

[0028] in, The set of L-UAVs associated with the j-th H-UAV and effectively offloaded within time slot t; d i，j (t) represents the distance between the i-th L-UAV and the j-th H-UAV; R LH The effective communication distance from L-UAV to H-UAV.

[0029] The number of L-UAVs in the set is:

[0030]

[0031] When the number of L-UAVs in the set is greater than 0, the four bandwidth modes are defined as follows:

[0032]

[0033] Among them, B i，j (t) represents the A2A link bandwidth allocated from the j-th H-UAV to the i-th L-UAV; B H For the total bandwidth budget of H-UAV; η B For conservative mode bandwidth factor; χ i，j (t)=1 / (d i，j (t)+ε d ) is a channel quality proxy, ε d The constant term is u; u is a set. Virtual indexes in summation; for items not belonging to the set The link, B i，j (t) = 0.

[0034] Furthermore, the feasible offloading action mask is applied to the discrete offloading action branches of the L-UAV. For the positive offloading ratio candidate action of the i-th L-UAV to the j-th H-UAV, the reference A2A uplink rate for feasibility judgment is first calculated. The reference A2A uplink rate is calculated based on the total bandwidth budget of the H-UAV, and the effects of instantaneous shadow fading and small-scale fading are ignored in the calculation, specifically expressed as:

[0035]

[0036] in, The reference received power used for mask determination is:

[0037]

[0038] in, This refers to the L-UAV transmit power; Let Ni be the distance-dependent path loss between the i-th L-UAV and the j-th H-UAV; N0 be the noise power spectral density.

[0039] The maximum allowable offloading ratio of the mask from the i-th L-UAV to the j-th H-UAV is:

[0040]

[0041] in, n represents the locally estimated serviceable backlog of tasks for the i-th L-UAV; m The feasibility coefficient; ∈ β For proportional tolerance; ∈ D This is a minimal constant term.

[0042] Based on the available backlog of tasks, communication distance, reference A2A uplink rate, and maximum allowed offload ratio, determine whether a candidate offload action can be selected. The result of the determination is expressed as follows:

[0043]

[0044] Where, m i，j，κ (t) represents the mask value for the candidate unloading action; R min The minimum available link rate; β κ For candidate unloading ratios; when m i，j，κ When (t) = 1, it means that the i-th L-UAV unloads at a rate of β in time slot t. κ The candidate action for unloading to the j-th H-UAV is allowed to be selected; when m i，j，κ When (t) = 0, it means that the candidate action was masked before action sampling.

[0045] Let the set of discrete unloading actions that the i-th L-UAV can choose in time slot t be denoted as . This set includes local processing actions and unloading processing actions. To facilitate a unified representation of all discrete candidate actions, let l denote the set. Any candidate action in; when l = 0, m i，l (t) = 1, indicating that the local processing action is always allowed to be selected; when l = (j, β) κ When m i，l (t)=m i，j，κ (t) represents the mask value corresponding to the candidate unloading action. The policy network is a set. The original selection probability of each candidate action in the algorithm is p. i,e (t). Before action sampling, based on the mask value m i，l (t) Mask out infeasible candidate actions and renormalize the probabilities of the remaining feasible candidate actions:

[0046]

[0047] Furthermore, the heterogeneous hybrid action multi-agent proximal policy optimization algorithm employs a branching policy output for both continuous and discrete actions. For any agent in either the L-UAV or H-UAV role, its hybrid action policy can be expressed as:

[0048]

[0049] Where r is a UAV role, r∈{L,H}; L and H represent L-UAV roles and H-UAV roles, respectively; n is the UAV index under role r; For the complete mixed action of the nth UAV, For continuous motion components, For discrete action components; Let θ be the feature vector output by the observation encoder for the nth UAV under role r in time slot t; r The policy network parameters for role r; and These are the complete hybrid strategy, continuous action strategy, and discrete action strategy for role r, respectively. When r = L, the discrete action strategy corresponds to the link-level offloading ratio decision; when r = H, the discrete action strategy corresponds to the bandwidth allocation mode. Attached Figure Description

[0050] Figure 1 This is a diagram of a multi-layered unmanned aerial vehicle (UAV) MEC network system architecture.

[0051] Figure 2 This is a structural diagram of a heterogeneous hybrid action multi-agent proximal policy optimization algorithm;

[0052] Figure 3This is a comparison chart of the total throughput of the present invention and existing methods under different flow pressures;

[0053] Figure 4 This is a comparison chart of the energy efficiency ratios of the present invention and existing methods under different flow rates and pressures;

[0054] Figure 5 This is a comparison chart of the task latency of the present invention and existing methods under different flow pressures;

[0055] Figure 6 This is a comparison chart of the task overflow of the present invention and existing methods under different flow pressures. Detailed Implementation

[0056] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0057] In the following description, this specification will refer to the UAV edge computing scheduling method based on multi-agent reinforcement learning proposed in this invention as HHA-MAPPO (Heterogeneous Hybrid Action Multi-Agent ProximalPolicy Optimization).

[0058] HHA-MAPPO first sets the following operating conditions:

[0059] 1. As attached Figure 1 As shown, the multi-layered UAV MEC network includes GUG, L-UAV, H-UAV, and RZ, and is denoted as:

[0060]

[0061] in, and These are GUG, L-UAV, H-UAV, and RZ sets, respectively. GUG generates computational tasks to be processed, L-UAV is responsible for tracking GUG, collecting data and aggregating computational tasks, and performing local computations or offloading tasks to H-UAV. H-UAV, as an airborne edge server, provides auxiliary computational processing and bandwidth coordination services to L-UAV.

[0062] 2. The system is deployed in a three-dimensional Cartesian coordinate system. UAV motion control operates within the permitted movement area. L-UAV and H-UAV maintain preset fixed flight altitudes, as shown below:

[0063]

[0064] Where, q i (t) and q j (t) represents the three-dimensional coordinates of the i-th L-UAV and the j-th H-UAV in time slot t, respectively; zL and z U These represent the fixed flight altitudes for L-UAV and H-UAV, respectively; Ω represents the permitted movement area for the UAV; x min x max y min and y max These represent the boundaries of the regions.

[0065] 3. The continuous motion time is divided into equal-length discrete time slots, and the set of time slots is denoted as:

[0066]

[0067] 4. Each GUG consists of M ground nodes, and the position of each ground node is represented by q. k，m (t)=[x k，m (t), y k，m (t), 0] T The GUG moves according to a constrained Gaussian-Markov model, and the velocity of the k-th GUG's centroid is updated according to the following formula:

[0068]

[0069] in, Let be the velocity of the k-th GUG centroid in time slot t; α be the time correlation coefficient; and w(t) be the two-dimensional Gaussian random velocity perturbation. The GUG centroid position update and intra-group node aggregation constraints are as follows:

[0070]

[0071] in, R represents the centroid position of the k-th GUG in time slot t; coh The aggregation radius within the GUG group.

[0072] The task arrival model for ground nodes is as follows:

[0073]

[0074] Among them, A k，m (t) represents the newly generated task quantity of the m-th node in the k-th GUG; u is the burst task quantity amplification factor; λ(t) is the basic task quantity intensity; Δt is the time slot length; P burst The probability of a sudden flow is represented by u = 1, indicating normal flow intensity, and u = {5, 10, 20}, indicating higher flow pressure.

[0075] The node cache update expression is:

[0076]

[0077] in, The cached task size of the m-th node in the k-th GUG at the start of time slot t, Q max ψ is the maximum cache size for a node. k，m (t) represents the amount of tasks served by this node in time slot t, [·] + It is a positive part function.

[0078] The expression for the number of tasks discarded due to cache overflow caused by untimely task processing is:

[0079]

[0080] The horizontal center coordinate of the i-th RZ is represented as The radius of the RZ region is R RZ .

[0081] 5. An A2A link is used between L-UAVs and H-UAVs. The three-dimensional distance between the i-th L-UAV and the j-th H-UAV is expressed as:

[0082]

[0083] in, and z represents the two-dimensional coordinates of the i-th L-UAV and the j-th H-UAV in the horizontal plane, respectively; U and z L These are the fixed flight altitudes for H-UAV and L-UAV, respectively. The distance-dependent path loss is:

[0084]

[0085] in, α is the distance-dependent path loss between the i-th L-UAV and the j-th H-UAV; PL0 is the path loss at the reference distance; PL Let be the path loss exponent. Considering large-scale shadow fading and Ricean small-scale fading, the A2A uplink transmission rate from the i-th L-UAV to the j-th H-UAV is:

[0086]

[0087] Among them, B i，j (t) represents the bandwidth allocated to the i-th L-UAV by the j-th H-UAV; Let Nj be the signal power received by the j-th H-UAV from the i-th L-UAV; N0 be the noise power spectral density. The downlink feedback rate can be defined similarly by switching the roles of the transmitter and receiver.

[0088] Based on the offloading actions of the L-UAVs, the set of L-UAVs associated with the j-th H-UAV and having offloading actions is determined, and the link bandwidth is calculated based on the bandwidth allocation mode selected by the j-th H-UAV. The associated set of L-UAVs is:

[0089]

[0090] The number of L-UAVs in the set is:

[0091]

[0092] The j-th H-UAV selects the bandwidth allocation mode in each time slot. and These correspond to four modes: average allocation, channel priority, backlog priority, and conservative allocation. For the set... The four bandwidth allocation methods for L-UAV within the system are represented as follows:

[0093]

[0094] The current backlog of tasks that the i-th L-UAV can serve is:

[0095]

[0096] Where g(i) is the GUG index of the i-th L-UAV auxiliary; Let be the set of ground nodes within the coverage area of ​​the i-th L-UAV in time slot t; The cached task volume for the m-th node in the bound GUG. The total backlog of tasks for the i-th L-UAV-assisted GUG is:

[0097]

[0098] The processing volume of an L-UAV task is determined by both local processing volume and remote offloading processing volume.

[0099]

[0100] The single-target unloading constraint of the i-th L-UAV in time slot t is defined as:

[0101]

[0102] The index of the H-UAV selected by the i-th L-UAV in time slot t is:

[0103]

[0104] The effective processing volume is expressed as:

[0105]

[0106] UAV energy consumption consists of flight energy consumption, communication energy consumption, and computing energy consumption. For a rotorcraft UAV with a horizontal flight speed of V, its propulsion power is expressed as:

[0107]

[0108] Among them, P fly (V) represents the propulsion power of the rotor UAV at a horizontal flight speed of V; P0 and P i These are the blade profile power and induced power in hovering state, respectively; U tip v0 is the rotor blade tip velocity; v0 is the average rotor induced velocity in hovering; d0 is the fuselage drag ratio. ρ is air density; s is rotor solidity; A is rotor disk area. The L-UAV local CPU calculated power is expressed as:

[0109]

[0110] in, Calculate power locally for L-UAV; κ L F is the effective switched capacitor coefficient. L This provides local computing capabilities for L-UAV.

[0111] The energy consumption of the i-th L-UAV per time slot t is expressed as:

[0112]

[0113] in, Let i be the total energy consumption of the i-th L-UAV in time slot t; and These are flight energy consumption, communication energy consumption, and local computing energy consumption. The energy consumption of the H-UAV also consists of flight, communication, and remote computing-related components. The flight energy consumption of the H-UAV is determined by its horizontal speed u. j The propulsion power corresponding to (t) is determined, and the communication energy consumption and remote computing-related energy consumption are determined by the L-UAV set it serves, the selected bandwidth allocation mode, and the amount of remote computing tasks.

[0114] The system optimization objective is to maximize the long-term overall system reward over the entire mission cycle. The overall system reward is used to characterize the total reward of the multi-layer UAV MEC network in terms of mission processing, energy consumption control, backlog mitigation, trajectory tracking, and cross-layer collaboration, and is expressed as:

[0115]

[0116] Wherein, U(t) is the system's comprehensive reward for time slot t, which is composed of the rewards of each L-UAV and each H-UAV; The reward for the i-th L-UAV is determined by comprehensively considering task processing volume, normalized energy consumption, backlog pressure, GUG tracking quality, and unloading success rate. The reward for the j-th H-UAV is determined by comprehensively considering factors such as service volume, normalized energy consumption, demand pressure, load balancing quality, and coverage support quality for emergency L-UAVs. The joint optimization problem is expressed as follows:

[0117]

[0118] Where T is the task cycle length.

[0119] 6. During UAV movement, the movement area, maximum speed, safe distance at the same level, and RZ collision avoidance constraints need to be considered; each UAV performs distributed decision-making based on local observations under partially observable conditions, and global state-assisted training strategies are allowed during the centralized training phase.

[0120] Based on the above conditions, the HHA-MAPPO proposed in this invention has undergone performance verification on a multi-layer UAV edge computing algorithm training and simulation verification platform. This platform implements a multi-layer UAV edge computing network simulation environment using Python, and employs the PyTorch framework to complete the training, updating, and testing of the policy network and evaluation network in the HHA-MAPPO algorithm. Simulation results demonstrate the effectiveness of the proposed method. The specific implementation steps of HHA-MAPPO are as follows:

[0121] Step 1: Model the joint trajectory planning, task offloading, and bandwidth coordination problems in the multi-layer UAV MEC network as Dec-POMDP, determine the global state space for evaluating the network during the centralized training phase, and construct the role-based observation spaces for L-UAV and H-UAV respectively for UAV to make online decisions during the distributed execution phase.

[0122] The global state s(t) is provided to the evaluation network only during the intensive training phase, and its composition is represented as follows:

[0123]

[0124] Among them, Q L (t) and Q H (t) represent the precise coordinate sets of L-UAV and H-UAV, respectively, V L (t) and U H (t) represent the sets of speeds for the corresponding characters. Represents the set of centroid coordinates of GUG. and Let C represent the total backlog of tasks and the set of serviceable backlog tasks, respectively. H (t) represents the computational availability state of the H-UAV, A A2A (t) represents the global A2A channel state, M H (t) represents the H-UAV bandwidth mode and load information.

[0125] During the distributed execution phase, the UAV acquires role-related local observations through interaction with onboard sensors and limited neighborhood information, and performs distributed decision-making based on these local observations. For the i-th L-UAV, its local observation in time slot t is denoted as:

[0126]

[0127] in, Let be the local observation space of the i-th L-UAV; the local observation of L-UAV consists of its own motion and boundary features, the safety features of RZ and adjacent L-UAVs within the observation range, the auxiliary GUG service features, and the optional offloadable H-UAV communication features; the auxiliary GUG service features include the location of the GUG prediction center, the number of covered nodes, the noisy backlog estimation, and the serviceable backlog proxy; the optional offloadable H-UAV communication features include the relative location of the optional offloadable H-UAV, the A2A link rate, and the optional H-UAV communication mode information.

[0128] For the j-th H-UAV, its local observation in time slot t is denoted as:

[0129]

[0130] in, Let j be the local observation space of the j-th H-UAV; the local observation of H-UAV consists of its own motion and boundary perception characteristics, the safety characteristics of RZ and adjacent H-UAVs within the observation range, and the service pressure and load information of the L-UAVs that can provide services.

[0131] Step 2: Construct the hybrid action space and multi-objective reward function for L-UAV and H-UAV respectively. L-UAV actions include continuous horizontal velocity actions and discrete unloading target and unloading ratio actions. H-UAV actions include continuous horizontal velocity actions and discrete bandwidth allocation mode actions. L-UAV rewards include mission success reward, energy consumption penalty, mission backlog penalty, following GUG reward, and successful unloading reward. H-UAV rewards include assistance service reward, energy consumption penalty, mission backlog penalty within service range, load balancing reward, and service communication link reward. The final reward for both types of UAVs is a linear coupling of the above components.

[0132] The action of the i-th L-UAV is represented as:

[0133]

[0134] The action of the j-th H-UAV is represented as:

[0135]

[0136] Different reward functions are set for different roles. For L-UAVs, rewards include successful task processing rewards, energy consumption penalties, task backlog penalties, following GUG rewards, and successful offloading rewards; for H-UAVs, rewards include assistance service rewards, energy consumption penalties, task backlog penalties within the service area, load balancing rewards, and service communication link rewards. The reward functions for different roles are expressed as follows:

[0137]

[0138] in, The reward for the i-th L-UAV character; The reward for the j-th H-UAV is w1, w2, w3, w4, and w5 are the weighting coefficients in the reward for the L-UAV. Γ is the weighting coefficient in H-UAV character rewards; i (t) represents the total processing workload of the i-th L-UAV in time slot t; and These are the normalized energy consumption penalties for the i-th L-UAV and the j-th H-UAV, respectively; and These are the normalized task backlog penalties for the i-th L-UAV and the j-th H-UAV, respectively; Track i (t) represents the follow-up reward for the i-th L-UAV to the auxiliary GUG; OffSucc i (t) represents the reward for successfully uninstalling the i-th L-UAV; Assist j (t) is the reward for the assistance services provided to the j-th H-UAV; Bal j (t) represents the load balancing reward for the j-th H-UAV; Cov j (t) represents the communication link reward for the j-th H-UAV to the L-UAV within its service range.

[0139] Therefore, the rewards per slot t for L-UAV and H-UAV are expressed as follows:

[0140]

[0141] Where, r i L (i) represents the reward for the i-th L-UAV; The reward for the j-th H-UAV; r i safe (t) and These are the safety penalties for different UAV roles. Penalties are applied when a collision occurs between neighbors, when a collision occurs with an RZ (Rapid Ranger), or when a UAV goes out of its movement area; otherwise, the safety penalty is 0.

[0142] Step 3: Determine the set of L-UAVs associated with each H-UAV and perform effective offloading based on the offloading actions of L-UAVs, calculate the corresponding A2A link bandwidth according to the bandwidth allocation mode selected by the H-UAVs, and construct feasible offloading action masks for the discrete offloading action branches of L-UAVs. Determine whether the candidate offloading actions are feasible based on the amount of backlog tasks that can be served, the communication distance between L-UAVs and H-UAVs, the reference A2A uplink rate, and the maximum allowable offloading ratio. Before action sampling, infeasible offloading actions are masked out.

[0143] In real-world environments, not all discrete unloading actions are executable. To improve training efficiency, this invention introduces a feasible unloading action mask for the L-UAV discrete unloading action branch. For the i-th L-UAV, the local processing action is always reserved as a feasible backup action, denoted as m. i，0 (t) = 1. For candidate offloading actions, when the candidate offloading ratio is positive, the reference A2A uplink rate, excluding transient shadows and small-scale fading, is first calculated based on path loss, expressed as:

[0144]

[0145] in, Reference A2A uplink rate for use in action masking; The reference received power is used. Based on the locally estimated backlog of tasks and the number of tasks that can be transmitted within a time slot, the maximum allowable mask offload ratio is calculated, specifically expressed as follows:

[0146]

[0147] Based on the available backlog of tasks, communication distance, reference A2A uplink rate, and maximum allowed offload ratio, determine whether a candidate offload action can be selected. The result of the determination is expressed as follows:

[0148]

[0149] Step 4: Construct and train a heterogeneous hybrid action multi-agent proximal policy optimization algorithm model, including L-UAV and H-UAV role-based policy networks, role-based evaluation networks, observation encoders, GRUs, multi-head attention modules, continuous action output heads, and discrete action output heads. Obtain the global state and role-related local observations of each UAV in the simulation environment. Input the local observations into the policy network to generate hybrid actions of L-UAV and H-UAV, and execute the hybrid actions in the simulation environment. Obtain role-based rewards based on environmental feedback. Write the local observations, actions, rewards, value estimates, old action probabilities, cyclic hidden states, and action masks into the trajectory cache. Update the policy network and evaluation network using generalized advantage estimation and proximal policy optimization until the reward function converges or reaches the preset number of training rounds to obtain the trained decision model.

[0150] As attached Figure 2 As shown, HHA-MAPPO adopts a Centralized Training with Decentralized Execution (CTDE) paradigm, and sets up role-based policy networks and role-based evaluation networks for L-UAVs and H-UAVs respectively. All L-UAVs share a set of L-UAV policy network and evaluation network parameters, and all H-UAVs share another set of H-UAV policy network and evaluation network parameters. For any role r, the policy network first divides the local observations into P based on the feature source. r Each role-related feature block is represented as:

[0151]

[0152] Among them, P r The number of feature blocks corresponding to character r. The feature block is the Pth local observation feature block of agent n under role r in time slot t; the feature block includes self-motion features, neighbor features, GUG features, A2A link features and RZ features.

[0153] Each feature block undergoes a linear transformation and bias summation by the observation encoder, followed by processing with a nonlinear activation function to obtain the encoded features, specifically represented as follows:

[0154]

[0155] in, and σ(·) represents the weight matrix and bias vector of the corresponding encoder; σ(·) is the activation function.

[0156] The pooled digest of encoded features is input into a GRU to obtain a temporal memory representation containing historical information, specifically as follows:

[0157]

[0158] in, The parameter is θ r Gated loop unit; θ r The policy network parameters for role r; To encode feature pooling summaries; The policy network side is a loop of hidden states; This is the hidden state of the policy network side in the previous time slot.

[0159] Subsequently, the policy network employs a multi-head attention mechanism to aggregate the features of the current spatial entities and candidate links, and fuses the attention spatial features with temporal memory representations to obtain latent features, represented as:

[0160]

[0161] in, Features of the attention space; This represents the concatenation operation. Based on this latent feature representation, the role blending strategy is decomposed into a continuous action strategy and a discrete action strategy, as follows:

[0162]

[0163] in, For the complete hybrid strategy of role r, For continuous action strategy, For discrete action strategies; For the complete mixed action of the nth UAV, and These represent continuous actions and discrete actions, respectively. Let be the latent feature representation of the nth UAV under role r.

[0164] During training, the simulation environment and trajectory cache are first initialized. Local observations of each UAV are acquired in each time slot and input into the policy network to generate hybrid actions. These hybrid actions are then executed in the simulation environment. Role-based rewards are obtained according to a multi-objective reward function. Local observations, actions, rewards, value estimates, probabilities of old actions, cyclic hidden states, and action masks are written into the trajectory cache. After each training round, advantage estimates and objective rewards are calculated based on the trajectory cache. The policy network and evaluation network are updated using a proximal policy optimization method until the reward function converges or the preset number of training rounds is reached, resulting in the trained HHA-MAPPO decision model.

[0165] Step 5: Deploy the trained L-UAV model and H-UAV model into the multi-layer UAV MEC simulation system, enabling each UAV to output trajectory control, task offloading, and bandwidth allocation decisions online based on its own local observations, thereby realizing joint trajectory planning and task offloading of the multi-layer UAV MEC network.

[0166] The performance of the proposed multi-layer UAV mobile edge computing network joint trajectory planning and task offloading method based on heterogeneous hybrid action multi-agent reinforcement learning has been verified through simulation on a multi-layer UAV edge computing algorithm training and simulation verification platform built on Python and the PyTorch framework. In the simulation experiment, the task area size was 200×200, the system contained 6 GUGs, each GUG contained 4 ground nodes, and 6 L-UAVs, 2 H-UAVs, and 2 RZs were configured. The L-UAV flight altitude was 50m, the H-UAV flight altitude was 100m, and the communication range between the L-UAVs and H-UAVs was 100m. The training budget was 3,200,000 rounds, the round step size was 200, the GAE decay parameter was 0.95, and 10 rounds of PPO updates were performed per round. (See attached...) Figure 3 , 4 As shown in Figures 5 and 6, under different flow pressure multiples, HHA-MAPPO maintains higher system throughput and energy efficiency, as well as lower task latency and task overflow compared to existing methods such as Heterogeneous-Agent Proximal Policy Optimization (HAPPO), Heterogeneous-Agent Trust Region Policy Optimization (HATRPO), Multi-Agent Soft Actor-Critic (MASAC), and Multi-Agent Deep Deterministic Policy Gradient (MADDPG).

[0167] The contents not described in detail in this application are existing technologies known to those skilled in the art.

Claims

1. A method for UAV edge computing scheduling based on multi-agent reinforcement learning, comprising the following steps: Step 1: Model the joint trajectory planning, task offloading, and bandwidth coordination problem in the multi-layer UAV MEC network as Dec-POMDP, determine the global state space for evaluating the network during the centralized training phase, and construct the role-based observation spaces for L-UAV and H-UAV respectively for UAV to make online decisions during the distributed execution phase; The global state s(t) is provided to the evaluation network only during the intensive training phase, and its composition is represented as follows: in, Q L (t) and Q H (t) represent the precise coordinate sets of L-UAV and H-UAV, respectively, V L (t) and U H (t) represent the sets of speeds for the corresponding characters. Represents the set of centroid coordinates of GUG. and Let C represent the total backlog of tasks and the set of serviceable backlog tasks, respectively. H (t) represents the computational availability state of the H-UAV, A A24 (t) represents the global A2A channel state, M H (t) represents the H-UAV bandwidth mode and load information; During the distributed execution phase, the UAV acquires role-related local observations through interaction with onboard sensors and limited neighborhood information, and performs distributed decision-making based on these local observations; for the i-th L-UAV, its local observation in time slot t is denoted as: in, Let i be the local observation space of the i-th L-UAV. The local observation of L-UAV consists of its own motion and boundary features, RZ and adjacent L-UAV safety features within the observation range, auxiliary GUG service features, and optional offloadable H-UAV communication features. The auxiliary GUG service features include the location of the GUG prediction center, the number of covered nodes, the noisy backlog estimation, and the serviceable backlog proxy. The optional offloadable H-UAV communication features include the relative position of the optional offloadable H-UAV, the A2A link rate, and the optional H-UAV communication mode information. For the j-th H-UAV, its local observation in time slot t is denoted as: in, For the local observation space of the j-th H-UAV, the local observation of H-UAV consists of its own motion and boundary perception characteristics, the safety characteristics of RZ and adjacent H-UAVs within the observation range, and the service pressure and load information of the L-UAVs that can provide services; Step 2: Construct the hybrid action space and multi-objective reward function for L-UAV and H-UAV respectively. The actions of L-UAV include continuous horizontal velocity actions and discrete unloading target and unloading ratio actions. The actions of H-UAV include continuous horizontal velocity actions and discrete bandwidth allocation mode actions. The rewards of L-UAV include task success reward, energy consumption penalty, task backlog penalty, following GUG reward, and unloading success reward. The rewards of H-UAV include assistance service reward, energy consumption penalty, task backlog penalty within the service range, load balancing reward, and service communication link reward. The final reward of both types of UAVs is a linear coupling of the above-mentioned components. The action of the i-th L-UAV is represented as: in, For the mixed action of the i-th L-UAV in time slot t, v i (t) represents the continuous horizontal velocity motion of the i-th L-UAV, v x,i (t) and v y，i (t) represents the velocity components in the x and y directions, respectively, and b i (t) is the link-level offloading ratio action vector of the i-th L-UAV towards the candidate H-UAV, a i，j (t) represents the unloading ratio action of the i-th L-UAV towards the j-th H-UAV; The action of the j-th H-UAV is represented as: in, For the j-th H-UAV's hybrid operation in time slot t, u j (t) represents the continuous horizontal velocity motion of the j-th H-UAV, u x，j (t) and u y，j (t) represents the velocity components in the x and y directions, respectively. For the j-th H-UAV discrete bandwidth mode component, v j (t) represents the discrete action v of H-UAV. j (t)∈{0,1,2,3}, corresponding to average allocation, channel priority, backlog priority and conservative allocation respectively; Different reward functions are set for different roles. For L-UAV, rewards include successful task processing reward, energy consumption penalty, task backlog penalty, following GUG reward, and successful unloading reward. For H-UAV, rewards include assistance service reward, energy consumption penalty, task backlog penalty within service range, load balancing reward, and service communication link reward. The reward functions for different roles are expressed as follows: in, For the i-th L-UAV character reward, Let w1, w2, w3, w4, and w5 be the character rewards for the j-th H-UAV, and w1, w2, w3, w4, and w5 be the weighting coefficients in the character rewards for the L-UAV. Γ represents the weighting coefficient in H-UAV character rewards. i (t) represents the total processing workload of the i-th L-UAV in time slot t. These are the normalized energy consumption penalties for the i-th L-UAV and the j-th H-UAV, respectively. The Track represents the normalized task backlog penalty for the i-th L-UAV and the j-th H-UAV, respectively. i (t) represents the follow-up reward for the i-th L-UAV to the assisted GUG, OffSucc i (t) represents the reward for successfully uninstalling the i-th L-UAV, Assist j (t) is the reward for the assistance services provided to the j-th H-UAV, Bal j (t) represents the load balancing reward for the j-th H-UAV, Cov j (t) represents the communication link reward for the j-th H-UAV to the L-UAV within its service range; Therefore, the rewards per slot t for L-UAV and H-UAV are expressed as follows: in, The reward for the i-th L-UAV, For the reward of the j-th H-UAV, These are the safety penalties for different UAV roles. The penalties are given when a collision occurs between neighbors, when a collision occurs with an RZ, or when the UAV goes out of its movement area. Otherwise, the safety penalty is 0. Step 3: Determine the set of L-UAVs associated with each H-UAV and perform effective offloading based on the offloading actions of L-UAVs; calculate the corresponding A2A link bandwidth based on the bandwidth allocation mode selected by the H-UAVs; construct feasible offloading action masks for the discrete offloading action branches of L-UAVs; determine the feasibility of candidate offloading actions based on the backlog of serviceable tasks, the communication distance between L-UAVs and H-UAVs, the reference A2A uplink rate, and the maximum allowable offloading ratio; and mask infeasible offloading actions before action sampling. In real-world environments, not all discrete unloading actions are executable. To improve training efficiency, this invention introduces a feasible unloading action mask for the L-UAV discrete unloading action branch. For the i-th L-UAV, the local processing action is always retained as a feasible backup action, denoted as m. i,0 (t) = 1. For candidate offloading actions, when the candidate offloading ratio is positive, the reference A2A uplink rate, excluding transient shadows and small-scale fading, is first calculated based on path loss, expressed as: in, For reference received power, This refers to the L-UAV transmit power. The distance-related path loss between the i-th L-UAV and the j-th H-UAV is... For the reference A2A uplink rate used in the action mask, B H Given the total bandwidth budget for H-UAV, N0 is the noise power spectral density. Based on the locally estimated backlog of tasks and the amount of tasks that can be transmitted within a time slot, the maximum allowable mask offloading ratio is calculated, specifically as follows: in, η is the locally estimated serviceable backlog of tasks for the i-th L-UAV. m ε is the feasibility coefficient. β For proportional tolerance, ε D It is a minimal constant term; Based on the available backlog of tasks, communication distance, reference A2A uplink rate, and maximum allowed offload ratio, determine whether a candidate offload action can be selected. The result of the determination is expressed as follows: Where, m i，j，κ (t) represents the mask value for the candidate unloading action, d i，j (t) represents the distance between the i-th L-UAV and the j-th H-UAV, R LH R represents the effective communication distance from L-UAV to H-UAV. min For the minimum available link rate, β k For candidate unloading ratios, when m i，j，κ When (t) = 1, it means that the i-th L-UAV unloads at a rate of β in time slot t. k The candidate action for unloading to the j-th H-UAV is allowed to be selected when m i，j，κ When (t) = 0, it means that the candidate action was masked before action sampling. Step 4: Construct and train a heterogeneous hybrid action multi-agent proximal policy optimization algorithm model, including a role-based policy network for L-UAV and H-UAV, a role-based evaluation network, an observation encoder, a GRU, a multi-head attention module, a continuous action output head, and a discrete action output head. In the simulation environment, acquire the global state and the role-related local observations of each UAV. Input the local observations into the policy network to generate hybrid actions for L-UAV and H-UAV, and execute the hybrid actions in the simulation environment. Obtain role-based rewards based on environmental feedback. Write the local observations, actions, rewards, value estimates, old action probabilities, cyclic hidden states, and action masks into the trajectory cache. Update the policy network and evaluation network using generalized advantage estimation and proximal policy optimization until the reward function converges or reaches the preset number of training rounds to obtain the trained decision model. HHA-MAPPO adopts the CTDE paradigm and sets up role-based policy networks and role-based evaluation networks for L-UAVs and H-UAVs respectively. All L-UAVs share a set of L-UAV policy network and evaluation network parameters, and all H-UAVs share another set of H-UAV policy network and evaluation network parameters. For any role r, the policy network first divides the local observations into P based on the feature source. r Each role-related feature block is represented as: Among them, P r The number of feature blocks corresponding to character r. The p-th local observation feature block of agent n under role r in time slot t includes its own motion features, neighbor features, GUG features, A2A link features and RZ features; Each feature block undergoes a linear transformation and bias summation by the observation encoder, followed by processing with a nonlinear activation function to obtain the encoded features, specifically represented as follows: in, and Let σ(·) be the weight matrix and bias vector of the corresponding encoder, and σ(·) be the activation function. The pooled digest of encoded features is input into a GRU to obtain a temporal memory representation containing historical information, specifically as follows: in, The parameter is θ r The gated loop unit, θ r For the policy network parameters of role r, To encode feature pooling summaries, The hidden state is a loop on the policy network side. This is the hidden state of the policy network side in the previous time slot; Subsequently, the policy network employs a multi-head attention mechanism to aggregate the features of the current spatial entities and candidate links, and fuses the attention spatial features with temporal memory representations to obtain latent features, represented as: in, For attention space features, The concatenation operation is represented by this latent feature representation. Based on this, the role blending strategy is decomposed into a continuous action strategy and a discrete action strategy, as follows: in, For the complete hybrid strategy of role r, For continuous action strategy, For discrete action strategies, For the complete mixed action of the nth UAV, These are continuous actions and discrete actions, respectively. Let r be the latent feature representation of the nth UAV under role r; During training, the simulation environment and trajectory cache are first initialized. Local observations of each UAV are acquired in each time slot and input into the policy network to generate hybrid actions. The hybrid actions are then executed in the simulation environment. Role-based rewards are obtained according to the multi-objective reward function. Local observations, actions, rewards, value estimates, probabilities of old actions, cyclic hidden states, and action masks are written into the trajectory cache. After each training round, advantage estimates and objective rewards are calculated based on the trajectory cache. The policy network and evaluation network are updated using the proximal policy optimization method until the reward function converges or the preset number of training rounds is reached, resulting in the trained HHA-MAPPO decision model. Step 5: Deploy the trained L-UAV model and H-UAV model into the multi-layer UAV MEC simulation system, so that each UAV can output trajectory control, task offloading and bandwidth allocation decisions online based on its own local observations, thereby realizing joint trajectory planning and task offloading of the multi-layer UAV MEC network.

Images