A layered reinforcement learning-based method for three-dimensional trajectory planning and joint resource scheduling of a UAV cluster

Abstract

The application discloses a kind of unmanned aerial vehicle cluster three-dimensional trajectory planning and joint resource scheduling method based on layered reinforcement learning.For the strong coupling optimization problem of unmanned aerial vehicle three-dimensional maneuver, relay topology, user association and uplink resource scheduling in low-altitude intelligent networking, a double-time-scale decision model of air-ground cooperative communication scenario is constructed, and the observation, action space and reward function of the strategic layer and the tactical layer are defined.A layered hybrid endogenous intelligent framework is constructed: the strategic layer uses topology-aware graph attention network and delayed incremental delay acceptance algorithm to realize three-dimensional trajectory planning and user association, and the tactical layer uses dynamic attention-Sinkhorn matching network to realize channel allocation and uplink power control.The multi-agent proximal policy optimization algorithm is used as the backbone, and the end-to-end collaborative training and deployment of the two-layer strategy are realized under the centralized training-distributed execution framework.Simulation shows that the method is superior to existing methods in terms of access rate, logarithmic utility and other indicators, and has good scalability and generalization performance.

Description

Technical Field

[0001] This invention belongs to the field of low-altitude intelligent network unmanned aerial vehicle (UAV) swarm collaborative communication, and specifically relates to a method for UAV swarm three-dimensional trajectory planning and joint resource scheduling based on hierarchical reinforcement learning. Background Technology

[0002] Low-Altitude Aerial Intelligent Networks (LAINs), as a product of the deep integration of low-altitude economy and air-to-ground collaborative communication, need to provide on-demand coverage, flexible access, and distributed autonomy for mission-oriented low-altitude services such as emergency communication, temporary large-scale event coverage, infrastructure inspection, and industrial monitoring. Within the overall architecture of the Space-Air-Ground Integrated Network (SAGIN), autonomous aerial vehicle (AAV) swarms, with their high mobility, rapid deployment capabilities, and favorable line-of-sight propagation conditions, have become a key means of providing temporary aerial access and relay services when ground infrastructure is damaged, insufficient, or restricted by terrain. Therefore, researching intelligent joint optimization methods for trajectory planning and resource scheduling of AAV swarms in LAINs is of great significance.

[0003] In low-altitude intelligent networks, the online control of UAV swarm-assisted air-to-ground cooperative communication networks faces severe challenges due to the strong coupling between physical domain maneuvering and information domain resource scheduling: building obstruction, terrain undulations, and altitude changes lead to fluctuations in path loss and line-of-sight / non-line-of-sight state transitions; unevenly distributed service traffic causes heterogeneous uplink and downlink loads; the three-dimensional position of the UAV directly determines the relay topology, line-of-sight probability, and co-channel interference, while service quality and relay connectivity constraints, in turn, limit the feasible maneuvers of the UAV. This "position-channel" coupling, coupled with the slow maneuvering of UAVs and the rapid changes in wireless channels, makes the problem a multi-timescale optimization problem with mixed continuous-discrete variables.

[0004] Deep Reinforcement Learning (DRL), as a data-driven method for policy design, has become an important tool for handling the aforementioned sequentially coupled trajectory and resource decision-making problems, demonstrating strong adaptability in high-dimensional continuous decision spaces and dynamic multi-agent coordination. However, existing research still has several shortcomings in solving complex air-ground cooperative network problems: First, many multi-agent methods rely on global state aggregation or centralized coordination, which conflicts with the distributed autonomy requirements in low signaling environments; second, many studies focus on access links or downlink throughput, lacking unified treatment for uplink, bidirectional service coupling, and relay bottlenecks; third, objectives centered on throughput or spectral efficiency tend to favor strong channel users, weakening fairness to edge or weak link users; fourth, discrete user association and orthogonal channel allocation are usually completed by heuristic methods, external matching, or post-processing, which makes it difficult to balance stable matching, differentiability optimization, and online convergence performance in dynamic multi-user scenarios.

[0005] Furthermore, macroscopic trajectory and topology reconstruction are essentially slow-timescale decisions, while channel allocation and power control must respond to rapidly changing wireless environments. Incorporating both into a single-layer policy can easily lead to state / action space explosion and worsen credit allocation across timescales. Therefore, decoupling slow-timescale trajectory-topology control from fast-timescale resource scheduling through a hierarchical mechanism while maintaining distributed execution capabilities has become a critical issue that urgently needs to be addressed. However, existing hierarchical reinforcement learning methods typically still rely on centralized coordination or solver assistance, and rarely explicitly model relay capacity constraints and end-to-end proportional fairness objectives, making it difficult to simultaneously meet the joint optimization requirements of three-dimensional movement, stable user association, orthogonal channel allocation, and uplink / downlink power control.

[0006] In summary, there is an urgent need to propose a method that can coordinate and optimize the slow-time-scale three-dimensional trajectory planning and user association with the fast-time-scale channel allocation and uplink / downlink power control end-to-end within a distributed execution framework, so as to achieve a synergistic improvement in the maneuver control capability and communication performance of UAV swarms in low-altitude intelligent networks. Summary of the Invention

[0007] The purpose of this invention is to address the strongly coupled joint optimization problem of UAV 3D maneuvering, relay topology, user association, and uplink / downlink resource scheduling in low-altitude intelligent networks. It proposes a hierarchical reinforcement learning-based method for UAV swarm 3D trajectory planning and joint resource scheduling, thereby achieving end-to-end collaborative learning of two time-scale policies within a distributed execution framework. This collaboratively improves the maneuver control and communication performance of UAV swarms in complex communication environments. To achieve this objective, the steps employed in this invention are as follows:

[0008] Step 1: Define the scenarios in which the method is applicable and construct the observation space and action space of the strategic and tactical layers.

[0009] The method is applicable to low-altitude air-to-ground cooperative communication scenarios consisting of K rotary-wing UAVs and M ground mobile users. The UAV swarm is deployed in a square mission area with a side length of L. At the start of the mission, the UAVs are randomly initialized and deployed in the lower left corner of the initial area. The time axis is divided into strategic periods n = 1, 2, ..., N. step With tactical time slot t, each strategic cycle has a length of ΔT and includes T. tac The method employs a two-timescale collaborative decision-making architecture, consisting of equal-length tactical time slots. The strategic layer performs UAV 3D trajectory updates and user association during the strategic period n, while the tactical layer performs channel allocation and uplink / downlink power control during the tactical time slot t.

[0010] The local observation of the k-th UAV at the strategic level is defined as follows:

[0011]

[0012] in To normalize the three-dimensional position, To normalize the average power of the previous period, and These are the normalized load of served users and the normalized load of physical coverage, respectively. To normalize the hotspot displacement, For the most recent N nb A masked neighbor observation set for each neighboring drone, where each drone is represented by a normalized relative 3D position, service load, and coverage load, and invalid positions are removed by masking.

[0013] Strategic-level actions are defined as three-dimensional velocity vectors:

[0014] a k，H [n] = v k [n] (2)

[0015] Where v k [n] is the output of the policy network, compressed with tanh and scaled to the maximum horizontal / vertical velocity, and then projected onto the feasible velocity set.

[0016] The local observation of the k-th UAV at the tactical level is defined as:

[0017] o k，L [t] = [s] AAV，k [t],F user,k [t], m k [t]] (3)

[0018] Where sAAV，k [t] represents dimension D s =3+2N ch The UAV-level aggregated state vector is obtained by concatenating the current downlink power occupancy rate, uplink relay load ratio, downlink relay load ratio, occupancy counts of neighboring UAVs on each channel, and interference summaries on each channel, where N ch F is the number of orthogonal subchannels in the system. user，k [t] represents the size. The fixed-capacity user feature matrix, where The upper limit of the number of users that can be served, i.e., the capacity of the fixed-capacity service set, is represented by the i-th row corresponding to the feature vector of the i-th user in the local service set, with feature dimension D. u Includes access channel gain, previous time slot uplink power coefficient, logarithmically scaled downlink interference power, historical average rate, normalized previous time slot channel index, and the number of co-channel users Q. k，· [t] Derived from the co-channel conflict flag between cells; m k [t] represents the length. The validity mask has a value of 1 for the i-th component, which indicates that the current data position is a valid user and a value of 0 indicates that the position is a placeholder.

[0019] Tactical actions are defined as follows:

[0020]

[0021] in To satisfy the downlink power share vector of the simplex constraint, c is the uplink power coefficient vector. k [t] is the channel allocation index vector for each service slot.

[0022] Step 2: Design the hierarchical reward functions for the strategic and tactical levels. The specific design principles for the reward functions are as follows:

[0023] The strategic layer reward consists of seven parts: access reward, tactical utility feedback, topology reward, hotspot guidance reward, energy consumption penalty, collision penalty, and boundary penalty. The total reward for the k-th UAV in strategic period n is:

[0024] r k，H [n] = R acc，k [n]+R util，k [n]+R topo [n]+R hot，k [n]-P eng，k [n]-P col，k [n]-P out，k [n] (5)

[0025] Among them, the access reward R acc，k[n] is the weighted combination of local service load and overall cluster access rate; tactical utility feedback R util，k [n] is determined by T within the current strategic cycle. tac The logarithmic utility accumulated per tactical time slot, expressed as an effective relay coefficient α eff,k [t] and β eff，k [t] is obtained by averaging after correction, and its value is in the interval [0, 1]. When the relay load ratio exceeds the threshold, it is inversely reduced to prevent the rate exceeding the relay bottleneck from being over-included; the topology reward is R. topo [n] = w topo ·λ2(L A2A [n]), where L A2A [n] is the Laplacian matrix corresponding to the distance-weighted adjacency matrix between UAVs, λ2(·) is the algebraic connectivity, and w topo Topology reward weight; Hotspot guidance reward R hot，k [n] is a non-positive value, determined by the normalized Euclidean distance between the UAV's horizontal position and the assigned hotspot center; energy penalty P eng，k [n] Penalizes energy consumption exceeding the hovering baseline; collision penalty P col，k [n] when the distance to the neighbor is less than the safe distance d safe Time activation; boundary penalty P out,k [n] is composed of hard outbound terms and soft repulsion terms.

[0026] Tactical layer rewards consist of three parts: uplink utility, downlink utility, and interference penalty. The total reward for the k-th UAV in tactical time slot t is:

[0027]

[0028] Among them, the upward and downward utility items and The forms are symmetrical, each being the logarithmic sum of the access rates after correction for effective relay coefficients, multiplied by an overload shaping function g(ρ) = min(1, 2 - ρ) based on the relay load ratio, where ρ is the relay load ratio, i.e., the ratio of actual relay load to relay capacity; interference penalty P interf，k [t] includes two terms: time-division multiplexing congestion penalty within the same cell and inter-cell co-channel interference penalty. The former is determined by the number of co-channel users n. k，c share factor (n) k，c -1) / n k，c The weighted average is activated when the per-user co-channel interference power exceeds a threshold.

[0029] Step 3: Construct a hierarchical hybrid AI-Native (H2N) network structure, including a topology-aware graph attention network (Topo-GAT) and a hysteresis-based incremental deferred acceptance algorithm (HIDAA) at the strategic layer, and a dynamic attentive-sinkhorn matching network (DASM) at the tactical layer. The specific construction method is as follows:

[0030] The topology-aware graph attention network includes two parallel paths: a topology-aware graph branch and a spatial target branch. The topology-aware graph branch is constructed with the k-th UAV as the central node, containing N... g =1+N nb A self-centered local graph of N nodes, where N nb The number of nearest neighbor drones for the drone at this center, with node characteristics including relative 3D displacement, service load, coverage load, and average power, and edge characteristics. This includes the distance between drones, altitude difference, and service load difference, which are then fed into L after linear coding and layer normalization. gat Layer edge-aware multi-head attention encoder, wherein the edge-aware multi-head attention is for the first... The attention weight of the h-th attention head in layer h is calculated using the following formula:

[0031]

[0032] Where i is the center node of the egocentric local graph, j and j′ are the neighbor nodes of i, and the summation of the denominator is used to traverse all the neighbor nodes of i; The number of node attention logs in the h-th head is the scaled dot product of the query vector of the center node i and the key vector of the neighbor node j. W is the edge offset term derived from and truncated by the edge features. e b e These are the weight matrix and bias vector of the edge-encoded linear layer, respectively. max To truncate the upper bound of the bias, restrict the edge bias to [-b]. max b max To suppress extreme values; Adj ijThe adjacency mask is set to 1 when neighbor node j is valid for the central node i, and 0 otherwise, thus normalizing the attention only on valid neighbors. The node state is updated by residual multi-head aggregation and then read from the central node i to obtain the topological branch representation z of the k-th UAV. topo，k Then, the spatial target representation z output by the multilayer perceptron of the spatial target branch is compared with the spatial target representation z. spatial，k By fusing the following formula, we obtain the strategic layer shared representation of the k-th UAV:

[0033] z k =LayerNorm(tanh(W fuse [z topo，k ||z spatial，k ]+b fuse (8)

[0034] Among them W fuse With b fuse These are the learnable weight matrix and bias vector of the fusion layer, respectively; the fusion representation z k As a shared input for the strategic layer Actor and Critic head, it outputs three-dimensional velocity motion and value estimation, respectively.

[0035] The hysteresis-based incremental delay-received user association algorithm consists of three stages: the first stage is hysteresis guarding, where each user m is based on the optimal feasible score Γ obtained from the current local channel measurement. best，m [n] Score of the original service drone If a comparison is made, Then add the user to k old The Waiting Collection S keep [k old Otherwise, add it to the active set S. active ,in The first stage is the hysteresis threshold; the second stage is capacity pruning, where each drone k starts from S. keep [k] is scored Γ k,· [n] Before selecting in descending order One user as the initial service set M k [n], the remaining waiting users join the active set; the third phase is a synchronous proposal-response incremental protocol, where each user in the active set makes a proposal to the highest-scoring drone in their feasible drone set that has not yet been rejected. Each drone merges its current service set with the new acceptance queue and then presses top- The program prunes and returns the accept / reject response. The rejected user is removed from the feasible sequence and the program continues to propose the next best drone, looping until no rejections are generated.

[0036] The dynamic attention-Sinkhorn matching network comprises three parts: a dual attention state encoder, a Sinkhorn channel matching head, and a probabilistic power head. The dual attention state encoder first processes the user feature matrix F... user,k [t]、AAV global state s AAV，k [t] and the channel state are mapped to a common dimension D. h The user embedding vector sequence, AAV global embedding vector, and channel embedding vector sequence are used to obtain the user internal representation H through user self-attention with validity mask. inter The AAV global embedding vector and the channel embedding vector are concatenated into a key-value sequence KV, and then H is used as the key value sequence. inter For querying, cross-attention is performed using key-value pairs, and finally, the result is output via linear projection:

[0037] H seq，k [t] = Proj(LN(MHA) cross (H inter ,KV,KV)+H inter (9)

[0038] The Sinkhorn channel matching header will H seq，k [t] and learnable channel embedding Perform a bilinear mapping to obtain the matching score matrix S ij [t] = <h seq,k,i [t],c j >, after setting the invalid line mask to -∞, perform T s The Sinkhorn iteration over the logarithmic field is performed, where τ is the Sinkhorn temperature coefficient used to control the sharpness of the soft allocation matrix. The iteration formula is as follows:

[0039]

[0040] Ultimately The magnitude is obtained by calculating the exponent element by element and normalizing it according to the valid rows. The soft allocation matrix P soft [t], For T s The logarithmic domain score matrix after Sinkhorn iteration in the logarithmic domain, T s P is the number of Sinkhorn iterations. soft The i-th row of [t] represents the allocation probability of the i-th effective service slot on each candidate channel; for P soft [t] The channel index vector c is obtained by sampling row-wise without replacement. k [t], with the help of The rectangular structure naturally ensures channel orthogonality within the cell during the sampling process.

[0041] The probabilistic power head is generated by a shared Gaussian multilayer perceptron f pow User internal representation H seq，k [t] Acts row-by-row, outputting the downlink Dirichlet concentration for the i-th valid service slot. and the Gaussian mean-standard deviation of uplink latent variables In the formula f pow The output of the downlink concentration logarithm, α min To prevent the lower limit of concentration degradation, and These are the mean and standard deviation of the Gaussian distribution followed by the hidden variable of uplink power, respectively.

[0042] Downlink power share is sampled by mask Dirichlet get, For the concentration of each time slot Concentration vector formed by stacking, m k [t] represents the length. The validity mask vector; the uplink power coefficient is obtained by sampling the Gaussian distribution to obtain the latent variables. After being compressed using Sigmoid and multiplied by the mask, that is... For the uplink power latent variable of the i-th effective service slot, The uplink power factor after compression and masking, m k，i [t] represents the mask m. k The i-th component of [t].

[0043] The final executable downlink and uplink powers of the i-th effective user m are respectively and For the downlink power budget after deducting the relay reserved power, The maximum downlink transmission power of the UAV, P A2A Reserve power for relay between drones This sets the maximum uplink transmit power limit for users.

[0044] Step 4: Construct a centralized training-distributed execution dual-timescale collaborative training framework based on the multi-agent proximal policy optimization algorithm to achieve end-to-end learning and deployment of strategies at the strategic and tactical levels. The specific construction method is as follows:

[0045] Each drone is deployed with one strategic layer H-Agent and one tactical layer L-Agent, with each layer maintaining its own parameter θ. Φ Actor Network And the parameter is φ Φ Critic Value Network Where Φ∈{H,L} is the layer index, H and L represent the strategic layer and tactical layer respectively, and θ Φ With φ Φ These are the learnable parameters of the Φ-th layer Actor and Critic networks, respectively, with each layer independently maintaining an experience buffer B. H and B L and the discount factor γ that matches their respective time scales H and γ L .

[0046] The two-layer Actor network updates the agent target by truncating the MAPPO with entropy regularization:

[0047]

[0048] in To calculate the expectation of the sampled trajectory, τ is the decision step index at the corresponding time scale. and These represent the local observation and the action performed at step τ of layer Φ, respectively. For the likelihood ratio of the new and old strategies, This is the old strategy prior to this update. For the advantage estimate, ∈ π β is the shear hyperparameter. ent S[·] is the entropy coefficient, and S[·] is the policy entropy. For mixed discrete-continuous actions, the log-likelihood is jointly evaluated by summing the log probability mass of the discrete component and the log probability density of the continuous component.

[0049] The two-layer Critic network is updated using value shearing regression loss:

[0050]

[0051] in Let Φ be the global state used for Critic in step τ. For current value estimation, This refers to the old value network prior to this round of updates. The value after cutting based on the old value, ∈ V For value shearing parameters, To achieve the goal of self-reward.

[0052] The training-execution process is as follows: At the beginning of each strategic cycle n, the H-Agent of each UAV is based on local observations o k，H [n] Samples 3D velocity and motion a using a topologically aware graph attention network. k，H [n] and update waypoint w k [n], then the hysteresis incremental delay acceptance algorithm is executed to obtain the user association for the current period. In each tactical time slot t, each UAV's L-Agent is based on local observations under the current topology. k，L [t] Sampling channel allocation and uplink / downlink power hybrid actions via dynamic attention-sinkhorn matching network a k，L [t], after execution, the reward r from L-Agent is obtained. k，L [t], the resulting tactical utility is calculated over the current period T. tac The accumulated data from each time slot is then used as tactical utility feedback and enters into the H-Agent's reward for the next cycle. k，H [n], for each N completed step Each strategic cycle is considered a round. After the round ends, generalized advantage is estimated in B. H and B L The dominance functions of the two layers are estimated separately, and the Actor and Critic parameters of the two layers are updated independently using the PPO cutoff loss. Finally, the buffer is cleared and the next round is entered. During the training phase, the Critic receives the global information corresponding to its layer, and during the execution phase, the Actor only relies on local observations, thus achieving a fully distributed deployment.

[0053] The beneficial effects of this invention are as follows: First, by using a hierarchical decision-making architecture with dual time scales, the three-dimensional trajectory planning and user association at the slow time scale are decoupled from the channel allocation and uplink / downlink power control at the fast time scale, avoiding the state / action space explosion and cross-scale credit allocation difficulties under a single-layer strategy; Second, the strategic layer Topo-GAT network incorporates relay geometry and load heterogeneity into the decision through edge-aware graph attention, and the HIDAA algorithm suppresses ping-pong handover and reduces control overhead through two mechanisms: hysteresis guardianship and incremental proposal; Third, the tactical layer DASM network jointly processes discrete channel allocation and continuous uplink / downlink power control through Sinkhom soft allocation and probabilistic power headers, and naturally supports intra-cell channel orthogonality and variable-length user sets through rectangular structure and validity mask; Fourth, the MAPPO algorithm is used to complete the end-to-end collaborative training of the two-layer strategy under the CTDE paradigm, utilizing global information during training and relying only on local observations during execution, balancing training efficiency and distributed deployment capabilities. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of a simulation scenario for the method of the present invention;

[0055] Figure 2 This is a diagram of the hierarchical hybrid endogenous intelligence framework proposed in this invention;

[0056] Figure 3 This is a comparison diagram of the convergence of H-layer reward training between the present invention and existing multi-agent reinforcement learning methods;

[0057] Figure 4This is a comparison diagram of the convergence of L-layer reward training between the present invention and existing multi-agent reinforcement learning methods;

[0058] Figure 5 This is a comparison chart of the user access rate training convergence between the present invention and existing multi-agent reinforcement learning methods;

[0059] Figure 6 This is a comparison diagram of the logarithmic utility training convergence of the present invention and existing multi-agent reinforcement learning methods;

[0060] Figure 7 This is a comparison chart of the logarithmic utility of ablation experiments using the H2N framework of this invention;

[0061] Figure 8 This is a comparison chart of the scalability logarithmic utility of the present invention and existing methods under different network scales;

[0062] Figure 9 This is a comparison chart of the generalization logarithmic utility of the present invention and the traditional baseline method. Detailed Implementation

[0063] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0064] The proposed method for UAV swarm 3D trajectory planning and joint resource scheduling based on hierarchical reinforcement learning first sets the following operating conditions:

[0065] 1. As attached Figure 1 As shown, the simulation scenario of this invention is a square task area with a side length of L = 500m, in which K = 10 rotary-wing UAVs and M = 100 ground mobile users are deployed. At the start of the task, the UAVs are randomly initialized and deployed in the initial area in the lower left corner. The ground users are initialized according to the Gaussian mixture prior driven by hotspots and their positions are updated according to the Gauss-Markov model on the strategic time scale. The UAV swarm provides access services to the ground users through directional antennas and builds a relay backbone through dedicated line-of-sight links between UAVs.

[0066] 2. The flight altitude range of the drone is [H]. min H max [30, 80] m, maximum horizontal speed Maximum vertical speed Minimum safe distance d safe =20m, communication range between drones Maximum number of users each drone can serve downlink maximum transmit power Reserved relay power P A2A =23dBm, maximum uplink transmit power for users

[0067] 3. Communication parameters are: carrier frequency f c =2.4GHz, total access bandwidth 40MHz, number of orthogonal sub-channels N ch =15, relay uplink bandwidth 20MHz, relay downlink bandwidth 6MHz; noise power spectral density -174dBm / Hz, noise figure 9dB; Rician channel K factor 3dB; UAV-ground channel adopts a probabilistic line-of-sight / non-line-of-sight model based on elevation angle and superimposed with log-normal shadowing and Rician small-scale fading, and UAV-to-UAV relay channel adopts a line-of-sight model.

[0068] 4. The dual-timescale structure is as follows: each training round contains N... step = 50 strategic cycles, each strategic cycle having a length ΔT = 1.0s and including T tac = 10 tactical time slots, each tactical time slot length δ t =0.1s; The strategic layer performs three-dimensional trajectory updates and user association at the beginning of each strategic cycle, and the tactical layer performs channel allocation and uplink / downlink power control in each tactical time slot.

[0069] 5. The training hyperparameter is: total number of training rounds N ep =2000, the learning rates for the strategic layer Actors and Critics are 1×10⁻⁶ respectively. -5 With 1×10 -4 The learning rates for tactical layer Actor and Critic are 5×10⁻⁶. -4 With 5×10 -3 Strategic layer discount factor γ H =0.99, Tactical layer discount factor γ L =0.95, PPO shearing parameter ∈ π =0.2, PPO training rounds 10, batch size 256, entropy regularization coefficient 0.001.

[0070] 6. The hierarchical reinforcement learning algorithms used to train the H2N framework are all implemented based on Python 3.11 and PyTorch, and integrate a dual-timescale simulation environment, including an air-to-ground channel model, a UAV propulsion energy consumption model, and a user association module.

[0071] Based on the above conditions, the proposed hierarchical reinforcement learning-based UAV swarm 3D trajectory planning and joint resource scheduling method has been implemented in a Python simulation environment, and simulation results demonstrate the effectiveness of the method. The specific implementation steps of the hierarchical reinforcement learning-based UAV swarm 3D trajectory planning and joint resource scheduling method are as follows:

[0072] Step 1: Define the applicable scenarios for the method, and construct the observation space and action space for the strategic and tactical layers. The specific steps are as follows:

[0073] The method is applicable to low-altitude air-to-ground cooperative communication scenarios consisting of K rotary-wing UAVs and M ground mobile users. The UAV swarm is deployed in a square mission area with a side length of L. At the start of the mission, the UAVs are randomly initialized and deployed in the lower left corner of the initial area. The time axis is divided into strategic periods n = 1, 2, ..., N. step With tactical time slot t, each strategic cycle has a length of ΔT and includes T. tac The method employs a two-timescale collaborative decision-making architecture, consisting of equal-length tactical time slots. The strategic layer performs UAV 3D trajectory updates and user association during the strategic period n, while the tactical layer performs channel allocation and uplink / downlink power control during the tactical time slot t.

[0074] The local observation of the k-th UAV at the strategic level is defined as follows:

[0075]

[0076] in To normalize the three-dimensional position, To normalize the average power of the previous period, and These are the normalized load of served users and the normalized load of physical coverage, respectively. To normalize the hotspot displacement, For the most recent N nb A masked neighbor observation set for each neighboring drone, where each drone is represented by a normalized relative 3D position, service load, and coverage load, and invalid positions are removed by masking.

[0077] Strategic-level actions are defined as three-dimensional velocity vectors:

[0078] a k，H [n] = v k [n] (14)

[0079] The velocity vector output by the strategy is compressed using tanh and then... After scaling, the projected velocity is determined by the feasible velocity set jointly defined by the flight altitude range, mission boundaries, and minimum safe distance constraints, ensuring the updated waypoint w k [n+1]=w k [n]+v k [n]ΔT strictly satisfies flight constraints; the local observation of the k-th UAV at the tactical level is defined as:

[0080] o k，L [t] = [S] AAV,k [t]F user，k [t], m k [t]] (15)

[0081] Where s AAV，k [t] represents dimension D s =3+2N ch The UAV-level aggregated state vector is obtained by concatenating the current downlink power occupancy rate, uplink relay load ratio, downlink relay load ratio, occupancy counts of neighboring UAVs on each channel, and interference summaries on each channel, where N ch F is the number of orthogonal subchannels in the system. user，k [t] represents the size. The fixed-capacity user feature matrix, where The upper limit of the number of users that can be served, i.e., the capacity of the fixed-capacity service set, is represented by the i-th row corresponding to the feature vector of the i-th user in the local service set, with feature dimension D. u Includes access channel gain, previous time slot uplink power coefficient, logarithmically scaled downlink interference power, historical average rate, normalized previous time slot channel index, and the number of co-channel users Q. k，· [t] Derived from the co-channel conflict flag between cells; m k [t] represents the length. The validity mask has a value of 1 for the i-th component, which indicates that the current data position is a valid user and a value of 0 indicates that the position is a placeholder.

[0082] Tactical actions are defined as follows:

[0083]

[0084] in To satisfy the downlink power share vector of the simplex constraint, This is the uplink power coefficient vector after Sigmoid compression and masking. Assign an index to the channel for each service slot.

[0085] Step 2: Design the hierarchical reward functions for the strategic and tactical layers; the specific design principles for the reward functions are as follows:

[0086] The strategic layer reward consists of seven parts: access reward, tactical utility feedback, topology reward, hotspot guidance reward, energy consumption penalty, collision penalty, and boundary penalty; the total reward for the k-th UAV in strategic period n is:

[0087] r k，H [n] = R acc，k [n]+R util，k [n]+R topo [n]+R hot，k [n]-P eng，k [n]-P col，k [n]-P out，k [n](17)

[0088] Among them, the access reward R acc，k [n] is the weighted combination of local service load and overall cluster access rate; tactical utility feedback R util，k [n] is determined by T within the current strategic cycle. tac The logarithmic utility accumulated per tactical time slot, expressed as an effective relay coefficient α eff，k [t] and β eff，k [t] is obtained by averaging after correction, and its value is in the interval [0, 1]. When the relay load ratio exceeds the threshold, it is inversely reduced to prevent the rate exceeding the relay bottleneck from being over-included; the topology reward is R. topo [n] = w topo ·λ2(L A2A [n]), where L A2A [n] is the Laplacian matrix corresponding to the distance-weighted adjacency matrix between UAVs, λ2(·) is the algebraic connectivity, and w topo Topology reward weight; Hotspot guidance reward R hot,k [n] is a non-positive value, determined by the normalized Euclidean distance between the UAV's horizontal position and the assigned hotspot center; energy penalty P eng，k [n] Penalizes energy consumption exceeding the hovering baseline; collision penalty P col，k [n] when the distance to the neighbor is less than the safe distance d safe Time activation; boundary penalty P out，k [n] is composed of hard outbound terms and soft repulsion terms.

[0089] Tactical layer rewards consist of three parts: uplink utility, downlink utility, and interference penalty. The total reward for the k-th UAV in tactical time slot t is:

[0090]

[0091] Among them, the upward and downward utility items Know The forms are symmetrical, each being the logarithmic sum of the access rates after correction for effective relay coefficients, multiplied by an overload shaping function g(ρ) = min(1, 2 - ρ) based on the relay load ratio, where ρ is the relay load ratio, i.e., the ratio of actual relay load to relay capacity; interference penalty P mterf，k [t] includes two terms: time-division multiplexing congestion penalty within the same cell and inter-cell co-channel interference penalty. The former is determined by the number of co-channel users n. k，c share factor (n) k，c -1) / n k,c The weighted approach, which activates when the per-user co-channel interference power exceeds a threshold, takes into account both intra-cell time-division multiplexing congestion and inter-cell co-channel interference, enabling the learned strategy to both reduce self-collisions and suppress inter-cell interference.

[0092] Step 3: Construct a hierarchical hybrid endogenous intelligent network structure; including a topology-aware graph attention network at the strategic layer, an incremental delayed acceptance user association algorithm based on hysteresis, and a dynamic attention-sinkhorn matching network at the tactical layer. The specific construction method is as follows:

[0093] The topology-aware graph attention network includes two parallel paths: a topology-aware graph branch and a spatial target branch. The topology-aware graph branch is constructed with the k-th UAV as the central node, containing N... g =1+N nb A self-centered local graph of N nodes, where N nb The number of nearest neighbor drones for the drone at this center, with node characteristics including relative 3D displacement, service load, coverage load, and average power, and edge characteristics. This includes the distance between drones, altitude difference, and service load difference, which are then fed into L after linear coding and layer normalization. gat Layer edge-aware multi-head attention encoder, wherein the edge-aware multi-head attention is for the first... The attention weight of the h-th attention head in layer h is calculated using the following formula:

[0094]

[0095] Where i is the center node of the egocentric local graph, j and j′ are the neighbor nodes of i, and the summation of the denominator is used to traverse all the neighbor nodes of i; The number of node attention logs in the h-th head is the scaled dot product of the query vector of the center node i and the key vector of the neighbor node j. W is the edge offset term derived from and truncated by the edge features. e b e These are the weight matrix and bias vector of the edge-encoded linear layer, respectively. max To truncate the upper bound of the bias, restrict the edge bias to [-b]. max b max To suppress extreme values; Adj ij The adjacency mask is set to 1 when neighbor node j is valid for the central node i, and 0 otherwise, thus normalizing the attention only on valid neighbors. The node state is updated by residual multi-head aggregation and then read from the central node i to obtain the topological branch representation z of the k-th UAV. topo，k Then, the spatial target representation z output by the multilayer perceptron of the spatial target branch is compared with the spatial target representation z. spatial，k By fusing the following formula, we obtain the strategic layer shared representation of the k-th UAV:

[0096] z k =LayerNorm(tanh(W fuse [z topo，k ||z spatial，k ]+bfuse (20)

[0097] Among them W fuse With b fuse These are the learnable weight matrix and bias vector of the fusion layer, respectively; the fusion representation z k As a shared input for the strategic layer Actor and Critic head, it outputs three-dimensional velocity motion and value estimation, respectively.

[0098] The hysteresis-based incremental delay-received user association algorithm consists of three stages: the first stage is hysteresis guarding, where each user m is based on the optimal feasible score Γ obtained from the current local channel measurement. best，m [n] Score of the original service drone If a comparison is made, Then add the user to k old The Waiting Collection S keep [k old Otherwise, add it to the active set S. active ,in The first stage is the hysteresis threshold; the second stage is capacity pruning, where each drone k starts from S. keep [k] is scored Γ k，· [n] Before selecting in descending order One user as the initial service set M k [n], the remaining waiting users join the active set; the third phase is a synchronous proposal-response incremental protocol, where each user in the active set makes a proposal to the highest-scoring drone in their feasible drone set that has not yet been rejected. Each drone merges its current service set with the new acceptance queue and then presses top- The program prunes and returns the accept / reject response. The rejected user is removed from the feasible sequence and the program continues to propose the next best drone, looping until no rejections are generated.

[0099] The dynamic attention-Sinkhom matching network comprises three parts: a dual attention state encoder, a Sinkhom channel matching head, and a probabilistic power head. The dual attention state encoder first processes the user feature matrix F... user，k [t]、AAV global state s AAV，k [t] and the channel state are mapped to a common dimension D. h The user embedding vector sequence, AAV global embedding vector, and channel embedding vector sequence are used to obtain the user internal representation H through user self-attention with validity mask. inter The AAV global embedding vector and the channel embedding vector are concatenated into a key-value sequence KV, and then H is used as the key value sequence. inter For querying, cross-attention is performed using key-value pairs, and finally, the result is output via linear projection:

[0100] H seq，k[t] = Proj(LN(MHA) cross (H inter ,KV,KV)+H inter )) (twenty one)

[0101] The Sinkhorn channel matching header will H seq,k [t] and learnable channel embedding Perform a bilinear mapping to obtain the matching score matrix S ij [t] = <h seq，k,i [t],c j After setting the invalid line mask to -∞, perform T. s The Sinkhorn iteration over the logarithmic field is performed, where τ is the Sinkhorn temperature coefficient used to control the sharpness of the soft allocation matrix. The iteration formula is as follows:

[0102]

[0103] Ultimately The magnitude is obtained by calculating the exponent element by element and normalizing it according to the valid rows. The soft allocation matrix P soft [t], For T s The logarithmic domain score matrix after Sinkhorn iteration in the logarithmic domain, T s P is the number of Sinkhorn iterations. soft The i-th row of [t] represents the allocation probability of the i-th effective service slot on each candidate channel; for P soft [t] The channel index vector c is obtained by sampling row-wise without replacement. k [t], with the help of The rectangular structure naturally ensures channel orthogonality within the cell during the sampling process.

[0104] The probabilistic power head is generated by a shared Gaussian multilayer perceptron f pow User internal representation H seq,k [t] Acts row-by-row, outputting the downlink Dirichlet concentration for the i-th valid service slot. and the Gaussian mean-standard deviation of uplink latent variables In the formula f pow The output of the downlink concentration logarithm, α min To prevent the lower limit of concentration degradation, and These are the mean and standard deviation of the Gaussian distribution followed by the hidden variable of uplink power, respectively.

[0105] Downlink power share is sampled by mask Dirichlet get, For the concentration of each time slot Concentration vector formed by stacking, m k [t] represents the length. The validity mask vector; the uplink power coefficient is obtained by sampling the Gaussian distribution to obtain the latent variables. After being compressed using Sigmoid and multiplied by the mask, that is... For the uplink power latent variable of the i-th effective service slot, The uplink power factor after compression and masking, m k,l [t] represents the mask m. k The i-th component of [t].

[0106] The final executable downlink and uplink powers of the i-th effective user m are respectively and For the downlink power budget after deducting the relay reserved power, The maximum downlink transmission power of the UAV, P A2A Reserve power for relay between drones This sets the maximum uplink transmit power limit for users.

[0107] Step 4: Construct a centralized training-distributed execution dual-timescale collaborative training framework based on the multi-agent proximal policy optimization algorithm to achieve end-to-end learning and deployment of strategies at the strategic and tactical levels. The specific construction method is as follows:

[0108] Each UAV independently deploys a strategic layer H-Agent and a tactical layer L-Agent. The Actor input of the H-Agent is local strategic observation, and the output is three-dimensional velocity action and triggers HIDAA user association. The Actor input of the L-Agent is local tactical observation, and the output is a mixed action including Dirichlet downlink share, Gaussian-Sigmoid uplink coefficient and Sinkhorn channel allocation. The Critic network of H-Agent and L-Agent receives global information corresponding to the current layer during the training phase, and does not participate in policy inference during the execution phase.

[0109] Each layer maintains its own parameter θ. Φ Actor Network And the parameter is φ Φ Critic Value Network Where Φ∈{H,L} is the layer index, H and L represent the strategic layer and tactical layer respectively, and θ Φ With φ Φ These are the learnable parameters of the Φ-th layer Actor and Critic networks, respectively, with each layer independently maintaining an experience buffer B. H and BL and the discount factor γ that matches their respective time scales H and γ L .

[0110] The two-layer Actor network updates the agent target by truncating the MAPPO with entropy regularization:

[0111]

[0112] in To calculate the expectation of the sampled trajectory, τ is the decision step index at the corresponding time scale. and These represent the local observation and the action performed at step τ of layer Φ, respectively. For the likelihood ratio of the new and old strategies, This is the old strategy prior to this update. For the advantage estimate, ∈ π β is the shear hyperparameter. ent S[·] is the entropy coefficient, and S[·] is the policy entropy. For mixed discrete-continuous actions, the log-likelihood is jointly evaluated by summing the log probability mass of the discrete component and the log probability density of the continuous component.

[0113] The two-layer Critic network is updated using the value shearing regression loss shown in the following equation:

[0114]

[0115] in Let Φ be the global state used for Critic in step τ. For current value estimation, This refers to the old value network prior to this round of updates. The value after cutting based on the old value, ∈ V For value shearing parameters, To achieve the goal of self-reward.

[0116] The training-execution process is as follows: At the beginning of each strategic cycle n, the H-Agent of each UAV is based on local observations o k，H [n] Samples 3D velocity and motion a using a topologically aware graph attention network. k，H [n] and update waypoint w k [n], then the hysteresis incremental delay acceptance algorithm is executed to obtain the user association for the current period. In each tactical time slot t, each UAV's L-Agent is based on local observations under the current topology. k，L [t] Sampling channel allocation and uplink / downlink power hybrid actions via dynamic attention-sinkhorn matching network a k，L[t], after execution, the reward r from L-Agent is obtained. k，L [t], the resulting tactical utility is calculated over the current period T. tac The accumulated data from each time slot is then used as tactical utility feedback and enters into the H-Agent's reward for the next cycle. k，H [n], for each N completed step Each strategic cycle is considered a round. After the round ends, generalized advantage is estimated in B. H and B L The dominance functions of the two layers are estimated separately, and the Actor and Critic parameters of the two layers are updated independently using the PPO cutoff loss. Finally, the buffer is cleared and the next round is entered. During the training phase, the Critic receives the global information corresponding to its layer, and during the execution phase, the Actor only relies on local observations, thus achieving a fully distributed deployment.

[0117] Based on the above implementation methods, the proposed hierarchical reinforcement learning-based UAV swarm 3D trajectory planning and joint resource scheduling method has completed the entire training and simulation testing process under the Python 3.11 and PyTorch framework, and is based on the attached... Figure 3 - Appendix Figure 9 The effectiveness of the method was verified from multiple dimensions.

[0118] As attached Figure 3 and attached Figure 4 As shown, a comparison of the training convergence process of this invention with four representative multi-agent reinforcement learning baseline methods—IPPO, MASAC, QMIX, and VDN—shows that H2N converges to a higher plateau value faster and more stably on both the H-layer and L-layer reward curves. IPPO is the closest baseline, but its final reward is still significantly lower than that of this invention. MASAC, QMIX, and VDN either fluctuate greatly or converge to a lower level, reflecting the differentiated contributions of the Topo-GAT and DASM modules to policy learning on the two time scales.

[0119] As attached Figure 5 and attached Figure 6 As shown, in terms of the two key performance indicators of user access rate and logarithmic utility, the present invention also enters the high-level platform at an earlier training stage, and the final user access rate is significantly higher than all baselines, and the logarithmic utility leads by a particularly significant margin. This indicates that the strategy learned by the present invention not only has an advantage in the process reward level, but also outperforms existing methods in actual communication performance.

[0120] As attached Figure 7As shown, ablation experiments were conducted on four variants: the full version "FULL" containing both Topo-GAT and DASM, the variant "TOPO-GAT" containing only Topo-GAT, the variant "DASM" containing only DASM, and the basic version "BASE" containing neither. The logarithmic utility ranking was FULL > TOPO-GAT > DASM > BASE. This indicates that Topo-GAT and DASM each perform their respective functions and complement each other at both the strategic and tactical scales, and that relying solely on either module cannot achieve a complete cross-layer performance improvement.

[0121] As attached Figure 8 As shown, under four progressively increasing network scales of K5M50L350, K10M100L500, K15M150L650 and K20M200L800, the total logarithmic utility of this invention consistently leads all baselines, and the advantage increases with the scale, verifying the scalability of the dual-time-scale hierarchical structure under higher user density and stronger coupling pressure.

[0122] As attached Figure 9 As shown, the H2N policy trained only in the K5M50L500 scenario is directly transferred to a larger-scale scenario in the range K∈{5,...,25}, and compared with three traditional baselines: "PSO+WF+HA" (Particle Swarm Optimization for Trajectory, Water-filling Algorithm for Power, and Hungarian Algorithm for Channel Allocation), "Greedy" (based on instantaneous channel gain), and "Random" (random action). The present invention demonstrates positive generalization ability, with the logarithmic utility peaking around K=10-13, and still significantly higher than all traditional baselines at K=25. This proves that the cross-layer policy learned by the present invention does not overfit to the training scenario and can stably transfer to a larger-scale scenario that has not been seen before.

[0123] In summary, the proposed method for UAV swarm 3D trajectory planning and joint resource scheduling based on hierarchical reinforcement learning effectively solves the problem of strong coupling optimization between UAV swarm communication and maneuvering in low-altitude intelligent networks by using the H2N framework and CTDE dual-timescale collaborative training under the MAPPO algorithm. It significantly outperforms existing methods in key indicators such as user access rate and logarithmic utility, and has good scalability and cross-scenario generalization performance.

[0124] The contents not described in detail in this application are existing technologies known to those skilled in the art.

Claims

1. A method for 3D trajectory planning and joint resource scheduling of UAV swarms based on hierarchical reinforcement learning, comprising the following steps: Step 1: Define the scenarios in which the method is applicable, and construct the observation space and action space of the strategic and tactical layers; The method is applicable to low-altitude air-to-ground cooperative communication scenarios consisting of K rotary-wing UAVs and M ground mobile users. The UAV swarm is deployed in a square mission area with a side length of L. At the start of the mission, the UAVs are randomly initialized and deployed in the lower left corner of the initial area. The time axis is divided into strategic periods n = 1, 2, ..., N. step With tactical time slot t, each strategic cycle has a length of ΔT and includes T. tac The method employs a two-timescale collaborative decision-making architecture, namely, a strategic layer and a tactical layer, with equal-length tactical time slots. The strategic layer performs UAV 3D trajectory updates and user association during the strategic period n, while the tactical layer performs channel allocation and uplink / downlink power control during the tactical time slot t. The local observation of the k-th UAV at the strategic level is defined as follows: in To normalize the three-dimensional position, To normalize the average power of the previous period, These are the normalized load of served users and the normalized load of physical coverage, respectively. To normalize the hotspot displacement, For the most recent N nb A masked neighbor observation set for each neighboring drone, where each drone is represented by a normalized relative 3D position, service load, and coverage load, and invalid positions are removed by masking. Strategic-level actions are defined as three-dimensional velocity vectors: a k，H [n]=v k [n] (2) in The output of the policy network is compressed with tanh and scaled according to the maximum horizontal / vertical velocity before being projected onto the feasible velocity set; The local observation of the k-th UAV at the tactical level is defined as: o k，L [t]=[s AAV，k [t]F user，k [t],m k [t]] (3) Where S AAV，k [t] represents dimension D s =3+2N ch The UAV-level aggregated state vector is obtained by concatenating the current downlink power occupancy rate, uplink relay load ratio, downlink relay load ratio, occupancy counts of neighboring UAVs on each channel, and interference summaries on each channel, where N ch F is the number of orthogonal subchannels in the system. user，k [t] represents the size. The fixed-capacity user feature matrix, where The upper limit of the number of users that can be served, i.e., the capacity of the fixed-capacity service set, is represented by the i-th row corresponding to the feature vector of the i-th user in the local service set, with feature dimension D. u Includes access channel gain, previous time slot uplink power coefficient, logarithmically scaled downlink interference power, historical average rate, normalized previous time slot channel index, and the number of co-channel users Q. k [t] Derived co-channel conflict flag between cells; m k [t] represents the length. The validity mask, where the i-th component takes a value of 1 to indicate that the current data position is a valid user, and a value of 0 to indicate that the position is a placeholder; Tactical actions are defined as follows: in To satisfy the downlink power share vector of the simplex constraint, c is the uplink power coefficient vector. k [t] is the channel allocation index vector for each service slot; Step 2: Design the hierarchical reward functions for the strategic and tactical layers; the specific design principles for the reward functions are as follows: The strategic layer reward consists of seven parts: access reward, tactical utility feedback, topology reward, hotspot guidance reward, energy consumption penalty, collision penalty, and boundary penalty. The total reward for the k-th UAV in strategic period n is: r k，H [n]=R acc，k [n]+R util，k [n]+R topo [n]+R hot，k [n]-P eng,k [n]-P col，k [n]-P out，k [n] (5) Among them, the access reward R acc，k [n] is the weighted combination of local service load and overall cluster access rate; tactical utility feedback R util，k [n] is determined by T within the current strategic cycle. tac The logarithmic utility accumulated per tactical time slot, expressed as an effective relay coefficient α eff，k [t] and β eff，k [t] is obtained by averaging after correction, and its value is in the interval [0, 1]. When the relay load ratio exceeds the threshold, it is inversely reduced to prevent the rate exceeding the relay bottleneck from being over-included; the topology reward is R. topo [n] = w topo ·λ2(L A2A [n]), where L A2A [n] is the Laplacian matrix corresponding to the distance-weighted adjacency matrix between UAVs, λ2(·) is the algebraic connectivity, and W topo Topology reward weight; Hotspot guidance reward R hot，k [n] is a non-positive value, determined by the normalized Euclidean distance between the UAV's horizontal position and the assigned hotspot center; energy penalty P eng，k [n] Penalizes energy consumption exceeding the hovering baseline; collision penalty P col，k [n] when the distance to the neighbor is less than the safe distance d safe Time activation; boundary penalty P out,k [n] is composed of hard outbound terms and soft repulsion terms; Tactical layer rewards consist of three parts: uplink utility, downlink utility, and interference penalty. The total reward for the k-th UAV in tactical time slot t is: Among them, the upward and downward utility items and The forms are symmetrical, each being the logarithmic sum of the access rates after correction for effective relay coefficients, multiplied by an overload shaping function g(ρ) = min(1, 2-p) based on the relay load ratio, where ρ is the relay load ratio, i.e., the ratio of actual relay load to relay capacity; interference penalty P mterf，k [t] includes two terms: time-division multiplexing congestion penalty within the same cell and inter-cell co-channel interference penalty. The former is determined by the number of co-channel users n. k，c share factor (n) k，c -1) / n k，c The weighted average, which is activated when the per-user co-channel interference power exceeds a threshold; Step 3: Construct a hierarchical hybrid AI-Native (H2N) network structure, including a topology-aware graph attention network (Topo-GAT) and a hysteresis-based incremental deferred acceptance algorithm (HIDAA) at the strategic layer, and a dynamic attentive-sinkhorn matching network (DASM) at the tactical layer. The specific construction method is as follows: The topology-aware graph attention network includes two parallel paths: a topology-aware graph branch and a spatial target branch. The topology-aware graph branch is constructed with the k-th UAV as the central node, containing N... g =1+N nb A self-centered local graph of N nodes, where N nb The number of nearest neighbor drones for the drone at this center, with node characteristics including relative 3D displacement, service load, coverage load, and average power, and edge characteristics. This includes the distance between drones, altitude difference, and service load difference, which are then fed into L after linear coding and layer normalization. gat The edge-aware multi-head attention encoder calculates the attention weight of the h-th attention head in the l-th layer using the following formula: Where i is the center node of the egocentric local graph, j and j′ are the neighbor nodes of i, and the summation of the denominator is used to traverse all the neighbor nodes of i; The number of node attention logs in the h-th head is the scaled dot product of the query vector of the center node i and the key vector of the neighbor node j. W is the edge offset term derived from and truncated by the edge features. e b e These are the weight matrix and bias vector of the edge-encoded linear layer, respectively. max To truncate the upper bound of the bias, restrict the edge bias to [-b]. max b max To suppress extreme values; Adj ij The adjacency mask is set to 1 when neighbor node j is valid for the center node i, and 0 otherwise, so that the attention is normalized only on the valid neighbors. The node state is updated by residual multi-head aggregation and read from the central node i to obtain the topological branch representation z of the k-th UAV. topo，k Then, the space target representation Z output by the multilayer perceptron of the space target branch is compared with the space target representation Z. spatial，k By fusing the following formula, we obtain the strategic layer shared representation of the k-th UAV: With k =LayerNorm(tanh(W fuse [With topo，k ||with spatial，k] +b fuse )) (8) Among them W fuse With b fuse These are the learnable weight matrix and bias vector of the fusion layer, respectively; the fusion representation z k As a shared input for the strategic layer Actor and Critic head, it outputs three-dimensional velocity action and value estimation respectively; The hysteresis-based incremental delay-received user association algorithm consists of three stages: the first stage is hysteresis guarding, where each user m is based on the optimal feasible score Γ obtained from the current local channel measurement. best，m [n] Score of the original service drone If a comparison is made, Then add the user to k old The Waiting Collection S keep [k old Otherwise, add it to the active set S. active ,in The hysteresis threshold; The second phase is capacity reduction, where each drone k is reduced from S... keep [k] is scored Γ k [n] Select the first [n] in descending order One user as the initial service set M k [n], the remaining waiting users join the active set; the third phase is a synchronous proposal-response incremental protocol, where each user in the active set makes a proposal to the highest-scoring drone in their feasible drone set that has not yet been rejected, and each drone merges its current service set with the new acceptance queue and then presses the button again. The program prunes and returns the accept / reject response. The rejected user is removed from the feasible sequence and the program continues to propose the next best drone. This process is repeated until no rejections are generated. The dynamic attention-Sinkhorn matching network comprises three parts: a dual attention state encoder, a Sinkhorn channel matching head, and a probabilistic power head. The dual attention state encoder first processes the user feature matrix F... user，k [t]、AAV global state s AAV，k [t] and the channel state are mapped to a common dimension D. h The user embedding vector sequence, AAV global embedding vector, and channel embedding vector sequence are used to obtain the user internal representation H through user self-attention with validity mask. inter The AAV global embedding vector and the channel embedding vector are concatenated into a key-value sequence KV, and then H is used as the key value sequence. inter For querying, cross-attention is performed using key-value pairs, and finally, the result is output via linear projection: H seq，k [t]=Proj(LN(MHA cross (H inter ,KV,KV)+H inter )) (9) The Sinkhorn channel matching header will H seq，k [t] and learnable channel embedding Perform a bilinear mapping to obtain the matching score matrix S ij [t] = <h seq，k，i [t],c j > Set the invalid line mask to ∞ and then perform T s The Sinkhorn iteration over the logarithmic field is performed, where τ is the Sinkhorn temperature coefficient used to control the sharpness of the soft allocation matrix. The iteration formula is as follows: Ultimately The magnitude is obtained by calculating the exponent element by element and normalizing it according to the valid rows. The soft allocation matrix P soft [t], For T s The logarithmic domain score matrix after Sinkhorn iteration in the logarithmic domain, T s P is the number of Sinkhorn iterations. soft The i-th row of [t] represents the allocation probability of the i-th effective service slot on each candidate channel; for P soft [t] The channel index vector c is obtained by sampling row-wise without replacement. k [t], with the help of The rectangular structure naturally ensures channel orthogonality within the cell during the sampling process; The probabilistic power head is generated by a shared Gaussian multilayer perceptron f pow User internal representation H seq，k [t] Acts row-by-row, outputting the downlink Dirichlet concentration for the i-th valid service slot. and the Gaussian mean-standard deviation of uplink latent variables In the formula f pow The output of the downlink concentration logarithm, α min To prevent the lower limit of concentration degradation, and These are the mean and standard deviation of the Gaussian distribution followed by the hidden variable of uplink power, respectively. Downlink power share is sampled by mask Dirichlet get, For the concentration of each time slot Concentration vector formed by stacking, m k [t] represents the length. The validity mask vector; the uplink power coefficient is obtained by sampling the Gaussian distribution to obtain the latent variables. After being compressed using Sigmoid and multiplied by the mask, that is... For the uplink power latent variable of the i-th effective service slot, The uplink power factor after compression and masking, m k，i [t] represents the mask m. k The i-th component of [t]; The final executable downlink and uplink powers of the i-th effective user m are respectively and For the downlink power budget after deducting the relay reserved power, The maximum downlink transmission power of the UAV, P A2A Reserve power for relay between drones This represents the maximum uplink transmit power limit for users. Step 4: Construct a centralized training-distributed execution dual-timescale collaborative training framework based on the multi-agent proximal policy optimization algorithm to achieve end-to-end learning and deployment of strategies at the strategic and tactical levels. The specific construction method is as follows: Each drone is deployed with one strategic layer H-Agent and one tactical layer L-Agent, with each layer maintaining its own parameter θ. Φ Actor Network And the parameter is φ Φ Critic Value Network Where Φ∈{H,L} is the layer index, H and L represent the strategic layer and tactical layer respectively, and θ Φ With φ Φ These are the learnable parameters of the Φ-th layer Actor and Critic networks, respectively, with each layer independently maintaining an experience buffer B. H and B L and the discount factor γ that matches their respective time scales H and γ L ; The two-layer Actor network updates the agent target by truncating the MAPPO with entropy regularization: in To calculate the expectation of the sampled trajectory, τ is the decision step index at the corresponding time scale. and These represent the local observation and the action performed at step τ of layer Φ, respectively. For the likelihood ratio of the new and old strategies, This is the old strategy prior to this update. For advantage estimation, ε π β is the shear hyperparameter. ent S[·] is the entropy coefficient, and S[·] is the policy entropy. For mixed discrete-continuous actions, the log-likelihood is jointly evaluated by summing the log probability mass of the discrete component and the log probability density of the continuous component. The two-layer Critic network is updated using value shearing regression loss: in Let Φ be the global state used for Critic in step τ. For current value estimation, This refers to the old value network prior to this round of updates. The value after cutting around the old value, ε V For value shearing parameters, To achieve the goal of self-reward; The training-execution process is as follows: At the beginning of each strategic cycle n, the H-Agent of each UAV is based on local observations o k,H [n] Samples 3D velocity and motion a using a topologically aware graph attention network. k，H [n] and update waypoint w k [n], then the hysteresis incremental delay acceptance algorithm is executed to obtain the user association for the current period. In each tactical time slot t, each UAV's L-Agent is based on local observations under the current topology. k，L [t] Sampling channel allocation and uplink / downlink power hybrid actions via dynamic attention-sinkhorn matching network a k，L [t], after execution, the reward r from L-Agent is obtained. k，L [t], the resulting tactical utility is calculated over the current period T. tac The accumulated data from each time slot is then used as tactical utility feedback and enters into the H-Agent's reward for the next cycle. k，H [n], for each N completed step Each strategic cycle is considered a round. After the round ends, generalized advantage is estimated in B. H and B L The dominance functions of the two layers are estimated separately, and the Actor and Critic parameters of the two layers are updated independently using the PPO cutoff loss. Finally, the buffer is cleared and the next round is entered. During the training phase, the Critic receives the global information corresponding to its layer, and during the execution phase, the Actor only relies on local observations, thus achieving a fully distributed deployment.

Images