Macro-micro dual-loop based space-frequency energy joint optimization method for heterogeneous UAV cluster

Abstract

The application provides a heterogeneous UAV cluster space-frequency-energy joint optimization method based on a macro-micro double loop, which comprises the following steps: dividing UAV cluster nodes into cluster heads CH and cluster members CM, constructing a heterogeneous MAC layer model, and periodically updating statistical channel distribution information CDI; the CH determines UAV flight trajectories, spectrum allocation and target signal-to-interference-and-noise ratio of each CM through MAPPO; the CM uses IMU to assist EKF to predict link retention probability in a fast time scale, performs access or switching based on the link retention probability according to double thresholds, and adopts an incremental PID to track the target signal-to-interference-and-noise ratio and adjust the transmission power; the CM feeds back the fast time scale execution statistics to the CH, the CH updates the next slow time scale decision accordingly, and trains the MAPPO strategy by using a centralized training and distributed execution architecture to jointly optimize the spatial position, spectrum resource and transmission energy of the UAV cluster; by decoupling the double scales, the CM end only needs to run a light-weight PID and EKF, thereby avoiding high energy consumption and high delay caused by running a deep network on an embedded terminal.

Description

Technical Field

[0001] This invention relates to the field of UAV swarm communication and resource scheduling, and provides a method for joint optimization of space-frequency energy of heterogeneous UAV swarms based on macro-micro dual-loop. Background Technology

[0002] With the widespread application of UAV swarms in military and civilian fields, heterogeneous swarms (composed of a small number of high-computing-power cluster heads (CHs) and a large number of low-computing-power cluster members (CMs)) have become the mainstream architecture. However, existing resource management technologies face severe challenges in practical deployment: First, the high-speed maneuvering of UAVs leads to severe Doppler shift, causing traditional resource allocation algorithms based on instantaneous CSI to fail due to the "acquisition-and-expiration" nature of channel information, resulting in frequent disconnections. Second, while existing deep reinforcement learning schemes for the entire network have superior theoretical performance, they neglect the computational bottleneck of CMs. Running complex Actor-Critic networks on embedded terminals struggles to meet the millisecond-level real-time requirements of the MAC layer and consumes enormous amounts of energy. Finally, flight trajectory, power control, and topology connectivity are highly coupled three-dimensional resources. Traditional step-by-step optimization methods struggle to achieve global optimum, while multi-agent synchronous game theory is prone to "policy oscillations." Therefore, developing a joint optimization method that can decouple time scales and balance engineering feasibility with global performance is an urgent need to improve the practical effectiveness of heterogeneous swarms. Summary of the Invention

[0003] To address the problems existing in the prior art, this invention proposes a joint optimization method for the space-frequency-energy of heterogeneous UAV clusters based on macro-micro dual-loop systems. This method includes:

[0004] S1. Divide the UAV cluster nodes into cluster heads (CH) and cluster members (CM), construct a heterogeneous MAC layer model, and periodically update the statistical channel distribution information (CDI).

[0005] S2 and CH determine the UAV flight trajectory, spectrum allocation, and target signal-to-interference-plus-noise ratio of each CM through MAPPO based on their own state, neighbor states, user density, and feedback from the previous CM cycle on a slow time scale.

[0006] S3 and CM use IMU-assisted EKF to predict link hold probability on a fast time scale. Based on the link hold probability, they perform access or handover according to dual thresholds and use incremental PID to track the target signal-to-interference-plus-noise ratio and adjust the transmit power.

[0007] S4 and CM feed back the fast timescale execution statistics to CH. CH updates the decision for the next slow timescale based on this and uses a centralized training and distributed execution architecture to train the MAPPO strategy in order to jointly optimize the spatial location, spectrum resources and transmission energy of the UAV cluster.

[0008] The beneficial effects of this invention are:

[0009] This invention employs dual-scale decoupling, requiring only lightweight PID and EKF execution at the CM end, thus avoiding the high power consumption and latency associated with running deep networks on embedded terminals. It utilizes statistical features and IMU-assisted prediction to address the frequent disconnections caused by CSI expiration in highly maneuverable scenarios. Through stationarity constraints in the loss function, it effectively reduces mechanical fatigue and RF device losses caused by frequent UAV course changes and power switching. Finally, through closed-loop control of target SINR commands, it improves overall capacity while ensuring connectivity of edge task nodes. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating the overall implementation of a heterogeneous UAV cluster joint optimization method according to the present invention.

[0011] Figure 2 This is a block diagram of the dual-loop control logic of the present invention;

[0012] Figure 3 This is a flowchart of the fast time-scale link survivability prediction and access decision-making process at the CM end of the present invention.

[0013] Figure 4 This is a schematic diagram of the CH-terminal MAPPO deep reinforcement learning network structure and reward mechanism in the slow timescale of this invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] A method for joint optimization of space-frequency energy of heterogeneous UAV clusters based on macro-micro dual-loop, such as... Figures 1-4 As shown, the method includes:

[0016] S1. Divide the UAV cluster nodes into cluster heads (CH) and cluster members (CM), construct a heterogeneous MAC layer model, and periodically update the statistical channel distribution information (CDI).

[0017] S2 and CH determine the UAV flight trajectory, spectrum allocation, and target signal-to-interference-plus-noise ratio of each CM through MAPPO based on their own state, neighbor states, user density, and feedback from the previous CM cycle on a slow time scale.

[0018] S3 and CM use IMU-assisted EKF to predict link hold probability on a fast time scale. Based on the link hold probability, they perform access or handover according to dual thresholds and use incremental PID to track the target signal-to-interference-plus-noise ratio and adjust the transmit power.

[0019] S4 and CM feed back the fast timescale execution statistics to CH. CH updates the decision for the next slow timescale based on this and uses a centralized training and distributed execution architecture to train the MAPPO strategy in order to jointly optimize the spatial location, spectrum resources and transmission energy of the UAV cluster.

[0020] The core technical solution of this invention is to deploy a MAPPO network at the CH end and a lightweight EKF and PID controller at the CM end. Through physical layer feature extraction, link survivability prediction, two-layer joint decision-making, and closed-loop feedback updates, the specific steps are as follows:

[0021] S1: Constructing a heterogeneous MAC layer model and perceptual statistical CDI.

[0022] By dividing the cluster into cluster heads (CH) responsible for global scheduling and cluster members (CM) responsible for specific tasks, the system constructs a master-slave architecture. Addressing the challenge of fleeting Channel State Information (CSI) in highly dynamic scenarios, a statistical channel model based on the Nakagami-m distribution is introduced. This model replaces high-frequency instantaneous measurements with periodic maintenance of statistical tuples, effectively reducing signaling overhead.

[0023] S11: Divide the drone swarm into two types of nodes: The cluster head (CH) acts as a mobile air base station, responsible for global resource scheduling on a slow time scale. Cluster member CM, acting as a task node, is responsible for fast timescale access and power control. After identifying the entities in the network, in order to quantify the communication quality between nodes, step S12 is required to establish a suitable channel fading model.

[0024] S12: For UAV air-to-air (A2A) channels, outdated instantaneous CSIs are discarded, and statistical characteristics are used to describe channel quality. CM With CH Channel gain between The probability density function is modeled as follows:

[0025]

[0026] in, For fading shape parameters, For average channel power, This is the Gamma function.

[0027] After the theoretical model is constructed, the system must obtain the above parameters in real time through measurement methods during actual operation, so step S13 needs to be executed.

[0028] S13: The system periodically senses and updates statistical tuples. ,in The background interference noise level is set for local measurements.

[0029] After completing the construction of the statistical environment perception model in step S1, given that the statistical information is insufficient to cope with millisecond-level topological changes and the computing power of the CM is limited, the system needs to enter step S2 to execute micro-closed-loop control with a fast time scale.

[0030] S2: Fast time scale - CM performs link prediction and PID access based on IMU-assisted EKF.

[0031] The CM (Mobile Controller) utilizes an onboard inertial measurement unit (IMU) and Doppler characteristics to accurately predict the link's survivability probability through an extended Kalman filter (EKF). This prediction mechanism, combined with dual-threshold hysteresis access decision, effectively reduces invalid handovers. Simultaneously, an incremental PID algorithm enables the CM to instantaneously track the target signal-to-interference-plus-noise ratio, ensuring rapid and robust power regulation even in complex electromagnetic environments.

[0032] S21: CM uses real-time acceleration data read from the onboard IMU. Using the Doppler frequency shift calculated by the physical layer as the control input, and combining it with the observed value, the state equation is constructed as follows:

[0033]

[0034] Position covariance matrix based on EKF output Calculate future time windows Link maintenance probability within :

[0035]

[0036] in, Here is the state transition matrix. To control the input matrix, For process noise, Let be the system state vector. To maintain the minimum receiving distance or power threshold required for communication, The mean of future distance / state predicted by EKF. The standard deviation derived from the position covariance matrix output by EKF. This is the tail probability function of the standard normal distribution.

[0037] After obtaining the link survival probability, in order to avoid the signaling overhead caused by frequent handover, it is necessary to proceed to step S22 to perform the delayed access decision.

[0038] S22: Set access high threshold With low threshold cut Only when Access is initiated only when... The switch is triggered at any time.

[0039] After the topology connection is established, in order to achieve stable transmission in an interference environment, it is necessary to proceed to step S23 to execute servo power control.

[0040] S23: CM employs an incremental PID algorithm to quickly track the target signal-to-interference-plus-noise ratio (SIR / NDR) sent by CH. Specifically, the CH (CH-based transmission controller) sends the target signal-to-interference-plus-noise ratio (SIR) and PID parameter sets to the CM (CM) on a slow timescale; the CM calculates the SIR error on a fast timescale, updates the transmit power using incremental PID control, and feeds back the error, power, and link interruption statistics to the CH at the end of the slow timescale period. The CM adjusts the transmit power based on the current measurements. :

[0041]

[0042] Among them, the signal-to-interference-plus-noise ratio error , These are the proportional, integral, and derivative gains of the PID controller, respectively. This mechanism ensures that the CM can respond to sudden disturbances in milliseconds.

[0043] Step S2 achieves rapid steady-state at the underlying level. To achieve optimal global performance, we need to proceed to step S3, where the CH terminal performs intelligent adjustment of macroscopic parameters on a slow time scale.

[0044] S3: Slow timescale – CH-end executes joint intelligent decision-making based on MAPPO

[0045] The macro-level decision-making layer utilizes CH's high-performance computing power for cross-dimensional resource scheduling. Based on user density, interference levels, and neighbor states within the service area, the CH agent uses the Multi-Agent Proximity Policy Optimization (MAPPO) algorithm to uniformly plan the UAV's flight trajectory, frequency bandwidth, and interference temperature threshold within a second-level cycle. By incorporating a guaranteed rate constraint into the reward function, the system maximizes total capacity while also ensuring fairness among edge task nodes.

[0046] S31: CH agent constructs observation vector This includes: the user's own and its neighbors' status, and the user density distribution within the service area. The average interference level of the fast loop feedback in the previous cycle is also considered. After obtaining the global observation information, in order to dynamically adapt to the network load and interference environment, step S32 is required to generate joint actions using a neural network.

[0047] S32: The CH agent outputs a hybrid action vector through the MAPPO network. It is not a pure numerical value, but refers to the control command of a physical entity. Adjustments to the yaw angle and thrust vector of the corresponding flight control system; The mask corresponding to the OFDMA subcarrier allocation matrix; A closed-loop gain compensation target for the physical layer RF link is set. To guide the agent to evolve towards the global optimum, step S33 is required to design a reward function with fairness constraints.

[0048] S33: The following reward function is used for evaluation:

[0049]

[0050] in, These are the weighting coefficients for each indicator; The total system capacity (spectral efficiency). For indicator functions; For task nodes transmission rate This is the minimum guaranteed rate threshold. Minimum rate constraint, when the rate of edge users is lower than Punishments should be imposed in a timely manner to ensure fairness; This is a penalty for mechanical energy consumption.

[0051] S34: Employs a fast and slow dual-timescale longitudinal collaborative mechanism, when CH predicts the user density distribution in a certain area... During a surge, by reducing the CM in that region This mechanism forces the underlying PID controller to reduce its transmit power (Ptx) to lower the overall network interference temperature threshold, thereby freeing up spectrum space for high-priority links. This mechanism achieves vertical integration of "sensing (CH) - decision (CH) - constraint (CM) - execution (CM)".

[0052] Step S3 completes a single macro decision. In order for this two-layer mechanism to continuously iterate and adapt to environmental changes, we need to proceed to step S4 to build a closed-loop training and execution architecture.

[0053] In this embodiment, the CH terminal performs joint intelligent decision-making based on MAPPO, including:

[0054] Step 1: CH constructs the observation vector based on its own state, neighbor states, user density, and CM feedback from the previous slow time scale period;

[0055] Step 2: Input the observation vector into the CH-end Actor network, and output the joint action including flight trajectory adjustment, spectrum allocation and target signal-to-interference-plus-noise ratio through MAPPO;

[0056] Step 3: CH maps the combined actions into flight control, spectrum allocation and microloop control commands, and CM performs link prediction, access or handover and PID power adjustment on a fast time scale;

[0057] Step 4: The CM will feed back the execution statistics to the CH, which will then calculate the periodic reward and update the observation vector for the next slower timescale period.

[0058] The fast and slow dual-timescale longitudinal coordination mechanism includes: the CH periodically sends the target signal-to-interference-plus-noise ratio (SIR) and interference temperature thresholds on a slow timescale, while the CM performs multiple fast timescale control steps within the period; when the CH predicts an increase in load in the target area, it reduces the target SIR of low-priority CMs in that area, causing them to reduce their transmit power, in order to release interference margin for high-priority links; after the period ends, the CM feeds back the SIR error, interruption probability, and energy consumption to the CH as observation information for the next period.

[0059] S4: Dual timescale closed loop and CTDE training.

[0060] Employing a "Centralized Training, Distributed Execution (CTDE)" model, the system incorporates a stationarity constraint term in the loss function to suppress power oscillations and reduce mechanical fatigue. Through a fast-slow loop interaction protocol, the CM feeds back its operational summary to the CH for policy iteration, while the CH issues refined control commands to guide the CM, thereby constructing a continuously evolving, dynamic optimization ecosystem encompassing four dimensions of resources: space, time, frequency, and energy.

[0061] S41: Construct a centralized training distributed execution (CTDE) architecture based on mixed time scales, adopting a centralized critic and decentralized actor model.

[0062] The Critic network utilizes a global state space that includes all macroscopic observations of the CH layer and aggregation performance statistics (such as average SINR bias and PID saturation) within a slow time window above the CM layer. This allows us to assess the value of the current strategy and effectively perceive the following ability of the underlying PID.

[0063] A stationarity constraint term is specifically introduced in the construction of the loss function. The core technology is to limit the mechanical fatigue of the drone's actuators and the nonlinear distortion of the radio frequency power amplifier. By suppressing abrupt changes in control commands, it ensures that the underlying topology will not frequently disconnect due to power oscillations during dynamic switching of heterogeneous networks. The specific loss function model is as follows:

[0064]

[0065] in, It represents the change in the target interference temperature threshold (or closed-loop gain compensation target) over two consecutive slow time periods, used to limit sudden changes in commands; For time step The corresponding advantage function; For policy network parameters, The probability ratio between the old and new strategies. Let be the cutting constant. For the error of the value function, For policy entropy, and These are the weighting coefficients.

[0066] After completing the training architecture and loss function construction in step S41, in order to apply the trained policy to the actual heterogeneous network interaction and clarify the specific collaboration process of the two-layer loop at different time scales, it is necessary to proceed to step S42 to carry out the fast and slow loop interaction protocol in the execution phase.

[0067] S42: Execution phase, CH occurs every slow cycle. Sending target signal-to-interference-plus-noise ratio And a set of PID parameters matched according to the Doppler frequency shift level. During the period, CM performs EKF prediction and PID power adjustment multiple times, and calculates a statistical summary (such as average) for that period. With interruption probability At the end of the cycle, CM packages and feeds back the statistical summary to CH as part of the macroscopic observations at the next time step.

[0068] To address the issue of low data utilization caused by the mismatch between fast and slow paces, and to leverage feedback data for continuous iterative optimization of the strategy, step S43 is required to update the strategy and replay the experience.

[0069] S43: System maintenance layered experience replay buffer, storing tuples Aggregated rewards Defined as the weighted sum of the system capacity integral and the violation penalty over the entire slow cycle:

[0070]

[0071] in, This represents the number of fast cycle / micro-control steps contained within a slow cycle.

[0072] This mechanism ensures that MAPPO optimizes long-term average performance rather than instantaneous peak performance, thus improving the system's robustness in time-varying channels.

[0073] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for joint optimization of space-frequency energy of heterogeneous UAV clusters based on macro-micro dual-loop systems, characterized in that, include: S1. Divide the UAV cluster nodes into cluster heads (CH) and cluster members (CM), construct a heterogeneous MAC layer model, and periodically update the statistical channel distribution information (CDI). S2 and CH determine the UAV flight trajectory, spectrum allocation, and target signal-to-interference-plus-noise ratio of each CM through MAPPO based on their own state, neighbor states, user density, and feedback from the previous CM cycle on a slow time scale. S3 and CM use IMU-assisted EKF to predict link hold probability on a fast time scale. Based on the link hold probability, they perform access or handover according to dual thresholds and use incremental PID to track the target signal-to-interference-plus-noise ratio and adjust the transmit power. S4 and CM feed back the fast timescale execution statistics to CH. CH updates the decision for the next slow timescale based on this and uses a centralized training and distributed execution architecture to train the MAPPO strategy in order to jointly optimize the spatial location, spectrum resources and transmission energy of the UAV cluster.

2. The method for joint optimization of space-frequency energy of heterogeneous UAV clusters based on macro-micro dual-loop as described in claim 1, characterized in that, Constructing a heterogeneous MAC layer model includes: using statistical features to describe channel quality for UAV air-to-air channels; calculating CM With CH Channel gain between Based on channel gain A probability density function is constructed and used as a channel fading model; the model periodically senses and updates the statistical tuples.

3. The method for joint optimization of space-frequency energy of heterogeneous UAV clusters based on macro-micro dual-loop as described in claim 1, characterized in that, The CM-side performs IMU-assisted EKF-based link prediction and PID access, including: Step 1: CM uses the onboard IMU to read the real-time acceleration. As the control input, the Doppler frequency shift calculated by the physical layer is used as the observation value to construct the state equation; the position covariance matrix output by the EKF is used as the control input. Calculate future time windows Link maintenance probability within ; Step 2: Set the access high threshold With low threshold cut ;when Access is initiated only when... Switching is triggered on time; Step 3: CH sends the target signal-to-interference-plus-noise ratio (SIR) and PID parameter set to CM according to the slow time scale period; CM calculates the SIR error according to the fast time scale period, updates the transmit power using incremental PID, and feeds back the error, power, and link interruption statistics to CH at the end of the slow time scale period.

4. The method for joint optimization of space-frequency energy of heterogeneous UAV clusters based on macro-micro dual loops according to claim 3, characterized in that, Future Time Window The link maintenance probability within is: ； in, To maintain the minimum receiving distance or power threshold required for communication, The mean of future distance / state predicted by EKF. The standard deviation derived from the position covariance matrix output by EKF. This is the tail probability function of the standard normal distribution.

5. The method for joint optimization of space-frequency energy of heterogeneous UAV clusters based on macro-micro dual-loop as described in claim 3, characterized in that, CM adjusts the transmit power based on the current measurement value. for: ； ； in, These are the proportional, integral, and derivative gains of the PID controller. This refers to the signal-to-interference-plus-noise ratio (SINR) error.

6. The method for joint optimization of space-frequency energy of heterogeneous UAV clusters based on macro-micro dual-loop as described in claim 1, characterized in that, CH-side execution of MAPPO-based joint intelligent decision-making includes: Step 1: CH constructs the observation vector based on its own state, neighbor states, user density, and CM feedback from the previous slow time scale period; Step 2: Input the observation vector into the CH-end Actor network, and output the joint action including flight trajectory adjustment, spectrum allocation and target signal-to-interference-plus-noise ratio through MAPPO; Step 3: CH maps the combined actions into flight control, spectrum allocation and microloop control commands, and CM performs link prediction, access or handover and PID power adjustment on a fast time scale; Step 4: The CM will feed back the execution statistics to the CH, which will then calculate the periodic reward and update the observation vector for the next slower timescale period.

7. The method for joint optimization of space-frequency energy of heterogeneous UAV clusters based on macro-micro dual-loop as described in claim 6, characterized in that, The fast and slow dual-timescale longitudinal coordination mechanism includes: the CH periodically sends the target signal-to-interference-plus-noise ratio (SIR) and interference temperature thresholds on a slow timescale, while the CM performs multiple fast timescale control steps within the period; when the CH predicts an increase in load in the target area, it reduces the target SIR of low-priority CMs in that area, causing them to reduce their transmit power, in order to release interference margin for high-priority links; after the period ends, the CM feeds back the SIR error, interruption probability, and energy consumption to the CH as observation information for the next period.

8. The method for joint optimization of space-frequency energy of heterogeneous UAV clusters based on macro-micro dual loops according to claim 6, characterized in that, The reward function includes: ； in, These are the weighting coefficients for each indicator; Total system capacity; For indicator functions; For task nodes transmission rate This is the minimum guaranteed rate threshold. This is a penalty for mechanical energy consumption.

9. The method for joint optimization of space-frequency energy of heterogeneous UAV clusters based on macro-micro dual-loop as described in claim 1, characterized in that, Joint training of distributed execution architectures based on dual-timescale closed-loop decision-making includes: Step 1: Construct a centralized training and distributed execution CTDE architecture, with the CH-end network as the Actor and the network that obtains the global state as the Critic; Step 2: Using a slow time scale period as the training step, aggregate the execution results of the CM fast loop into periodic rewards and store the training tuples; Step 3: Update the Actor and Critic using a loss function that includes a policy pruning term, a value error term, a policy entropy term, and a stationarity constraint term; Step 4: During the execution phase, deploy the Actor in CH and the IMU-assisted EKF, dual-threshold access or switching, and incremental PID in CM. Step 5: Training is complete when the loss function converges.

Images