A method and system for collaborative optimization of unmanned aerial vehicle cluster assisted hierarchical federated learning

Abstract

This invention discloses a hierarchical federated learning collaborative optimization method and system for UAV swarms, belonging to the fields of wireless communication, UAV swarm collaboration, and distributed machine learning. In this application, the ground base station performs global model aggregation based on the edge models uploaded by each UAV and broadcasts global model parameters to the UAVs. Each UAV acts as an edge server, simultaneously providing wireless energy transmission and edge model aggregation services to ground equipment. Each UAV has a built-in multi-head output strategy network, which synchronously outputs continuous displacement actions, discrete equipment scheduling actions, and continuous uplink time slot allocation actions based on local observation states, realizing distributed autonomous collaborative flight and scheduling of multiple UAVs. The ground equipment receives wireless energy and performs model training, uploading local model parameters to associated UAVs. This invention solves the two major problems of communication lag and limited equipment energy, and can improve equipment scheduling rate and ensure global model accuracy in multi-UAV collaborative scenarios.

Description

Technical Field

[0001] This invention belongs to the fields of wireless communication, UAV swarm collaboration and distributed machine learning technology, specifically relating to a UAV swarm-assisted hierarchical federated learning collaborative optimization method and system. Background Technology

[0002] Federated learning, a key paradigm and important branch of distributed machine learning, achieves multi-node collaborative modeling by training models locally on devices and uploading only the model parameters rather than the raw data, thus protecting data privacy. This technology has shown broad application prospects in scenarios such as IoT sensing, smart city management, and industrial internet monitoring. However, deploying federated learning in the complex environment of real wireless communication networks still faces two major challenges:

[0003] First, there is the communication bottleneck problem. The inherent time-varying and random fading characteristics of wireless channels cause some devices to require a long time to upload model parameters due to poor channel quality and high path loss. The parameter server has to wait for these lagging devices for a long time, which greatly increases the model aggregation latency.

[0004] Secondly, there is the issue of energy constraints. The multi-round global iterations of federated learning require devices to repeatedly perform local model training and wireless parameter uploading, both of which are energy-intensive operations. In scenarios involving battery-powered IoT terminals and embedded sensor nodes, device energy is easily depleted, leading to a gradual decrease in the number of devices participating in the iteration and a continuous deterioration in model convergence accuracy.

[0005] Drones, with their high mobility, flexible deployment, and inherent air-to-ground line-of-sight propagation advantages, can transform non-line-of-sight channels into line-of-sight channels, significantly improving communication link quality and effectively alleviating communication lag. Simultaneous data and energy transmission technology utilizes the physical characteristic that radio frequency signals can simultaneously carry information and energy, enabling network nodes to replenish energy while receiving data, providing a foundation for the sustainable operation of low-power devices. The hierarchical federated learning architecture, by adding an intermediate aggregation layer at the edge, reduces frequent communication with the central cloud, lowers system latency, and alleviates the load on the central server.

[0006] In existing technical solutions, there are basically no solutions that combine drones with data and energy transmission for application in federated learning scenarios. Therefore, how to deeply integrate drone swarms, data and energy transmission with hierarchical federated learning, especially for multi-drone collaborative scenarios, and achieve efficient joint optimization of drone collaborative flight trajectories, equipment scheduling, and uplink time slot allocation while considering the coupled constraints of collision avoidance constraints, equipment energy constraints, and scheduling conflicts among drones, is an important technical problem that urgently needs to be solved in this field. Summary of the Invention

[0007] To address the problems mentioned in the background section, this invention provides a method and system for collaborative optimization in hierarchical federated learning assisted by unmanned aerial vehicle (UAV) swarms, thereby solving the problems of communication lag and limited device energy in existing wireless federated learning.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A drone swarm-assisted hierarchical federated learning collaborative optimization method includes the following steps:

[0010] S1: Deploy a three-layered federated learning system consisting of one ground base station, J drones, and I ground devices. Configure the ground base station as the central server responsible for global model aggregation and distribution, configure each drone as an edge server responsible for edge model aggregation and wireless energy transmission, and configure each ground device as a local training node.

[0011] S2: Divide the total working time of the federated learning system into N time slots. At the beginning of each time slot, each UAV infers the current local observation state through the built-in policy network and synchronously outputs displacement actions, equipment scheduling actions and uplink time slot allocation actions.

[0012] S3: Each UAV updates its own flight position based on displacement actions, determines the scheduled ground equipment according to equipment scheduling actions, and plans the upload time slots for each ground equipment according to the uplink time slot allocation actions; within the current time slot, the UAV first transmits wireless power signals to the ground equipment within its coverage area; after receiving the wireless power replenishment, the scheduled ground equipment completes local model training using the local dataset, and uploads the updated local model parameters to the associated UAV within the allocated uplink time slot; after receiving the local model parameters uploaded by the scheduled equipment, the UAV performs edge aggregation on the received local model parameters to obtain the edge model, and finally broadcasts the edge model back to all ground equipment in the cluster;

[0013] S4: After completing a preset number of edge aggregations, each UAV uploads its current edge model to the ground base station; the ground base station performs global aggregation on the edge models uploaded by all UAVs to obtain a global model, and broadcasts the global model parameters to all UAVs; after receiving the global model parameters, each UAV uses them as the initial model for the next round of edge aggregation iteration and sends them to the corresponding ground equipment.

[0014] Preferably, the strategy network of each drone adopts a multi-head output architecture, simultaneously outputting three sets of action parameters and generating corresponding actions:

[0015] The first set consists of 4-dimensional displacement parameters, including the mean x-axis displacement, the logarithmic standard deviation x-axis displacement, the mean y-axis displacement, and the logarithmic standard deviation y-axis displacement. The policy network samples the two-dimensional displacement from the corresponding Gaussian distribution and uses the product of the maximum flight speed and the time slot duration as the maximum displacement threshold for cropping. If the magnitude of the sampled displacement exceeds the threshold, it is scaled proportionally to the threshold range to obtain the final displacement action.

[0016] The second group consists of I-dimensional equipment scheduling action parameters, corresponding to the scheduling tendency score of each ground device, used to generate equipment scheduling decisions;

[0017] The third group assigns weight parameters to I-dimensional time slots. The weights are normalized using Softmax to obtain the communication duration ratio of each device. Then, the ratio is multiplied by the total uplink communication duration available in the current time slot to obtain the exclusive uplink time slot length of each associated device. The total uplink communication duration is equal to the total time slot duration minus the fixed local training duration of the device.

[0018] Preferably, the training and execution of equipment scheduling action parameters are implemented in the following manner:

[0019] During the training phase, differentiable sampling is performed using Gumbel-Softmax. The policy network outputs a logit value for each device. Random noise following a Gumbel distribution is added to each logit value, which is then divided by the temperature parameter and subjected to a Softmax operation to obtain a continuous approximate device connection probability.

[0020] During the training process, the temperature parameter is gradually annealed from its initial value to near zero, so that the continuous approximation results gradually approach the 0-1 discrete decision.

[0021] During the execution phase, in conjunction with the Top-α strategy, no more than α devices are selected to establish connections based on their scheduling preference scores from high to low, where α is the maximum number of devices that a single drone can connect to.

[0022] Preferably, hierarchical federated learning employs a two-layer aggregation mechanism, and multiple drones collaborate using a multi-agent proximal policy optimization algorithm framework with centralized training and decentralized execution.

[0023] The two-layer aggregation mechanism is as follows: each ground device performs local model training using stochastic gradient descent. After each λ1 local model update, the UAV serving it performs an edge federated average aggregation on the local models of all devices under its jurisdiction. After each λ2 edge aggregation, the ground base station performs a global federated average aggregation on the edge models uploaded by each UAV.

[0024] The centralized training and decentralized execution framework is as follows: In the centralized training phase, a central trainer is configured to collect the local observation states of all UAVs, the joint actions consisting of displacement actions, equipment scheduling actions, and uplink time slot allocation actions output by each UAV, and the team-shared rewards fed back by the system. Based on the global state consisting of the local observation states of all UAVs, a centralized commentator network is trained to evaluate the value of the joint actions. Each UAV independently updates the parameters of its own policy network based on the advantage function output by the commentator network. In the decentralized execution phase, each UAV generates flight, scheduling, and time slot allocation decisions based solely on its own policy network and local observation states, without needing to obtain complete state information from other UAVs.

[0025] A drone swarm-assisted hierarchical federated learning collaborative optimization system includes a ground base station, J drones, and I ground equipment;

[0026] The ground base station acts as a central server, communicating with each UAV via a wireless channel. It is configured to receive edge model parameters uploaded by each UAV, perform global model aggregation calculations, and broadcast global model parameters to all UAVs.

[0027] Each UAV is equipped with a wireless power transmission module, a data communication module, and a built-in policy network. The data communication module establishes wireless communication connections with ground base stations and ground equipment. The policy network adopts a multi-head output architecture, configured to synchronously output displacement actions, equipment scheduling actions, and uplink time slot allocation actions based on local observation status. The UAV acts as an edge server and wireless power source, adjusting its flight position based on displacement actions, transmitting wireless power signals to ground equipment via the wireless power transmission module, receiving local model parameters uploaded by ground equipment via the data communication module and performing edge aggregation to obtain an edge model, as well as uploading edge model parameters to the ground base station and transmitting the edge model back to all ground equipment within the cluster.

[0028] Each ground device is equipped with a local dataset, a model training module, an energy receiving module, and a data communication module. It is wirelessly connected to the UAV via the data communication module. The model training module is used to perform local model training based on the local dataset. The energy receiving module receives the radio frequency energy emitted by the UAV and converts it into usable electrical energy. The data communication module also uploads local model parameters to the associated UAV.

[0029] Compared with the prior art, the beneficial effects of the present invention are:

[0030] 1. By deeply integrating drone swarms, data and energy transmission, and hierarchical federated learning, a three-layer collaborative optimization system is constructed, which simultaneously solves the dual bottleneck problems of communication lag and limited device energy in wireless federated learning.

[0031] 2. A multi-agent near-end strategy optimization algorithm is adopted to achieve joint optimization of multi-UAV collaborative trajectory planning, equipment scheduling and resource allocation. The collaborative and competitive relationships among UAVs are effectively handled through a centralized training and decentralized execution framework.

[0032] 3. A multi-head output strategy network architecture is designed for each UAV. A single network simultaneously outputs three types of heterogeneous actions: continuous displacement, discrete scheduling, and continuous time slot allocation. End-to-end differentiable training of discrete scheduling decisions is achieved through Gumbel-Softmax, resulting in a compact and efficient structure. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the architecture of the UAV swarm-assisted hierarchical federated learning system of the present invention;

[0034] Figure 2 This is a schematic diagram of the working stages within a single time slot of the present invention;

[0035] Figure 3 This is a comparative diagram of traditional federated learning and hierarchical federated learning in this invention;

[0036] Figure 4 This is a framework diagram of the multi-UAV trajectory, scheduling, and resource joint optimization algorithm based on the multi-agent near-end policy optimization algorithm of this invention. Detailed Implementation

[0037] To facilitate understanding of the technical content of this invention by those skilled in the art, the invention will be further described in detail below with reference to the accompanying drawings and specific examples. It should be understood that the specific examples described herein are merely illustrative and not intended to limit the scope of the invention.

[0038] A hierarchical federated learning collaborative optimization system assisted by drone swarms, the system architecture of which is as follows: Figure 1 As shown, the system proposed in this invention consists of a three-layered federated learning architecture, with one ground base station as the central server, J drones as edge servers, and I ground devices.

[0039] The ground base station is responsible for global model aggregation, broadcasting global model parameters to all UAVs, and receiving edge model parameters uploaded by each UAV. A global aggregation module runs within the ground base station, which performs federated averaging after collecting the edge model parameters from all J UAVs.

[0040] Each drone is equipped with a wireless power transmission module and a data communication module. It can act as a mobile edge server to perform edge aggregation of local models of devices within its service range, and as a wireless power source to transmit radio frequency power signals to devices. Each drone has a built-in policy network, which adopts a multi-head output architecture and is configured to simultaneously output three types of actions based on the local observation status.

[0041] Each ground device includes: a local dataset, such as the MNIST handwritten digit dataset; a model training module for performing stochastic gradient descent updates using local data and current model parameters; an energy receiving module for receiving radio frequency energy signals emitted by the UAV and converting them into usable DC power; and a data communication module for uploading local model parameters to the associated UAV via a wireless channel within allocated uplink time slots.

[0042] The system employs a two-layer aggregation hierarchical federated learning mechanism to replace the traditional direct cloud aggregation method, such as... Figure 3 As shown, in traditional federated learning, after each round of local training, the data is directly aggregated with the cloud. Frequent centralized aggregation significantly increases communication latency and the load on the central server. In contrast, in hierarchical federated learning, after each device completes λ1 local gradient descent updates, edge aggregation is performed by an edge server (i.e., a drone). After λ2 edge aggregations, global aggregation is then performed by a central server (i.e., a ground base station). Edge aggregation has lower propagation latency and higher communication efficiency compared to cloud aggregation, and it also avoids the local model from becoming overly biased towards the optimal solution of the local loss function, thus deviating from the global optimum.

[0043] The local model, marginal model, and global model in this application are lightweight classification models with completely consistent structures. They are softmax regression models without hidden layers used for the MNIST handwritten digit ten classes, that is, a single-layer fully connected linear classifier with 784-dimensional input and 10-dimensional output.

[0044] Specifically, in this embodiment, the relevant hardware environment configuration of the system is as follows:

[0045] One ground base station is deployed at the coordinate origin. A total of three drones are involved, flying at a fixed altitude of H=20m and with a maximum speed of V. max =20m / s, the drone takes off from the ground base station. Each drone is equipped with a wireless power transmission module (transmit power 10W) ​​and a data communication module (operating frequency band 2MHz bandwidth). There are 20 ground devices, randomly distributed within a square area of ​​[-100,100]m × [-100,100]m centered on the ground base station. Each device has an initial energy of 0.01J or 0.005J, and the devices are equipped with energy receiving modules to convert radio frequency energy into DC power. The device CPU frequency is 1GHz, and the CPU cycles required to process a unit of data are 20 cycles / bit.

[0046] The system's relevant software and algorithm modules are configured as follows:

[0047] The federated learning task is based on the MNIST handwritten digit dataset. Each device locally holds 60,000 training samples. The model parameter vector has a dimension of 28×28×10 (corresponding to 28×28 image input and 10 classification outputs). Each parameter is stored in 16-bit floating-point format, and the model parameter data size is 28×28×10×16 bits. The learning rate is set to 0.01. The hierarchical federated learning is configured with λ1 = 2 local stochastic gradient descent updates followed by one edge aggregation, and λ2 = 2 edge aggregations followed by one global aggregation.

[0048] Regarding communication parameters, the channel gain at a reference distance d0=1m is -50dB, the path loss factor γ=2, and the Gaussian noise power spectral density per unit bandwidth is -174dBm / Hz.

[0049] A drone swarm-assisted hierarchical federated learning collaborative optimization method includes the following steps:

[0050] A1. System initialization; such as Figure 1 As shown, a ground base station is deployed at the origin position. Three drones take off from the base station and hover at a height of H=20m. Twenty ground devices are randomly distributed within a designated area. The local datasets, initial global model parameters (which can be randomly initialized or pre-trained model parameters), and data transmission parameters for each device are initialized. The time slot counter n=0 corresponding to the total system operating time T, and the global iteration counter n1=0.

[0051] A2. Multi-UAV cooperative trajectory planning and scheduling decision-making; such as Figure 4 As shown, at the beginning of each time slot, each UAV collects local observation data (its current position, the positions of other UAVs, relative distances to various ground devices, the current remaining energy of each device, and the cumulative number of times each device has been scheduled), and inputs this data into the built-in multi-head policy network. The policy network synchronously outputs the mean and logarithmic standard deviation of displacements, the device scheduling logit vector, and the time slot allocation weight vector. Displacements in the x and y axes are sampled from a Gaussian distribution and pruned by maximum velocity constraints to update the UAV position. During the execution phase, the Top-α strategy (α being the preset maximum number of devices each UAV can connect to) is used in conjunction with the scheduling logit vector to select devices for connection from high to low. The time slot allocation weights are normalized using Softmax and multiplied by the available communication time to obtain the dedicated uplink time slot for each scheduled device. This step enables multiple UAVs to achieve distributed cooperative flight and scheduling based solely on local observations without real-time coordination by a centralized controller.

[0052] A3. Wireless Power Transfer and Device Energy Renewal: During the wireless power transfer phase of the current time slot, each UAV simultaneously broadcasts a radio frequency (RF) energy signal to all devices within the area at a fixed transmit power of 10W. The devices receive RF energy from all three UAVs via their energy receiving modules, rectify it into DC power, and then renew their local remaining energy. The duration of the power transfer is the total time slot duration minus the sum of the device's local training time and the uplink communication time of all scheduled devices.

[0053] A4. Local model training on the device; scheduled device (scheduling variable a) ij Devices with [n]=1 perform stochastic gradient descent updates during the local training phase, utilizing their respective local MNIST datasets and currently held local model parameters. Unscheduled devices (a ij Devices with [n]=0 do not perform training operations in this time slot.

[0054] A5. Local Model Parameter Upload and Edge Model Aggregation: Scheduled devices, according to their allocated dedicated uplink time slots, sequentially upload updated local model parameters to their associated drones via wireless channels. After receiving the local model parameters uploaded by all scheduled devices, the drone performs a federated average aggregation operation to generate the edge model parameters for the current edge aggregation round, and then broadcasts the edge model to all devices within the cluster via a broadcast channel.

[0055] A6. Edge Aggregation and Global Aggregation Iteration: The system counts the number of edge aggregations completed so far. If the number of edge aggregations has not yet reached λ², return to step A2 and continue to the next time slot. Each device uses the newly received edge model as the initial parameters for the next round of training. If the number of edge aggregations has reached λ², each UAV uploads its own edge model parameters to the ground base station. After receiving the edge models from all three UAVs, the ground base station performs global federated average aggregation to generate global model parameters and broadcasts them to all UAVs via the downlink channel. The global iteration counter n1 is incremented by 1.

[0056] A7. Convergence Judgment and Iteration Termination: After each round of global aggregation, check whether the loss change or gradient norm of the global model on the validation set meets the preset convergence condition, or whether the number of global iterations n1 has reached the preset maximum number of rounds. If the convergence condition is met or the maximum number of rounds is reached, terminate training and output the final global model, the flight trajectories of each UAV, the equipment scheduling records of each time slot, and the uplink time slot allocation records; otherwise, return to step A2 and continue to the next round of global iteration.

[0057] 1. In the method of this application, the time slot structure and workflow are as follows:

[0058] like Figure 2As shown, the total system operating time is T, divided into N time slots, with the fixed duration of the nth time slot denoted as δ[n]. A Cartesian coordinate system is established with the ground base station as the origin, and the UAV flies at an altitude H, which is typically set to around 20m.

[0059] The work within each time slot δ[n] is divided into the following five stages:

[0060] (a) Wireless Power Transfer Phase. Each UAV simultaneously broadcasts a wireless power signal to the device at a fixed transmit power. Since ground equipment cannot simultaneously communicate and receive power, the wireless power transfer is scheduled after all scheduled equipment has completed its uplink communication and before the start of the next time slot. The device simultaneously receives wireless radio frequency power from all J UAVs and converts it into usable DC power through the power receiving module.

[0061] For example, if the total time slot duration is set to δ[n] = 0.1s, the device's local training consumes 0.02s, and the cumulative uplink communication time allocated to each scheduled device is within 0.05s, then approximately 0.03s remains for wireless power transmission. Assuming the UAV's transmit power is 10W and the channel gain is -50dB, the device can receive power replenishment in the order of millijoules.

[0062] (b) Local Model Training Phase. The scheduled device performs stochastic gradient descent training using the local dataset and the currently held model parameters. (The last sentence appears to be incomplete and requires further context.) i For example, its local dataset size is D. i One sample, the device uses a fixed CPU frequency f u Perform c on each sample u The computation takes one CPU cycle, and the training time is... The energy consumed during training is proportional to the square of the CPU frequency.

[0063] (c) Local Model Parameter Upload Phase. The scheduled device uploads its updated local model parameters to the associated UAV via a wireless channel within its dedicated uplink time slot allocated by the policy network. The upload rate depends on the device's transmit power, channel gain, and communication bandwidth. For example, with a communication bandwidth of 2MHz, the device transmits at lower power, resulting in a higher upload rate and lower energy consumption when the device and UAV are close together.

[0064] (d) Edge Model Aggregation Stage. The UAV uses a federated averaging algorithm to perform weighted averaging of the local model parameters received from each scheduled device. Specifically, for UAV v j The set of devices serving a location, the edge model is equal to the weighted sum of the local model parameters of each device according to the scheduling coefficient, and then divided by the total number of scheduled devices.

[0065] (e) Edge Model Backhaul Phase. The UAV distributes the aggregated edge model to all devices in the cluster via a broadcast channel. Each device uses this edge model as the initial model parameters for the next round of local training.

[0066] After completing λ² edge aggregation, each UAV uploads its latest edge model parameters to the ground base station in a designated time slot. Upon receiving the edge models from all J UAVs, the ground base station performs global federated average aggregation to obtain a global model, and then broadcasts the global model parameters to all UAVs via the downlink channel, completing one full global iteration.

[0067] 2. In the method of this application, the multi-head policy network and action generation are specifically as follows:

[0068] State space;

[0069] Since the multi-agent proximal policy optimization algorithm adopts a paradigm of centralized training and decentralized execution, the state space is divided into two categories: agent local observation state and centralized commentator global state.

[0070] Drone intelligent agent v j The local observation state o observed in time slot δ[n] j [n] is defined as:

[0071] ;

[0072] in, The three-dimensional position coordinates of the drone VJ in the current time slot. Let be the set of relative position vectors between the drone and the UI of various ground devices. This represents the current remaining energy of the UI devices in each location. This represents the cumulative number of times each ground device UI has been scheduled up to the current time slot. The global state s[n] of the centralized commentator network is defined as the set of local observation states of all J UAVs, i.e.:

[0073] ;

[0074] Each drone v j It has a built-in policy network that adopts a multi-head output architecture. In a single forward propagation, it outputs three sets of action parameters, which correspond to flight displacement decision, equipment scheduling decision and uplink time slot allocation decision, respectively.

[0075] In the multi-agent proximal policy optimization algorithm, both the actor network and the centralized critic network for each drone employ a two-layer hidden layer structure, with 128 neurons in each layer. The initial temperature of Gumbel-Softmax is set to 1.0, and gradually annealed to 0.01 during training. The generalized advantage estimation parameter λ=0.95, and the PPO truncation parameter ε=0.2. The experience replay buffer size is set to 2048 experience trajectories. The total number of training epochs is K=10000, with each epoch containing N=300 time slots, each time slot lasting 0.1s.

[0076] Mixed motion space;

[0077] Each drone agent outputs a coordinated action in each time slot. This joint action consists of three types of heterogeneous actions, which are uniformly represented as:

[0078] ;

[0079] Displacement motion generation;

[0080] The policy network outputs a 4-dimensional displacement head vector, containing the mean displacement μ along the x-axis. x y-axis displacement mean μ y x-axis logarithmic standard deviation logσ x and the logarithm of the y-axis (logσ) y The two-dimensional displacement Δx is obtained by sampling from the corresponding Gaussian distribution. j [n] and Δy j [n].

[0081] ;

[0082] After sampling, the displacement is clipped by a maximum velocity constraint: the magnitude of the displacement vector is calculated, and if the magnitude exceeds V... max ·δ[n](V max If the maximum flight speed is 20 m / s, then the displacement is scaled proportionally to the maximum allowable range. The clipped displacement is used to update the drone's position, resulting in new flight coordinates:

[0083] ;

[0084] Equipment scheduling action generation;

[0085] The policy network outputs an I-dimensional scheduling head logit vector z. j [n], where each element z ij [n] corresponds to the original score of the scheduling preference for the i-th device.

[0086] During the training phase, the Gumbel-Softmax technique is employed to achieve differentiable relaxation for discrete scheduling decisions. Specifically, random noise g sampled from the Gumbel(0,1) distribution is added to each logit value. i Then, divide by the temperature parameter ι, and perform a Softmax operation to obtain the continuous approximate connection probability of each device:

[0087] ;

[0088] In the early stages of training, the temperature parameter ι is set relatively large (e.g., ι=1.0), resulting in a smoother connection probability distribution, which is beneficial for policy exploration. As training progresses, the temperature parameter is gradually annealed and reduced (e.g., reduced to ι=0.01), allowing the continuous approximation to gradually approach the true 0-1 discrete decision.

[0089] During the execution phase, Gumbel-Softmax is no longer used; instead, a Top-α strategy is directly employed for selection. The drone selects the top α devices for connection based on the values ​​of the elements in its logit vector, sorting them from highest to lowest. This means that an α-value is assigned to these α devices. ij [n]=1, set a for the remaining devices ij [n]=0. Here, α is the maximum number of devices that a single drone can connect to simultaneously. In actual execution, fewer than α devices can be connected depending on the scenario requirements.

[0090] To prevent multiple drones from simultaneously selecting the same device (scheduling conflict), a scheduling conflict penalty is introduced into the reward function during the training phase to guide policy learning and avoid conflict.

[0091] Uplink time slot allocation action generation;

[0092] The policy network outputs an I-dimensional time slot allocation head weight vector w. j [n]. To meet the total time slot duration constraint, the weight vector is first normalized using Softmax to obtain the communication time ratio of each device. Then, it is multiplied by the total time available for uplink communication within the current time slot to obtain the actual uplink time slot length allocated to each device. The total available communication time is equal to the total time slot duration δ[n] minus the fixed time τ consumed by the device's local training. comp [n]. Through the above operations, it can be ensured that the sum of the uplink time slots allocated to all scheduled devices does not exceed the total available time, specifically:

[0093] ;

[0094] The reward function is designed to closely align with the system optimization goals, while incorporating various constraints as penalty terms.

[0095] The basic reward for each drone agent in time slot δ[n] is defined as a trade-off between equipment scheduling reward and equipment energy consumption:

[0096] ;

[0097] Team rewards are shared and equal to the sum of all drone base rewards:

[0098] ;

[0099] In addition to the basic rewards, four types of penalties are introduced for violating constraints:

[0100] Collision penalty: When any two drones v j With v k Spatial distance between ||q j [n]-q k When [n]||2 is less than the preset safety distance Δ, a negative constant penalty -C is applied to each of the two drones. c ;

[0101] Negative energy penalty: When device u i Current remaining energy E i [n] is insufficient to cover the energy consumption required to complete its training. With communication energy consumption When the sum is still scheduled, a negative penalty -C is imposed on the drone connected to the device. e ;

[0102] Illegal Area Penalty: When a drone v j Position q j [n] When the drone exceeds the prescribed legal flight area, a negative penalty is imposed on the drone -C b ;

[0103] Scheduling conflict penalty: When multiple drones target the same device in the same time slot... i Set a ij When [n]=1, apply a negative penalty of -C to all drones participating in the scheduling. s .

[0104] The final instant reward is the team's base reward minus the sum of all penalties. The environment updates the policy network parameters based on this signal, guiding each drone agent to learn safe, efficient, and compliant collaborative strategies.

[0105] 3. In the method of this application, the multi-UAV collaborative training mechanism is specifically as follows:

[0106] The J UAVs of this invention achieve efficient collaboration through a multi-agent proximal policy optimization algorithm framework involving centralized training and distributed execution. The overall framework is as follows: Figure 4 As shown.

[0107] During the intensive training phase, a central trainer is deployed in the system. In each time slot, the central trainer collects the local observation states of all J UAVs, the actions output by each UAV, and the reward signals fed back by the system. The central trainer maintains a centralized critic network, which takes the global state as input (defined as the concatenated set of local observation states of all UAVs) and outputs a value estimate of the current global state. Through the global state, the critic network can accurately evaluate the value of joint actions, effectively mitigating the non-stationarity problem caused by changes in the policies of other agents in a multi-agent environment.

[0108] Each drone's policy network updates its parameters independently based on the advantage function fed back by the critic network. The advantage function is calculated from the team reward signal using a generalized advantage estimation method, reflecting the superiority or inferiority of the current action relative to the average level. The policy network employs a truncated objective function of proximal policy optimization to constrain the update magnitude, preventing drastic policy changes from causing training instability.

[0109] During the distributed execution phase, the central trainer is deactivated, and each UAV relies solely on its own onboard policy network for decision-making. In each time slot, each UAV generates displacement actions, scheduling actions, and time slot allocation actions synchronously through the multi-head output architecture of the policy network based on its local observation status (including its own position, the positions of other UAVs, its relative position with each device, the remaining energy of each device, and the cumulative number of scheduling attempts) obtained by its own sensors and communication modules. This achieves fully distributed autonomous cooperative flight and scheduling without the need for real-time coordination by a central controller or explicit communication with other UAVs.

[0110] The team reward is designed as the sum of all drones' basic rewards minus penalties for violating various constraints. The basic rewards reflect the optimization goal: encouraging more devices to be scheduled for federated learning while keeping device energy consumption at a low level. Penalties include four categories: collision penalty (a negative reward is applied to drones involved in a collision when the spatial distance between any two drones is less than a preset safe distance Δ (e.g., Δ=5m); negative energy penalty (a penalty is applied when a device's remaining energy is insufficient to support the current task but it is still scheduled); illegal area penalty (a penalty is applied when a drone flies out of a preset legal flight area (e.g., a range of [-100,100]m × [-100,100]m); and scheduling conflict penalty (a penalty is applied when multiple drones connect to the same device in the same time slot).

[0111] This application has the following advantages:

[0112] 1. Multi-technology deep integration architecture: For the first time, it deeply integrates three technologies: drone swarm collaboration, data and energy transmission wireless energy replenishment, and hierarchical federated learning, to build a three-layer collaborative optimization system of "ground base station-drone swarm-ground equipment", and solves the dual bottleneck problems of communication lag and equipment energy limitation in wireless federated learning.

[0113] 2. Multi-agent collaborative decision-making mechanism: The multi-agent near-end policy optimization algorithm is adopted. Through centralized training and decentralized execution framework, multiple UAVs can achieve distributed autonomous collaborative flight and scheduling with only local observation, effectively handling the cooperation and competition relationship between UAVs.

[0114] 3. Integrated output of hybrid action space: A multi-head strategy network architecture is designed for each UAV. A single network synchronously outputs three types of heterogeneous actions: continuous displacement, discrete scheduling, and continuous time slot allocation. The structure is compact and efficient, and end-to-end differentiable training of discrete scheduling decisions is achieved through the Gumbel-Softmax technique.

[0115] 4. Multi-layer aggregation reduces communication overhead: A two-layer aggregation mechanism of edge aggregation and global aggregation is adopted. Frequent aggregation at the edge reduces the number of communications with the central end, while periodic global aggregation at the central end ensures the global convergence of the model, balancing communication efficiency and model accuracy.

[0116] 5. Constraint-aware safety strategy learning: Unmanned aerial vehicle (UAV) collision avoidance constraints, device energy causal constraints, flight area constraints, and scheduling conflict constraints are incorporated into the reward function design as penalty terms. This drives the agent to autonomously learn safe and compliant collaborative strategies while optimizing performance, ensuring the feasibility and security of the system in actual deployment.

[0117] This invention deeply integrates drone swarms, data and energy transmission, and hierarchical federated learning. It leverages a multi-agent near-end policy optimization algorithm to achieve joint optimization of multi-drone collaborative trajectories, equipment scheduling, and resource allocation. This effectively overcomes the dual bottlenecks of limited device energy and communication lag in wireless federated learning, and is suitable for low-power wireless network scenarios such as the Internet of Things and the Industrial Internet that require distributed collaborative model training.

[0118] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A hierarchical federated learning collaborative optimization method assisted by unmanned aerial vehicle (UAV) swarms, characterized in that, Includes the following steps: S1: Deploy a three-layered federated learning system consisting of one ground base station, J drones, and I ground devices. Configure the ground base station as the central server responsible for global model aggregation and distribution, configure each drone as an edge server responsible for edge model aggregation and wireless energy transmission, and configure each ground device as a local training node. S2: Divide the total working time of the federated learning system into N time slots. At the beginning of each time slot, each UAV infers the current local observation state through the built-in policy network and synchronously outputs displacement actions, equipment scheduling actions and uplink time slot allocation actions. S3: Each UAV updates its own flight position based on displacement actions, determines the scheduled ground equipment according to equipment scheduling actions, and plans the upload time slots for each ground equipment according to the uplink time slot allocation actions; within the current time slot, the UAV first transmits wireless power signals to the ground equipment within its coverage area; after receiving the wireless power replenishment, the scheduled ground equipment completes local model training using the local dataset, and uploads the updated local model parameters to the associated UAV within the allocated uplink time slot; after receiving the local model parameters uploaded by the scheduled equipment, the UAV performs edge aggregation on the received local model parameters to obtain the edge model, and finally broadcasts the edge model back to all ground equipment in the cluster; S4: After completing a preset number of edge aggregations, each UAV uploads its current edge model to the ground base station; the ground base station performs global aggregation on the edge models uploaded by all UAVs to obtain a global model, and broadcasts the global model parameters to all UAVs; after receiving the global model parameters, each UAV uses them as the initial model for the next round of edge aggregation iteration and sends them to the corresponding ground equipment.

2. The UAV swarm-assisted hierarchical federated learning collaborative optimization method according to claim 1, characterized in that, Each drone's policy network adopts a multi-head output architecture, simultaneously outputting three sets of action parameters and generating corresponding actions: The first set consists of 4-dimensional displacement parameters, including the mean x-axis displacement, the logarithmic standard deviation x-axis displacement, the mean y-axis displacement, and the logarithmic standard deviation y-axis displacement. The policy network samples the two-dimensional displacement from the corresponding Gaussian distribution and uses the product of the maximum flight speed and the time slot duration as the maximum displacement threshold for cropping. If the magnitude of the sampled displacement exceeds the threshold, it is scaled proportionally to the threshold range to obtain the final displacement action. The second group consists of I-dimensional equipment scheduling action parameters, corresponding to the scheduling tendency score of each ground device, used to generate equipment scheduling decisions; The third group assigns weight parameters to I-dimensional time slots. The weights are normalized using Softmax to obtain the communication duration ratio of each device. Then, the ratio is multiplied by the total uplink communication duration available in the current time slot to obtain the exclusive uplink time slot length of each associated device. The total uplink communication duration is equal to the total time slot duration minus the fixed local training duration of the device.

3. The UAV swarm-assisted hierarchical federated learning collaborative optimization method according to claim 2, characterized in that, The training and execution of equipment scheduling action parameters are implemented in the following manner: During the training phase, differentiable sampling is performed using Gumbel-Softmax. The policy network outputs a logit value for each device. Random noise following a Gumbel distribution is added to each logit value, which is then divided by the temperature parameter and subjected to a Softmax operation to obtain a continuous approximate device connection probability. During the training process, the temperature parameter is gradually annealed from its initial value to near zero, so that the continuous approximation results gradually approach the 0-1 discrete decision. During the execution phase, in conjunction with the Top-α strategy, no more than α devices are selected to establish connections based on their scheduling preference scores from high to low, where α is the maximum number of devices that a single drone can connect to.

4. The UAV swarm-assisted hierarchical federated learning collaborative optimization method according to claim 1, characterized in that, In S3, the workflow within each time slot is divided into five stages in sequence: Wireless power transfer phase: Each UAV broadcasts a wireless power signal to the coverage area at a fixed transmission power. Ground equipment simultaneously receives radio frequency power from all UAVs within the coverage area and converts it into usable electrical energy. The energy transmission duration is equal to the total time slot duration minus the sum of the device's local training fixed duration and the cumulative uplink communication duration of all scheduled devices; Local model training phase: The scheduled ground equipment performs stochastic gradient descent training using the local dataset and current model parameters, and updates the local model parameters; Local model upload phase: The scheduled ground equipment uploads the updated local model parameters to the associated UAV via wireless channel within the allocated dedicated uplink time slot; Edge model aggregation stage: The UAV uses a federated averaging algorithm to perform a weighted average of the local model parameters uploaded by each scheduled device to generate an edge model; Edge model backhaul phase: The UAV distributes the aggregated edge model to all ground devices in the cluster via the broadcast channel as the initial model parameters for the next round of local training of the devices.

5. The UAV swarm-assisted hierarchical federated learning collaborative optimization method according to claim 1, characterized in that, Hierarchical federated learning employs a two-layer aggregation mechanism, and multiple drones collaborate using a multi-agent proximal policy optimization algorithm framework with centralized training and decentralized execution. The two-layer aggregation mechanism is as follows: each ground device performs local model training using stochastic gradient descent. After each λ1 local model update, the UAV serving it performs an edge federated average aggregation on the local models of all devices under its jurisdiction. After each λ2 edge aggregation, the ground base station performs a global federated average aggregation on the edge models uploaded by each UAV. The centralized training and decentralized execution framework is as follows: During the centralized training phase, a central trainer is configured. The central trainer collects the local observation states of all UAVs, the joint actions consisting of displacement actions, equipment scheduling actions, and uplink time slot allocation actions output by each UAV, as well as the team-shared rewards fed back by the system. Based on the global state consisting of the local observation states of all UAVs, a centralized commentator network is trained to evaluate the value of the joint actions. Each UAV independently updates the parameters of its own policy network based on the advantage function output by the commentator network. During the distributed execution phase, each UAV generates flight, scheduling, and time slot allocation decisions solely based on its own policy network and local observation status, without needing to acquire complete status information from other UAVs.

6. The UAV swarm-assisted hierarchical federated learning collaborative optimization method according to claim 5, characterized in that, The team-shared reward equals the sum of the base rewards of all drones minus the total value of constraint violation penalties. The base reward of a single drone is weighted by a positive incentive term related to the number of scheduled devices and a trade-off term related to device energy consumption, used to control device energy consumption levels while increasing device scheduling participation. The constraint violation penalties include collision penalties, negative energy penalties, illegal area penalties, and scheduling conflict penalties: a collision penalty is applied when the distance between any two drones is less than a preset safe distance; a negative energy penalty is applied when the remaining energy of a device is insufficient to complete training and communication tasks but it is still scheduled; an illegal area penalty is applied when a drone flies out of a preset legal flight area; and a scheduling conflict penalty is applied when multiple drones simultaneously schedule the same device.

7. A drone swarm-assisted hierarchical federated learning collaborative optimization system, used to implement the method described in any one of claims 1-6, characterized in that, It includes one ground base station, J drones, and I ground equipment; The ground base station acts as a central server, communicating with each UAV via a wireless channel. It is configured to receive edge model parameters uploaded by each UAV, perform global model aggregation calculations, and broadcast global model parameters to all UAVs. Each UAV is equipped with a wireless power transmission module, a data communication module, and a built-in policy network. The data communication module establishes wireless communication connections with ground base stations and ground equipment. The policy network adopts a multi-head output architecture, configured to synchronously output displacement actions, equipment scheduling actions, and uplink time slot allocation actions based on local observation status. The UAV acts as an edge server and wireless power source, adjusting its flight position based on displacement actions, transmitting wireless power signals to ground equipment via the wireless power transmission module, receiving local model parameters uploaded by ground equipment via the data communication module and performing edge aggregation to obtain an edge model, as well as uploading edge model parameters to the ground base station and transmitting the edge model back to all ground equipment within the cluster. Each ground device is equipped with a local dataset, a model training module, an energy receiving module, and a data communication module. It is wirelessly connected to the UAV via the data communication module. The model training module is used to perform local model training based on the local dataset. The energy receiving module receives the radio frequency energy emitted by the UAV and converts it into usable electrical energy. The data communication module also uploads local model parameters to the associated UAV.

8. The UAV swarm-assisted hierarchical federated learning collaborative optimization system according to claim 7, characterized in that, The multi-head output architecture of each UAV's policy network includes a displacement output head, a scheduling output head, and a time slot allocation output head. The displacement output head outputs 4-dimensional displacement action parameters, including the mean and logarithmic standard deviation of displacement in the x and y directions, which are used to sample 2-dimensional displacement quantities from a Gaussian distribution and prune them according to the maximum flight speed constraint to generate displacement actions. The scheduling output head outputs 1-dimensional scheduling logit parameters. During the training phase, continuous connection probabilities are generated through Gumbel-Softmax. During the execution phase, the Top-α strategy is combined to select no more than α devices to generate device scheduling actions. The time slot allocation output head outputs 1-dimensional time slot allocation weights, which are mapped to the exclusive uplink time slot length of each associated device after Softmax normalization.

9. The UAV swarm-assisted hierarchical federated learning collaborative optimization system according to claim 7, characterized in that, The local observation states input to the policy network include: the current position of the UAV itself, the current positions of other UAVs, the relative positions of the UAV and ground equipment in various locations, the current remaining energy of each ground equipment, and the cumulative number of times each equipment has been scheduled.

10. The UAV swarm-assisted hierarchical federated learning collaborative optimization system according to claim 7, characterized in that, The system implements a two-layer aggregation mechanism: the UAV acts as the execution subject of edge aggregation, and performs an edge federated average aggregation once after each λ1 local model update of the device under its jurisdiction; the ground base station acts as the execution subject of global aggregation, and performs a global federated average aggregation once for the edge models uploaded by all UAVs after each λ2 edge aggregation.

Images