An information age-sensitive federated reinforcement learning method for drone swarms

Abstract

The present invention discloses an information age-sensitive federated reinforcement learning method for a swarm of unmanned aerial vehicles (UAVs), comprising the following steps: step 1, determining the number and initial position information, the maximum number of execution rounds, and the cycle time of UAVs, ground sensors, and base stations; step 2, performing clustering; after all UAVs in the same cluster complete local training, the UAVs in the cluster send a federated learning local model to their neighbors through a pruned D2D network to obtain an intra-cluster federated learning aggregation model; the cluster head UAV sends the intra-cluster federated learning aggregation model and information of all UAVs in the cluster to the base station; step 3, the base station obtains an out-of-cluster federated learning aggregation model, calculates a path planning strategy, and sends the model to the corresponding cluster head UAV; step 4, the cluster head UAV sends information to other UAVs in the cluster; the UAVs in the cluster execute the path planning strategy to collect data; if the current round does not reach the set maximum number of execution rounds, the method returns to step 2: if the set maximum number of execution rounds is reached, the method terminates.

Description

Technical Field

[0001] The present invention belongs to the technical field of cross-integration of artificial intelligence and machine learning, drone communication technology, cloud computing and edge computing, and particularly relates to a federated reinforcement learning method for drone swarms that is sensitive to information age. Background Art

[0002] In recent years, drones have played an increasingly important role in wireless communication networks and the Internet of Things (IoT). In wireless communication networks, drones offer the advantages of high flexibility and maneuverability. They can dynamically adjust their flight paths based on actual needs, enabling rapid deployment to areas requiring coverage. This eliminates the geographical constraints of ground-based base stations, which often hinder their deployment. In the IoT, drones, acting as flying base stations, can support the connection and data transmission of IoT devices, providing stable communication services. In many fields, drones can serve as relay nodes in IoT networks, enabling real-time data collection and transmission. Since IoT data is often real-time, IoT systems must be able to process and respond quickly and efficiently. Furthermore, IoT data often contains sensitive industry data, requiring IoT systems to process and analyze it securely and reliably. Summary of the Invention

[0003] The purpose of the present invention is to provide an information age-sensitive UAV swarm federated reinforcement learning method, which can improve the real-time performance of the system, reduce system overhead, and make the system efficient and reliable.

[0004] The technical solution provided by the present invention is:

[0005] An information age-sensitive federated reinforcement learning method for drone swarms, including:

[0006] Step 1: Determine the number and initial location information of drones, ground sensors, and base stations; determine the maximum number of execution rounds and the cycle time of each execution round; the base station initializes and distributes the global model parameters of federated learning;

[0007] Step 2: All the drones are clustered according to their geographic location and local training time; the drones receive the parameters of the federated learning global model or the federated learning aggregation model outside the cluster and perform local training to obtain a federated learning local model; after all drones in the same cluster complete local training, a D2D network is established within the cluster, and a cluster head drone is selected. The drones in the cluster send the federated learning local model to their neighbors through the pruned D2D network, and an intra-cluster federated learning aggregation model is obtained based on the intra-cluster federated learning aggregation weight; the cluster head drone sends the intra-cluster federated learning aggregation model and information about all drones in the cluster to the base station;

[0008] Step 3: After receiving the intra-cluster federated learning aggregation model and information about all drones in any cluster, the base station directly obtains the extra-cluster federated learning aggregation model based on the extra-cluster aggregation weights without waiting for other drone clusters, and calculates the corresponding path planning strategies for all drones in the cluster; the base station sends the extra-cluster federated learning aggregation model and the path planning strategy to the corresponding cluster head drone;

[0009] Step 4: The cluster head UAV sends information to other UAVs in the cluster; the UAVs in the cluster execute the path planning strategy to collect data from the ground sensors during the remaining execution time of this round;

[0010] If the current round does not reach the set maximum number of execution rounds, return to step 2 to continue executing the next round; if the current round reaches the set maximum number of execution rounds, terminate.

[0011] Preferably, each execution round includes a training phase and a data collection phase.

[0012] Preferably, the clustering process is:

[0013] Step 1, in the Before the start of an execution cycle, the distance between all the drones is:

[0014]

[0015] in, For drones With drones the distance between them; For the The execution cycle starts at the moment when the drone coordinates of For the The execution cycle starts at the moment when the drone coordinates of

[0016] Based on the principle of ensuring that any two UAVs in the same cluster are within the maximum D2D communication distance, the hierarchical clustering method is used for the first clustering;

[0017] Step 2: All the drones are in the The local training time for each execution cycle is:

[0018]

[0019] in, For drones In the Local training time per execution cycle; For drones In the The number of local iterations per execution cycle; For drones In the The number of data samples collected during the execution cycle, For drones CPU cycles to process one data sample; For drones CPU frequency for processing one data sample;

[0020] Based on the principle of ensuring that the difference in training time required by any two UAVs in the same cluster during this execution cycle is within the maximum time range, the kernel density estimation method is used for the second clustering.

[0021] Preferably, the optimization goal of the local training process for all the drones is:

[0022]

[0023] in, are the model parameters of the federated learning global model; For drone swarms In the The out-of-cluster federated learning aggregation model obtained through execution cycles; is the total number of samples; is the loss function when the drone is used as a client; is the proximal term, is a regularization parameter used to control the impact of global model parameter deviation on the local objective function.

[0024] Preferably, the local training process is:

[0025] ;

[0026] ;

[0027] ;

[0028] in, For drones In the The number of local training rounds per execution cycle; Represents a proxy for local iteration, the smaller the value, the higher the accuracy; is the loss function, For drones In the A federated learning local model with 1 execution cycle; is the gradient operator, which is used to calculate the derivative of a function at a certain point.

[0029] Preferably, the D2D network needs to perform adaptive link pruning:

[0030] Step 1: Set the signal-to-noise ratio threshold , when the signal-to-noise ratio is lower than When , pruning is required, and pruning is performed if the following formula is met:

[0031]

[0032] in, Assuming that the communication link between UAVs is mainly LoS link, the UAV cluster Medium UAV and drones The channel gain between For drones Send to drone The transmission power; For drones With drones The channel bandwidth of the transmission link between them; is the noise power density; Interference from other drones in the same cluster; is the signal-to-noise ratio threshold;

[0033] Step 2: Prune the relevant links of drones that are overfitted or underfitted in the local federated learning model after local training. Prune if the following formula is met:

[0034] ;

[0035] ;

[0036] ;

[0037] in, represents the parameter vector of the logistic regression model; is the training error; is the cross validation error; Indicates the error threshold when the model is overfitting or underfitting; is the minimum ratio for underfitting; is the maximum ratio that is judged as overfitting.

[0038] Preferably, the intra-cluster federated learning aggregation model is:

[0039]

[0040] in, For drone swarms Medium UAV In the Intra-cluster federated learning aggregation model with execution cycles; Represents a drone cluster In the The out-of-cluster federated learning aggregation model obtained through execution cycles; For the The out-degree of each vertex; It is a local model for federated learning.

[0041] Preferably, the out-of-cluster federated learning aggregation model is:

[0042]

[0043] in, Aggregate model for out-of-cluster federated learning; For drone swarms In sequence Number; For the The number of drone clusters per execution cycle, Represents the number of vertices in a directed graph.

[0044] Preferably, the base station uses a reinforcement learning algorithm to calculate the path planning strategy of all drones in the corresponding cluster during the data collection phase within the remaining execution time of this round:

[0045] Step 1: Set the environment and the initial state information of multiple drones;

[0046] Step 2: Select an action based on the current path planning strategy;

[0047] Step 3: Perform the selected action in the environment and observe its results;

[0048] Step 4: Update the path planning strategy based on the new reward and new status;

[0049] If the maximum number of iterations of the reinforcement learning algorithm is not reached, return to step 1, and then repeat steps 1 to 4; if the maximum number of iterations of the reinforcement learning algorithm is reached, stop and send the path planning strategy to the corresponding cluster head UAV.

[0050] Preferably, the formula required to obtain the path planning strategy is:

[0051] ;

[0052] ;

[0053] ;

[0054] in, For a given time slot The state space of the system below; For drones in time slots Location information below; is the location information of the ground sensor; For the time slot The number of clusters that applied to enter the data collection phase; For drone swarms The number of the drone; The time when the ground sensor was last used to collect data by the drone; For ground sensors in time slots The amount of data under For ground sensors in time slots Total information age under ; For a given time slot The action space of the lower system; For the time slot The drone in the data collection phase of the execution round The number of the ground sensor to be traversed; For the time slot The drone in the data collection phase of the execution round The number of ground sensors to be traversed; For the time slot At the end of the data collection phase of the execution round, the drone The final position coordinates of For the time slot The reward function of the following system is: For the time slot The total information age of the data collected by the drone in the execution round; For the time slot The total amount of data collected by the drone in the execution round; is the weight between the total information age and the total data volume.

[0055] The beneficial effects of the present invention are:

[0056] The information age-sensitive UAV swarm federated reinforcement learning method provided by the present invention can improve the real-time performance of the system, reduce system overhead, make the system efficient and reliable, and ensure the accuracy and convergence speed of federated learning while minimizing the average information age of the system while minimizing the energy consumption of the UAV cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 Schematic diagram of asynchronous semi-distributed federated learning of drone swarms described in the present invention.

[0058] Figure 2 This is a flowchart of the information age-sensitive UAV swarm federated reinforcement learning method described in the present invention. DETAILED DESCRIPTION

[0059] The present invention will be described in further detail below in conjunction with the accompanying drawings so that those skilled in the art can implement the invention with reference to the description.

[0060] like Figure 1 As shown, in this embodiment, there is a drones, A system consisting of 10 ground sensors and 1 base station, whose data is both real-time and sensitive. To improve system real-time performance, reduce system overhead, and enhance system security, this paper proposes an information-age-sensitive federated reinforcement learning method for drone swarms. Within the framework of asynchronous semi-distributed federated learning, the drone clustering algorithm is optimized by combining the characteristics of federated learning training. To reduce system overhead, a D2D network adaptive pruning algorithm is proposed to optimize internal communication within the drone cluster. Addressing the interaction level and local neglect issues faced by federated learning model aggregation within and outside clusters, corresponding aggregation weighting schemes are designed to ensure system accuracy and iteration speed. To prevent drones from falling into a situation of persistent advantage or disadvantage, the present invention adopts a method that sets the drone training time and data collection time as one execution cycle. Since the training time is proportional to the amount of data collected by the drones, the present invention allows drones with less training time in the current round to have more data collection time, thereby ensuring training time for the next execution cycle. Since the collected data is real-time and used for federated learning training, a reinforcement learning algorithm is used to optimize the paths and trajectories of all drones in the drone cluster that apply for out-of-cluster aggregation, with the goal of ensuring information age and data volume. Therefore, the proposed information age-sensitive UAV swarm federated reinforcement learning method can achieve the goal of minimizing the average information age of the system while ensuring the accuracy and convergence speed of federated learning and minimizing the energy consumption of the UAV cluster.

[0061] like Figure 2 As shown, the present invention provides an information age-sensitive UAV swarm federated reinforcement learning method, and the specific implementation process is as follows:

[0062] Step 1: Determine the number and initial location information of the components in the system, including drones, ground sensors, and base stations; determine the maximum number of execution rounds As the system's stop condition and fixed time As the cycle time of each execution round, each execution round includes two parts: model training phase and data collection phase. The time of the two parts needs to be adjusted in combination with the current execution round system status; the base station initializes and distributes the federated learning global model parameters to the UAV.

[0063] Step 2: Before each execution cycle begins, all the drones are clustered according to their geographical locations and local training times. The specific process is as follows:

[0064] Step 1: Cluster all the drones according to their geographical locations. At the beginning of each execution cycle, the distances between all people are calculated. :

[0065]

[0066] Among them, the UAV is in a hovering state during the learning phase of each execution round, that is, the position coordinates do not change; For drones With drones the distance between them; For the The execution cycle starts at the moment when the drone coordinates of For the The execution cycle starts at the moment when the drone coordinates of

[0067] Limited by the transmission power of the UAV, to ensure that any two UAVs in the same cluster are within the maximum D2D communication range Based on the principle of internal clustering, the hierarchical clustering method is used for the first clustering;

[0068] Step 2: All the drones are clustered according to their local training time, and the number of all the drones in the first Local training time per execution cycle :

[0069]

[0070] in, For drones In the Local training time per execution cycle; For drones In the The number of local iterations per execution cycle; For drones In the The number of data samples collected during the execution cycle, For drones CPU cycles to process one data sample; For drones CPU frequency for processing one data sample;

[0071] In order to adapt to the real-world scenario where equipment is constantly changing, and to ensure that the difference in training time required for any two drones in the same cluster is within the maximum time range during this execution cycle, the kernel density estimation method is used to perform a second clustering based on the first clustering.

[0072] All the UAVs receive the federated learning global model parameters or the out-of-cluster federated learning aggregate model sent by the cluster head UAV, and use the collected data to perform local training to obtain a federated learning local model; the optimization goal of the local training process for all the UAVs is:

[0073]

[0074] in, ;

[0075] Where, are the model parameters of the federated learning global model; For drone swarms In the The out-of-cluster federated learning aggregation model obtained through execution cycles; is the total number of samples; is the loss function when the drone is used as a client; For the input samples; For the output labels; is the proximal term, is a regularization parameter used to control the impact of global model parameter deviation on the local objective function; It is a common loss function symbol used to represent the difference or error between the predicted value and the true value;

[0076] All of the drones passing After a round of local training, an inaccurate solution is obtained, namely the federated learning local model of the UAV ; Use the neighboring gradient descent algorithm to learn the local model of federated learning To perform a partial update:

[0077] ;

[0078] ;

[0079] ;

[0080] in, For drones In the The number of local training rounds per execution cycle; Represents a proxy for local iteration, the smaller the value, the higher the accuracy; is the loss function, is the gradient operator, which is used to calculate the derivative of a function at a certain point.

[0081] After all the UAVs in the same cluster have completed local training, a D2D network is established in the cluster and adaptive link pruning is performed on the D2D network. The specific process is as follows:

[0082] Step 1: To ensure transmission quality, it is necessary to prune links with poor communication quality. Considering the mutual interference between all drones in the same cluster, a signal-to-noise ratio threshold is set. ; When the signal-to-noise ratio is lower than When , the link communication quality is considered to be poor and pruning is required; if the following formula is satisfied, the drone and drones The links between them need to be pruned:

[0083] ;

[0084] in,

[0085] ;

[0086] in, Assuming that the communication link between UAVs is mainly LoS link, the UAV cluster Medium UAV and drones The channel gain between For drones Send to drone The transmission power; The reference distance is The channel power gain when For drones With drones The channel bandwidth of the transmission link between them; is the noise power density; Interference from other drones in the same cluster; is the signal-to-noise ratio threshold;

[0087] Step 2: In the process of machine learning, in order to ensure the balance between the learning ability and generalization ability of the model, the relevant links of the UAVs with overfitting or underfitting in the local federated learning model after local training are pruned; considering that in actual use, overfitting and underfitting are relatively subjective evaluations, the training error is often used. and cross validation error To determine whether the model is overfitting or underfitting; when the cross-validation error is large, if the cross-validation error is close to the training error and the training error is relatively large, the model is judged to be underfitting; if the training error is much smaller than the cross-validation error, the model is judged to be overfitting;

[0088] The cross-validation error is large when the formula is as follows:

[0089] ;

[0090] If the cross-validation error is close to the training error and the training error is relatively large, the federated learning local model is underfitting. The underfitting situation is as follows:

[0091] ;

[0092] If the training error is much smaller than the cross-validation error, the federated learning local model is overfitting. The overfitting situation is as follows:

[0093] ;

[0094] If the underfitting or overfitting formula is met, the drone The relevant links need to be pruned;

[0095] in,

[0096] ;

[0097] ;

[0098] in, is the parameter vector of the logistic regression model; is the training error; is the cross validation error; is the sample data size of the training set; is the sample data size of the cross-validation set; is the training set; is the cross validation set; For the training set Sample input values; The cross validation set Sample input values; The model is trained on the The output value predicted by each sample; For the model pair cross validation set The output value predicted by each sample; For the training set The actual output value of each sample; The cross validation set The actual output value of each sample; The error threshold when the model is overfitting or underfitting; is the minimum ratio for underfitting; is the maximum ratio that is judged as overfitting.

[0099] With the goal of minimizing the energy consumption of sending the federated learning aggregation model within the cluster to the base station, a cluster head UAV is selected within the cluster. The specific process is as follows:

[0100] Step 1: Calculate the During the training phase of the execution cycle, the number of drones in the cluster Distance to base station :

[0101]

[0102] in, For the During the training phase of the execution cycle, the number of drones in the cluster Distance to the base station; is the coordinate of the base station;

[0103] Step 2: Calculate the number of drones in the cluster Energy consumption of sending the cluster federated learning aggregation model to the base station :

[0104] ;

[0105] in, For drones within a cluster The energy consumption of sending the cluster federated learning aggregation model to the base station; is the model parameter size, which is a constant; Assigned to drones Channel bandwidth; For drones The transmission power; For drones Channel gain between the BS and the base station; is the propagation rate of the signal on the channel; For drones Transmit power sent to the base station;

[0106] Step 3: Select the UAV with the lowest energy consumption as the cluster head UAV, and mark the cluster head UAV as .

[0107] Swarming drones In the The D2D network after execution cycle pruning is modeled as a directed graph:

[0108] ;

[0109] in, is a directed graph; For drone swarms In the The vertex set of the directed graph of execution cycles; For drone swarms In the The edge set of the directed graph of execution cycles;

[0110] If the directed graph There is a line from vertex To the top , then the drone and drones There is a communication link between the drones. For the drone's predecessor, the drone For drones Considering the communication cost and the stability of the learning process, after waiting for all the drones in the same cluster to complete the local training, the drone cluster The UAV in the cluster sends the federated learning local model to its subsequent neighbors through the pruned D2D network, and obtains the intra-cluster federated learning aggregation model based on the designed intra-cluster federated learning aggregation weight. The UAV in the cluster must wait for all local training of other UAVs in the cluster to be completed before sending the federated learning local model to its neighbors, without waiting for other UAVs outside the cluster, thereby realizing synchronous federated learning training among UAVs in each cluster. The specific process is as follows:

[0111] Step 1: Considering the cooperative relationship between drones, all drone clusters select their cluster head drones in each execution cycle; After the local training of all the UAVs is completed, each UAV transmits the updated gradient of its current round to its subsequent neighbors and sends its subsequent neighbor information to the cluster head UAV. ;

[0112] Step 2: Based on the information sent by the drone, the cluster head drone The out-degree sequence of each vertex is obtained as:

[0113] ;

[0114] in, is the out-degree sequence; Indicates the number of vertices of the directed graph;

[0115] When the cluster head UAV When the cluster head drone receives the subsequent neighbor information of all the drones in the cluster, the cluster head drone Broadcast the end signal of this round of communication and the out-degree sequence to all drones in the cluster ;

[0116] Step 3: Drone According to the updated gradient received from the previous neighbor, we get The predecessor neighbor information table at the execution cycle After receiving the communication end signal, the drone performs weighted aggregation on the received update gradients and expresses the drone cluster by the following formula: Medium UAV In the Intra-cluster aggregation results for execution cycles:

[0117] ;

[0118] in, Local models for federated learning; For the The out-degree of each vertex; Local models for federated learning;

[0119] After the intra-cluster aggregation of any cluster is completed, the cluster head UAV sends the intra-cluster federated learning aggregation model and information of all UAVs in the cluster to the base station;

[0120] Among them, all drone information is ,in, is the number sequence of the UAVs in the cluster; is the position sequence of the UAVs in the cluster.

[0121] Step 3. Each drone cluster operates at its own speed, and the drones in the cluster are aggregated in a synchronous manner; after the base station receives the intra-cluster federated learning aggregation model and all drone information in the cluster of any cluster, it does not need to wait for the intra-cluster federated learning aggregation model and all drone information in the cluster of other drone clusters, and directly obtains the extra-cluster federated learning aggregation model based on the extra-cluster aggregation weight; asynchronous federated learning training is performed between the drone cluster and the base station; as a logical, coarse-grained training entity, each drone cluster updates a federated learning global model in an asynchronous manner; the federated learning global model is the extra-cluster aggregation model returned by the base station to the last corresponding cluster after all clusters are aggregated in each execution cycle; the extra-cluster aggregation model is the model returned by the base station to other clusters except the last cluster that applied for aggregation with the base station in each execution cycle; each cluster is sorted according to the local training time, and the sequence is defined as ; Located in the sequence The front clusters have shorter training time, which can promote faster convergence of global model training; The latter clusters have a longer training time, which can further improve the prediction performance of the model. To prevent biased training, higher weights are assigned to drone clusters with longer training time during asynchronous aggregation. The following formula is used to express the drone clusters: All drones in Out-of-cluster aggregation results for execution cycles:

[0122] ;

[0123] in, Aggregate model for out-of-cluster federated learning; For drone swarms In sequence Number; For the The number of drone clusters per execution cycle;

[0124] The information age is the time from when the ground sensor data packet is generated to when the data packet is uploaded to the drone. In the time slot The information age of the data packet generated below is:

[0125] ;

[0126] in, is the time when the data packet is collected by the drone;

[0127] Based on the real-time and dynamic adjustment capabilities of reinforcement learning, the optimal path planning strategy is obtained by using a model-free deep reinforcement learning algorithm. While ensuring the freshness of the drone-collected data, as much data as possible is collected to participate in the training of federated learning. The multi-UAV path planning strategy is expressed as follows:

[0128] ;

[0129] ;

[0130] ;

[0131] in, For a given time slot The state space of the system below; For drones in time slots Location information below; is the location information of the ground sensor; For the time slot The number of clusters that applied to enter the data collection phase; For drone swarms The number of the drone; The time when the ground sensor was last used to collect data by the drone; For ground sensors in time slots The amount of data under For ground sensors in time slots Total information age under ; For a given time slot The action space of the lower system; For the time slot The drone in the data collection phase of the execution round The number of the ground sensor to be traversed; For the time slot The drone in the data collection phase of the execution round The number of ground sensors to be traversed; For the time slot At the end of the data collection phase of the execution round, the drone The final position coordinates of For the time slot The reward function of the following system is: For the time slot The total information age of the data collected by the drone in the execution round; For the time slot The total amount of data collected by the drone in the execution round; is the weight between the total information age and the total data volume;

[0132] According to the reward function Determine whether the action selection is correct. A large reward function indicates that the action selection is correct. A small reward function indicates that the action selection is wrong and needs to be reselected.

[0133] The base station uses a reinforcement learning algorithm to combine the information of all drones in the corresponding cluster and the status information of the ground sensors to calculate the path planning strategy for all drones in the corresponding cluster during the data collection phase within the remaining execution time of this round. The specific process is as follows:

[0134] Step 1: Set the environment and the initial state information of multiple drones;

[0135] Step 2: Select an action based on the current path planning strategy;

[0136] Step 3: Perform the selected action in the environment and observe its results;

[0137] Step 4: Update the path planning strategy based on the new reward and new status;

[0138] If the maximum number of iterations of the reinforcement learning algorithm is not reached, return to step 1 and repeat steps 1 to 4. If the maximum number of iterations of the reinforcement learning algorithm is reached, stop.

[0139] The base station sends the out-of-cluster federated learning aggregation model and the path planning strategy to the corresponding cluster head UAV.

[0140] Step 4: The cluster head UAV receives the federated learning aggregation model and the path planning strategy outside the cluster, and sends information to other UAVs in the cluster; the UAVs in the cluster execute the path planning strategy to collect data from the ground sensors during the remaining execution time of this round; if the current round does not reach the set maximum number of execution rounds, , then return to step 2 to continue executing the next round; if the current round reaches the set maximum execution round, the system is deemed to be terminated.

[0141] After the out-of-cluster aggregation of all clusters is completed, the federated learning global model is obtained; the federated learning global model can reflect the learning and understanding of the data by the entire system during the training process and can be used for subsequent data analysis, prediction or decision-making tasks.

[0142] The information age-sensitive UAV swarm federated reinforcement learning method provided by the present invention establishes a multi-objective joint optimization model that ensures the accuracy and convergence speed of federated learning while minimizing the average information age of the system, while minimizing the energy consumption of the UAV cluster. First, the clustering information of the UAV before each round of federated learning training is determined. Then the UAV and the base station are trained according to the designed aggregation mechanism. Finally, the base station uses the reinforcement learning algorithm to calculate the path planning strategy of the UAV and sends it to the corresponding UAV cluster. A federated reinforcement learning method is designed under the premise of meeting data privacy security and ensuring real-time performance, so that the Internet of Things system can use its own data more efficiently and accurately. In order to solve the problems of high risk of single point failure of centralized federated learning, communication overhead and difficulty in convergence of decentralized federated learning models, and uneven resource allocation, a semi-decentralized federated learning method is used to combine the traditional client-to-server communication paradigm with client-to-client communication. Clustering, as an important way to organize and manage nodes, plays an important role in improving management efficiency and optimizing resource utilization. The clustering algorithm is used to divide the UAVs into Clusters are formed, and drones within the cluster communicate with each other as clients, while drones between different clusters do not communicate with each other. To solve the problem of wanderers and the problem of system transmission bottleneck, a second clustering is performed based on the training time required for this round on the basis of the traditional clustering based on geographical location. Synchronous federated learning training is performed between drones within each cluster, and asynchronous federated learning training is performed between drone clusters and base stations. Communication between drones within a cluster will aggregate their scaled cumulative gradients with their neighbors. Considering the communication cost and stability of the learning process, an adaptive pruning algorithm for the communication network is designed, and the intra-cluster federated learning aggregation weight is designed based on the degree of interaction between drones and the consensus of model parameters. Since the drone cluster model with fast response speed undergoes significantly more training rounds than the drone cluster with slow response speed, the traditional federated averaging method will make it difficult for the drone cluster with slow response to affect the global model. Therefore, the out-of-cluster aggregation weight is designed based on the response time of each drone cluster. At the same time, the execution cycle time is designed, and the response time and data collection time of the drone are set to one cycle, so that the drones whose updates are ignored due to the long response time in the current execution round will have a shorter response time in the next round due to the small remaining data collection time, thereby solving the problem that the model update is easily ignored due to the slow response speed from the perspective of the drone cluster and the individual drone; due to the real-time and strong dynamic adjustment capabilities of reinforcement learning, the reinforcement learning algorithm is used to optimize the path trajectories of all drones in the drone cluster that applies for out-of-cluster aggregation with the goal of ensuring information age and data volume; in summary, the present invention can achieve the premise of minimizing the average information age of the system, while ensuring the accuracy and convergence speed of federated learning and minimizing the energy consumption of the drone cluster.

[0143] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the description and implementation methods. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A federated reinforcement learning method for drone swarms that is sensitive to information age, characterized by: The steps include: Step 1: Determine the number and initial location information of drones, ground sensors, and base stations; determine the maximum number of execution rounds and the cycle time of each execution round; the base station initializes and distributes the federated learning global model parameters; each execution round includes a training phase and a data collection phase; Step 2: All the drones are clustered according to their geographical location and local training time; the drones receive the parameters of the federated learning global model or the federated learning aggregation model outside the cluster, and perform local training to obtain a federated learning local model; after all the drones in the same cluster complete local training, a D2D network is established in the cluster, and the drone with the lowest energy consumption is selected as the cluster head drone. The drones in the cluster send the federated learning local model to their neighbors through the pruned D2D network, and obtain the cluster federated learning aggregation model based on the cluster federated learning aggregation weight; the cluster head drone sends the cluster federated learning aggregation model and all drone information in the cluster to the base station; wherein, the D2D network needs to perform adaptive link pruning: set the signal-to-noise ratio threshold , when the signal-to-noise ratio is lower than Pruning is required when the local model of federated learning is overfitted or underfitted. Pruning is also required for the relevant links of drones that are overfitted or underfitted in the local model of federated learning after local training. Step 3: After receiving the intra-cluster federated learning aggregation model and information about all drones in any cluster, the base station directly obtains the extra-cluster federated learning aggregation model based on the extra-cluster aggregation weights without waiting for other drone clusters, and calculates the path planning strategy for all drones in the corresponding cluster using the reinforcement learning algorithm; the base station sends the extra-cluster federated learning aggregation model and the path planning strategy to the corresponding cluster head drone; Step 4: The cluster head UAV sends information to other UAVs in the cluster; the UAVs in the cluster execute the path planning strategy to collect data from the ground sensors during the remaining execution time of this round; If the current round does not reach the set maximum number of execution rounds, return to step 2 to continue executing the next round; if the current round reaches the set maximum number of execution rounds, terminate.

2. The information age-sensitive UAV swarm federated reinforcement learning method according to claim 1 is characterized in that: The clustering process is as follows: Step 1, in the Before the start of an execution cycle, the distance between all the drones is: ； in, For drones With drones the distance between them; For the The execution cycle starts at the moment when the drone coordinates of For the The execution cycle starts at the moment when the drone coordinates of Based on the principle of ensuring that any two UAVs in the same cluster are within the maximum D2D communication distance, the hierarchical clustering method is used for the first clustering; Step 2: All the drones are in the The local training time for each execution cycle is: ； in, For drones In the Local training time per execution cycle; For drones In the The number of local iterations per execution cycle; For drones In the The number of data samples collected during the execution cycle, For drones CPU cycles to process one data sample; For drones CPU frequency for processing one data sample; Based on the principle of ensuring that the difference in training time required by any two UAVs in the same cluster during this execution cycle is within the maximum time range, the kernel density estimation method is used for the second clustering.

3. The information age-sensitive UAV swarm federated reinforcement learning method according to claim 2, characterized in that: The optimization goal of the local training process for all the drones is: ； in, are the model parameters of the federated learning global model; For drone swarms In the The out-of-cluster federated learning aggregation model obtained through execution cycles; is the total number of samples; is the loss function when the drone is used as a client; is the proximal term, is a regularization parameter used to control the impact of global model parameter deviation on the local objective function.

4. The information age-sensitive UAV swarm federated reinforcement learning method according to claim 3 is characterized in that: The local training process is: ； ； ； in, For drones In the The number of local training rounds per execution cycle; Represents a proxy for local iteration, the smaller the value, the higher the accuracy; is the loss function; For drones In the A federated learning local model with 1 execution cycle; is the gradient operator, which is used to calculate the derivative of a function at a certain point.

5. The information age-sensitive UAV federated reinforcement learning method according to claim 1 is characterized in that: The D2D network needs to perform adaptive link pruning: Step 1: Set the signal-to-noise ratio threshold , when the signal-to-noise ratio is lower than When , pruning is required, and pruning is performed if the following formula is met: ； in, Assuming that the communication link between UAVs is mainly LoS link, the UAV cluster Medium UAV and drones The channel gain between For drones Send to drone The transmission power; For drones With drones The channel bandwidth of the transmission link between them; is the noise power density; Interference from other drones in the same cluster; is the signal-to-noise ratio threshold; Step 2: Prune the relevant links of drones that are overfitted or underfitted in the local federated learning model after local training. Prune if the following formula is met: ； ； ； in, represents the parameter vector of the logistic regression model; is the training error; is the cross validation error; Indicates the error threshold when the model is overfitting or underfitting; is the minimum ratio for underfitting; is the maximum ratio that is judged as overfitting.

6. The information age-sensitive UAV swarm federated reinforcement learning method according to claim 1, characterized in that: The intra-cluster federated learning aggregation model is: ； in, For drone swarms Medium UAV In the Intra-cluster federated learning aggregation model with execution cycles; Represents a drone cluster In the The out-of-cluster federated learning aggregation model obtained through execution cycles; For the The out-degree of each vertex; It is a local model for federated learning.

7. The information age-sensitive UAV federated reinforcement learning method according to claim 6 is characterized in that: The out-of-cluster federated learning aggregation model is: ； in, Aggregate model for out-of-cluster federated learning; For drone swarms In sequence Number; For the The number of drone clusters per execution cycle; Represents the number of vertices in a directed graph.

8. The information age-sensitive UAV swarm federated reinforcement learning method according to claim 1, characterized in that: The base station uses the reinforcement learning algorithm to calculate the path planning strategy for all drones in the corresponding cluster during the data collection phase within the remaining execution time of this round: Step 1: Set the environment and the initial state information of multiple drones; Step 2: Select an action based on the current path planning strategy; Step 3: Perform the selected action in the environment and observe its results; Step 4: Update the path planning strategy based on the new reward and new status; If the maximum number of iterations of the reinforcement learning algorithm is not reached, return to step 1, and then repeat steps 1 to 4; if the maximum number of iterations of the reinforcement learning algorithm is reached, stop and send the path planning strategy to the corresponding cluster head UAV.

9. The information age-sensitive UAV swarm federated reinforcement learning method according to claim 8, characterized in that: The formula required to obtain the path planning strategy is: ； ； ； in, For a given time slot The state space of the system below; For drones in time slots Location information below; is the location information of the ground sensor; For the time slot The number of clusters that applied to enter the data collection phase; For drone swarms The number of the drone; The time when the ground sensor was last used to collect data by the drone; For ground sensors in time slots The amount of data under For ground sensors in time slots Total information age under ; For a given time slot The action space of the lower system; For the time slot The drone in the data collection phase of the execution round The number of the ground sensor to be traversed; For the time slot The drone in the data collection phase of the execution round The number of ground sensors to be traversed; For the time slot At the end of the data collection phase of the execution round, the drone The final position coordinates of For the time slot The reward function of the following system is: For the time slot The total information age of the data collected by the drone in the execution round; For the time slot The total amount of data collected by the drone in the execution round; is the weight between the total information age and the total data volume.

Images