Federal reinforcement learning method for information age sensitive unmanned aerial vehicle group

Abstract

The invention discloses an information age sensitive unmanned aerial vehicle group federal reinforcement learning method. The method comprises the steps of 1, determining the number and initial position information of unmanned aerial vehicles, ground sensors and base stations, the maximum execution round number and the cycle time; step 2, clustering is carried out; after all the unmanned aerial vehicles in the same cluster finish local training, the unmanned aerial vehicles in the cluster send a federated learning local model to neighbors through the pruned D2D network to obtain a federated learning aggregation model in the cluster; the cluster head unmanned aerial vehicle sends the intra-cluster federated learning aggregation model and information of all the unmanned aerial vehicles in the cluster to the base station; 3, the base station obtains an out-cluster federated learning aggregation model, calculates a path planning strategy and sends the path planning strategy to the corresponding cluster head unmanned aerial vehicle; 4, the cluster head unmanned aerial vehicle sends information to other unmanned aerial vehicles in the cluster; the in-cluster unmanned aerial vehicle executes a path planning strategy to collect data; and if the current round does not reach the set maximum execution round number, returning to the step 2, and if the set maximum execution round number is reached, terminating.

Description

Technical Field

[0001] The present invention belongs to the technical field of cross-integration of artificial intelligence and machine learning, unmanned aerial vehicle communication technology, cloud computing and edge computing, and in particular relates to a federated reinforcement learning method for unmanned aerial vehicle swarms that is sensitive to information age. Background Art

[0002] In recent years, drones have played an increasingly important role in wireless communication networks and the Internet of Things. In terms of wireless communication networks, drones have the advantages of high flexibility and maneuverability. They can dynamically adjust their flight trajectories according to actual needs to quickly deploy to areas that require communication coverage, while avoiding the problem that ground base stations are difficult to deploy due to geographical restrictions. In terms of the Internet of Things, drones, as flying base stations, can support the connection and data transmission of IoT devices and provide stable communication services for IoT devices. In many fields, drones can be used as relay nodes of the IoT network to achieve real-time data collection and transmission. Since IoT data is usually real-time perception data, the IoT system is required to be able to process and respond to data quickly and effectively. At the same time, there is no lack of sensitive industry data in IoT data, which requires the IoT system to be able to process and analyze data safely and reliably. Summary of the invention

[0003] The purpose of the present invention is to provide an information age-sensitive UAV swarm federated reinforcement learning method, which can improve the real-time performance of the system, reduce system overhead, and make the system efficient and reliable.

[0004] The technical solution provided by the present invention is: An information age-sensitive federated reinforcement learning method for drone swarms, including: Step 1: Determine the number and initial location information of drones, ground sensors, and base stations; determine the maximum number of execution rounds and the cycle time of each execution round; the base station initializes and distributes the global model parameters of federated learning; Step 2: All the drones are clustered according to their geographical locations and local training time; the drones receive the parameters of the federated learning global model or the federated learning aggregation model outside the cluster, and perform local training to obtain a federated learning local model; after all the drones in the same cluster complete local training, a D2D network is established in the cluster, and a cluster head drone is selected. The drones in the cluster send the federated learning local model to their neighbors through the pruned D2D network, and obtain the federated learning aggregation model in the cluster according to the federated learning aggregation weight in the cluster; the cluster head drone sends the federated learning aggregation model in the cluster and information of all drones in the cluster to the base station; Step 3: After receiving the intra-cluster federated learning aggregation model and the information of all drones in the cluster of any cluster, the base station directly obtains the extra-cluster federated learning aggregation model based on the extra-cluster aggregation weight without waiting for other drone clusters, and calculates the path planning strategy of all drones in the corresponding cluster; the base station sends the extra-cluster federated learning aggregation model and the path planning strategy to the corresponding cluster head drone; Step 4: The cluster head UAV sends information to other UAVs in the cluster; the UAVs in the cluster execute the path planning strategy to collect data from the ground sensors during the remaining execution time of this round; If the current round does not reach the set maximum number of execution rounds, return to step 2 to continue executing the next round; if the current round reaches the set maximum number of execution rounds, terminate.

[0005] Preferably, each execution round includes a training phase and a data collection phase.

[0006] Preferably, the clustering process is: Step 1: Before the execution cycle starts, the distance between all the drones is:

[0007] in, For drones With drones The distance between For the The execution cycle starts at the moment when the drone The coordinates of For the The execution cycle starts at the moment when the drone The coordinates of Based on the principle of ensuring that any two UAVs in the same cluster are within the maximum D2D communication distance, a hierarchical clustering method is used for the first clustering; Step 2: All the drones are in the The local training time for each execution cycle is:

[0008] in, For drones In the Local training time per execution cycle; For drones In the The number of local iterations per execution cycle; For drones In the The number of data samples collected during each execution cycle, For drones CPU cycles to process one data sample; For drones The CPU frequency to process one data sample; Based on the principle of ensuring that the difference in training time required for any two UAVs in the same cluster during this execution cycle is within the maximum time range, the kernel density estimation method is used for the second clustering.

[0009] Preferably, the optimization goal of the local training process of all the drones is:

[0010] in, Model parameters for the federated learning global model; For drone swarms In the The out-of-cluster federated learning aggregation model obtained through execution cycles; is the total number of samples; is the loss function when the drone is used as a client; is the proximal term, is a regularization parameter used to control the impact of global model parameter deviation on the local objective function.

[0011] Preferably, the local training process is: ;

[0012] ;

[0013] ;

[0014] in, For drones In the The number of local training rounds per execution cycle; Represents a proxy for local iteration, the smaller the value, the higher the accuracy; is the loss function, For drones In the A local model of federated learning with execution cycles; is the gradient operator, which is used to calculate the derivative of a function at a certain point.

[0015] Preferably, the D2D network needs to perform adaptive link pruning: Step 1: Set the signal-to-noise ratio threshold , when the signal-to-noise ratio is lower than When , pruning is required, and pruning is performed if the following formula is met:

[0016] in, Assuming that the communication link between UAVs is mainly LoS link, the UAV cluster Medium UAV and drones The channel gain between For drones Send to drone The transmission power of For drones With drones The channel bandwidth of the transmission link between them; is the noise power density; for interference from other drones in the same cluster; is the signal-to-noise ratio threshold; Step 2: Prune the relevant links of the UAVs that are overfitted or underfitted in the local federated learning model after local training. Prune if the following formula is met: ;

[0017] ;

[0018] ;

[0019] in, represents the parameter vector of the logistic regression model; is the training error; is the cross validation error; Indicates the error threshold when the model is overfitting or underfitting; is the minimum ratio for underfitting; is the maximum ratio that is judged as overfitting.

[0020] Preferably, the intra-cluster federated learning aggregation model is:

[0021] in, For drone swarms Medium UAV In the Intra-cluster federated learning aggregation model for execution cycles; Represents a drone cluster In the The out-of-cluster federated learning aggregation model obtained through execution cycles; For the The out-degree of each vertex; Local models for federated learning.

[0022] Preferably, the out-of-cluster federated learning aggregation model is:

[0023] in, Aggregate model for out-of-cluster federated learning; For drone swarms In sequence Serial number; For the The number of drone clusters per execution cycle, Represents the number of vertices in a directed graph.

[0024] Preferably, the base station uses a reinforcement learning algorithm to calculate the path planning strategy of all drones in the corresponding cluster during the data collection phase within the remaining execution time of this round: Step 1: Set the initial state information of the environment and multiple drones; Step 2: Select an action based on the current path planning strategy; Step 3: Perform the selected action in the environment and observe its results; Step 4: Update the path planning strategy based on the new rewards and new status; If the maximum number of iterations of the reinforcement learning algorithm is not reached, return to step 1, and then repeat steps 1 to 4; if the maximum number of iterations of the reinforcement learning algorithm is reached, stop and send the path planning strategy to the corresponding cluster head drone.

[0025] Preferably, the formula required to obtain the path planning strategy is: ;

[0026] ;

[0027] ;

[0028] in, For a given time slot The state space of the following system; For drones in time slots The following location information; is the location information of the ground sensor; For the time slot The number of clusters that applied to enter the data collection phase; For drone swarms The serial number of the drone; The time when the ground sensor was last used to collect data by the drone; For ground sensors in time slots The amount of data under For ground sensors in time slots Total information age under; For a given time slot The action space of the lower system; For the time slot The next round of data collection phase drone The number of the ground sensor to be traversed; For the time slot The next round of data collection phase drone The number of ground sensors to be traversed; For the time slot At the end of the data collection phase of the execution round, the drone The final position coordinates of For the time slot The reward function of the following system is: For the time slot The total information age of the data collected by the drone in the execution round; For the time slot The total amount of data collected by the drone in the execution round; is the weight between the total information age and the total data volume.

[0029] The beneficial effects of the present invention are: The information age-sensitive UAV swarm federated reinforcement learning method provided by the present invention can improve the real-time performance of the system, reduce system overhead, make the system efficient and reliable, and, under the premise of minimizing the average information age of the system, ensure the accuracy and convergence speed of federated learning while minimizing the energy consumption of the UAV cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Schematic diagram of asynchronous semi-distributed federated learning of drone swarms described in the present invention.

[0031] Figure 2 This is a flow chart of the information age-sensitive UAV swarm federated reinforcement learning method described in the present invention. DETAILED DESCRIPTION

[0032] The present invention is further described in detail below in conjunction with the accompanying drawings so that those skilled in the art can implement the invention with reference to the description.

[0033] like Figure 1 As shown, in this embodiment, there is a drones, The system consists of a ground sensor and a base station. The data of the ground sensor has the characteristics of real-time and sensitivity. In order to achieve the purpose of improving the real-time performance of the system, reducing the system overhead and improving the system security, this paper proposes a federated reinforcement learning method for UAV swarms with information age sensitivity. Under the framework of asynchronous semi-distributed federated learning, the UAV clustering algorithm is optimized in combination with the characteristics of federated learning training; in order to reduce the system overhead, a D2D network adaptive pruning algorithm is proposed to optimize the internal communication of the UAV cluster. Aiming at the degree of interaction and local neglect faced by the aggregation of federated learning models within the cluster and the aggregation of federated learning models outside the cluster, the corresponding aggregation weight method is designed to ensure the accuracy and iteration speed of the system; in order to prevent the UAV from falling into a situation of continuous advantage or continuous disadvantage, the present invention adopts a method of setting the training time of the UAV to one execution cycle. Since the training time is proportional to the amount of data collected by the UAV, the present invention allows the UAV with less training time in this round to have more data collection time, thereby ensuring the training time of the next execution cycle of the UAV; since the collected data has the characteristics of real-time and the collected data is used for the training of federated learning, the reinforcement learning algorithm is used to ensure the information age and data volume, and the path trajectory of all UAVs in the UAV cluster applying for cluster aggregation outside the cluster is optimized. Therefore, the proposed information age-sensitive UAV swarm federated reinforcement learning method can achieve the accuracy and convergence speed of federated learning while minimizing the average information age of the system while minimizing the energy consumption of the UAV cluster.

[0034] like Figure 2 As shown, the present invention provides an information age-sensitive UAV swarm federated reinforcement learning method, and the specific implementation process is as follows: Step 1: Determine the number and initial location information of the components in the system, including drones, ground sensors, and base stations; determine the maximum number of execution rounds As the system stop condition and fixed time As the cycle time of each execution round, each execution round includes two parts: model training phase and data collection phase. The time of the two parts needs to be adjusted in combination with the current execution round system status; the base station initializes and distributes the federated learning global model parameters to the UAV.

[0035] Step 2: Before each execution cycle starts, all the drones are clustered according to their geographical locations and local training times. The specific process is as follows: Step 1: Cluster all the drones according to their geographical locations. At the beginning of each execution cycle, the distances between all people are calculated. :

[0036] Among them, the drone is in a hovering state during the learning phase of each execution round, that is, the position coordinates do not change; For drones With drones The distance between For the The execution cycle starts at the moment when the drone The coordinates of For the The execution cycle starts at the moment when the drone The coordinates of Limited by the transmission power of the drone, to ensure that any two drones in the same cluster are within the maximum D2D communication range Based on the principle of internal clustering, the hierarchical clustering method is used for the first clustering; Step 2: Cluster all the drones according to their local training time, and calculate the number of all the drones in the first Local training time per execution cycle :

[0037] in, For drones In the Local training time per execution cycle; For drones In the The number of local iterations per execution cycle; For drones In the The number of data samples collected during each execution cycle, For drones CPU cycles to process one data sample; For drones The CPU frequency to process one data sample; In order to adapt to the real-world scenarios where equipment is constantly changing, and to ensure that the difference in training time required for any two drones in the same cluster is within the maximum time range during this execution cycle, the kernel density estimation method is used to perform a second clustering based on the first clustering.

[0038] All the drones receive the federated learning global model parameters or the out-of-cluster federated learning aggregation model sent by the cluster head drone, and use the collected data to perform local training to obtain a federated learning local model; the optimization goal of the local training process for all the drones is:

[0039] in, ; In the formula, Model parameters for the federated learning global model; For drone swarms In the The out-of-cluster federated learning aggregation model obtained through execution cycles; is the total number of samples; is the loss function when the drone is used as a client; For the input samples; For the output labels; is the proximal term, is a regularization parameter used to control the impact of global model parameter deviation on the local objective function; It is a common loss function symbol used to represent the difference or error between the predicted value and the true value; All of the drones passing After a round of local training, an inaccurate solution is obtained, namely, the federated learning local model of the UAV ; Use the neighbor gradient descent algorithm to learn the local model of federated learning To perform a partial update: ;

[0040] ;

[0041] ;

[0042] in, For drones In the The number of local training rounds per execution cycle; Represents a proxy for local iteration, the smaller the value, the higher the accuracy; is the loss function, is the gradient operator, which is used to calculate the derivative of a function at a certain point.

[0043] After waiting for all the drones in the same cluster to complete local training, a D2D network is established in the cluster and adaptive link pruning is performed on the D2D network. The specific process is as follows: Step 1: In order to be realistic and ensure transmission quality, it is necessary to prune links with poor communication quality. Considering the mutual interference between all drones in the same cluster, a signal-to-noise ratio threshold is set. ; When the signal-to-noise ratio is lower than When , the link communication quality is considered to be poor and pruning is required; if the following formula is satisfied, the drone and drones The links between them need to be pruned: ;

[0044] in, ;

[0045] in, Assuming that the communication link between UAVs is mainly LoS link, the UAV cluster Medium UAV and drones The channel gain between For drones Send to drone The transmission power of The reference distance is The channel power gain at ; For drones With drones The channel bandwidth of the transmission link between them; is the noise power density; for interference from other drones in the same cluster; is the signal-to-noise ratio threshold; Step 2: In the process of machine learning, in order to ensure the balance between the learning ability and generalization ability of the model, prune the relevant links of the drones that are overfitted or underfitted in the local federated learning model after local training; considering that in actual use, overfitting and underfitting are a relatively subjective evaluation, often using training error and the cross validation error To determine whether the model is overfitting or underfitting; when the cross-validation error is large, if the cross-validation error is close to the training error and the training error is relatively large, the model is judged to be underfitting; if the training error is much smaller than the cross-validation error, the model is judged to be overfitting; The cross-validation error is large when the formula is as follows: ;

[0046] If the cross-validation error is close to the training error and the training error is relatively large, the local federated learning model is underfitting. The underfitting situation is as follows: ;

[0047] If the training error is much smaller than the cross-validation error, the local federated learning model is overfitted. The overfitting situation is as follows: ;

[0048] If the underfitting or overfitting formula is met, the drone The relevant links need to be pruned; in, ;

[0049] ;

[0050] in, is the parameter vector of the logistic regression model; is the training error; is the cross validation error; is the sample data size of the training set; is the sample data size of the cross-validation set; is the training set; is the cross validation set; For the training set Sample input values; The cross validation set Sample input values; The model is trained on the The output value predicted by each sample; For the model, cross validation set The output value predicted by each sample; For the training set The actual output value of each sample; The cross validation set The actual output value of each sample; It is the error threshold when the model is overfitting or underfitting; is the minimum ratio for underfitting; is the maximum ratio that is judged as overfitting.

[0051] With the goal of minimizing the energy consumption of sending the cluster federated learning aggregation model to the base station, a cluster head drone is selected in the cluster. The specific process is as follows: Step 1: Calculate the The training phase of the execution cycle is Distance to base station :

[0052] in, For the The training phase of the execution cycle is Distance to the base station; is the coordinate of the base station; Step 2: Calculate the number of drones in the cluster Energy consumption of sending the cluster federated learning aggregation model to the base station : ;

[0053] in, For drones in clusters The energy consumption of sending the cluster federated learning aggregation model to the base station; is the model parameter size, which is a constant; Assigned to drones The channel bandwidth; For drones The transmission power of For drones Channel gain between the BS and the base station; is the propagation rate of the signal on the channel; For drones Transmit power sent to the base station; Step 3: Select the UAV with the lowest energy consumption as the cluster head UAV, and mark the cluster head UAV as .

[0054] Swarming drones In the The D2D network after execution cycle pruning is modeled as a directed graph: ;

[0055] in, is a directed graph; For drone swarms In the The vertex set of the directed graph of execution cycles; For drone swarms In the The edge set of the directed graph of execution cycles; If a directed graph There is a line from vertex To the top , then the drone and drones There is a communication link between the two drones. For the drone's predecessor, the drone For drones Considering the communication cost and the stability of the learning process, after waiting for all the drones in the same cluster to complete the local training, the drone cluster The UAV in the cluster sends the federated learning local model to the subsequent neighbors of the UAV through the pruned D2D network, and obtains the cluster federated learning aggregation model according to the designed cluster federated learning aggregation weight; wherein, the UAV in the cluster has to wait for all local training of other UAVs in the cluster to be completed before sending the federated learning local model to its neighbors, without waiting for other UAVs outside the cluster, so as to realize synchronous federated learning training between UAVs in each cluster; the specific process is as follows: Step 1: Considering the cooperative relationship between drones, all drone clusters select their cluster head drones in each execution cycle; After the local training of all the drones is completed, each drone passes its updated gradient of the current round to its successor neighbor, and sends its successor neighbor information to the cluster head drone ; Step 2: According to the information sent by the drone, the cluster head drone The out-degree sequence of each vertex is obtained as: ;

[0056] in, is the out-degree sequence; Represents the number of vertices of a directed graph; When the cluster head UAV When receiving the subsequent neighbor information of all the drones in the cluster, the cluster head drone Broadcast the end signal of this round of communication and the out-degree sequence to all drones in the cluster ; Step 3: Drone According to the update gradient received from the previous neighbor, we get The predecessor neighbor information table at the execution cycle After receiving the communication end signal, the drone performs weighted aggregation on the received update gradients, and the drone cluster is expressed by the following formula: Medium UAV In the The cluster aggregation results for execution cycles are: ;

[0057] in, Local models for federated learning; For the The out-degree of each vertex; Local models for federated learning; After the intra-cluster aggregation of any cluster is completed, the cluster head drone sends the intra-cluster federated learning aggregation model and information of all drones in the cluster to the base station; Among them, all drone information is ,in, is the serial number sequence of the drones in the cluster; is the position sequence of the UAVs in the cluster.

[0058] Step 3, each drone cluster operates at its own speed, and the drones in the cluster are aggregated in a synchronous manner; after the base station receives the intra-cluster federated learning aggregation model and all drone information in the cluster of any cluster, it does not need to wait for the intra-cluster federated learning aggregation model and all drone information in the cluster of other drone clusters, and directly obtains the extra-cluster federated learning aggregation model based on the extra-cluster aggregation weight; asynchronous federated learning training is performed between the drone cluster and the base station; as a logical, coarse-grained training entity, each drone cluster updates a federated learning global model in an asynchronous manner; the federated learning global model is the extra-cluster aggregation model returned by the base station to the last corresponding cluster after all clusters are aggregated in each execution cycle; the extra-cluster aggregation model is the model returned by the base station to other clusters except the last cluster that the base station applies for aggregation with in each execution cycle; each cluster is sorted according to the local training time, and the sequence is defined as ; Located in the sequence The front clusters have shorter training time, which can promote faster convergence of global model training; The latter clusters have a longer training time, which can further improve the prediction performance of the model. To prevent biased training, higher weights are assigned to drone clusters with longer training time during asynchronous aggregation. The following formula is used to represent the drone clusters: All drones in Out-of-cluster aggregation results for execution cycles: ;

[0059] in, Aggregate model for out-of-cluster federated learning; For drone swarms In sequence Serial number; For the The number of drone clusters per execution cycle; The information age is the time from when the ground sensor data packet is generated to when the data packet is uploaded to the drone. In time slot The information age of the data packet generated below is: ;

[0060] in, is the time when the data packet is collected by the drone; According to the real-time and dynamic adjustment characteristics of reinforcement learning, the optimal path planning strategy is obtained by using the model-free deep reinforcement learning algorithm. On the basis of ensuring the freshness of the drone collected data, as much data as possible is collected to participate in the training of federated learning; the multi-drone path planning strategy is expressed as follows: ;

[0061] ;

[0062] ;

[0063] in, For a given time slot The state space of the following system; For drones in time slots The following location information; is the location information of the ground sensor; For the time slot The number of clusters that applied to enter the data collection phase; For drone swarms The serial number of the drone; The time when the ground sensor was last used to collect data by the drone; For ground sensors in time slots The amount of data under For ground sensors in time slots Total information age under; For a given time slot The action space of the lower system; For the time slot The next round of data collection phase drone The number of the ground sensor to be traversed; For the time slot The next round of data collection phase drone The number of ground sensors to be traversed; For the time slot At the end of the data collection phase of the execution round, the drone The final position coordinates of For the time slot The reward function of the following system is: For the time slot The total information age of the data collected by the drone in the execution round; For the time slot The total amount of data collected by the drone in the execution round; is the weight between the total information age and the total data volume; According to the reward function Determine whether the action selection is correct. If the reward function is large, it means the action selection is correct. If the reward function is small, it means the action selection is wrong and needs to be reselected. The base station uses a reinforcement learning algorithm, combined with the information of all drones in the corresponding cluster and the status information of the ground sensor, to calculate the path planning strategy of all drones in the corresponding cluster during the data collection phase within the remaining execution time of this round. The specific process is as follows: Step 1: Set the initial state information of the environment and multiple drones; Step 2: Select an action based on the current path planning strategy; Step 3: Perform the selected action in the environment and observe its results; Step 4: Update the path planning strategy based on the new rewards and new status; If the maximum number of iterations of the reinforcement learning algorithm is not reached, return to step 1, and then repeat steps 1 to 4; if the maximum number of iterations of the reinforcement learning algorithm is reached, stop; The base station sends the out-of-cluster federated learning aggregation model and the path planning strategy to the corresponding cluster head UAV.

[0064] Step 4: The cluster head UAV receives the federated learning aggregation model and the path planning strategy outside the cluster, and sends information to other UAVs in the cluster; the UAVs in the cluster execute the path planning strategy to collect data from the ground sensors during the remaining execution time of this round; if the current round does not reach the set maximum number of execution rounds , then return to step 2 to continue executing the next round; if the current round reaches the set maximum execution round, the system is deemed to be terminated.

[0065] After the out-of-cluster aggregation of all clusters is completed, the federated learning global model is obtained; the federated learning global model can reflect the learning and understanding of the data by the entire system during the training process, and can be used for subsequent data analysis, prediction or decision-making tasks.

[0066] The information age-sensitive UAV swarm federated reinforcement learning method provided by the present invention establishes a multi-objective joint optimization model that ensures the accuracy and convergence speed of federated learning while minimizing the average information age of the system, while minimizing the energy consumption of the UAV cluster. First, the clustering information of the UAV before each round of federated learning training is determined, and then the UAV and the base station are trained according to the designed aggregation mechanism. Finally, the base station uses the reinforcement learning algorithm to calculate the path planning strategy of the UAV and sends it to the corresponding UAV cluster. A federated reinforcement learning method is designed under the premise of satisfying data privacy security and ensuring real-time performance, so that the Internet of Things system can use their respective data more efficiently and accurately. In order to solve the problems of high risk of single point failure in centralized federated learning, communication overhead and difficulty in convergence of decentralized federated learning models, and uneven resource allocation, a semi-decentralized federated learning method is used to combine the traditional client-to-server communication paradigm with client-to-client communication. Clustering, as an important way to organize and manage nodes, plays an important role in improving management efficiency and optimizing resource utilization. The clustering algorithm is used to divide the UAVs into The UAVs in the cluster communicate with each other as clients, and the UAVs in different clusters do not communicate with each other. In order to solve the problem of wanderers and the system transmission bottleneck problem, a second clustering is performed based on the training time required for this round on the basis of the traditional clustering based on geographical location. Synchronous federated learning training is performed between the UAVs in each cluster, and asynchronous federated learning training is performed between the UAV cluster and the base station. The communication between the UAVs in the cluster will aggregate their scaled cumulative gradients with their neighbors. Considering the communication cost and the stability of the learning process, an adaptive pruning algorithm for the communication network is designed, and the aggregation weight of the intra-cluster federated learning is designed according to the degree of interaction between the UAVs and the consensus of the model parameters. Since the training rounds of the UAV cluster model with a fast response speed are much more than those of the UAV cluster model with a slow response speed, the traditional federated averaging method will make it difficult for the UAV cluster model with a slow response to affect the global model. Therefore, the out-of-cluster aggregation weight is designed according to the response time of each UAV cluster. At the same time, the execution cycle time is designed, and the response time and data collection time of the drone are set to one cycle, so that the drone whose update is ignored due to the long response time in the current execution round will have a shorter response time in the next round due to the small remaining data collection time, thereby solving the problem that the model update is easily ignored due to the slow response speed from the perspective of the drone cluster and the individual drone; due to the real-time and strong dynamic adjustment capabilities of reinforcement learning, the reinforcement learning algorithm is used to optimize the path trajectories of all drones in the drone cluster that applies for out-of-cluster aggregation with the goal of ensuring the information age and data volume; in summary, the present invention can achieve the premise of minimizing the average information age of the system, while ensuring the accuracy and convergence speed of federated learning and minimizing the energy consumption of the drone cluster.

[0067] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. An information age-sensitive UAV swarm federated reinforcement learning method, characterized in that: The steps include: Step 1: Determine the number and initial location information of drones, ground sensors, and base stations; determine the maximum number of execution rounds and the cycle time of each execution round; the base station initializes and distributes the global model parameters of federated learning; Step 2: All the drones are clustered according to their geographical locations and local training time; the drones receive the parameters of the federated learning global model or the federated learning aggregation model outside the cluster, and perform local training to obtain a federated learning local model; after all the drones in the same cluster complete local training, a D2D network is established in the cluster, and a cluster head drone is selected. The drones in the cluster send the federated learning local model to their neighbors through the pruned D2D network, and obtain the federated learning aggregation model in the cluster according to the federated learning aggregation weight in the cluster; the cluster head drone sends the federated learning aggregation model in the cluster and information of all drones in the cluster to the base station; Step 3: After receiving the intra-cluster federated learning aggregation model and the information of all drones in the cluster of any cluster, the base station directly obtains the extra-cluster federated learning aggregation model based on the extra-cluster aggregation weight without waiting for other drone clusters, and calculates the path planning strategy of all drones in the corresponding cluster; the base station sends the extra-cluster federated learning aggregation model and the path planning strategy to the corresponding cluster head drone; Step 4: The cluster head UAV sends information to other UAVs in the cluster; the UAVs in the cluster execute the path planning strategy to collect data from the ground sensors during the remaining execution time of this round; If the current round does not reach the set maximum number of execution rounds, return to step 2 to continue executing the next round; if the current round reaches the set maximum number of execution rounds, terminate.

2. The information age-sensitive UAV swarm federated reinforcement learning method according to claim 1, characterized in that: Each execution round includes a training phase and a data collection phase.

3. The information age-sensitive UAV swarm federated reinforcement learning method according to claim 1, characterized in that: The clustering process is: Step 1: Before the execution cycle starts, the distance between all the drones is: ； in, For drones With drones The distance between For the The execution cycle starts at the moment when the drone The coordinates of For the The execution cycle starts at the moment when the drone The coordinates of Based on the principle of ensuring that any two UAVs in the same cluster are within the maximum D2D communication distance, a hierarchical clustering method is used for the first clustering; Step 2: All the drones are in the The local training time for each execution cycle is: ； in, For drones In the Local training time per execution cycle; For drones In the The number of local iterations per execution cycle; For drones In the The number of data samples collected during each execution cycle, For drones CPU cycles to process one data sample; For drones The CPU frequency to process one data sample; Based on the principle of ensuring that the difference in training time required for any two UAVs in the same cluster during this execution cycle is within the maximum time range, the kernel density estimation method is used for the second clustering.

4. The information age-sensitive UAV swarm federated reinforcement learning method according to claim 1, characterized in that: The optimization goal of the local training process of all the drones is: ； in, Model parameters for the federated learning global model; For drone swarms In the The out-of-cluster federated learning aggregation model obtained through execution cycles; is the total number of samples; is the loss function when the drone is used as a client; is the proximal term, is a regularization parameter used to control the impact of global model parameter deviation on the local objective function.

5. The information age-sensitive UAV swarm federated reinforcement learning method according to claim 4, characterized in that: The local training process is: ； ； ； in, For drones In the The number of local training rounds per execution cycle; Represents a proxy for local iteration, the smaller the value, the higher the accuracy; is the loss function; For drones In the A local model of federated learning with execution cycles; is the gradient operator, which is used to calculate the derivative of a function at a certain point.

6. The information age-sensitive UAV swarm federated reinforcement learning method according to claim 1, characterized in that: The D2D network needs to perform adaptive link pruning: Step 1: Set the signal-to-noise ratio threshold , when the signal-to-noise ratio is lower than When , pruning is required, and pruning is performed if the following formula is met: ； in, Assuming that the communication link between UAVs is mainly LoS link, the UAV cluster Medium UAV and drones The channel gain between For drones Send to drone The transmission power of For drones With drones The channel bandwidth of the transmission link between them; is the noise power density; for interference from other drones in the same cluster; is the signal-to-noise ratio threshold; Step 2: Prune the relevant links of the UAVs that are overfitted or underfitted in the local federated learning model after local training. Prune if the following formula is met: ； ； ； in, represents the parameter vector of the logistic regression model; is the training error; is the cross validation error; Indicates the error threshold when the model is overfitting or underfitting; is the minimum ratio for underfitting; is the maximum ratio that is judged as overfitting.

7. The information age-sensitive UAV swarm federated reinforcement learning method according to claim 1, characterized in that: The intra-cluster federated learning aggregation model is: ； in, For drone swarms Medium UAV In the Intra-cluster federated learning aggregation model for execution cycles; Represents a drone cluster In the The out-of-cluster federated learning aggregation model obtained through execution cycles; For the The out-degree of each vertex; Local models for federated learning.

8. The information age-sensitive UAV swarm federated reinforcement learning method according to claim 7, characterized in that: The out-of-cluster federated learning aggregation model is: ； in, Aggregate model for out-of-cluster federated learning; For drone swarms In sequence Serial number; For the The number of drone clusters per execution cycle; Represents the number of vertices in a directed graph.

9. The information age-sensitive UAV swarm federated reinforcement learning method according to claim 2, characterized in that: The base station uses the reinforcement learning algorithm to calculate the path planning strategy of all drones in the corresponding cluster during the data collection phase within the remaining execution time of this round: Step 1: Set the initial state information of the environment and multiple drones; Step 2: Select an action based on the current path planning strategy; Step 3: Perform the selected action in the environment and observe its results; Step 4: Update the path planning strategy based on the new rewards and new status; If the maximum number of iterations of the reinforcement learning algorithm is not reached, return to step 1, and then repeat steps 1 to 4; if the maximum number of iterations of the reinforcement learning algorithm is reached, stop and send the path planning strategy to the corresponding cluster head drone.

10. The information age-sensitive UAV swarm federated reinforcement learning method according to claim 9, characterized in that: The formula required to obtain the path planning strategy is: ； ； ； in, For a given time slot The state space of the following system; For drones in time slots The following location information; is the location information of the ground sensor; For the time slot The number of clusters that applied to enter the data collection phase; For drone swarms The serial number of the drone; The time when the ground sensor was last used to collect data by the drone; For ground sensors in time slots The amount of data under For ground sensors in time slots Total information age under; For a given time slot The action space of the lower system; For the time slot The next round of data collection phase drone The number of the ground sensor to be traversed; For the time slot The next round of data collection phase drone The number of ground sensors to be traversed; For the time slot At the end of the data collection phase of the execution round, the drone The final position coordinates of For the time slot The reward function of the following system is: For the time slot The total information age of the data collected by the drone in the execution round; For the time slot The total amount of data collected by the drone in the execution round; is the weight between the total information age and the total data volume.

Images