Neural network modeling method for training a drone swarm based on a federated learning framework

Abstract

The application discloses a neural network modeling method for training a UAV cluster based on a framework of federal learning, establishes a double-layer UAV communication architecture of a top-layer UAV and a group of bottom-layer UAVs, considers that the trajectory of the UAV when performing a task is changed following a target and has strong randomness, adopts a random walk model to represent the trajectory of the bottom-layer UAV, and is suitable for most scenes of unknown trajectory such as target tracking and reconnaissance detection. Meanwhile, considering that the flight energy consumption of the UAV in a mobile scene is much greater than the calculation energy consumption and the transmission energy consumption, each round is divided into unequal time slots and an unfixed number of time slots, under the constraints of the performance and convergence speed of the federal learning model, the speed and flight direction of the top-layer UAV and the local iteration number are jointly optimized, a flight energy consumption minimization problem of the top-layer UAV is established, and a deep reinforcement learning algorithm is adopted to solve the problem, the state space and action space design are simplified, and the algorithm can be directly deployed on a UAV platform with limited computing power and energy.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle (UAV) communication technology, and in particular to a neural network modeling method for training UAV clusters based on a federated learning framework. Background Art

[0002] Drone swarms have been widely used in both military and civilian applications, such as target identification and traffic monitoring. Traditional centralized machine learning requires transmitting raw data to a central entity for processing, such as a cloud computing center. However, these traditional solutions are not suitable for drone networks. First, transmitting large amounts of raw data requires high bandwidth and consumes significant energy on the drones. Second, the high latency associated with this raw data transmission makes it unsuitable for real-time applications, such as target tracking. Finally, the transmission of raw data can lead to privacy leaks, such as the drone's location.

[0003] As a distributed learning framework, federated learning transmits model parameters instead of raw data, protecting data privacy while reducing the amount of data transmitted. This makes it suitable for drones with limited communication capabilities and energy. In addition, data is not concentrated in one drone but is distributed in a fragmented manner across each drone, which reduces the requirements for computing power and is also suitable for drones with limited computing power.

[0004] In the prior art, Solution 1 (Y.Shen, Y.Qu, C.Dong, F.Zhou and Q.Wu, "Joint Training and Resource Allocation Optimization for Federated Learning in UAV Swarm," in IEEE Internet of Things Journal, doi:10.1109 / JIOT.2022.3152829) describes a scenario in which a swarm of drones performs federated learning training based on collected labeled data, such as intruder vehicles, while performing patrol missions such as campus monitoring along a fixed trajectory. By allocating bandwidth and computing resources and optimizing local iteration accuracy and number, the transmission and computing energy consumption of the drones during model training during the entire federated learning period is optimized within the constraints of each round of computing time and transmission time. This solution divides the optimization problem into three subproblems. First, the local iteration number (integer variable) is relaxed to a continuous variable, and successive convex approximation (SCA) is used to convert the non-convex constraints into convex constraints. Then, the CVX package in MATLAB is used to iteratively solve the local iteration accuracy, number of local iterations, bandwidth, and computation frequency. However, this solution considers data collection and model training of drone clusters under fixed trajectories or even fixed formations, which is not suitable for most scenarios with unknown trajectories such as target tracking and reconnaissance detection.

[0005] In the prior art, Solution 2 (CN114997737A) uses hierarchical federated learning to train a network slice resource allocation model for scenarios where drone clusters serve as access base stations to provide services to ground users, thereby improving the isolation performance of network service slices. Layering reduces communication costs, clustering reduces data heterogeneity, data augmentation increases data volume, and attention mechanisms are used for model aggregation. However, this solution relies on base stations, which are often lacking or far away in many remote areas, resulting in high communication costs. It is not suitable for scenarios such as target tracking, border patrols, forest fire detection, air quality monitoring, and resource exploration. In addition, this solution requires preset trajectories when deploying federated learning in drone clusters. It does not take into account that the trajectory of drones follows the changes of targets and is highly random when performing tasks. It only dynamically deploys drone positions, and is therefore not suitable for most scenarios where the trajectory is unknown, such as target tracking and reconnaissance detection.

[0006] Therefore, it is necessary to propose a federated learning method for drone swarm systems to adapt to most scenarios with unknown trajectories, such as target tracking and reconnaissance detection. At the same time, it is necessary to solve the problem that the deep neural network model is extremely large and the model parameters in the tens of millions require extremely high computing power of the unmanned platform, making it difficult to directly deploy deep learning models on unmanned platforms with limited computing power and energy. SUMMARY

[0007] The present application is directed to the problem that the prior art is not suitable for most scenarios with unknown trajectories and deep neural network models are difficult to be directly deployed on unmanned platforms with limited computing power and energy, and proposes a neural network modeling method for training a UAV cluster based on a federated learning framework.

[0008] To achieve the above purpose, the present application provides the following technical solutions:

[0009] The present application proposes a neural network modeling method for training a UAV cluster based on a federated learning framework, comprising the following steps:

[0010] S1, a two-layer UAV communication architecture composed of a top-layer UAV and a group of bottom-layer UAVs is established, wherein the top-layer UAV is used to complete parameter aggregation of the federated learning model and global model training control, and the group of bottom-layer UAVs is used to complete local model training and execute intelligent tasks, and the movement model of each bottom-layer UAV is modeled as a random walk model;

[0011] S2, a system model including a transmission model, a calculation model and a flight model is established, and based on the system model, a top-layer UAV flight energy minimization problem is constructed, and under the constraints of global model convergence time and convergence accuracy, the flight energy of the top-layer UAV is minimized by jointly optimizing the top-layer UAV speed and direction and the local model iteration accuracy and iteration number;

[0012] S3, a deep reinforcement learning algorithm is used to solve the proposed top-layer UAV flight energy minimization problem, which is a non-convex mixed integer optimization problem.

[0013] Further, in step S1, the specific process of one global model iteration of federated learning is as follows:

[0014] S11, the top-layer UAV broadcasts the current model parameters and federated learning decision parameters, including the local iteration number of the bottom-layer UAVs;

[0015] S12, the bottom-layer UAVs perform local model training according to the received model parameters, federated learning decision parameters and collected data, update the model parameters using a full gradient descent algorithm, and perform local iteration until the local convergence requirement is met;

[0016] S13, the trained bottom-layer UAVs upload the model parameters and the current flight state of the UAVs in turn;

[0017] S14, after the model parameters of all bottom-layer UAVs are successfully uploaded, the top-layer UAV performs weighted aggregation of the model parameters, and inputs the current cluster state into a policy model trained by a reinforcement learning algorithm to output the current decision;

[0018] S15. The top-level UAV broadcasts model parameters and federated learning decision parameters, and implements flight control based on the current decision.

[0019] S16. Repeat steps S11-S15 until the global convergence requirement is met.

[0020] Furthermore, in step S12, the local iteration process adopts the distributed approximate Newton algorithm. Before the local iteration begins, the local gradient is calculated once and then uploaded to the top-level UAV to obtain the global gradient.

[0021] Furthermore, in step S12, the local model is iterated until convergence is expressed as:

[0022]

[0023] That is, the current loss function of the underlying drone k and the optimal loss function The difference is less than the initial loss function and the optimal loss function The difference is η times, η is defined as the convergence accuracy of the local model, w i is the current global model parameter, are the initial, current and optimal local model parameter changes respectively;

[0024] Furthermore, the local iterative loss function is designed as:

[0025]

[0026] Among them, F k (w i +h k ) is the original underlying UAV k loss function, is the global gradient before training, is the local gradient of the underlying drone k before training, and ξ is the adjustable global gradient weight.

[0027] Furthermore, in step S16, the global model iteration reaches convergence as:

[0028] F(w i )-F(w * )≤ε(F(w 0 )-F(w * ))

[0029] Among them, F(w 0 ), F(w i ), F(w * ) are the initial, current, and optimal global model loss functions, respectively, and ε is the global model accuracy.

[0030] Further, assume that the local model loss function F k (w) is L-continuous and γ-strongly convex, i.e. The relationship between the local iteration number I and the local iteration accuracy η is:

[0031]

[0032] The relationship between the global iteration accuracy ε and the local iteration accuracy η and the global iteration round number N is:

[0033]

[0034] where I is the identity matrix, λ is defined as the model learning rate, L and γ are constants related to the loss function.

[0035] Further, the system model establishment process in step S2 is:

[0036] The entire federated learning process is divided into N time slots, and in each time slot represents each global round, and the number of time slots and the length of the time slots are not fixed. In each time slot, the position and flight speed of all UAVs are fixed. The position of the top-level UAV in time slot i is denoted as q T (i) = (X(i), Y(i), H), and the speed is denoted as v(i); the position of the bottom-level UAV k in time slot i is denoted as q k (i) = (x k (i), y k (i), h).

[0037] (1) Transmission of the model

[0038] It is assumed that the communication link between the top-level UAV and the bottom-level UAV is mainly a wireless link with line-of-sight characteristics, in which the channel quality only depends on the current communication distance between the top-level UAV and the bottom-level UAV k According to the Shannon formula, the uplink and downlink transmission time of the top-level UAV and the bottom-level UAV k is:

[0039]

[0040]

[0041] where S1 is the data amount transmitted by the bottom-level UAV to the top-level UAV, and S2 is the data amount transmitted by the top-level UAV to the bottom-level UAV; B sub is the subchannel bandwidth, and M is the number of subchannels; if the mthsubchannel is allocated to the bottom-level UAV k, then the indicator factor χ k,m = 1, otherwise χ k,m = 0; B is the broadcast bandwidth of the top-level UAV; p T , p krespectively, are the transmit power of the top-layer UAV and the bottom-layer UAV k; l T,k is the path loss of the top-layer UAV and the bottom-layer UAV k, which is related to the communication distance and the carrier frequency; σ 2 is the noise power;

[0042] (2) Calculation model

[0043] When the local iteration number I(i) is determined, the calculation time of the bottom-layer UAV k is:

[0044]

[0045] where C is the number of CPU cycles required to train each sample data, D k ,f k respectively, are the data volume and the calculation frequency of the bottom-layer UAV k;

[0046] (3) Flight model

[0047] The flight power consumption of the top-layer UAV is a one-dimensional function of the flight speed when the flight speed is greater than 5 m / s, and the specific modeling is:

[0048]

[0049] where P0, P1 represent the blade profile power and induced power in the hovering state, U tip is the rotor tip speed, v0 is the average rotor induced speed when hovering, ζ and s are the fuselage drag ratio and the rotor solidity respectively, and ρ and A are the air density and the rotor disc area respectively.

[0050] Further, in step S2, the construction process of the top-layer UAV flight energy minimization problem is:

[0051] The total time of federated learning is represented as:

[0052]

[0053] The total flight energy consumption of the top-layer UAV is represented as:

[0054]

[0055] Thus, the optimization problem is obtained:

[0056]

[0057] C1:

[0058] C2:

[0059] C3:

[0060] C4:

[0061] C5:X(i+1)=X(i)+v(i)t(i)cos(θ(i))

[0062] C6:Y(i+1)=Y(i)+v(i)t(i)sin(θ(i))

[0063] C7:

[0064] Among them, C1 is the constraint of the local iteration accuracy of federated learning on the number of local iterations; C2 is the constraint of the global accuracy of federated learning on the local accuracy and the number of global iterations; C3 and C4 are the flight direction and flight speed constraints of the top-level UAV respectively; C5 and C6 are the coordinate constraints of the top-level UAV; C7 is the total time constraint of federated learning.

[0065] Furthermore, step S3 uses a model-free off-policy deep reinforcement learning Soft Actor Critic (SAC) algorithm to optimize the trajectory in the continuous state and action space. The state space, action space and reward function are designed as follows:

[0066] State space: A relative coordinate system is established with the top-level drone position as the origin. The horizontal distance and azimuth angle of each bottom-level drone relative to the top-level drone are used instead of their absolute positions. The logarithm of the current global accuracy is also added to the state space. Specifically:

[0067]

[0068] Among them, d Tk (i),θ Tk (i) are the horizontal distance and azimuth of the top UAV and bottom UAV k at time slot i, respectively, lnε cur (i) is the logarithm of the current global model accuracy at time slot i;

[0069] Action space: According to the C1 constraint and C2 constraint, the action space is designed as:

[0070]

[0071] Reward function: The daily reward is designed to be the opposite of the flight energy consumption of the current global round, that is, -t(i)P(v(i)). For the C7 constraint, the settlement penalty is designed to be:

[0072]

[0073] Where r1 is a positive constant.

[0074] Compared with the prior art, the present application has the following beneficial effects:

[0075] 1. The neural network modeling method for training a UAV cluster based on a federated learning framework provided by the present application is used for executing tasks such as target tracking, boundary patrol, forest fire detection, air quality monitoring and resource detection as a whole, and needs to deploy a neural network model to execute intelligent tasks such as target detection and trajectory prediction. In many remote areas, there is a lack of base stations or the distance to the base stations is far. In addition to considering the unavailability of base stations, the interaction with the base stations will also increase the communication cost. Therefore, the present application does not consider the base stations, and establishes a two-layer UAV communication architecture of a top-layer UAV and a group of bottom-layer UAVs, wherein the top-layer UAV completes federated learning model aggregation and overall model training control, and the group of bottom-layer UAVs completes local model training and executes intelligent tasks. Considering that the trajectory of the UAV executing the task is changed following the target and has strong randomness, a random walk model is used to represent the trajectory of the bottom-layer UAV, which is suitable for most scenes such as target tracking and reconnaissance detection where the trajectory is unknown.

[0076] 2. The neural network modeling method for training a UAV cluster based on a federated learning framework provided by the present application considers that the flight energy consumption of the UAV in a mobile scenario is much greater than the calculation energy consumption and the transmission energy consumption. Each round is divided into unequal time slots and an unfixed number of time slots. The time slot length dynamically changes to achieve the optimization goal. Under the constraints of the federated learning model performance and the convergence speed, the flight energy consumption minimization problem of the top-layer UAV is established by jointly optimizing the speed and flight direction of the top-layer UAV and the local iteration number. The problem has non-convex constraints and integer variable local iteration number, and the randomness of the position of the bottom-layer UAV causes the dynamic change of the channel state between the top-layer UAV and the bottom-layer UAV. The present application is not aimed at a specific federated learning task, but is more based on the optimization of this federated learning algorithm, including trajectory planning and federated training parameter optimization, so as to reduce the UAV flight energy consumption (which is the main energy consumption) and reduce the global model training time under the constraint of model precision.

[0077] 3. In terms of solving method, the present application uses a deep reinforcement learning algorithm to solve the proposed non-convex mixed integer optimization problem, and simplifies the state space and action space design, such as reducing the position input of the top-layer UAV in the state space, and directly solving the local iteration precision based on the inequality constraint according to the local iteration number I, reducing the dimension of the action space. This method can be directly deployed on a UAV platform with limited computing power and energy. BRIEF DESCRIPTION OF DRAWINGS

[0078] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments described in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0079] Figure 1 A diagram of the federated learning framework for training drone swarm neural networks provided by the present invention.

[0080] Figure 2 A global model iteration flow chart provided by the present invention.

[0081] Figure 3 This is a training curve diagram based on the DDPG and SAC optimization algorithms provided by the present invention.

[0082] Figure 4 Comparison of global accuracy requirements and total energy consumption of T-UAV under different numbers of B-UAVs provided by the present invention. DETAILED DESCRIPTION

[0083] In order to better understand the present technical solution, the method of the present invention is described in detail below with reference to the accompanying drawings.

[0084] This paper proposes a neural network modeling method for training drone clusters based on a federated learning framework. The overall system architecture is as follows: Figure 1 As shown in the figure, in this system, a swarm of drones performs target tracking. To achieve better tracking, a machine learning model, such as YoloV7, is deployed. First, the drones perform preliminary inference based on their inherent labeled prior datasets to locate the target and expand the labeled data by taking photos from multiple angles. The swarm then uses the collected labeled datasets to train a model using federated learning. Finally, the swarm uses the federated learning-trained model to detect the target.

[0085] The present invention proposes a neural network modeling method for training drone clusters based on a federated learning framework, comprising the following steps:

[0086] S1. Establish a two-layer UAV communication architecture consisting of a top-level UAV and a group of bottom-level UAVs. The top-level UAV is used to complete parameter aggregation and overall model training control of the federated learning model, and the group of bottom-level UAVs is used to complete local model training and perform intelligent tasks. The mobility model of each bottom-level UAV is modeled as a random walk model.

[0087] The present invention adopts the distributed learning framework of federated learning, in which the bottom unmanned aerial vehicles (B-UAVs) performing tasks are at the same height h, and a top unmanned aerial vehicle (T-UAV) is deployed at a higher height H for parameter aggregation of the federated learning model. When tracking a target, it will fly according to the trajectory prediction algorithm required to perform the task, and at the same time collect relevant image data of the tracking target. Due to the differences in tracking targets and flight trajectories, the amount of data collected by each underlying drone is D k Please note that we are not targeting a specific target tracking scenario here, so the target trajectory and the underlying drone trajectory are unknown. We model the underlying drone's mobility model as a random walk model.

[0088] The process of federated learning a global model iteration is as follows Figure 2 As shown, the specific process is:

[0089] S11, the top-level UAV broadcasts the model parameters and federated learning decision parameters for this round, including the local iteration number I of the bottom-level UAV;

[0090] S12. The bottom-level UAV performs local model training based on the received model parameters, federated learning decision parameters, and collected data, and uses the full gradient descent algorithm to update the model parameters. The local iteration is repeated 1 times until the local convergence requirement is reached.

[0091] In order to better guide the local iteration process, the local iteration process adopts the distributed approximate Newton algorithm. Before the local iteration starts, the local gradient is calculated once and then uploaded to the top-level drone to obtain the global gradient.

[0092] S13, the bottom-level UAVs that have completed training upload the model parameters and the current flight status of the UAV (such as location information) in turn using Orthogonal Frequency Division Multiple Access (OFDMA);

[0093] S14. After all the bottom-level drones’ model parameters are uploaded successfully, the top-level drone performs weighted aggregation of the model parameters and inputs the current cluster state into the policy model trained by the reinforcement learning algorithm to output the current decision.

[0094] S15. The top-level UAV broadcasts model parameters and federated learning decision parameters, and implements flight control based on the current decision.

[0095] S16. Repeat steps S11-S15 until global convergence is achieved.

[0096] For this federated learning algorithm:

[0097] Local model solution:

[0098]

[0099] That is, the current loss function of the underlying drone k and the optimal loss function The difference is less than the initial loss function and the optimal loss function The difference is η times, η is defined as the convergence accuracy of the local model. Obviously, the smaller η is, the higher the accuracy of the local model is. i is the current global model parameter, are the initial, current and optimal local model parameter changes respectively.

[0100] For the distributed approximate Newton DANE algorithm, the local iterative loss function is designed as:

[0101]

[0102] Among them, F k (w i +h k ) is the original underlying UAV k loss function, is the global gradient before training, is the local gradient of the underlying drone k before training, and ξ is the adjustable global gradient weight.

[0103] Global model solution:

[0104] F(w i )-F(w * )≤ε(F(w 0 )-F(w * ))

[0105] Among them, F(w 0 ), F(w i ), F(w * ) are the initial, current, and optimal global model loss functions, respectively, and ε is the global model accuracy. Similar to η, the smaller ε is, the higher the global model accuracy is. For example, the present invention sets the relative global model accuracy to 0.001, and the initial loss function F(w 0 ) is very small (less than 100), especially the pre-trained model will be even smaller, and when the accuracy reaches 0.001, the current global model is basically the optimal global model.

[0106] Assume that the local model loss function F k (w) is L-continuous and γ-strongly convex, that is, I is the unit matrix, λ is defined as the model learning rate, then the relationship between the number of local iterations I and the local iteration accuracy η is:

[0107]

[0108] The relationship between the global iteration accuracy ε, local iteration accuracy η, and the number of global iteration rounds N is:

[0109]

[0110] Among them, L and γ are constants related to the loss function.

[0111] S2. Establish a system model that includes a transmission model, a calculation model, and a flight model. Based on the system model, construct the top-level UAV flight energy consumption minimization problem. Under the constraints of the global model convergence time and convergence accuracy, minimize the top-level UAV flight energy consumption by jointly optimizing the top-level UAV speed and direction with the local model iteration accuracy and number of iterations.

[0112] The process of establishing the system model is as follows:

[0113] The entire federated learning process is divided into N time slots, each time slot Represents each global round, the number of time slots and the length of time slots are not fixed, and the positions and flight speeds of all drones in each time slot are fixed. The position of the top drone in time slot i is recorded as q T (i)=(X(i),Y(i),H), the velocity is recorded as v(i); the position of the bottom drone k is recorded as q k (i)=(x k (i),y k (i), h).

[0114] (1) Transmission model

[0115] Assume that the communication link between the top UAV and the bottom UAV is mainly a wireless link with Line-of-Sight (LoS) characteristics, where the channel quality only depends on the current communication distance between the top UAV and the bottom UAV k. According to Shannon's formula, the uplink and downlink transmission time between the top-level UAV and the bottom-level UAV k is:

[0116]

[0117]

[0118] Among them, S1 is the amount of data transmitted from the bottom UAV to the top UAV, and S2 is the amount of data transmitted from the top UAV to the bottom UAV; B sub is the subchannel bandwidth length, M is the number of subchannels; if the mth subchannel is assigned to the underlying drone k, then the indicator factor χk,m =1, otherwise χ k,m = 0; B is the broadcast bandwidth of the top-level drone; p T ,p k are the transmission powers of the top-level UAV and the bottom-level UAV k respectively; l T,k is the path loss between the top-level UAV and the bottom-level UAV k, which is related to the communication distance and carrier frequency; σ 2 is the noise power.

[0119] (2) Computational model

[0120] When the number of local iterations I(i) is determined, the computation time of the underlying UAV k is:

[0121]

[0122] Among them, C is the number of CPU cycles required to train each sample data, D k ,f k are the data volume and calculation frequency of the underlying drone k, respectively.

[0123] (3) Flight model

[0124] The flight power consumption of the top-level UAV is a univariate function of the flight speed when the flight speed is greater than 5m / s. The specific model is:

[0125]

[0126] Among them, P0, P1 represent the blade profile power and induced power in the hovering state, U tip is the rotor tip speed, v0 is the average rotor induced speed in hover, ζ and s are the fuselage drag ratio and rotor solidity, ρ and A are the air density and rotor disk area, respectively.

[0127] The construction process of the top-level UAV flight energy consumption minimization problem is as follows:

[0128] The total time of federated learning is expressed as:

[0129]

[0130] The total flight energy consumption of the top-level UAV is expressed as:

[0131]

[0132] This leads to the optimization problem:

[0133]

[0134] C1:

[0135] C2:

[0136] C3:

[0137] C4:

[0138] C5:X(i+1)=X(i)+v(i)t(i)cos(θ(i))

[0139] C6:Y(i+1)=Y(i)+v(i)t(i)sin(θ(i))

[0140] C7:

[0141] Among them, C1 is the constraint of the local iteration accuracy of federated learning on the number of local iterations; C2 is the constraint of the global accuracy of federated learning on the local accuracy and the number of global iterations; C3 and C4 are the flight direction and flight speed constraints of the top-level UAV respectively; C5 and C6 are the coordinate constraints of the top-level UAV; C7 is the total time constraint of federated learning.

[0142] S3. Deep reinforcement learning algorithm is used to solve the proposed top-level UAV flight energy consumption minimization problem, which is a non-convex mixed integer optimization problem.

[0143] In the proposed model, the trajectory selection of the T-UAV has Markov characteristics, so a deep reinforcement learning algorithm can be used to optimize the trajectory in a continuous state and action space. The proposed model-free off-policy deep reinforcement learning Soft Actor-Critic (SAC) algorithm is used to optimize the trajectory in a continuous state and action space. The state space, action space, and reward function are designed as follows:

[0144] State space: To simplify the state space, unlike the general use of the absolute positions of all drones to describe the state space, this invention designs a relative coordinate system with the top drone position as the origin, and replaces the absolute position of each bottom drone with the horizontal distance and azimuth angle of the top drone. This eliminates the need for T-UAV coordinate input, reduces the state space dimension and algorithm complexity, and facilitates faster algorithm convergence. In addition, to guide the agent to better learn, this invention adds the logarithm of the current global accuracy to the state space, specifically:

[0145]

[0146] Among them, d Tk (i),θ Tk (i) are the horizontal distance and azimuth of the top UAV and bottom UAV k at time slot i, respectively, lnε cur(i) is the logarithm of the current global model accuracy at time slot i;

[0147] Action space: Write the C1 inequality constraint as:

[0148] C1:

[0149] According to the C2 inequality constraint, η(i) should take the minimum value, and the C1 inequality constraint can be transformed into an equality constraint:

[0150] C1:

[0151] The local accuracy η is uniquely determined by the number of local iterations I, so the local accuracy is ignored and the action space is designed as:

[0152]

[0153] Reward function: Based on the optimization goal, the daily reward is designed to be the opposite of the flight energy consumption of the current round, that is, -t(i)P(v(i)). To guide the agent to learn better, the settlement reward is designed to be a fixed value r0, but the final reward will be subtracted from r0. For the C7 constraint, the settlement penalty is designed as:

[0154]

[0155] Where r1 is a positive constant.

[0156] According to the learning objectives of reinforcement learning, the top-level UAV flight energy consumption during the entire federated learning training period can be optimized.

[0157] The present invention conducts simulation experiments based on the convergence performance of the Deep Deterministic Policy Gradient (DDPG) and SAC-based optimization algorithms. The results are as follows: Figure 3 As shown in the figure, both algorithms converge after approximately 200 episodes, with the DDPG algorithm converging slightly faster. Notably, the SAC algorithm converges to lower values ​​with less fluctuation as training progresses. This is because the SAC algorithm, based on maximum entropy learning, has better exploration capabilities, discovering more optimal actions and learning more robust policies. On the other hand, the DDPG algorithm, due to its poor exploration mechanism, may become trapped in local optima, and its deterministic actions cannot cope well with highly dynamic environments, resulting in greater fluctuations.

[0158] In addition, the present invention compares the impact of global accuracy requirements on the average total energy consumption of top-level UAVs after convergence under different numbers of bottom-level UAVs. The results are as follows: Figure 4As shown in the figure, as global accuracy requirements increase, the number of global iterations also increases, resulting in longer T-UAV FL execution time and greater energy consumption. Furthermore, the number of B-UAVs also affects T-UAV energy consumption. As the number of B-UAVs increases, the corresponding subchannel allocation decreases, while the amount of data processed by the T-UAV increases, resulting in increased transmission and computation time. Furthermore, a larger number of B-UAVs increases randomness, which has a certain impact on the T-UAV's decision-making.

[0159] In summary, the present invention proposes a neural network modeling method for training drone clusters based on a federated learning framework, which models the movement trajectory of the underlying drones as a random walk model. Specifically, the present invention selects the speed of each time slot of the underlying drones as (v min ,v max ), and the direction is selected as a random value of (-π / 2,π / 2). This is consistent with many practical scenarios, such as target tracking, traffic monitoring, reconnaissance detection, etc., and the strategy trained in this way is highly robust, and can also be used as a pre-training strategy for other mobile models or fixed-trajectory scenarios, and is highly portable. At the same time, in mobile scenarios with unfixed trajectories, considering that the flight energy consumption of T-UAV is much greater than the transmission energy consumption and computing energy consumption, the present invention takes the main energy consumption of the UAV - flight energy consumption as the optimization target, models the problem, and minimizes the energy consumption of the UAV by selecting the speed and direction of T-UAV and optimizing the local accuracy of federated learning, thereby transforming the non-convex mixed integer programming problem into an MDP problem. In view of the randomness of the random walk model, the maximum entropy reinforcement learning algorithm SAC is adopted to solve it, and the federated learning parameters are optimized to accelerate the convergence of the model.

[0160] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A neural network modeling method for training drone clusters based on a federated learning framework, characterized by: The following steps are involved: S1. Establish a two-layer UAV communication architecture consisting of a top-level UAV and a group of bottom-level UAVs. The top-level UAV is used to complete parameter aggregation and global model training control of the federated learning model, while the bottom-level UAVs are used to complete local model training and perform intelligent tasks. The mobility model of each bottom-level UAV is modeled as a random walk model. S2. Establish a system model that includes a transmission model, a computational model, and a flight model. Based on the system model, formulate a top-level UAV flight energy consumption minimization problem. Under the constraints of the global model convergence time and convergence accuracy, minimize the top-level UAV flight energy consumption by jointly optimizing the top-level UAV speed and direction with the local model iteration accuracy and number of iterations. S3, a deep reinforcement learning algorithm is used to solve the proposed top-level UAV flight energy consumption minimization problem, which is a non-convex mixed integer optimization problem; The system model building process in step S2 is: The entire federated learning process is divided into N time slots, each time slot Represents each global round, the number of time slots and the length of time slots are not fixed, and the positions and flight speeds of all drones in each time slot are fixed. The position of the top drone in time slot i is recorded as q T (i)=(X(i),Y(i),H), the velocity is recorded as v(i); the position of the bottom drone k is recorded as q k (i)=(x k (i),y k (i),h); (1) Transmission model Assume that the communication link between the top UAV and the bottom UAV is mainly a wireless link with line-of-sight characteristics, where the channel quality only depends on the current communication distance between the top UAV and the bottom UAV k. According to Shannon's formula, the uplink and downlink transmission time between the top-level UAV and the bottom-level UAV k is: Among them, S1 is the amount of data transmitted from the bottom UAV to the top UAV, and S2 is the amount of data transmitted from the top UAV to the bottom UAV; B sub is the subchannel bandwidth, M is the number of subchannels; if the mth subchannel is assigned to the underlying drone k, then the indicator factor χ k,m =1, otherwise χ k,m = 0; B is the broadcast bandwidth of the top-level drone; p T ,p k are the transmission powers of the top-level UAV and the bottom-level UAV k respectively; l T,k is the path loss between the top-level UAV and the bottom-level UAV k, which is related to the communication distance and carrier frequency; σ 2 is the noise power; (2) Computational model When the number of local iterations I(i) is determined, the computation time of the underlying UAV k is: Among them, C is the number of CPU cycles required to train each sample data, D k ,f k are the data volume and computation frequency of the underlying UAV k, respectively; (3) Flight model The flight power consumption of the top-level UAV is a univariate function of the flight speed when the flight speed is greater than 5m / s. The specific model is: Among them, P0, P1 represent the blade profile power and induced power in the hovering state, U tip is the rotor tip speed, v0 is the average rotor induced speed in hover, ζ and s are the fuselage drag ratio and rotor solidity, ρ and A are the air density and rotor disk area, respectively.

2. The neural network modeling method for training a drone cluster based on a federated learning framework according to claim 1 is characterized in that: In step S1, the specific process of federated learning for a global model iteration is as follows: S11, the top-level UAV broadcasts the model parameters and federated learning decision parameters for this round, including the local iteration number of the bottom-level UAV; S12. The bottom-level drone performs local model training based on the received model parameters, federated learning decision parameters, and collected data, and uses the full gradient descent algorithm to update the model parameters, performing local iterations until the local convergence requirements are met. S13, the bottom-level drone that has completed training uploads the model parameters and the current flight status of the drone in turn; S14. After all the bottom-level drones’ model parameters are uploaded successfully, the top-level drone performs weighted aggregation of the model parameters and inputs the current cluster state into the policy model trained by the reinforcement learning algorithm to output the current decision. S15. The top-level UAV broadcasts model parameters and federated learning decision parameters, and implements flight control based on the current decision. S16. Repeat steps S11-S15 until the global convergence requirement is met.

3. The neural network modeling method for training a drone cluster based on a federated learning framework according to claim 2 is characterized in that: In step S12, the local iteration process uses the distributed approximate Newton algorithm. Before the local iteration begins, the local gradient is calculated once and then uploaded to the top-level UAV to obtain the global gradient.

4. The neural network modeling method for training a drone cluster based on a federated learning framework according to claim 3 is characterized in that: In step S12, the local model is iterated until convergence is expressed as: That is, the current loss function of the underlying drone k and the optimal loss function The difference is less than the initial loss function and the optimal loss function The difference is η times, η is defined as the convergence accuracy of the local model, w i is the current global model parameter, are the initial, current and optimal local model parameter changes respectively.

5. The method for training a neural network modeling of a UAV cluster based on a federated learning framework according to claim 4 is characterized in that: The local iterative loss function is designed as: Among them, F k (w i +h k ) is the original underlying UAV k loss function, is the global gradient before training, is the local gradient of the underlying drone k before training, and ξ is the adjustable global gradient weight.

6. The neural network modeling method for training a drone cluster based on a federated learning framework according to claim 5 is characterized in that: Step S16, the global model iteration reaches convergence as: F(w i )-F(w * )≤ε(F(w 0 )-F(w * )) Among them, F(w 0 ), F(w i ), F(w * ) are the initial, current, and optimal global model loss functions, respectively, and ε is the global model accuracy.

7. The neural network modeling method for training a drone cluster based on a federated learning framework according to claim 6 is characterized in that: Assume that the local model loss function F k (w) is L-continuous and γ-strongly convex, that is, Then the relationship between the number of local iterations I and the local iteration accuracy η is: The relationship between the global iteration accuracy ε, local iteration accuracy η, and the number of global iteration rounds N is: Where I is the identity matrix, λ is defined as the model learning rate, and L and γ are constants related to the loss function.

8. The neural network modeling method for training a drone cluster based on a federated learning framework according to claim 1 is characterized in that: The construction process of the top-level UAV flight energy consumption minimization problem is as follows: The total time of federated learning is expressed as: The total flight energy consumption of the top-level UAV is expressed as: This leads to the optimization problem: C5:X(i+1)=X(i)+v(i)t(i)cos(θ(i)) C6:Y(i+1)=Y(i)+v(i)t(i)sin(θ(i)) Among them, C1 is the constraint of the local iteration accuracy of federated learning on the number of local iterations; C2 is the constraint of the global accuracy of federated learning on the local accuracy and the number of global iterations; C3 and C4 are the flight direction and flight speed constraints of the top-level UAV, respectively; C5 and C6 are the coordinate constraints of the top-level UAV; C7 is the total time constraint of federated learning; θ(i) is the azimuth angle of the top-level UAV in time slot i.

9. The method for training a neural network modeling of a UAV cluster based on a federated learning framework according to claim 8 is characterized in that: Step S3 uses a model-free off-policy deep reinforcement learning soft actor-critic algorithm to optimize the trajectory in the continuous state and action space. The state space, action space and reward function are designed as follows: State space: A relative coordinate system is established with the top-level drone position as the origin. The horizontal distance and azimuth angle of each bottom-level drone relative to the top-level drone are used instead of their absolute positions. The logarithm of the current global accuracy is also added to the state space. Specifically: Among them, d Tk (i),θ Tk (i) are the horizontal distance and azimuth of the top UAV and bottom UAV k at time slot i, respectively, lnε cur (i) is the logarithm of the current global model accuracy at time slot i; Action space: According to the C1 constraint and C2 constraint, the action space is designed as: Reward function: The daily reward is designed to be the opposite of the flight energy consumption of the current global round, that is, -t(i)P(v(i)). For the C7 constraint, the settlement penalty is designed to be: Where r1 is a positive constant.

Images