Unmanned aerial vehicle deployment optimization method based on multi-agent federated reinforcement learning

Abstract

The present application relates to a kind of large-scale UAV deployment optimization method based on multi-agent federal reinforcement learning (K-MAFRL), by optimizing the real-time deployment position of multiple UAVs, the minimization of system average task response time is realized.First, each UAV is modeled as an agent, and can independently make deployment decisions according to local information.Then, the decision model is trained using the K-MAFRL algorithm, the local model of each agent is trained by interacting with the environment, and the model parameters are federated aggregated to generate a global model, and then the model parameters are transmitted back to all agents to update their local network.Finally, through feedback control and multi-UAV cooperation, an effective large-scale UAV deployment scheme is gradually found.The present application realizes efficient and adaptive large-scale UAV deployment.

Description

Technical Field

[0001] This invention relates to the field of drone deployment, and more specifically to a drone deployment optimization method based on multi-agent federated reinforcement learning. Background Technology

[0002] Unmanned Aerial Vehicles (UAVs) can be combined with Mobile Edge Computing (MEC) to form a UAV-assisted network to address the shortage of computing resources on mobile devices. This approach pushes cloud-centric computing power down to the network edge by deploying edge servers, leveraging the abundant computing resources of edge servers to compensate for the insufficient computing power of mobile devices (MDs). Furthermore, UAV-assisted MEC networks are particularly suitable for high-density public emergency response, special emergency situations, and other on-demand services. In dynamically changing runtime environments, UAV deployment schemes need to be scheduled in a timely manner to achieve ideal performance. This section will review and analyze relevant work on UAV deployment optimization in UAV-assisted MEC systems. Current approaches to adaptive UAV deployment mainly include heuristic and learning-based optimization methods.

[0003] Existing research on UAV deployment optimization largely employs heuristic algorithms. However, the selection of UAV deployment schemes faces the combinatorial explosion problem, which these heuristic algorithms cannot solve in a short time. Because a large number of iterations are required to obtain an approximate optimal solution, heuristic-based methods are generally unsuitable for real-time MEC applications in dynamic environments. To address this issue, some deep reinforcement learning (DRL)-based methods have been applied to UAV scheduling strategies. These methods typically adopt a centralized approach, modeling all UAVs as a single agent, which means a control center is needed to collect global information and control all agents. This approach is difficult to scale because the system state and action space grow exponentially with the number of UAVs or MDs, and requires high communication overhead during execution. Another approach to applying single-agent DRL to multi-agent scenarios is to simply train each agent independently using DRL methods. This method generates an unstable environment because each agent's policy changes during training. Therefore, distributed DRL methods are more suitable for large-scale UAV-assisted MEC communication systems in dynamic environments. However, existing distributed methods generally give less consideration to UAV deployment optimization and latency constraints. Furthermore, the extensive communication and interaction required between agents can increase risks to data security and privacy. Federated learning allows agents to train on local data, collecting and distributing only the parameters of the neural network, thereby protecting the agents' data security.

[0004] Given the large number of drones and mobile devices in the system, traditional centralized algorithms model all UAVs as a single agent. This leads to problems such as unscalable models and additional communication overhead, making it difficult to effectively achieve adaptive deployment of large-scale drones. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a drone deployment optimization method based on multi-agent federated reinforcement learning, so as to achieve efficient and adaptive drone deployment.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A method for optimizing drone deployment based on multi-agent federated reinforcement learning includes the following steps:

[0008] Step S1: Model each UAV in the multi-UAV assisted MEC system as an intelligent agent. Each UAV makes independent decisions based on local information collected in the runtime environment. By designing the state space, action space, transition function, and reward function, the dynamic deployment problem of UAVs is modeled as a Markov problem model.

[0009] Step S2: Train the Q-value prediction federated model for UAV deployment operations using the K-MAFRL algorithm. Step S3: During the execution phase, each UAV uses the UAV deployment operation Q-value prediction federated model obtained in Step S2 to predict the Q-value of different deployment operations based on local information in its runtime environment. By comparing the Q-values, a suitable UAV deployment operation is selected. Through feedback control and multi-UAV collaboration, an effective large-scale UAV deployment scheme is gradually found.

[0010] Furthermore, the multi-UAV-assisted MEC system consists of a two-layer structure: a ground layer composed of M MDs and an air layer composed of N UAVs; all MDs and UAVs in the multi-UAV-assisted MEC system are represented by M = {1,2,…,M} and N = {1,2,…,N}, respectively; a three-dimensional Euclidean coordinate system is used to represent the positions of MDs and UAVs within the multi-UAV-assisted MEC system;

[0011] The coordinates of MD m∈M are represented as in and Used to indicate the position of MD m on the horizontal plane; the deployment positions of all UAVs in the multi-UAV assisted MEC system are used as... It means that, among them For the deployment location of UAV n, and This indicates the deployment location of UAV n on the horizontal plane, where H represents the flight altitude of the UAV.

[0012] Furthermore, the target service area is divided into I×J equal-area units, with each UAV deployed above the center of each unit at a height of H, and the UAV only moves between the units.

[0013] Computational task A of MD m∈M m Use a positive tuple m D m O m > indicates that I m A represents m The size of the input data, in bits; D m A represents m The computational complexity, i.e., the number of CPU cycles required to compute each bit of input data, is expressed in cycles per bit; O m A represents m The size of the calculation result is in bits.

[0014] The wireless channel between MD m∈M and UAVn∈N is simulated using a free-space path loss model, denoted as:

[0015]

[0016] in Let φm represent the spatial distance between MDm and UAVn, and φ0 represent the wireless channel gain per unit spatial distance; the transmission rate of the wireless channel between MDm and UAVn is φ0.

[0017]

[0018] Where β represents the wireless channel bandwidth, P m Let σ represent the transmit power of the m-th MD. 2 This represents the Gaussian white noise power in the wireless channel.

[0019] Considering that all UAVs and MDs provide computing services, computing tasks on the MD can be offloaded to the UAV for execution or executed locally; define a binary variable ξ = {ξ m,n |m∈{1,...,M},n∈{0,1,...,N}} represents the access relationship between MD and UAV, where ξ m,n ξ ∈{0,1} represents the access relationship between MD m and UAV n. When MD m decides to offload the computation task to UAV n for execution, ξ m,n =1, otherwise ξ m,n =0; in particular, when ξ m,0 ​When = 1, the computation task is executed locally; assume that each MD offloads the computation task to only one UAV for execution, i.e.

[0020]

[0021] Assume that the number of computational tasks received by each UAV within the considered time slot cannot exceed the maximum number of parallel tasks N. max That is, the following constraints apply.

[0022]

[0023] Furthermore, the access strategy for the MD and UAVs in the multi-UAV assisted MEC system is as follows: For each MD, if the execution time of the computation task locally is less than the computation time offloaded to the nearest UAV, then the task is executed locally; otherwise, MD m attempts to offload the task to UAV n; if this offloading operation exceeds the service limit N of the UAV... max Then, search for the MD that is furthest from UAVn that has been connected and execute it locally. The final MD-UAV access relationship ξ can be obtained by applying a greedy strategy to each user.

[0024] Furthermore, the objective function of the multi-UAV-assisted MEC system is constructed as follows:

[0025] Define the CPU clock frequency f during the local execution of the computation task m∈M. m L Then calculate task A m The local computation latency of the input data is quantized as follows:

[0026]

[0027] Computation Task A m The total latency calculated on UAV n is:

[0028]

[0029] Where I m / R m,n Indicates the transmission delay of task input data from MD m to UAVn; I m D m / f m,n Represents computation task A m Execution latency on UAV; f n U Indicates that UAVn is executing task A m CPU clock frequency at that time;

[0030] Computation Task A m The execution completion time is:

[0031]

[0032] Minimizing the average task response time is taken as the optimization objective, and the objective function is defined as follows:

[0033]

[0034]

[0035]

[0036]

[0037]

[0038]

[0039] Among them, constraints C1 and C2 represent the deployment scope constraints of UAVs; constraint C3 indicates that the task adopts a complete offloading strategy; constraint C4 indicates that each MD will only offload the computing task to one UAV for execution or execute it locally; constraint C5 indicates that the number of tasks received by each UAV cannot exceed its maximum parallel processing capacity.

[0040] Furthermore, the Markov problem model is defined as a 4-tuple, using...<S、A、R、P> Let S be the state space, A be the action space, R be the reward function, and P be the state transition function. Based on the problem formalization described in Section 3, the state space S, action space A, state transition function P, and reward function R for each agent are defined as follows:

[0041] State space: The state space of the UAVn agent is denoted by s. n Indicates; s n Defined as a 3-tuple meaning <f n U B n C n >, which includes the computing power f of UAVn n U Distribution of other drones within the current UAV's line of sight (B) n and the distribution of ground users C n；

[0042] Action Space: The action space of the UAV is discretized and five actions are defined. Therefore, the action space of UAVn is defined as a. n ={0,1,2,3,4}, which includes actions that horizontally distribute the direction of motion to the four flight directions of east, south, west and north, and a vertical hovering action. The hovering action allows the UAV to maintain its current state after finding a suitable position.

[0043] State transition function: State transition function P(a n ,s n ,s' n Record the given current state s n and applied to s n The current action a n Next state s' n The probability; the specific transition function is expressed as

[0044] P(a n ,s n ,s' n )=Pr(s' n |s n ,a n )

[0045] Reward Function: In the multi-UAV deployment optimization problem, each UAV has the same objective, namely minimizing the system's average response time. Therefore, the reward function for UAVn is defined as follows:

[0046] r n =TT'

[0047] Where r n This indicates that UAVn is in its current state s. n Next, execute action a n The reward obtained by the agent is then given, where T and T' represent the system response time before and after the deployment scheme is executed, respectively.

[0048] Furthermore, during the training process of the Markov problem model, each agent obtains the state s from the MEC environment at runtime, and then uses the optimal policy π. * Choose action a; the optimal strategy π for the MDP problem. * Represented as

[0049] π * (s,a)=argmaxQ(s,a;ω)

[0050] After the agent performs action 'a', it receives a corresponding reward value 'r' and the next state 's' from the environment. Next, based on the DQN algorithm, the (s,a,r,s') obtained at each step are stored in the experience pool. When the storage threshold is reached, the neural network parameters are updated. The loss function of the neural network is as follows:

[0051] Loss=(r+γmaxQ(s',a';ω')-Q(s,a;ω)) 2

[0052] Where r is the reward value, γ is the discount factor, maxQ(s',a';ω') is the output of TargetNet, used to calculate the maximum Q value obtained by choosing action a' in the next state s', and Q(s,a;ω) is the output of EvalNet, used to calculate the Q value of the current state-action pair.

[0053] Furthermore, the K-MAFRL algorithm comprises two basic parts:

[0054] (1) Distributed training: Each UAV agent uses the standard DQN algorithm to interact with the environment to train its local model; after local training is completed, network parameters are extracted from each agent and sent to the central aggregation unit.

[0055] (2) Federated aggregation: When the central unit receives the network parameters of each agent, it aggregates these network parameters to generate a global model, and then transmits its model parameters back to all agents to update their local networks.

[0056] Furthermore, model aggregation is performed using the FedAvg method:

[0057]

[0058] Where, ω g and ω n These represent the global network parameters and the local network parameters of the nth UAV agent, respectively.

[0059] Furthermore, step S3 specifically includes:

[0060] Randomly initialize the network weights of EvalNet and TargetNet for each UAV agent, and initialize the starting position of the UAV in each round;

[0061] During local model training, each agent interacts with the runtime environment in different scenarios, observes the local state around the current UAV, and uses policy π. * Select flight action a n Then calculate the execution action a. n The reward r obtained later n And obtain the next state s' n ;

[0062] Next, (s) n ,a n ,r n ,s n The samples are stored in an experience pool, where a minimum batch of samples is randomly drawn and the target Q-value is calculated. These samples may come from different runtime environments to ensure sufficient learning.

[0063] Then, gradient descent is performed according to the mean square loss function to update the weights ω of the EvalNet neural network, and the weights of the TargetNet neural network are updated after reaching the set C rounds of iterations;

[0064] Next, the central aggregation unit receives network parameters from all agents and trains the global network. Specifically, the uploaded parameters are fitted to the global Q-network using a federated averaging algorithm.

[0065] Finally, the central aggregation unit distributes the parameters of the global Q network to each agent, and each agent updates its local network accordingly.

[0066] Compared with the prior art, the present invention has the following advantages:

[0067] This invention uses reinforcement learning combined with federated learning to study the deployment optimization problem of UAVs in large-scale MEC scenarios under dynamic environments, thereby achieving efficient and adaptive UAV deployment to minimize the average task response time of the system. Attached Figure Description

[0068] Figure 1 This is the framework of the method of the present invention;

[0069] Figure 2 This is a multi-UAV-assisted MEC system in one embodiment of the present invention;

[0070] Figure 3 This is a performance comparison of K-MAFRL and the classical algorithm under varying MDs scale in one embodiment of the present invention;

[0071] Figure 4 This is an embodiment of the invention showing the impact of the number of drones on the average task response time;

[0072] Figure 5 This is an embodiment of the invention showing the impact of the number of MDs on the average task response time;

[0073] Figure 6 This is an embodiment of the invention showing the impact of the UAV initial setting algorithm on the average task response time. Detailed Implementation

[0074] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0075] Please refer to Figure 1 This invention provides a method for optimizing drone deployment based on multi-agent federated reinforcement learning, comprising the following steps:

[0076] Step S1: Model each UAV in the multi-UAV assisted MEC system as an intelligent agent. Each UAV makes independent decisions based on local information collected in the runtime environment. By designing the state space, action space, transition function, and reward function, the dynamic deployment problem of UAVs is modeled as a Markov problem model.

[0077] Step S2: Train the Q-value prediction federated model for UAV deployment operations using the K-MAFRL algorithm. Step S3: During the execution phase, each UAV uses the UAV deployment operation Q-value prediction federated model obtained in Step S2 to predict the Q-value of different deployment operations based on local information in its runtime environment. By comparing the Q-values, a suitable UAV deployment operation is selected. Through feedback control and multi-UAV collaboration, an effective large-scale UAV deployment scheme is gradually found.

[0078] refer to Figure 2 In this embodiment, the multi-UAV-assisted MEC system consists of a two-layer structure: a ground layer composed of M MDs and an airborne layer composed of N UAVs. All MDs and UAVs in the system are represented by M = {1, 2, ..., M} and N = {1, 2, ..., N}, respectively. We use a three-dimensional Euclidean coordinate system to represent the positions of the MDs and UAVs within the system. The coordinates of an MD m ∈ M can be represented as... in and This is used to represent the position of MD m on the horizontal plane. The deployment locations of all UAVs in the system are represented by... It means that, among them For the deployment location of UAV n, and The location of UAV n on the horizontal plane is represented by H, where H represents the flight altitude of the UAV. Since the available deployment options for UAVs increase exponentially with their number, to simplify the problem model, we divide the target service area into I×J equal-area cells. Each UAV is deployed above the center of its respective cell at an altitude of H, and the UAV only moves between cells. We assume that the UAV hovers at a fixed altitude H>0 during the computational task processing, where H>0 is the minimum altitude at which the UAV adapts to the working terrain and avoids obstacles, and that the UAV does not require frequent ascents and descents during flight.

[0079] Computational task A of MD m∈M m A positive tuple can be used m D m O m > indicates that I m A represents m The size of the input data (such as program code and input parameters), in bits; D​m A represents m The computational complexity, i.e., the number of CPU cycles required to compute each bit of input data, is expressed in cycles per bit; O m A represents m The size of the computation result is measured in bits. The characteristics of a computation task are determined by the device that generates it; that is, the input data size, computational complexity, and result size are all different for each computation task.

[0080] The wireless channel between MD m∈M and UAVn∈N can be simulated using a free-space path loss model, expressed as:

[0081]

[0082] in Let φm represent the spatial distance between MDm and UAVn, and φ0 represent the wireless channel gain per unit spatial distance. The transmission rate of the wireless channel between MDm and UAVn is...

[0083]

[0084] Where β represents the wireless channel bandwidth, P m Let σ represent the transmit power of the m-th MD. 2 This represents the Gaussian white noise power in the wireless channel.

[0085] Considering that all UAVs and MDs can provide computing services, computing tasks on MDs can be offloaded to UAVs for execution or executed locally. Define a binary variable ξ = {ξ m,n |m∈{1,...,M},n∈{0,1,...,N}} represents the access relationship between MDs and UAVs, where ξ m,n ξ ∈{0,1} represents the access relationship between MD m and UAVn. When MD m decides to offload the computing task to UAVn for execution, ξ m,n =1, otherwise ξ m,n =0. Specifically, when ξ m,0 When = 1, the computation task is executed locally. We assume that each MD offloads the computation task to only one UAV for execution, i.e.

[0086]

[0087] Furthermore, due to the limited parallel computing capabilities of UAVs, we assume that the number of computing tasks received by each UAV within the considered time slot cannot exceed the maximum number of parallel tasks N. max That is, the following constraints apply.

[0088]

[0089] In this embodiment, preferably, the access strategy for MDs and UAVs assumes the following scheme: After the deployment location of UAVs is determined, a greedy offloading strategy is used to determine the MD-UAV access relationship, offloading the computing tasks of MDs to the UAV closest to the current MD to reduce offloading time. Specifically, the process is described as follows: traverse each MD; if the execution time of a computing task locally is less than the computing time offloaded to the nearest UAV, then the task is executed locally; otherwise, MD m attempts to offload the task to UAV n. Considering the limited computing power of UAVs, they can only provide offloading services for a certain number of MDs. If the offloading operation exceeds the service limit N of the UAVs... max Then, search for the MD that is furthest from UAV n that has been connected and execute it locally. The final MD-UAV connection relationship ξ can be obtained by executing a greedy strategy for each user.

[0090] Assuming all MDs employ a full offload strategy, the computational tasks of MDs can be offloaded to UAVs for execution or executed locally. We define the CPU clock frequency during the local execution of the computational task of MD m∈M as... (in Hz), then calculate task A m The local computation latency of the input data can be quantified as

[0091]

[0092] When the MD (Multi-Target Machine) offloads the input data of the computation task to the UAV (User Auxiliary Vehicle) for execution, the resulting latency consists of three parts: 1) the communication latency incurred by transmitting the input data from the MD to the UAV; 2) the computation latency incurred by executing the input data on the UAV; and 3) the communication latency incurred by transmitting the computation results back from the UAV to the MD. In particular, compared to the input data, the size of the computation results is often very small, and the communication latency incurred by outputting the computation results is very small, so we ignore the communication latency incurred in this process. Ignoring the process of transmitting the computation results back can greatly reduce the complexity of the model, which has been adopted by many research institutes. Therefore, computation task A... m The total latency calculated on UAV n is

[0093]

[0094] Where I m / R m,n Indicates the transmission delay of task input data from MD m to UAVn; I m D m / f m,n Represents computation task A m Execution latency on UAV; fn U Indicates that UAVn is executing task A m CPU clock frequency at that time.

[0095] In summary, computational task A m The execution completion time is

[0096]

[0097] Different UAV and user locations will result in different task execution times. To achieve better computational offloading in a multi-UAV assisted MEC environment, we take minimizing the average task response time as the optimization objective, and define the objective function as follows:

[0098]

[0099] Among them, constraints C1 and C2 represent the deployment scope constraints of UAVs; constraint C3 indicates that the task adopts a complete offload strategy; constraint C4 indicates that each MD will only offload the computing task to one UAV for execution or execute it locally; constraint C5 indicates that the number of tasks received by each UAV cannot exceed its maximum parallel processing capacity.

[0100] The local information of UAV n includes the deployment location of the UAV (represented as...). The drone's computing power (denoted as f) n U The distribution of other UAVs within the visual range of this drone (denoted as B). n ) and the distribution of ground mobile equipment (represented as C) n As shown in Table 1. Specifically, we divide the visible area of ​​UAV n into L×L units, assuming... in Let (i,j) represent the number of other UAVs distributed within cell (i,j); in This represents the number of MDs distributed within unit (i,j). Furthermore, T represents the average system task response time of the current overall UAVs deployment scheme; a smaller T indicates a better deployment scheme. T' represents the time taken to execute deployment action a on UAV n. n The average task response time of the system after that.

[0101] Table 1. Local information and performance metrics of UAVn in the runtime environment.

[0102]

[0103] In this embodiment, the Deep Q-Networks (DQN) algorithm is used to evaluate the Q-values ​​of different UAV deployment operations to explore UAV deployment schemes. Based on the DQN algorithm, each UAV is modeled as an agent and can make deployment decisions based on local information in the runtime environment. By training the DQN agent, the deployment operation with the highest Q-value under different system states can be found, where the DQN algorithm is used to control the Q-value evaluation process. The goal of Reinforcement Learning (RL) is to maximize cumulative rewards, which is typically modeled using Markov Decision Processes (MDPs). More specifically, an MDP can be defined as a set of four tuples...<S、A、R、P> Let S be the state space, A be the action space, R be the reward function, and P be the state transition function. Based on the problem formalization described in Section 3, the state space S, action space A, state transition function P, and reward function R for each agent are defined as follows:

[0104] State space: The state space of the UAVn agent is denoted by s. n This indicates that, in order to comprehensively consider the characteristics of the operating environment and the drone deployment scheme, s n Defined as a 3-tuple meaning <f n U B n C n >, which includes the computing power f of UAVn n U Distribution of other drones within the current UAV's line of sight (B) n and the distribution of ground users C n .

[0105] Action Space: The action space of UAVs is discretized and five actions are defined. Therefore, the action space of UAVn is defined as a. n ={0,1,2,3,4}, which includes actions that distribute the direction of motion horizontally to the four flight directions of east, south, west, and north, and a vertical hovering action. The hovering action allows the UAV to maintain its current state after finding a suitable position.

[0106] State transition function: State transition function P(a n ,s n ,s' n Record the given current state s n and applied to s n The current action a n Next state s' n The probability of that. The specific transition function can be expressed as:

[0107] P(a n ,sn ,s' n )=Pr(s' n |s n ,a n ) Formula (3-1)

[0108] Reward Function: In the multi-UAV deployment optimization problem, each UAV has the same objective, namely minimizing the system's average response time. Therefore, the reward function for UAV n is defined as follows:

[0109] r n =TT' formula (3-2)

[0110] Where r n This indicates that UAVn is in its current state s. n Next, execute action a n The reward received by the agent is then calculated, where T and T' represent the system response time before and after the deployment action, respectively. Therefore, T – T' is a negative value. To maximize the cumulative reward, the algorithm aims to achieve the minimum response time.

[0111] During training, each DQN agent acquires state s from the MEC environment at runtime, and then uses the optimal policy π. * Choose action 'a'. Typically, the optimal strategy π for the MDP problem... * It can be represented as

[0112] π * (s,a)=argmaxQ(s,a;ω) Formula (3-3)

[0113] After a DQN agent performs action 'a', it receives a reward value 'r' and the next state 's' from the environment. The DQN algorithm then stores the (s, a, r, s') obtained at each step into an experience pool. Typically, the capacity of the experience pool in the DQN algorithm is pre-set. When the storage threshold is reached, the neural network parameters are updated. The loss function of the neural network is as follows:

[0114] Loss=(r+γmaxQ(s',a';ω')-Q(s,a;ω)) 2 Formula (3-4)

[0115] Where r is the reward value, γ is the discount factor, maxQ(s',a';ω') is the output of TargetNet, used to calculate the maximum Q value obtained by choosing action a' in the next state s', and Q(s,a;ω) is the output of EvalNet, used to calculate the Q value of the current state-action pair.

[0116] In this embodiment, during the training phase, the K-MAFRL algorithm allows agents to train their local models distributedly, and then send these local models to a central aggregation unit to build a global network. Unlike centralized algorithms, this communication only requires uploading the model parameters of each agent, which greatly reduces communication load and latency. All agents' local models in the K-MAFRL algorithm have the same network structure and are aggregated into a global model through federated averaging. The K-MAFRL algorithm consists of two basic parts:

[0117] (1) Distributed training. Each UAV agent uses the standard DQN algorithm to interact with the environment to train its local model. After local training is complete, network parameters are extracted from each agent and sent to the central aggregation unit.

[0118] (2) Federated Aggregation. When the central unit receives the network parameters from each agent, it aggregates these parameters to generate a global model, and then transmits its model parameters back to all agents to update their local networks. We use the FedAvg method to perform model aggregation:

[0119]

[0120] Where, ω g and ω n These represent the global network parameters and the local network parameters of the nth UAV agent, respectively.

[0121] Based on the above definitions, we use the K-MAFRL algorithm to train a federated model to evaluate the Q-value of all UAV deployment operations. The key steps of this algorithm are shown in Algorithm 1. First, the network weights of EvalNet and TargetNet for each UAV agent are randomly initialized (lines 3-5), and the starting position of the UAV is initialized in each round according to Algorithm 2. During the local model training process, each agent interacts with the runtime environment under different scenarios, observes the local state around the current UAV, and uses policy π. * Select flight action a n (Line 10). Then, calculate the action a using Formula 9. n The reward r obtained later n And obtain the next state s' n (Line 11). Next, (s n ,a n ,r n ,s nThe samples are stored in an experience pool, where a minimum batch of samples is randomly drawn and the target Q-value is calculated. These samples may come from different runtime environments to ensure sufficient learning (lines 12-13). Then, gradient descent is performed according to the mean squared loss function to update the EvalNet neural network weights ω, and the TargetNet neural network weights are updated after reaching the set C iterations (lines 14-16). Next, the central aggregation unit receives network parameters from all agents and trains the global network; specifically, the uploaded parameters are fitted to the global Q-network using a federated averaging algorithm (lines 18-19). Finally, the central aggregation unit distributes the parameters of the global Q-network to the individual agents, and each agent updates its local network accordingly (line 20).

[0122]

[0123] 1.1 Runtime Decision-Making Mechanism

[0124] In general, the density of MDs distribution determines the computational resource requirements of the area. Based on the current ground distribution of MDs, we use the K-means algorithm to initially position UAVs. The main steps are shown in Algorithm 2. First, set the number of K-means clusters to N and initialize the cluster centroids (line 3). Then, calculate the distance of each MD to the cluster centroid and assign it to the nearest cluster (lines 5-8). Next, update the cluster centroids according to the new clusters (lines 9-11). Iterate the above operations until the maximum number of iterations I, and initialize the starting position of UAVs with the final clustering result.

[0125]

[0126] During the execution phase, the deployment process of drones involves two steps: First, considering the impact of the initial position of the drones on the results, we use a clustering algorithm to generate the initial positions of the drones; then, using a model trained with the K-MAFRL algorithm, each drone can make independent decisions based on local information, and gradually find an effective deployment scheme through feedback control and multi-drone collaboration.

[0127] The decision-making process for UAV deployment operations is performed at runtime. This invention proposes a runtime decision-making algorithm that gradually finds a reasonable UAV deployment scheme through feedback control mechanisms and multi-UAV collaboration. Its main steps are shown in Algorithm 3. First, Algorithm 2 is used to initialize the UAV position based on the current distribution of MDs. Then, the local information of UAVn is updated (line 6). Since the local information of UAVn changes continuously at runtime, it should be re-acquired before each round of decision-making. The Q-value of each deployment operation of UAVn is evaluated by calling the Q-value prediction federated model, and the action corresponding to the maximum Q-value is obtained (lines 7-8). If the obtained action value is 0, UAVn remains hovering and no further adjustments are made (lines 9-10). Otherwise, the deployment position of UAVn will continue to be adjusted (line 12).

[0128]

[0129] Through an iterative feedback control process, large-scale drone deployment can be achieved incrementally in the runtime environment. Feedback control will continue until all drones reach their optimal deployment positions.

[0130] Example 1:

[0131] This embodiment evaluates the derived K-MAFRL through the following research questions (RQs).

[0132] RQ1: How much performance improvement does K-MAFRL offer compared to other commonly used baseline algorithms?

[0133] •RQ2: How do the number of drones and mobile devices of different sizes affect the methodology?

[0134] RQ3: When using K-MAFRL to solve the average response time optimization problem, is the initial positioning algorithm for UAVs effective?

[0135] For RQ1, experimental results show that the average task response time of the UAV deployment schemes obtained by the K-MAFRL method in different scenarios is reduced by an average of 2.47%, 4.22%, 5.76%, and 8.91% compared to commonly used benchmark methods PSO-GA, CDRL, Greedy, and Random, respectively. Furthermore, K-MAFRL can obtain UAV deployment schemes in a shorter time, with execution time reduced by an average of 99.3%, 13.4%, 67.7%, and 99.4% compared to PSO-GA, CDRL, Greedy, and Random algorithms, respectively. For RQ2, in large-scale experiments, the average response time of K-MAFRL is superior to other algorithms; however, the performance gap gradually increases with the number of UAVs and MDs. For RQ3, comparing K-MAFRL with MAFRL using random initialization, the average task response time of the UAV deployment schemes obtained by K-MAFRL is reduced by an average of 1.70% compared to MAFRL, and the execution time of K-MAFRL is reduced by an average of 24.3% compared to MAFRL.

[0136] In this embodiment, to simulate the diversity of application scenarios, we considered four different square experimental areas, each with a size of 400*400m. 2 600*600m 2 800*800m 2 1000*1000m 2 The microcontrollers (MDs) were randomly distributed within the experimental area. Furthermore, different sizes of UAVs and MDs were set for different experimental areas, as shown in the table. The CPU cycle frequency of the MDs distributed in this area was 0.8–1.2 GHz, and the CPU cycle frequency of all UAVs was set to 2.5–3.5 GHz. The input data size for all computational tasks in the system was randomly distributed within the range of 10–20 Mbit, and the computational task complexity was set to 80–120 cycles / bit. The transmit power of the MDs was set to 1W, the wireless channel gain per unit spatial distance was set to -20 dB, the wireless channel bandwidth was set to 10 MHz, and the Gaussian white noise power was -1 × 10⁻¹⁰. -5 dBm.

[0137] Table 2 Experimental Scale Settings

[0138] <![CDATA[Experimental area size / m 2 > Number of UAVs N Number of MDs M 400＊400 8 40 600＊600 18 90 800＊800 32 160 1000＊1000 50 250

[0139] The proposed K-MAFRL is implemented using TensorFlow 2.3.0. K-MAFRL uses a fully connected DNN network consisting of one input layer, two hidden layers, and one output layer, where the two hidden layers have 256 and 128 hidden neurons, respectively. The empirical pool size M...c The training batch size m, discount factor γ, and learning rate α of the Adam optimizer were set to 15000, 64, 0.9, and 0.001, respectively. Furthermore, we trained the federated model for 10000 epochs in four different experimental regions, demonstrating that the model is applicable to scenarios with varying region sizes.

[0140] To verify the performance advantages of the K-MAFRL algorithm, we compared it with other commonly used benchmark methods in UAV-assisted MEC systems. The relevant comparison algorithms are described below:

[0141] Centralized DQN: Centralized DQN makes centralized decisions for all UAVs based on global information collected from the runtime environment. The state space contains information about all UAVs and MDs, and the action space is set as the set of deployment actions of all UAVs. The learning rate α, discount factor β, experience pool size M, and number of training epochs are set to 0.001, 0.9, 10000, and 10000, respectively. Centralized DQN requires training the model according to different scenarios and experimental scales.

[0142] • PSO-GA: Improves the particle update strategy of the traditional PSO algorithm by introducing crossover and mutation operators from a genetic algorithm. Each particle represents a deployment scheme for all UAVs in the system, and each quantile of the particle encodes the deployment position of each UAV. The start and end values ​​of the two learning factors c1 and c2, and the maximum and minimum values ​​of the inertia weight w are set to 0.9, 0.2, 0.9, 0.4, 0.9, and 0.4, respectively. The number of iterations and the population size are set to 1000 and 50, respectively.

[0143] Greedy Algorithm: Based on the current deployment locations of UAVs, each UAV selects one action from all actions that minimizes the average task response time. The location of each UAV is continuously updated until no immediate action of any UAV can achieve a better response time.

[0144] • Random algorithm: All UAVs are randomly selected for deployment within the service area, and the optimal value from 20,000 repeated experiments is taken as the final result.

[0145] Figure 3 The figure shows a comparison of the average response time of the proposed K-MAFRL algorithm with other benchmark algorithms in different scenarios. As can be seen from the figure, in the four experimental regions, the response time of K-MAFRL is reduced by an average of 2.47% compared to PSO-GA. Specifically, this is achieved when the region size is 400*400m. 2In the first case, K-MAFRL's response time was on average 3.09% higher than PSO-GA's, while in the other three experimental regions of different sizes, K-MAFRL's response time was 2.07%–6.55% lower than PSO-GA's. This is because the quality of solutions obtained by PSO-GA can be affected by the size of the solution space. If the solution space is small, PSO-GA can obtain high-quality solutions through crossover and mutation operations. For example, when the region size is 400*400m... 2 In general, PSO-GA can search for optimal solutions in various scenarios. However, when faced with a large solution space, PSO-GA is prone to getting trapped in local optima and failing to reach the optimal solution. For example... Figure 3 As shown, K-MAFRL reduced the task response time by an average of 4.22% compared to CDRL in different experimental regions. Specifically, when the region size was 400*400m... 2 In the first experiment, K-MAFRL and CDRL achieved equivalent performance. However, in the other three experimental regions of different sizes, K-MAFRL outperformed CDRL by 3.14–8.12%. CDRL centrally deployed UAVs based on global information, and when the experimental scale was small, CDRL could cover more comprehensive state information during training. However, as the experimental scale increased, large-scale UAVs and mobile users meant a larger system state space and action space, which made the training task more difficult and thus made it difficult to show better performance. The figure also shows that K-MAFRL outperformed Greedy and Random in different experimental regions, reducing response time by 2.65–9.25% and 5.11–13.05% respectively. For the Greedy algorithm, since each optimization only considers the current optimal deployment operation, the calculated solution is generally a local optimum, thus its optimization effect on average task response time is limited. The Random algorithm simply deploys drones randomly, completely ignoring the impact of the location of MDs and the size of the computation task on the task response time, so its performance is poor.

[0146] As shown in Table 3, we evaluated the execution time required for five methods to obtain UAV deployment schemes in experimental areas of different sizes. The experimental results are the average time cost for the four scenarios. In comparison, the K-MAFRL algorithm takes the least time, reducing the time required to obtain UAV deployment schemes by an average of 99.3%, 13.4%, 67.7%, and 99.4% compared to PSO-GA, CDRL, Greedy, and Random algorithms, respectively. The CDRL algorithm uses global information as input, and as the experimental scale increases, the high-dimensional state space increases the complexity of the algorithm's execution. The Greedy algorithm needs to exhaustively enumerate the performance of deployment schemes corresponding to all UAV actions during execution until a local solution is reached only after all instantaneous actions of all UAVs fail to yield a better deployment scheme, thus requiring a longer execution time. Furthermore, the PSO-GA and Random algorithms require a large number of iterations to ensure the quality of the obtained solution, therefore their execution time is much longer than the other three algorithms. In summary, the proposed K-MAFRL algorithm can obtain higher-quality large-scale UAV deployment schemes in a shorter time and has higher application value.

[0147] Table 3. Time (s) required to find the target deployment plan

[0148] AreaSize K-MAFRL PSO-GA CDRL Greedy Random <![CDATA[400*400m 2 ]]> 1.993 274.135 1.765 2.773 289.027 <![CDATA[600*600m 2 ]]> 4.116 655.028 5.081 8.327 687.278 <![CDATA[800*800m 2 ]]> 8.033 1212.529 10.390 18.323 1346.536 <![CDATA[1000*1000m 2 ]]> 14.377 2286.316 18.121 44.537 2523.568

[0149] Next, we further evaluated the performance of the proposed method at different experimental scales (e.g., different scales of UAVs and MDs), using the same comparison algorithm settings as in RQ1. Figure 4 The graph shows the impact of different numbers of drones on the average task response time of the algorithm. As can be seen from the graph, when the number of MDs (M) remains constant, the average response latency decreases as the number of drones increases. This phenomenon is because when the number of MDs to be served remains constant while the number of drones increases, the computational tasks of the MDs can be offloaded to more drones. Simultaneously, the distance between the MDs and the UAVs providing the service also decreases, thus reducing the task offloading time accordingly. We can also see that within a 600*600m... 2 800*800m 2 1000*1000m 2In the experimental region, when the number of UAVs is greater than 15, the average response time of the proposed K-MAFRL is superior to other algorithms. Furthermore, the performance gap gradually widens as the number of UAVs increases. This is because in a large-scale UAV environment, each UAV in K-MAFRL makes deployment decisions independently based on local information, which can adapt to changes in the number of UAVs. In contrast, the state space and action space of CDRL grow exponentially with the number of UAVs, while the performance of other traditional methods gradually decreases due to the increase in solution space.

[0150] Figure 5 This diagram illustrates the relationship between different numbers of Management Devices (MDs) and average task response time. Experimental results show that when the number of UAVs (N) in the system remains constant, the average response latency increases with the increase in the number of MDs. This is because as the number of MDs increases, each UAV needs to provide computing services for more MDs. However, the limited computing resources of UAVs cause MD tasks to be offloaded to more distant UAVs or executed locally, thus increasing the average task response time. As can be seen from the figure, within a 600*600m... 2 800*800m 2 1000*1000m 2 In the experimental region, when the number of MDs is greater than 70, the average task response time of K-MAFRL is consistently lower than that of other methods. This is because in large-scale MD environments, each UAV in K-MAFRL only observes local information in the environment and then shares data through model aggregation. The state space setting is more adaptable to large-scale MD scenarios. In contrast, the performance of PSO-GA, CDRL, Greedy, and Random algorithms gradually lags behind K-MAFRL because the state space of CDRL increases with the number of MDs, which significantly reduces the convergence efficiency of the method; the complex environment of more MDs also greatly limits the performance of other traditional methods.

[0151] Finally, we compared K-MAFRL with MAFRL for UAVs using random initialization to evaluate the effectiveness of the initialization algorithm in the method, with the experimental environment being the same as the scenario in RQ1. Figure 6The average task response time of the two methods was compared under different system scenarios. As shown in the figure, the average task response time obtained by K-MAFRL is reduced by 1.70% compared to MAFRL, indicating that the method based on K-means initial positioning can find a more suitable UAV deployment scheme. Furthermore, we compared the average time cost of the two methods in finding the UAV deployment scheme, as shown in Table 4. Experimental results show that the execution time of K-MAFRL is reduced by an average of 24.3% compared to MAFRL. This is because after K-MAFRL uses K-Means for initial positioning of the UAV, it requires fewer feedback iterations to search for a better solution in the solution space. Therefore, the UAV initial positioning algorithm can enable the proposed method to obtain a better UAV deployment scheme in a shorter time.

[0152] Table 4. Time (s) required to find the target deployment plan

[0153] AreaSize <![CDATA[400*400m 2 ]]> <![CDATA[600*600m 2 ]]> <![CDATA[800*800m 2 ]]> <![CDATA[1000*1000m 2 ]]> K-MAFRL 1.924 3.788 7.485 14.590 MAFRL 2.229 5.352 10.401 19.793

[0154] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made within the scope of the claims of the present invention should be included in the scope of the present invention.

Claims

1. A method for optimizing UAV deployment based on multi-agent federated reinforcement learning, characterized in that, Includes the following steps: Step S1: Model each UAV in the multi-UAV assisted MEC system as an intelligent agent. Each UAV makes independent decisions based on local information collected in the runtime environment. By designing the state space, action space, transition function, and reward function, the dynamic deployment problem of UAVs is modeled as a Markov problem model. Step S2: Train a federated model for predicting the Q-value of UAV deployment operations using the K-MAFRL algorithm. Step S3: During the execution phase, each UAV uses the drone deployment operation Q-value prediction federated model obtained in step S2 to predict the Q-value of different deployment operations based on local information in its runtime environment. By comparing the Q-values, it selects the appropriate drone deployment operation and gradually finds an effective large-scale drone deployment scheme through feedback control and multi-drone collaboration. The Markov problem model is defined as a 4-tuple, using...<S、A、R、P> Let S be the state space, A be the action space, R be the reward function, and P be the state transition function. The state space S, action space A, state transition function P, and reward function R corresponding to each agent are defined as follows: State space: The state space of the UAV n-agent is represented by s. n Indicates; s n Defined as a 3-tuple meaning This includes the computing power of UAV n Distribution of other drones within the current UAV's line of sight (B) n and the distribution of ground users C n ; Action Space: The action space of the UAV is discretized and five actions are defined. Therefore, the action space of UAV n is defined as a. n ={0,1,2,3,4}, which includes actions that horizontally distribute the direction of motion to the four flight directions of east, south, west and north, and a vertical hovering action. The hovering action allows the UAV to maintain its current state after finding a suitable position. State transition function: State transition function P(a n ,s n ,s' n Record the given current state s n and applied to s n The current action a n Next state s' n The probability; the specific transition function is expressed as P(a n ,s n ,s' n )=Pr(s' n |s n ,a n ) Reward Function: In the multi-UAV deployment optimization problem, each UAV has the same objective, namely minimizing the system's average response time. Therefore, the reward function for UAV n is defined as follows: r n =T-T' Where r n This indicates that UAV n is in the current state s. n Next, execute action a n The reward obtained by the agent is then given, where T and T' represent the system response time before and after the deployment scheme is executed, respectively.

2. The UAV deployment optimization method based on multi-agent federated reinforcement learning according to claim 1, characterized in that, The multi-UAV-assisted MEC system consists of a two-layer structure: a ground layer composed of M MDs and an air layer composed of N UAVs; all MDs and UAVs in the multi-UAV-assisted MEC system are represented by ={1,2,…,M} and ={1,2,…,N}, respectively. The positions of the MD and UAV within a multi-UAV assisted MEC system are represented using a three-dimensional Euclidean coordinate system; The coordinates of MD m∈ are represented as in and Used to indicate the position of MD m on the horizontal plane; the deployment positions of all UAVs in the multi-UAV assisted MEC system are used as... It means that, among them For the deployment location of UAV n, and This indicates the deployment location of UAV n on the horizontal plane, where H represents the flight altitude of the UAV.

3. The UAV deployment optimization method based on multi-agent federated reinforcement learning according to claim 2, characterized in that, The target service area is divided into I×J equal-area units. Each UAV is deployed above the center of each unit at a height of H, and the UAV only moves between the units. Computational task A of MD m∈ m Use a positive tuple m D m O m > indicates that I m A represents m The size of the input data, in bits; D m A represents m The computational complexity, i.e., the number of CPU cycles required to compute each bit of input data, is expressed in cycles per bit; O m A represents m The size of the calculation result is in bits.​ The wireless channel between MD m∈ and UAV n∈ is simulated using a free-space path loss model, denoted as: in Let represent the spatial distance between MDm and UAVn, where 0 represents the wireless channel gain per unit spatial distance; the transmission rate of the wireless channel between MDm and UAVn is . Where β represents the wireless channel bandwidth, P m This represents the transmit power of the m-th MD. 2 The power of Gaussian white noise in the wireless channel; Considering that all UAVs and MDs provide computing services, computing tasks on the MD can be offloaded to the UAV for execution or executed locally; define a binary variable ξ = {ξ m,n |m∈{1,...,M},n∈{0,1,...,N}} represents the access relationship between MD and UAV, where ξ m,n ξ ∈{0,1} represents the access relationship between MD m and UAV n. When MD m decides to offload the computation task to UAV n for execution, ξ m,n =1, otherwise ξ m,n =0; in particular, when ξ m,0 When = 1, the computation task is executed locally; assume that each MD offloads the computation task to only one UAV for execution, i.e. Assume that the number of computational tasks received by each UAV within the considered time slot cannot exceed the maximum number of parallel tasks N. max That is, the following constraints apply.

4. The UAV deployment optimization method based on multi-agent federated reinforcement learning according to claim 2, characterized in that, The access strategy for the MD and UAVs in the multi-UAV assisted MEC system is as follows: For each MD, if the execution time of the computation task locally is less than the computation time offloaded to the nearest UAV, then the task is executed locally; otherwise, MD m attempts to offload the task to UAV n; if this offloading operation exceeds the UAV's service limit N... max Then, search for the MD that is farthest from UAV n that has been connected and execute it locally; by executing a greedy strategy for each user, the final MD-UAV access relationship ξ can be obtained.

5. The UAV deployment optimization method based on multi-agent federated reinforcement learning according to claim 4, characterized in that, The objective function of the multi-UAV-assisted MEC system is constructed as follows: Define the CPU clock frequency of MD m∈ during the local execution of the computation task as Then calculate task A m The local computation latency of the input data is quantized as follows: Computation Task A m The total latency calculated on UAV n is: Where I m / R m,n This indicates the transmission delay of task input data from MD m to UAV n; I m D m / f m,n Represents computation task A m Execution latency on UAV; Indicates that UAV n is executing task A m CPU clock frequency at that time; Computation Task A m The execution completion time is: Minimizing the average task response time is taken as the optimization objective, and the objective function is defined as follows: Among them, constraints C1 and C2 represent the deployment scope constraints of UAVs; constraint C3 indicates that the task adopts a complete offloading strategy; constraint C4 indicates that each MD will only offload the computing task to one UAV for execution or execute it locally; constraint C5 indicates that the number of tasks received by each UAV cannot exceed its maximum parallel processing capacity.

6. The UAV deployment optimization method based on multi-agent federated reinforcement learning according to claim 1, characterized in that, During the training process of the Markov problem model, each agent obtains the state s from the MEC environment at runtime, and then uses the optimal policy π. * Choose action a; the optimal strategy π for the MDP problem. * Represented as p * (s,a)=argmaxQ(s,a;ω) After the agent performs action 'a', it receives a corresponding reward value 'r' and the next state 's' from the environment. Next, based on the DQN algorithm, the (s,a,r,s') obtained at each step are stored in the experience pool. When the storage threshold is reached, the neural network parameters are updated. The loss function of the neural network is as follows: Loss=(r+γmaxQ(s',a';ω')-Q(s,a;ω)) 2 Where r is the reward value, γ is the discount factor, maxQ(s',a';ω') is the output of TargetNet, used to calculate the maximum Q value obtained by choosing action a' in the next state s', and Q(s,a;ω) is the output of EvalNet, used to calculate the Q value of the current state-action pair.

7. The UAV deployment optimization method based on multi-agent federated reinforcement learning according to claim 1, characterized in that, The K-MAFRL algorithm consists of two basic parts: (1) Distributed training: Each UAV agent uses the standard DQN algorithm to interact with the environment to train its local model; after local training is completed, network parameters are extracted from each agent and sent to the central aggregation unit. (2) Federated aggregation: When the central unit receives the network parameters of each agent, it aggregates these network parameters to generate a global model, and then transmits its model parameters back to all agents to update their local networks.

8. The UAV deployment optimization method based on multi-agent federated reinforcement learning according to claim 7, characterized in that, Perform model aggregation using the FedAvg method: Where, ω g and ω n These represent the global network parameters and the local network parameters of the nth UAV agent, respectively.

9. The UAV deployment optimization method based on multi-agent federated reinforcement learning according to claim 7, characterized in that, Step S3 specifically involves: Randomly initialize the network weights of EvalNet and TargetNet for each UAV agent, and initialize the starting position of the UAV in each round; During local model training, each agent interacts with the runtime environment in different scenarios, observes the local state around the current UAV, and uses policy π. * Select flight action a n Then calculate the execution action a. n The reward r obtained later n And obtain the next state s' n ; Next, (s) n ,a n ,r n ,s n The samples are stored in an experience pool, where minimum batch samples are randomly drawn and the target Q value is calculated; these samples may come from different runtime environments to ensure sufficient learning. Then, gradient descent is performed according to the mean square loss function to update the weights ω of the EvalNet neural network, and the weights of the TargetNet neural network are updated after reaching the set C rounds of iterations; Next, the central aggregation unit receives network parameters from all agents and trains the global network. Specifically, the uploaded parameters are fitted to the global Q-network using a federated averaging algorithm. Finally, the central aggregation unit distributes the parameters of the global Q network to each agent, and each agent updates its local network accordingly.

Images