A multi-unmanned aerial vehicle assisted edge computing task offloading method and system

Abstract

The application relates to a multi-unmanned aerial vehicle (UAV) assisted edge computing task offloading method and system, and belongs to the technical field of mobile edge computing task offloading. The method comprises the following steps: constructing a multi-UAV assisted edge computing task offloading system model and an optimization problem model; based on the above model, a Markov decision process under a multi-agent environment is constructed, a reward function taking queue stability and information age minimization as targets is defined, a deep deterministic policy gradient algorithm is adopted, the optimization problem is solved through interaction between the multi-agent and the environment, and an optimal offloading strategy of a ground communication device computing task is obtained. The system comprises a ground communication device and a UAV, the UAV is in communication connection with the ground communication device, and an edge computing server arranged on the UAV is used for realizing the above method. The application not only guarantees the stability of a long-term data queue, but also obtains an optimal information age in an online mode, and solves problems such as communication congestion and poor user experience quality.

Description

Technical Field

[0001] This invention belongs to the field of mobile edge computing task offloading technology, specifically relating to a multi-UAV-assisted edge computing task offloading method and system. Background Technology

[0002] With the increasing maturity of artificial intelligence and edge computing applications, countries have established high-performance computing systems based on shared data and applied them to telemedicine, traffic flow prediction, data-intensive hotspots, and emergency disaster relief scenarios. In emergency disaster relief scenarios, however, there is a passive competition for network resources during emergency command and dispatch, leading to bandwidth limitations and a surge in communication latency. Drones collect on-site information in real time, uploading images, videos, and mission data to the command center. Large-scale simulations are then used to accurately analyze various disaster situations, continuously improving capabilities in information sharing, remote dispatch, and collaborative rescue. Figure 1 This is a diagram of the overall architecture of the emergency communication system.

[0003] In mobile edge computing (MEC) tasks, traditional methods require iterative iterations every time the pre-defined environmental state changes, wasting significant communication and computational resources and greatly reducing efficiency. However, reinforcement learning (RL) methods offer greater flexibility and are suitable for dynamic scenarios. Deep learning (DL) neural networks also possess a certain level of perception capability. Combining the perception capabilities of DL with the decision-making capabilities of RL yields the Deep Reinforcement Learning (DRL) algorithm. DRL is an important research direction in artificial intelligence and is frequently used to address perception-decision problems in complex systems.

[0004] The proposal and implementation of Federated Learning (FL) are closely related to edge computing and deep reinforcement learning. Edge computing is a prerequisite for local training in Federated Learning, while deep reinforcement learning provides the theoretical basis and core technologies. The combination and complementarity of Federated Learning and edge computing is an inevitable trend. The purpose of edge computing is to offload computational tasks from the cloud center to the edge, which perfectly matches the computational model of Federated Learning, providing favorable conditions for its implementation.

[0005] The connection between federated learning and edge computing is: 1) improving data processing efficiency in mobile edge networks and protecting user data privacy; 2) edge computing provides federated learning with a large amount of computing resources and enriches application scenarios; 3) combining edge computing and federated learning as the basic architecture of artificial intelligence.

[0006] In terms of data transmission, due to the highly dynamic nature of the environment, the privacy data of drones is more easily leaked, which can then be further analyzed to infer other information. Due to the broadcast nature of wireless channels, the identity, location, and flight trajectory information of each legitimate Unmanned Aerial Vehicle (UAV) are sensitive information, and the task unloading process is easily eavesdropped on, leading to data security and privacy risks. Therefore, federated learning has emerged to improve system performance while protecting data privacy.

[0007] However, with the surge in computationally intensive and time-sensitive tasks in large-scale intelligent mobile devices, existing computation offloading methods based on federated deep reinforcement learning in drone mobile edge computing environments cannot guarantee queue stability and information freshness in mobile edge computing networks, resulting in communication congestion and poor user experience. Summary of the Invention

[0008] The purpose of this invention is to provide a method and system for offloading multi-UAV-assisted edge computing tasks, in order to solve the problems of communication congestion and poor user experience in existing mobile edge computing networks.

[0009] The multi-UAV assisted edge computing task offloading method provided by the present invention to solve the above-mentioned technical problems includes:

[0010] 1) Construct a multi-UAV-assisted edge computing task offloading system model and an optimization problem model;

[0011] 2) Based on the multi-UAV assisted edge computing task offloading system model and optimization problem model, a Markov decision process in a multi-agent environment is constructed, and a reward function with the objectives of queue stability and minimizing information age is defined. The deep deterministic policy gradient algorithm is adopted to solve the optimization problem through the interaction between multiple agents and the environment, thereby obtaining the optimal offloading strategy for the computing task of ground communication equipment.

[0012] Furthermore, the queue stability component in the reward function is calculated based on the backlog length of the UAV mission queue and the transmission rate between the UAV and the ground communication equipment.

[0013] Furthermore, the information age minimization part of the reward function is calculated based on the difference between the current time and the generation time of the latest data received by the server.

[0014] Furthermore, the queue stability part r in the reward function t1 The calculation formula is:

[0015]

[0016] Among them, Ys (t) represents the queue length of the s-th terrestrial communication device at the beginning of the t-th time slot, R k,s (t) represents the transmission rate between the UAV and the ground communication equipment in time slot t.

[0017] Furthermore, the information age minimization part r in the reward function t2 The calculation formula is:

[0018]

[0019] Where V is the weighting factor, S is the number of terrestrial communication devices, t represents the t-th time slot, and l s (t) is the generation time of the latest data received by the server.

[0020] Furthermore, a federated learning approach is employed to train the agents.

[0021] Furthermore, the deep deterministic policy gradient algorithm is a deep reinforcement learning algorithm guided by Lyapunov optimization.

[0022] Furthermore, the model parameters of the UAV are encrypted using the difference of Gaussian method, and the encrypted UAV model parameters are sent to the cloud center server.

[0023] The beneficial effects of this invention are as follows: Considering the queue stability problem in mobile edge computing networks, this invention defines a reward function with the goal of minimizing queue stability and information age, and adopts a deep reinforcement learning algorithm guided by Lyapunov optimization. Through the interaction between multiple agents and the environment, the optimization problem is solved. This not only ensures the stability of the long-term data queue, but also obtains the optimal information age in online mode, solving problems such as communication congestion and poor user experience quality.

[0024] The present invention provides a multi-UAV assisted edge computing task offloading system to solve the above-mentioned technical problems. The system includes a ground communication device and UAVs. The UAVs communicate with the ground communication device within their coverage area. An edge computing server is deployed on the UAVs. The edge computing server is used to implement the multi-UAV assisted edge computing task offloading method described above.

[0025] Furthermore, the system also includes a cloud center server, which communicates with the drone. The drone uses a federated learning method to train the agent, sending the drone model parameters to the cloud center server, which then aggregates them to obtain global model parameters and distributes them to the drone.

[0026] The beneficial effects of this invention are as follows: Considering the queue stability problem in mobile edge computing networks, this invention defines a reward function with the goal of minimizing queue stability and information age, and adopts a deep reinforcement learning algorithm guided by Lyapunov optimization. Through the interaction between multiple agents and the environment, the optimization problem is solved. This not only ensures the stability of the long-term data queue, but also obtains the optimal information age in online mode, solving problems such as communication congestion and poor user experience quality. Attached Figure Description

[0027] Figure 1 This is an overall architecture diagram of the emergency communication system according to an embodiment of the present invention;

[0028] Figure 2 This is an implementation scenario diagram of an embodiment of the present invention;

[0029] Figure 3 This is a general flowchart of an embodiment of the present invention;

[0030] Figure 4 This is a network architecture diagram according to an embodiment of the present invention;

[0031] Figure 5 This is an initial coverage area diagram according to an embodiment of the present invention;

[0032] Figure 6 This is the coverage map under the existing LYT algorithm;

[0033] Figure 7 Coverage map under the existing LYST algorithm;

[0034] Figure 8 This is a coverage map of the LYSTBFL algorithm according to an embodiment of the present invention;

[0035] Figure 9 This is the coverage map under the existing LYTFL algorithm. Detailed Implementation

[0036] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0037] Example of a method for offloading edge computing tasks assisted by multiple drones

[0038] The multi-UAV assisted edge computing task offloading method of the present invention includes:

[0039] 1) Construct a multi-UAV-assisted edge computing task offloading system model and an optimization problem model;

[0040] 2) Based on the multi-UAV assisted edge computing task offloading system model and optimization problem model, a Markov decision process in a multi-agent environment is constructed, and a reward function with the objectives of queue stability and minimizing information age is defined. The deep deterministic policy gradient algorithm is adopted to solve the optimization problem through the interaction between multiple agents and the environment, thereby obtaining the optimal offloading strategy for the computing task of ground communication equipment.

[0041] Preferably, the queue stability part of the reward function is calculated based on the UAV mission queue backlog length and the transmission rate between the UAV and the ground communication equipment. The information age minimization part of the reward function is calculated based on the difference between the current time and the generation time of the latest data received by the server. The deep deterministic policy gradient algorithm is a deep reinforcement learning algorithm guided by Lyapunov optimization. Federated learning is used to train the agent.

[0042] The technical solution of the present invention will be further illustrated below using an emergency rescue scenario as an example.

[0043] like Figure 1 , 2 As shown, the system network model of this invention consists of three entities: ground emergency communication vehicles, drones, and a cloud center server, considering the computational offloading scenario within a single time slot t. This invention represents a group of ground emergency communication vehicles as s = {1, 2, ..., S}, and a group of drones as k = {1, 2, ..., K}. Within time slot t, a binary computational offloading rule is adopted, with binary variables... This represents the unloading decision for the α-th data segment. This indicates that within time slot t, the α-th data segment is unloaded to the UAV for task processing. This indicates that the α-th data segment is processed locally within time slot t. Here, k represents the UAV's ID, K represents the total number of UAVs, s represents the emergency communication vehicle's ID, and S represents the total number of emergency communication vehicles.

[0044] I. Establishing a Multi-UAV Assisted Edge Computing Offloading System Model

[0045] 1. Establish an air-to-ground channel model

[0046] Assuming that the drones neither transmit data nor tasks to each other, what is the transmission rate R between the drones and the emergency communication vehicle within time slot t? k,s (t) is:

[0047]

[0048] Among them, R k,s(t) represents the transmission rate of the k-th UAV and the s-th communication vehicle at time t, ι is the number of computation cycles required to process one bit of raw data, ι>0; ξ is the total bandwidth, P t k L(t) represents the transmission power of the k-th UAV in the t-th time slot, and L(t) represents the average path loss of the channel. h is the noise power spectral density. t For channel parameters, f k,s The computing resources allocated to the emergency communication vehicle s for the drone k.

[0049] Within time slot t, E k,s The power consumption (t) is:

[0050]

[0051] Among them, E k,s (t) represents the power consumption generated by the k-th UAV and the s-th communication vehicle in the t-th time slot, and κ is the energy efficiency parameter for calculation, κ > 0; s (t) represents the energy consumption of the s-th drone in time slot t. Indicates the decision to unload. This indicates that the task will be unloaded onto the drone for processing. This indicates that the process is handled locally.

[0052] 2. Establish an edge movement model

[0053] The application scenario of this invention is that a drone operates above an emergency communication vehicle, in time slot t. Let be the initial position of the drone; therefore, the drone's mobility can be modeled as follows:

[0054]

[0055] in, Let be the distance traveled within time slot t, subject to the following constraints: It is the position of the drone in time slot t+1, u k It is the number of drones. These are the mobility constraints for the t-slot UAV.

[0056] 3. Establish a data scheduling model

[0057] On each drone, To collect data buffers, unprocessed data from the data source can be cached. The unloading strategy for the k-th drone in the t-th time slot can be expressed as:

[0058] in, It is the CPU-assigned tag for each data segment, and it meets the following conditions:

[0059]

[0060] in, This represents the total number of data segments.

[0061] 4. Establish a dynamic queue model

[0062] Within time slot t, the data task generated by the s-th emergency communication vehicle is denoted as... Y s (t) represents the queue length of the s-th emergency communication vehicle at the start of the t-th time slot. The UAV needs C to process the mission. s One CPU clock cycle. B represents the communication bandwidth, f ue This indicates the CPU cycle frequency.

[0063]

[0064] Y s (t+1)=max(0,Y s (t+1))

[0065]

[0066] Among them, B s (t) represents the length of the queue after execution is complete, υ s (t) represents the percentage of CPU resources allocated to the UAV for the data queue of the s-th emergency communication vehicle, ω s (t) represents the percentage of communication bandwidth that is offloaded from the data queue of the s-th emergency communication vehicle to the cloud center server.

[0067] 5. Establish a federated learning model

[0068] On the cloud center server side, a synchronous update strategy is used to obtain new global parameters. Each UAV (unmanned aerial vehicle) retains a weight of Parameters, global parameters As shown in the formula below:

[0069]

[0070] δ t =δ t-1 *β

[0071] in, It represents the number of cloud servers carried by the k-th drone. δ is the local parameter of the model for the k-th UAV in time slot t. t is the vector of the actor network for all UAVs in time slot t, and β is the joint update matrix.

[0072] II. Formulating Optimization Problems

[0073] The objective function of this invention is to minimize the total delay of the dual UAV-assisted unloading task while ensuring secure transmission, where T(k) represents the total delay of the UAV-assisted unloading task. The optimization problem can be expressed as:

[0074]

[0075] H min ≤H≤H max

[0076]

[0077] Constraint (13a) represents the offloading strategy O in time slot t. s The range of values ​​for (t);

[0078] Constraint (13b) indicates that the SINR of the A2G communication link is greater than γ. min Only then can an effective connection be established. k,e It is the interference-to-noise ratio between the UAV and the eavesdropping user, γ s,k It is the interference-to-noise ratio between the UAV and the emergency communication vehicle.

[0079] Constraint (13c) states that the total energy consumption of the UAV across all time slots is less than or equal to the maximum battery capacity of the UAV, E uav (t) represents the energy consumed by the UAV in time slot t, E battery This represents the maximum battery capacity of the UAV.

[0080] Constraint (13d) indicates that the UAV's flight distance is within the range given by the system. This is the x-coordinate of the UAV in the t-th time slot. This is the y-axis coordinate of the UAV in the t-th time slot. This is the x-axis coordinate of the emergency communication vehicle in the t-th time slot. It is the y-axis coordinate of the emergency communication vehicle in the t-th time slot, and L is the flight area of ​​the UAV.

[0081] Constraint (13e) indicates that the UAV's flight altitude H is within a specified threshold range, H min H is the lower limit of flight altitude. max This is the upper limit of flight altitude.

[0082] Constraint (13f) indicates that the UAV has a certain coverage range. If the emergency communication vehicle is not within the threshold range, the UAV will be unable to perform computational tasks. k It is the maximum overhead view of the drone.

[0083] III. MDP Modeling

[0084] To address the aforementioned optimization problem, the dynamic computational unloading decision problem is modeled as a reinforcement learning framework based on Markov Decision Process (MDP). The state space and dynamic space of the deep learning approach are defined, and the system's cost function is used as the reward function for the decision. Considering the queue stability problem in mobile edge computing networks, a reward function is designed with the objectives of minimizing queue stability and information age. Furthermore, a deep deterministic policy gradient algorithm is applied to seek the optimal policy, helping each UAV adaptively find the optimal unloading decision.

[0085] (1) State space

[0086] Drone status x uav (t) Includes local environmental observations, UAV status, task data queue length, and arrival queue length. Cloud center server status x c (t)θ includes all UAV states, data queue length, and arrival queue length. Therefore, the state at time slot t can be defined as:

[0087]

[0088] Among them, E res (t) represents the remaining energy of the UAV, Z ue (t) represents the coordinates of the emergency communication vehicle, Z uav (t) represents the coordinates of the UAV. For each emergency communication vehicle, a computational task size is randomly generated, A res (t) represents the remaining computational task size.

[0089] (2) Action space

[0090] The actions of a drone include movement Decision execution Uninstallation decision Right now:

[0091]

[0092] The cloud center server allocates optimal offload bandwidth to each drone; therefore, the cloud center server's action A... C (t) is the bandwidth ratio vector.

[0093] (3) Reward function

[0094] Will reward r t Set as the sum of two terms, i.e., the stable part of the queue r t1 And the information age minimization part r t2 The weighting factor is V, and the reward function is as follows:

[0095]

[0096]

[0097]

[0098] Among them, l s (t) is the generation time when the cloud center server receives the latest data, Y s (t) is the backlog length of the task queue, R k,s (t) represents the transmission rate between the drone and the emergency communication vehicle.

[0099] IV. Computational Unloading Algorithm Based on Deep Deterministic Policy Gradient

[0100] This invention solves the problem of continuous states and action space through the Actor-Critic (AC) architecture, specifically including the following steps:

[0101] a: At each time step, the actor network obtains current state information from the external environment;

[0102] b: Actor networks typically consist of multi-layer neural networks. They learn the mapping relationship between environmental states and actions, and use the softmax activation function to constrain the network output within a suitable range.

[0103] c: Select the action to be executed based on the action probability distribution or value output by the actor network in order to maximize long-term cumulative rewards.

[0104] d: After the action is executed, the agent interacts with the external environment, and the environment returns the next status and an immediate reward.

[0105] e: The immediate reward obtained by the agent and the next state are used together as input to the critic network to update the value function of the critic network.

[0106] f: The value estimate of the critic network's output state, i.e., the state-value function or action-value function, is used to evaluate the quality of the state, thereby influencing the probability distribution of the actor network's output actions, realizing an interactive learning process of action selection and value evaluation. Centralized training of deep reinforcement learning agents can lead to key issues related to agent privacy, scalability, and additional communication overhead. Federated learning, on the other hand, allows each agent to train its own local model using local data. These local models are then sent to an aggregation unit for model parameter aggregation; this process continues until a certain training accuracy is achieved, at which point the federated learning iteration ends. Federated learning eliminates the need to exchange large amounts of training data between different edge nodes; it only sends updated model parameters, rather than the original data, to the model owner for aggregation, significantly reducing communication overhead and ensuring the heterogeneity and data privacy of each local agent node.

[0107] Therefore, this invention establishes an edge federated computing offloading framework that combines centralized learning and distributed training by integrating data scheduling, edge mobility, and queue dynamic models. Employing a Lyapunov-guided DRL method, it identifies effective computing offloading strategies, such as... Figure 4 As shown, the specific steps include:

[0108] a: The central cloud server establishes a basic model and informs each drone carrying an edge computing server of the basic structure and parameters of the model.

[0109] b: Each UAV uses local data to train the model and uploads the neural network parameters to the central cloud server in a confidential manner through the uplink communication channel.

[0110] c: The central cloud server aggregates drone parameters within the coverage area to build a more accurate global model, thereby improving overall performance;

[0111] d: Broadcast the network parameters of the global model to the relevant drones and conduct the next round of training to improve the overall model performance and results;

[0112] e: Repeat the ad step until the model converges.

[0113] like Figure 4 As shown, this invention combines the advantages of Lyapunov optimization, DRL, and FL, and proposes an online offloading algorithm integrating the Lyapunov-DRL-FL framework (referred to as the LYSTBFL algorithm). The specific steps of this algorithm are as follows:

[0114] 1) Input: State space X, Action space A, Discount factor Learning rate lr, latency update frequency fr, target update network parameters τr;

[0115] 2) Initialization: Critic networks Q1 and Q2, Actor networks π1 and π2 and their random parameters θ1, θ2, φ1, φ2, memory bf, minimum experience pool size bm;

[0116] 3) Iterate through the sequence E times;

[0117] 4) Initialize simulation parameters;

[0118] 5) Iterate through the four drones in sequence;

[0119] 6) Mini-batch sampling refers to randomly selecting a small batch of data samples from the experience replay buffer for training neural networks.

[0120] 7) Based on π1 and π2, select the noisy action 'a', where the noise ε follows a normal distribution, i.e.

[0121] 8) Here, a' represents the next action. Indicates network policy, Let represent the network parameters, and c represent the policy noise. This method achieves a smoothed policy by adding pruned noise to each dimension of the action. By smoothing the different action variations of the Q-function, the policy becomes more difficult to exploit errors in the Q-function, thus facilitating the discovery of the optimal unloading policy.

[0122] 9) in, Let X' represent the j-th Q-value, X' represent the next state, and a' represent the next action. This represents the network parameters. Q'(x',a') represents the minimum Q value at the next state x' and the next action a'.

[0123] 10) The double-Q learning approach for pruning involves updating the same target for both Q functions, using the minimum value obtained from the target network for both Q functions. This method reduces the overestimation problem, making it easier to find the optimal unloading strategy.

[0124] In the formula, y i Let 'r' represent the target value for TD and 'r' represent the reward. Indicates the discount factor. This indicates that after taking an action in a certain state s', there is a certain probability of receiving a "done" signal, signifying the end of this MDP. The Temporal Difference Target (TD) is a key concept in reinforcement learning used to update the value function or policy, primarily manifested in Temporal Difference Learning (TD learning). It combines estimations of immediate and future rewards, serving as an effective approximation method for measuring the value of action choices under the current policy. Therefore, this embodiment employs the TD learning method.

[0125]

[0126] Among them, softmax is used ψ (Q'(x',·)) defines the softmax operator in the continuous action space. ψ These are the parameters of the softmax operator. p(a') is the probability density function of the Gaussian distribution.

[0127] 11) According to Updating the critic network means updating the parameters θ of the critic network. i N represents N empirical trajectories sampled, y i This represents the TD target value.

[0128] 12) Update the target network

[0129] Where θ' i Represents the neural network parameters at the next time step; It's the learning rate, used to control the step size for parameter updates, θ. i φ' represents the neural network parameters at this moment. i φ represents the neural network parameters at the next time step. i This represents the neural network parameters at this moment.

[0130] 13) Update the actor network:

[0131] Where N represents the number of empirical trajectories sampled, φ i The parameters representing the policy function are typically used to represent the weights of a neural network; π(x; φ) i Let represent the probability of taking action a given state s and policy parameters φ. i (x,a; θ) i ) represents the action-value function given state x and action a, with parameter θ. i These are typically also the weights of a neural network. This represents the gradient with respect to the policy parameter φ, used to average the samples to estimate the expectation. ∑s refers to the summation over all states x; in practice, it is usually summed over a set of sample states. It is a conditional expression that means that the gradient of state x will only be considered when behavior a is equal to the behavior determined by policy π in state x.

[0132] 14) The parameters of the UAV model are encrypted using the difference of Gaussian method and then sent to the cloud center server;

[0133] 15) Obtain global model parameters from the cloud center server through federated learning;

[0134] 16) Update the global model parameters for each UAV:

[0135] 17) End.

[0136] The LYSTBFL algorithm combines the advantages of Lyapunov optimization, DRL, and FL, ensuring not only the stability of the long-term data queue, average power constraints, and energy consumption constraints, but also obtaining the optimal offloading strategy online while protecting data privacy. This involves the following aspects: 1) Applying Lyapunov optimization theorems to transform the variable constraint objective into a queue stability problem; 2) Designing a reward function that aims to minimize queue stability and information age, helping each UAV adaptively find the optimal offloading decision; 3) The TD3+BC algorithm ensures the stability of Critic learning, mitigates Critic overestimation, smooths the target strategy, and eliminates unnecessary computational costs by simply adding a BC term and Normalizedstate at the code level; the SD3 algorithm utilizes the softmax operator of dual estimators to help smooth the environment, reducing overestimation bias on DDPG and improving underestimation bias on TD3+BC. The DRL in the LYSTBFL algorithm combines the advantages of both SD3 and TD3+BC algorithms, offering the following benefits: (a) it improves value estimation bias; (b) it uses an actor delayed update method to make actor training more stable; (c) it adds noise to the target actor network to improve algorithm stability; (d) it adds behavioral cloning to normalize the state matrix, reducing computational complexity; (e) it uses the difference of Gaussians method to encrypt the local network parameters of the UAV; and (f) the edge joint mode transmits only lightweight network parameters, improving system efficiency.

[0137] The overall flowchart of the multi-UAV assisted edge computing task offloading method of this invention is as follows: Figure 3 As shown, the process includes the following:

[0138] 1) Determine the location of the emergency communication vehicle.

[0139] 2) Locate the drone closest to the emergency communication vehicle;

[0140] 3) Determine whether the emergency communication vehicle is outside the coverage area of ​​the drone. If it is outside the coverage area, find the computing nodes of other available drones. If it is within the coverage area, the drone receives the computing task.

[0141] 4) The UAV that receives the computational task determines whether to unload the task. If so, the task is unloaded onto the UAV for processing; otherwise, the task is processed locally by the emergency communication vehicle.

[0142] 5) After unloading the task onto the drone, determine whether to adopt the federated learning mode. If the federated learning mode is adopted, upload the drone's model parameters to the cloud center server. The cloud center server will perform secure aggregation of the drone's model parameters and distribute the obtained model parameters to each drone. Each drone will replace the original drone's model parameters and use the new model to perform the calculation task and obtain the calculation result. If the federated learning mode is not adopted, the drone will directly perform the calculation task and obtain the calculation result.

[0143] 6) Determine if the calculation task being performed is the last task. If it is the last calculation task, send the calculation result back to the emergency communication vehicle and end the calculation process. If it is not the last calculation task, determine the location of the emergency communication vehicle after the calculation task is completed, and repeat steps 2) to 6) until the last task is completed, send the calculation result back to the emergency communication vehicle, and end the calculation process.

[0144] Example of a multi-UAV-assisted edge computing task offloading system

[0145] The system includes ground communication equipment, drones, and a cloud central server. The drones communicate with the ground communication equipment within their coverage area and with the cloud central server. The ground communication equipment identifies the nearest drone and decides whether to offload computational tasks to it, while also receiving the computation results from the drones. Each drone is equipped with an edge computing server, on which an agent is deployed. This edge computing server receives the computational tasks offloaded by the ground communication equipment, determines the computation method (whether to use federated learning), uploads the drone's model parameters to the cloud central server, receives model parameters from the cloud central server, replaces the original model parameters on the drone, obtains new model parameters, and uses the new model to execute the computational task. The cloud central server securely aggregates the drone model parameters and distributes the aggregated model parameters to each drone. Each drone updates its model, uses the new model to select the optimal offloading decision, and sends the result back to the ground communication equipment to assist in completing the task.

[0146] The agent deployed on the UAV in this invention is a DRL-based agent guided by Lyapunov, which is used to learn the optimal offloading strategy. The task offloading algorithm implemented by this agent is described in the multi-UAV assisted edge computing task offloading method in the above method embodiment, and will not be described in detail here.

[0147] Emergency communication vehicles are equipped with sensors, cameras, and other devices to collect environmental data, including images, videos, and audio. This data can be analyzed and processed using models for tasks such as environmental perception, target detection, and tracking.

[0148] The flight data of a drone includes information such as its flight status, attitude, and speed. By analyzing this data through models, functions such as flight path planning, obstacle avoidance, and attitude control can be achieved.

[0149] After the drone establishes a communication connection with the ground communication equipment, the two transmit communication data, including control commands and sensor data transmission.

[0150] When the system of the present invention is applied in emergency rescue scenarios, the ground communication equipment is an emergency communication vehicle.

[0151] The following simulation verification of the multi-UAV assisted edge computing task offloading method of the present invention is used to prove that the method of the present invention can alleviate network congestion and reduce latency.

[0152] In the multi-UAV-assisted mobile edge computing task offloading method, the UAV is an intelligent agent with computing and decision-making capabilities, and the simulation parameters are shown in Table 1.

[0153] Table 1: Simulation Parameter Table

[0154] parameter describe initial value <![CDATA[u k ]]> Number of drones 4 MC Cloud computing nodes 1 <![CDATA[χ s ]]> Number of emergency communication vehicles 30 <![CDATA[OR uav ]]> observation radius of drones 60m <![CDATA[CR uav ]]> drone coverage radius 40m <![CDATA[SP uav ]]> The flight speed of drones 6m / s N Noise density 1e-13W / Hz <![CDATA[M off ]]> Maximum power of offloading communication 0.2W <![CDATA[f uav ]]> The computing frequency of drones 2.5GHz ε Total offload bandwidth 100MHz <![CDATA[σ los ]]> Line-of-sight loss 1 <![CDATA[σ nlos ]]> Non-line-of-sight loss 20

[0155] The embodiments of the present invention have achieved some positive results during the research and development or use process, and have indeed great advantages compared with the prior art. The following content describes them in conjunction with the data, charts and other information of the experimental process.

[0156] This paper analyzes and compares the LYSTBFL algorithm of this invention with the existing LYTFL, LYST, and LYTB algorithms. The LYSTBFL algorithm combines the advantages of both SD3 and TD3+BC algorithms in its DRL. The LYTFL algorithm is also based on the Lyapunov-DRL-FL framework, but its DRL uses the TD3+BC algorithm. The LYST algorithm is based on the Lyapunov-DRL framework, where the DRL combines the advantages of both SD3 and TD3+BC algorithms. The LYTB algorithm is based on the Lyapunov-DRL framework, where the DRL uses the TD3+BC algorithm. TD3+BC is an enhancement algorithm that combines TD3 (Dual Delay Deep Deterministic Policy Gradient) and BC (Behavioral Cloning) algorithms, focusing on performance improvement in offline reinforcement learning scenarios. SD3 is an improvement on the TD3 algorithm, which optimizes the update of the value function and policy function by introducing SoftMax operations to smooth action selection and estimate calculation.

[0157] Drones are highly valuable in collecting information from disaster-stricken areas. The primary goal of edge computing offloading tasks is to ensure effective coverage of a given area, optimizing performance metrics such as information age, data queue stability, and privacy security while ensuring effective communication coverage and network connectivity. Effective coverage of target areas in urban areas includes... Figures 5-9 As shown, black dots represent the location of the drone, gray dots represent the location of the emergency communication vehicle, light gray circles represent the range that the drone can observe, and dark gray circles represent the range that the drone can effectively cover. Figure 5 This is the initial coverage area map. Figure 6 This is the coverage map trained 3000 times using the LYTB algorithm. Figure 7 This is the coverage map trained 3000 times using the LYST algorithm. Figure 8 This is the coverage map trained 3000 times using the LYSTBFL algorithm. Figure 9The image shows the coverage area obtained after 3000 training iterations of the LYTFL algorithm. It's clear from the image that the LYSTBFL and LYTFL algorithms, based on the edge federation framework, achieve a better offloading strategy than LYT and LYSTB, resulting in a larger effective coverage area for the drones in specific regions. Further comparison of the effective coverage areas of LYSTBFL and LYTFL reveals that although the LYTFL algorithm covers a slightly larger area, the amount of offloading tasks within the coverage area of ​​drone 3 is greater, potentially causing network congestion and higher latency. Conversely, when using the LYSTBFL algorithm, the number of emergency communication vehicles is balanced within the coverage areas of the four drones, the deployment of drones and emergency communication vehicles is optimal, the system has the lowest information age, and the data queue stability is the strongest. Therefore, this invention is suitable for multi-drone assisted edge computing offloading systems for emergency communications, especially for applications requiring low latency, low power consumption, and low-complexity transmission.

Claims

1. A method for offloading edge computing tasks assisted by multiple unmanned aerial vehicles (UAVs), characterized in that, The method includes: 1) Construct a multi-UAV-assisted edge computing task offloading system model and an optimization problem model; 2) Based on the multi-UAV assisted edge computing task offloading system model and optimization problem model, a Markov decision process in a multi-agent environment is constructed, and a reward function with the objectives of queue stability and minimizing information age is defined. The deep deterministic policy gradient algorithm is adopted to solve the optimization problem through the interaction between multiple agents and the environment, thereby obtaining the optimal offloading strategy for the computing task of ground communication equipment.

2. The multi-UAV assisted edge computing task offloading method according to claim 1, characterized in that, The queue stability component in the reward function is calculated based on the backlog length of the UAV mission queue and the transmission rate between the UAV and the ground communication equipment.

3. The multi-UAV assisted edge computing task offloading method according to claim 1, characterized in that, The information age minimization part of the reward function is calculated based on the difference between the current time and the generation time of the latest data received by the server.

4. The multi-UAV assisted edge computing task offloading method according to claim 2, characterized in that, Queue stability part r in reward function t1 The calculation formula is: Among them, Y s (t) represents the queue length of the s-th terrestrial communication device at the beginning of the t-th time slot, R k,s (t) represents the transmission rate between the UAV and the ground communication equipment in time slot t.

5. The multi-UAV assisted edge computing task offloading method according to claim 3, characterized in that, The information age minimization part r in the reward function t2 The calculation formula is: Where V is the weighting factor, S is the number of terrestrial communication devices, t represents the t-th time slot, and l s (t) is the generation time of the latest data received by the server.

6. The multi-UAV assisted edge computing task offloading method according to any one of claims 1 to 5, characterized in that, The federated learning method is used to train the intelligent agent.

7. The multi-UAV assisted edge computing task offloading method according to any one of claims 1 to 5, characterized in that, The deep deterministic policy gradient algorithm is a deep reinforcement learning algorithm guided by Lyapunov optimization.

8. The multi-UAV assisted edge computing task offloading method according to claim 6, characterized in that, The model parameters of the UAV are encrypted using the difference of Gaussian method, and the encrypted UAV model parameters are sent to the cloud center server.

9. A multi-UAV-assisted edge computing task offloading system, comprising ground communication equipment and UAVs, wherein the UAVs communicate with the ground communication equipment within their coverage area, and an edge computing server is deployed on the UAVs, characterized in that, The edge computing server is used to implement the multi-UAV assisted edge computing task offloading method according to any one of claims 1 to 5.

10. The multi-UAV assisted edge computing task offloading system according to claim 9, characterized in that, The system also includes a cloud center server, which communicates with the drone. The drone uses a federated learning method to train the agent, sending the drone model parameters to the cloud center server, which then aggregates them to obtain global model parameters and distributes them to the drone.

Images