Heterogeneous unmanned aerial vehicle task flow edge calculation unloading method based on multi-agent deep reinforcement learning

Abstract

The invention belongs to the technical field of unmanned aerial vehicle edge calculation, and discloses a heterogeneous unmanned aerial vehicle task flow edge calculation unloading method based on multi-agent deep reinforcement learning. In order to solve the problem of insufficient timeliness of complex task unloading in a heterogeneous unmanned aerial vehicle cooperative processing environment, a complex task generated on the ground is split into a plurality of serial sub-tasks, the serial sub-tasks are sequentially executed by a plurality of unmanned aerial vehicles with different computing capabilities, and an intermediate result is transmitted through an air link every time until the task is completed. According to the invention, task-oriented information age is introduced as a system optimization target, and a multi-agent deep reinforcement learning algorithm is utilized to realize autonomous scheduling and cooperative control between unmanned aerial vehicles, so that the information processing timeliness and the system calculation efficiency are improved, the overall delay is reduced, and the real-time processing requirement of complex tasks is met.

Description

Technical Field

[0001] This invention belongs to the field of drones and artificial intelligence, specifically involving a drone edge computing task offloading method based on multi-agent deep reinforcement learning, and particularly involving heterogeneous drone collaborative processing and task-oriented information age optimization strategies. Background Technology

[0002] With the rapid development of 6G, user needs are no longer limited to traditional communication services, but also include a large number of computationally intensive tasks involving environmental perception and image recognition. These complex applications are difficult to complete independently on resource-constrained terminal devices. To address the insufficient computing power of user devices, Mobile Edge Computing (MEC) has emerged as a cutting-edge solution. MEC offloads computing tasks to edge computing nodes closer to the data source, enabling users to complete complex tasks with lower latency and energy consumption, significantly reducing the computational burden on local devices. With the development of the low-altitude economy, Unmanned Aerial Vehicles (UAVs), with their high mobility and flexible deployment, are gradually becoming an important support for mobile edge computing networks. In UAV-assisted mobile edge computing networks, UAVs can serve as aerial edge computing platforms. Once ground equipment completes perception, the UAV can quickly collect data from the ground equipment and flexibly execute edge computing processes.

[0003] Several patents have been reported regarding task offloading solutions for drone-assisted edge computing. Patent CN118764886A proposes a management method for multi-drone edge computing networks, minimizing the total network latency of all computing tasks by optimizing task offloading and resource allocation strategies. Patent CN117596571A proposes a drone-assisted mobile edge computing offloading method, optimizing the proportion of tasks offloaded to drones based on resource allocation to balance computing load and reduce task execution latency and energy consumption. Patent CN120547631A proposes an edge computing method based on deep reinforcement learning, whereby, after receiving information from ground equipment, the drone can determine the offloading ratio of computing tasks based on the nature of the task and resource status.

[0004] Most of the existing patents mentioned above focus on formulating optimal computational offloading strategies. Their solutions all assume that a single drone possesses the computational capability to complete the entire task; that is, regardless of how the task is offloaded, the drone can complete it independently. This ignores the resource limitations of the drone itself and fails to consider the issues of collaborative processing between drones and segmented execution of task flows. In reality, a complex computational task typically has a modular processing flow, requiring the sequential completion of multiple different types of computational subtasks to obtain the final result. If a single drone is to carry all the computational functions and independently complete the entire task using the traditional offloading method, it will be inefficient due to limited onboard resources. Summary of the Invention

[0005] To address the aforementioned shortcomings, this invention proposes a multi-UAV collaborative task flow offloading strategy. This strategy fully considers the limited computing resources of UAVs themselves, breaking down complex tasks generated on the ground into linear sub-task flows executed sequentially according to their required computational functions. Multiple heterogeneous UAVs with different computing capabilities collaboratively process each sub-task stage. The entire strategy encompasses UAV trajectory planning, task data offloading decisions from ground equipment to UAVs, and task flow transmission scheduling between UAVs.

[0006] Specifically, each drone is equipped with only one specific type of computing function. Each complex task is divided into multiple subtasks, which are processed sequentially by drones of the corresponding function type. During task unloading, the drone handling the first subtask first retrieves task data from ground equipment and performs the first subtask calculation. Subsequently, this drone transmits the intermediate results via an air link to drones with the next function type to continue processing the next subtask. This process continues, with intermediate calculation results of subtasks being passed hop-by-hop to the next stage drone until all subtasks are completed sequentially and the final result is obtained. For this collaborative computing process, this invention dynamically optimizes the drone's flight trajectory and transmission strategy, rationally allocating tasks to suitable drones and promptly transmitting intermediate results, thereby ensuring the efficient operation of the task pipeline. To evaluate the timeliness of information processing in the system, this invention designs a Task-Oriented Age of Information (TAoI) index. Unlike the currently commonly used single-link latency index, TAoI quantifies the end-to-end latency of perceived data from its generation to the completion of each subtask calculation and finally to the acquisition of the calculation result, thus characterizing the freshness of information from a global task perspective. Furthermore, considering the dynamic characteristics of UAV networks and the need for intelligent collaborative scheduling, this invention employs a multi-agent deep reinforcement learning framework to achieve autonomous decision-making and scheduling optimization for UAVs.

[0007] The specific steps are as follows:

[0008] This invention considers a drone-assisted mobile edge computing network. The ground layer has N ground devices equipped with cameras, which periodically perceive the surrounding environment and generate different computing task requirements. Each task can be formed by permuting and combining K types of subtasks, which need to be computed sequentially by drones of the corresponding categories. The air layer is configured with M drones, divided into K functional clusters according to computing type. Drones in the k-th cluster can only handle subtasks of the k-th category. The entire operation cycle is divided into T discrete time slots. Each time slot contains two phases for the drone: a flight phase and a service phase. In the flight phase of the t-th time slot, drone m starts from its previous position... Move to the next position Upon reaching the target location, the drone enters the service phase, hovering and performing data transmission and computation. The drone needs to decide whether to establish an air-to-ground link with ground equipment within its coverage area to collect mission data, and whether there are intermediate results that need to be transmitted to other drones. After making these decisions, the drone will execute the data transmission strategy accordingly and complete the corresponding computation tasks when mission data arrives.

[0009] To achieve intelligent task scheduling, autonomous collaboration among UAVs, and maximize overall system efficiency, this invention employs a multi-agent deep reinforcement learning algorithm. In this system, each UAV is considered an agent. Within this framework, the definition of each agent's state representation, action space, and reward function, as well as the design of the deep neural network structure and training method, are crucial. The definitions and training steps of these elements will be elaborated in detail below.

[0010] Step 1: Define the drone's status

[0011] During time slot t, each UAV m observes the current state. Represented as:

[0012]

[0013] in, This indicates the position of drone m at the beginning of time slot t. and These represent the location of the ground equipment within the observation range of UAV m at the beginning of time slot t and the corresponding TAoI, respectively.

[0014] The TAoI of the task generated by ground device n at any time u is defined as follows:

[0015]

[0016] in, This represents the set of all valid tasks that have been generated but not yet successfully processed before time u. It is the generation time of task k;

[0017] In scenarios where ground equipment generates tasks periodically, three different task processing states may occur: (1) timely processing; (2) the old task has not been completed and the new task has arrived; (3) the new task has been completed and the old task has not yet ended.

[0018] (1) Timely processing: The current task has been completed and the next task has not yet been generated; the current task has been completed and there are no other unprocessed tasks in the system at that time, then the system is considered to have completely up-to-date information; when the current task is completed, TAoI is reset to zero, indicating that the information has been sensed, transmitted and processed in a timely manner;

[0019] (2) The old task has not been completed and the new task has arrived: the new task is generated while the current task is still being processed; the new task has been generated by the ground equipment while the current task is still being processed; the task generated later is suspended in the system after the current task is completed. In this case, the system is considered to not have completely up-to-date information, so the TAoI will not be reset to zero; instead, the system will be updated immediately to the time that has elapsed since the new task was generated.

[0020] (3) The new task has been completed, but the old task has not yet been completed: the old task has not been completed, the new task has been completed, and the old task is considered to be discarded; the old task has not been completed, so it is considered to be deleted and excluded from the TAoI calculation; considering that the new task has been completed, the system will continue to track the earliest valid pending task; therefore, TAoI will be updated according to the waiting time of the earliest valid pending task (calculated from its generation time); Step 2: Define the drone's actions

[0021] During time slot t, based on the observed state, each UAV needs to select an action to execute; the UAV's action consists of two parts: flight control and mission transfer decision, represented as:

[0022]

[0023] in, and Indicates horizontal and vertical flight speed decisions. For transmission decisions, where It is a binary decision variable, representing whether the drone m establishes a communication link with the ground equipment n; For transmission decisions, where It is a binary decision variable, representing whether drone m moves towards drone m. Send intermediate results;

[0024] To ensure the effectiveness and safety of action decisions, the following constraints are imposed on the behavior of the drone:

[0025] First, trajectory safety constraints: To prevent collisions, a minimum safe distance is established. To constrain trajectory design, the minimum interval between any two drones must be In addition, the maximum speed limit for each drone is specified. ;

[0026] Second, there is the air-to-ground link constraint: within a time slot, any ground device n can only establish a communication link with one UAV, and each UAV can only simultaneously establish a communication link with at most one UAV. Establish a communication link with each ground device;

[0027] Third, air-to-air transmission constraints: each drone can transmit data to at most one drone in each time slot; Step 3: Define the drone rewards

[0028] For each action performed by the drone in each time slot, a corresponding reward is designed to evaluate its performance; the reward function for drone m is defined as:

[0029]

[0030] The reward function is based on the changes in TAoI of ground equipment. This is the sum of the TAoI of the ground equipment, where N is the total number of ground equipment. A higher reward is given when a UAV's mission transmission decision lowers the overall TAoI level of all missions within the system; conversely, a penalty is given if it causes mission delays and increases the TAoI. Furthermore, additional penalties are applied if mission processing fails, intermediate result transmission is interrupted, or a mission is deemed invalid and abandoned due to resource conflicts. This invention effectively guides each intelligent agent to optimize its strategy in the direction of reducing the age of system information.

[0031] Step 4: Define a multi-agent deep reinforcement learning network

[0032] A centralized training and distributed execution Actor-Critic architecture is introduced, using a multi-agent deep deterministic policy gradient algorithm to train each UAV. Each UAV has its own Actor network and Critic network. During centralized training, each UAV's Critic network needs to input the state and action data generated by all UAVs in steps one and two to estimate the global Q-value. Subsequently, combined with the reward obtained in step three, the Bellman equation is used to iteratively update the Critic network to improve the accuracy of long-term action value estimation. At the same time, the Actor network updates the policy gradient based on the global Q-value of the Critic network, aiming to maximize the expected cumulative reward. Experience replay is introduced during training, storing the state, action, and reward values ​​obtained by the UAV in steps one, two, and three in a buffer, and conducting training through random sampling. In addition, a soft update strategy is adopted to gradually update the target network parameters, enhancing the stable convergence of training. After centralized training, each UAV downloads the trained Actor network parameters. In actual operation, each UAV only inputs its Actor network based on its own local observations and outputs decision actions, realizing distributed autonomous cooperative control.

[0033] The beneficial effects of this invention are:

[0034] (1) This invention addresses the application background of limited computing resources for UAVs by proposing an edge computing offloading strategy for heterogeneous UAV task flows, which efficiently processes complex computing tasks generated by ground terminals in the form of task flows. By decomposing complex tasks into linear sub-task flows according to computing functions and having multiple UAVs of different functional types process them in segments collaboratively, each type of UAV undertakes only a single type of computing function, which significantly reduces the computing load and hardware complexity of a single machine and effectively reduces the overall latency of task processing, thereby achieving better computing offloading performance under resource-constrained conditions.

[0035] (2) To characterize the timeliness of information, this invention proposes a task-oriented information age index, quantifying the timeliness of the entire process from data acquisition to result output. Using TAoI as the optimization objective, it ensures that the system can process and feedback the latest information in a timely manner, thereby improving the response efficiency of the entire edge computing system. To enhance the system's flexibility and adaptability in dynamic and uncertain environments, this invention utilizes multi-agent deep reinforcement learning to achieve distributed intelligent scheduling. Each UAV autonomously decides on trajectory planning and task allocation based on local observation, realizing autonomous collaborative control. Detailed Implementation

[0036] The specific embodiments of the present invention will be further described below in conjunction with the technical solution.

[0037] In actual operation, the ground layer has 18 ground devices equipped with cameras, which can periodically perceive the surrounding environment. Each task can be formed by arranging and combining three types of sub-tasks, which need to be calculated sequentially by drones of the corresponding categories. The air layer is equipped with 6 drones, divided into 3 functional clusters according to the calculation type, with each function containing 2 drones. The entire operation cycle is divided into 20 discrete time slots. At the beginning of each discrete time slot, referring to the state definition in step one, all drones will acquire their own state information, including their current position, the distribution of nearby ground devices, and the freshness of relevant task information. Based on the current state, each drone agent selects the corresponding action using the decision-making strategy obtained from offline training, referring to step two. During the training phase, in order to enable the drone swarm to learn the above-mentioned efficient collaborative processing strategy, this invention uses a reinforcement learning reward mechanism to optimize and adjust the strategy. Specifically, after each drone swarm completes a task in a time slot by executing a set of actions, the system calculates a reward value based on the task processing results. The reward value calculation follows step three: if the decisions made by each drone in this time slot significantly reduce the overall information age, a higher reward is given; conversely, if improper decisions lead to delays or failures in some task processing, increasing the system's information age, a negative reward or even a penalty is given. This reward design encourages each drone to continuously adjust its strategies during training to minimize task processing delays and maintain information freshness.

[0038] Based on step four and the aforementioned state, action, and reward design, this invention employs a multi-agent deep reinforcement learning algorithm to train the UAV swarm strategy offline. During centralized training, a multi-agent Actor-Critic architecture is introduced. All UAVs share environmental information to train the Critic network to evaluate the global Q-value, and each UAV updates its own Actor network strategy based on feedback. Through iterative training, the UAV agents gradually learn how to autonomously coordinate under different environmental conditions, optimizing their flight trajectories and task offloading decisions. After training, the converged strategy model is deployed to each UAV. In actual operation, each UAV relies solely on its local observations as input, and its Actor decision network outputs the execution actions in real time, achieving distributed autonomous collaborative control. With this well-trained strategy, the UAV swarm can continuously perform joint scheduling in a dynamically changing network environment with the goal of minimizing task-oriented information age, ensuring the entire system maintains low information lag and high task processing performance.

Claims

1. A heterogeneous unmanned aerial vehicle task flow edge computing offloading method based on multi-agent deep reinforcement learning, characterized in that, The steps are as follows: Step one: define the state of the UAV; During a time slot t Each drone m Observes the current state Is represented as: ； wherein, represents a UAV m at the beginning of a time slot, t the position at the beginning of a time slot, and respectively represent the position of a ground device and the corresponding TAoI within the observation range of the UAV t at the beginning of a time slot, m at the beginning of a time slot. at any time u underground equipment n The TAoI of a generated task is defined as follows: ； wherein, represents a set of all valid tasks that have been generated but not yet successfully processed before time u is the generation time of task k ;​ In the scenario of periodic task generation by the ground device, there are three different task processing states: (1) timely processing; (2) the old task has not been completed, and the new task has arrived; (3) the new task has been completed, and the old task is still not finished; (1) Timely processing: the current task has been completed, and the next task has not been generated; the current task has been completed, and at that time there is no other unprocessed task in the system, and the system is considered to have completely up-to-date information; at the time when the current task is completed, the TAoI is reset to zero, indicating that the information has been sensed, transmitted and processed in a timely manner; (2) The old task has not been completed, and the new task has arrived: the scenario of generating a new task while the current task is still in the process of processing; the current task is still in the process of processing, and the new task has been generated by the ground device; after the task being processed currently is completed, the newly generated task remains in a suspended state in the system, and the system is considered to not have completely up-to-date information, so the TAoI is not reset to zero; on the contrary, the system will be immediately updated to the time elapsed since the generation of the new task; (3) The new task has been completed, and the old task is still not finished: the scenario of the old task being completed and the new task being completed, and the old task being considered discarded; the old task is considered to be deleted and excluded from the TAoI calculation; considering that the new task has been completed, the system continues to track the earliest valid pending task; therefore, the TAoI will be updated according to the waiting time of the earliest valid pending task; Step two: define the action of the UAV; During a time slot t Each drone needs to select an action to perform based on the observed state; the action of a drone consists of two parts, flight control and task transmission decision, denoted as: ； wherein, and denote horizontal and vertical flight speed decisions, is a transmission decision, wherein is a binary decision variable indicating whether the UAV m establishes a communication link with the ground equipment; n is a transmission decision, wherein is a binary decision variable indicating whether the UAV m sends intermediate results to the UAV command center;​ The following constraints are imposed on the UAV behavior: First, trajectory safety constraints: establish a minimum safe distance. To constrain trajectory design, the minimum interval between any two drones must be In addition, the maximum speed limit for each drone is specified. ; Second, air-ground link constraint: in a time slot, any ground device n can only establish a communication link with one UAV, while each UAV can establish a communication link with at most one ground device simultaneously; Thirdly, air-to-air transmission constraint: each UAV can transmit data to at most one UAV in each time slot; Step three: define the reward of the UAV; For each time slot, the UAV performs an action, and the corresponding reward is designed to evaluate its pros and cons; the reward function of the UAV m is defined as: ； Wherein, the reward function takes the change of TAoI of ground devices as the core; is the sum of TAoI of ground devices, N is the total number of ground devices, when the task transmission decision of a certain UAV makes the overall TAoI level of each task in the system decrease, a higher reward is given; on the contrary, if it leads to task lag and makes TAoI increase, a penalty is given; in addition, if the situation of task processing failure, interruption of intermediate result transmission or the task being abandoned due to being judged as invalid due to resource conflict occurs, an additional penalty term will be applied ; Step four: define the multi-agent deep reinforcement learning network; An Actor-Critic architecture of centralized training and distributed execution is introduced, and a multi-agent deep deterministic policy gradient algorithm is used to train each UAV; each UAV has its own Actor network and Critic network; in the centralized training phase, the Critic network of each UAV needs to input the state and action data generated by all UAVs in steps one and two to estimate the global Q value; then, combined with the reward obtained in step three, the Critic network is iteratively updated using the Bellman equation to improve the accuracy of long-term value estimation of the action; at the same time, the Actor network updates the policy gradient according to the global Q value of the Critic network, aiming to maximize the expected cumulative reward; experience replay is introduced during training, and the state, action and reward value data obtained by the UAV in steps one, two and three are stored in a buffer, and training is carried out through random sampling; in addition, the target network parameters are gradually updated using a soft update strategy to enhance the stable convergence of training; after centralized training, each UAV downloads the trained Actor network parameters; in actual operation, each UAV only inputs its Actor network according to its local observation, and outputs the decision action to realize distributed autonomous cooperative control.

Images