Multi-uav edge computing oriented task offloading and trajectory joint optimization method

Abstract

The application discloses a task unloading and trajectory joint optimization method for multi-unmanned aerial vehicle edge computing, relates to the unmanned aerial vehicle mobile edge computing field, and collects system position, communication and task information first, and pre-distributes service target points for unmanned aerial vehicles through K-means clustering; after detecting a DAG ready subtask, a high-layer discrete unloading decision is completed based on an improved deep Q network (IDQN); each time slot outputs continuous trajectory actions of the unmanned aerial vehicle in reliance on an improved double-delay deep deterministic policy gradient (ITD3); the position, channel, task and energy consumption states of the unmanned aerial vehicle are updated and the experience update network is stored in layers; after training convergence, layered strategies are scheduled online, and the total computing amount of the system in a time window is maximized. The layered architecture of the application decouples discrete and continuous mixed decisions, relieves high-dimensional action space and training instability problems, adapts to multi-unmanned aerial vehicle cooperation, complex DAG delay-sensitive tasks, and improves the real-time performance of online decision and the system computing throughput.

Description

Technical Field

[0001] This invention relates to the field of mobile edge computing for unmanned aerial vehicles (UAVs), and in particular to a method for joint optimization of task offloading and trajectory for multi-UAV edge computing. It is designed for multi-UAV edge computing scenarios with task dependencies in directed acyclic graphs, and achieves joint optimization of task offloading and UAV trajectory through event-driven hierarchical reinforcement learning. Background Technology

[0002] With the development of mobile internet, IoT, and smart terminal technologies, computationally intensive and latency-sensitive services such as image processing, collaborative sensing, target recognition, and real-time control are constantly increasing. Mobile edge computing, by deploying computing resources closer to users, can effectively reduce task processing latency and alleviate core network load.

[0003] Unmanned aerial vehicles (UAVs) are characterized by high mobility, flexible deployment, and good line-of-sight communication capabilities, making them ideal for serving as aerial mobile edge nodes to provide on-demand computing services to ground users. In scenarios with limited infrastructure, dynamically changing user distribution, or sudden emergencies, UAV-assisted mobile edge computing can improve coverage flexibility and service accessibility.

[0004] In real-world computing scenarios, data dependencies and timing constraints are common. Typical applications such as collaborative perception, multi-stage video parsing, and closed-loop real-time control require breaking down into multiple subtasks with sequential execution relationships. The industry typically uses Directed Acyclic Graphs (DAGs) to model these tasks. For DAG-structured tasks, any subtask can only enter an executable state after all preceding subtasks have completed data interaction. The unloading strategy of a single subtask is propagated to all subsequent subtasks through the data dependency chain, directly affecting the overall task completion latency.

[0005] In multi-UAV edge computing systems, there is a tight coupling between UAV trajectory planning and task offloading. On the one hand, task offloading changes the UAV's computational load, queuing status, and onboard power consumption; on the other hand, the UAV's real-time flight position affects the air-to-ground channel gain, task uplink transmission rate, and offloading latency. UAV flight power consumption is also influenced by speed and trajectory; therefore, the system needs to make a comprehensive trade-off between task processing capacity, communication quality, and energy consumption.

[0006] Current joint optimization schemes for multi-UAV assisted edge computing mainly fall into two categories, both of which have significant limitations: The first category, traditional mathematical optimization methods, typically require simplification of the system model, and are prone to excessive computational complexity under conditions of multiple UAVs, multiple users, DAG task dependence, and online decision-making. The second category, single-layer deep reinforcement learning methods, while possessing adaptive learning capabilities, face challenges such as high-dimensional mixed action spaces, training instability, and low convergence efficiency when simultaneously handling discrete task unloading actions and continuous UAV trajectory control actions. Summary of the Invention

[0007] The purpose of this invention is to propose a task offloading and trajectory joint optimization method for multi-UAV edge computing, which addresses the problems of high action space dimension, unstable model training, and insufficient real-time decision-making efficiency in existing methods under complex DAG tasks, multi-UAV collaboration, and discrete-continuous hybrid decision-making scenarios.

[0008] To achieve the above objectives, the technical solution of this invention is as follows: A joint optimization method for task offloading and trajectory for multi-UAV edge computing, which decomposes the original joint optimization problem into two sub-problems with different time scales: high-level task offloading decision and low-level UAV trajectory control, by constructing a multi-UAV assisted mobile edge computing system model and a DAG task model; the high-level layer uses the DAG sub-task entering the ready state as the trigger event and adopts an improved deep Q network IDQN to output discrete offloading decisions; the low-level layer uses a fixed time slot as the trigger period and adopts an improved dual-delay deep deterministic strategy gradient ITD3 to output continuous speed and heading control actions; the high-level and low-level layers are trained through independent experience pools and achieve implicit cooperation through UAV position, channel quality, computational load, and remaining energy.

[0009] In this invention, the multi-UAV assisted mobile edge computing system includes K ground users and M UAVs, each flying at a fixed altitude and acting as a mobile edge server. The system time is divided into multiple discrete time slots. Within each time slot, the air-to-ground channel gain and transmission rate are calculated based on the three-dimensional distance between the UAVs and users, and the UAV energy consumption is calculated based on the rotorcraft UAV flight power consumption model and the onboard computing power consumption model.

[0010] In this invention, each user task consists of multiple subtask nodes and directed dependency edges. If a subtask is executed locally, the execution latency is calculated based on the user's local computing frequency; if a subtask is offloaded to a drone for execution, the total latency includes the input data upload latency, the drone computing latency, and the output data return latency.

[0011] Compared with existing technologies, the present invention has the following advantages: First, it reduces the learning complexity of the hybrid action space by modeling discrete unloading decisions and continuous trajectory control in layers; second, it reduces redundant unloading decisions in the no-task state by using DAG ready tasks as event triggering conditions; third, it improves the coverage efficiency of UAVs by pre-computing target points through K-means clustering; fourth, it enhances training stability through improved mechanisms of IDQN and ITD3; and fifth, it improves the task completion rate of multi-UAV edge computing systems under limited time and energy constraints through implicit collaboration between high and low layers. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of a multi-UAV assisted mobile edge computing system scenario in an embodiment of the present invention.

[0013] Figure 2 This is a flowchart of the task unloading and trajectory joint optimization method in an embodiment of the present invention.

[0014] Figure 3 This is a schematic diagram of the event-driven hierarchical reinforcement learning algorithm framework in an embodiment of the present invention.

[0015] Figure 4 This diagram illustrates the performance comparison between the IDQN-ITD3 algorithm provided in this embodiment of the invention and existing algorithms under a fixed unloading strategy. The performance metrics being compared are, in order, the amount of data to be computed, the unloading reward, and the average reward for drones.

[0016] Figure 5 This diagram illustrates the performance comparison between the IDQN-ITD3 algorithm provided in this embodiment of the invention and existing algorithms under a fixed trajectory strategy. The performance metrics being compared are, in order, the amount of data to be computed, the unloading reward, and the average reward for the UAV.

[0017] Figure 6 The diagram shows the simulation results of the system computational workload of the IDQN-ITD3 algorithm provided in the embodiments of the present invention and existing algorithms. (a) is a comparison diagram of the IDQN-ITD3 algorithm with different combinations of existing algorithms, and (b) is a comparison diagram of the IDQN-ITD3 algorithm with existing single-agent algorithms. Detailed Implementation

[0018] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0019] like Figure 1As shown, in one embodiment, the multi-UAV assisted mobile edge computing system includes K static ground users and M UAVs; the UAVs fly at a fixed altitude H and act as mobile edge servers, communicating with ground users via an air-to-ground line-of-sight link, possessing both wireless access and edge computing capabilities. The system time is discretized into T time slots, each time slot having a length of Δt, and the system state within each time slot is approximately quasi-static. Figure 2 As shown, the task offloading and trajectory joint optimization method for multi-UAV edge computing provided in this embodiment achieves joint optimization of task offloading and trajectory through the following steps S1 to S8.

[0020] Step S1: Obtain the set of ground users, the set of drones, communication environment information, and user computing tasks within the service area of ​​the multi-UAV assisted mobile edge computing system. In this embodiment, the horizontal coordinates of all ground users within the service area are collected. The horizontal coordinate of drone m in time slot t and obtain the remaining energy of the drone. Current computing load, total system bandwidth (B), and drone (m) in time slot. t Number of communication links for the service User transmit power Noise power spectral density Channel power gain at reference distance and path loss index α .

[0021] Based on the above communication environment parameters, the three-dimensional distance between user k and drone m is calculated. Air-to-ground channel power gain and uplink / downlink transmission rates This serves as the input for subsequent task unloading latency, energy consumption constraints, and the state of high and low layer reinforcement learning. , and As shown in formulas (1) to (3) respectively: (1) (2) (3) In this embodiment, the total bandwidth B is read from the configuration parameters of the wireless access system or resource scheduler and can be set to 10MHz; This is based on the number of users who established effective connections with drone m within time slot t; It can be set to 1W by reading from the user terminal physical layer power control configuration or transmit power report; It can be measured by the receiver in an idle frequency band or determined based on thermal noise and receiver noise figure, and can be taken as -130dBm / Hz; It can be calculated using the following formula; α By fitting RSSI / CSI measurement data at different distances, or based on preset values ​​for typical scenarios, a value of 2.5 can be used.

[0022]

[0023] In the formula, Represents the speed of light. Indicates the carrier frequency. This represents the reference distance for communication, which can be taken as 1 m.

[0024] Step S2: Before training begins, the service area is spatially clustered based on the locations of ground users to pre-calculate the corresponding service target points for each UAV. Specifically, this is done using the set of two-dimensional location coordinates of all ground users. As input, initialize M cluster centers according to the number of drones M, assign each user to the nearest cluster center and iteratively update each cluster center until convergence; the clustering objective is to minimize the sum of squared distances from all users to their respective cluster centers, as shown in formula (4); the converged m-th cluster center As the preset service target point for the m-th UAV, as shown in Formula (5), the low-level trajectory control obtains target guidance towards the task-intensive area in the early stage of training, rather than relying solely on random exploration.

[0025] (4) (5) In the formula, This represents the m-th cluster center. This represents the service target point assigned to the m-th UAV. This target point is used to provide guidance for low-level trajectory control in mission-intensive areas, without limiting the continuous speed and heading control actions output by the UAV during online decision-making.

[0026] Step S3 involves modeling the user's computation task as a directed acyclic graph (DAG) and completing the state detection ready subtask based on the predecessor subtasks. Specifically, the computation task of the k-th ground user is modeled as a directed acyclic graph. ,in Denotes the set of subtasks for user k. This represents the data dependencies and execution order constraints (i.e., data dependency edges) between subtasks. For example... Figure 1As shown in the rightmost legend, a directed acyclic graph of a certain user terminal contains nine subtasks. Starting from the initial subtask 1, they are executed sequentially according to the arrow link order until the final subtask 9 is reached. It is tentatively assumed that subtasks 1, 6, 7, and 9 are executed on the local user terminal, while subtasks 2, 4, 8, and 35 are assigned to different drones for execution.

[0027] Each subtask By triplet The descriptions represent the computational load, input data volume, and output data volume (in bits), respectively. Each time slot examines the set of predecessor subtasks for each subtask. The completion status is determined only when all predecessor subtasks are completed and the necessary data transmission is in place. In this case, the subtask is added to the ready set and a high-level unloading decision is triggered, as shown in formula (6): (6) In the formula, This represents the set of subtasks that user k has completed up to time slot t. Let be the set of ready subtasks for user k in time slot t. Only subtasks that meet this prerequisite condition are added. This triggers the event-driven high-level unloading decision in step S4.

[0028] Step S4: When the ready subtask is detected, a high-level offloading decision agent is triggered. This agent, based on an improved deep Q-network (IDQN), performs action value assessments on both local execution and offloading to various UAVs for execution of the ready subtask, to determine the execution node of the ready subtask. Specifically, when a ready subtask is detected... Time-constructed high-level state ,in Includes all drone locations, remaining energy, and current computing load, as well as ready subtask attributes. The user location that generated this subtask Design high-level action space Action 0 indicates local execution, action This indicates that the unloading will be performed on the m-th drone.

[0029] The latency of the ready subtask being executed locally and the latency of being unloaded to the UAV for execution are calculated separately, and a high-level reward function is constructed based on the difference between the two to guide IDQN to prioritize the execution node that can reduce the completion latency of the subtask. For the triggered ready subtask, the local execution latency, the unloading execution latency, and the high-level reward function are shown in Equations (7) to (9), respectively: (7) (8) (9) In the formula, Subtasks Local execution latency, This represents the local computation frequency of user k. This represents the number of CPU cycles required to process a unit of bit of data, measured in cycles / bit. It can take values ​​such as... , For drone sub-missions The computational load, This represents the local task execution queue for user k, with the absolute value indicating the number of tasks in the queue. This indicates the total latency from unloading the subtask to execution by the drone. and These represent the input and output data volumes of the subtask, respectively. The uplink transmission rate is calculated based on formula (4). This represents the calculation frequency of the m-th drone. This represents the computation task queue for drone m, with the absolute value indicating the number of tasks in the queue. This is an immediate reward for high-level operations, consisting of the difference between local execution latency and offload execution latency, used to encourage IDQN to select execution nodes that can reduce completion latency.

[0030] like Figure 3 As shown, the training process of IDQN specifically includes: establishing the current Q-network. With the target Q network And introduce three improvements. First, use the current Q network to select the optimal action in the next state, and use the target Q network to calculate the target Q value. Using double Q learning to construct the target value can alleviate the overestimation of Q value and reduce the overestimation error in traditional DQN. The target Q value is shown in formula (10); Second, use time difference error The Huber loss function updates the current Q network parameters. It is quadratic in the small error segment and linear in the large error segment to suppress the gradient of abnormal samples. The Huber loss function is shown in Equation (11). Thirdly, the action exploration is carried out by the ε-greedy annealing mechanism that decays exponentially with the training rounds. The adaptive exploration rate is shown in Equation (12).

[0031] (10) (11) (12) In the formula, This indicates the current state of the Q network at the next higher layer. The optimal action to choose is... Indicates the current high-level status The next higher-level state obtained after executing the uninstallation action and updating the task execution status. As a candidate action for the next state, The immediate reward for the high-level personnel corresponding to the e-th DAG ready event. As a discount factor, These are the current Q-network parameters. For the target Q network parameters, The target Q value; The error between the current Q estimate and the target Q value. This represents the current high-level state observed by the high-level IDQN when the e-th DAG ready event is triggered. This indicates the current high-level IDQN status. The actual uninstallation action selected and executed; This is the Huber loss threshold. This represents Huber's loss value. Let be the exploration rate in the t-th round of training. This is the initial exploration rate. To minimize the exploration rate, This represents the decay coefficient. Therefore, sufficient exploration is maintained in the early stages of training, gradually stabilizing to utilize the current unloading strategy in the later stages.

[0032] Step S5: A low-level motion decision agent is triggered in each fixed time slot. This agent outputs continuous trajectory control actions for each UAV based on the improved dual-delay deep deterministic policy gradient (ITD3) algorithm. Specifically, an independent low-level motion decision agent is set up for each UAV m to construct the low-level state. ,in To normalize the residual energy; assume continuous motion at the lower level. The output continuous trajectory control actions include speed adjustment and heading angle adjustment, used to adjust the flight speed of the UAV m. With heading Lower-level rewards The reward is composed of a weighted sum of the service target point distance reward, the heading alignment reward, the energy consumption penalty, and the boundary penalty. This reward design enables the UAV to avoid high energy consumption and outbound flight while approaching the service area, as shown in formula (13): (13) in: , , , .

[0033] In the above formula, This indicates the reward items related to the distance between the drone and the service target point. This is a reference distance, usually set according to the scale of the service area; This indicates a bonus item for aligning the course with the target direction. The maximum permissible flight speed for a drone; This indicates an energy consumption penalty item. This is a single-slot reference energy consumption. This represents the boundary constraint penalty term. , and These are the corresponding weight coefficients, and their sum is 1. In this embodiment, After normalization by the reference distance, the value range is [-1, 0]; The value range is [-1, 1]; The value range is [-1, 0]; when the drone is within the service area Set the value to 0, and set it to -10 when the drone flies out of the service area.

[0034] like Figure 3 As shown, the training process of the improved dual-delay deep deterministic policy gradient ITD3 algorithm specifically includes: deploying an actor network for each drone. Two commentator networks , The corresponding target network is introduced, and two improvements are made. The first is state-dependent adaptive exploration: the difference between the outputs of two commentators on the same state-action pair is defined as the value estimation uncertainty, as shown in Equation (14), and the exploration noise intensity is adaptively adjusted accordingly, as shown in Equation (15). This setting increases exploration when the estimation discrepancy is large, so that high-value target actions are kept less disturbed; the second is value-perceived target policy smoothing: the target policy smoothing noise intensity is dynamically adjusted according to the average value estimation of the target action by the target commentator network, as shown in Equations (16) to (18). The commentator network constructs a conservative target by taking the minimum value of the bi-target Q, as shown in Equation (19): (14) (15) (16) (17) (18) (19) In the above formulas, This represents the continuous trajectory control action currently being performed by the m-th UAV. Indicates the execution of an action And the next lower-level state obtained after updating the environment, This indicates that the target actor network is in the next lower-level state. The target trajectory action generated below, and For the corresponding two target critic networks, This indicates the uncertainty in value estimation reflected by the difference in outputs from the two commentators; Based on the exploration of noise scale, This is the uncertainty adjustment coefficient. To adaptively explore noise; This represents the two target commentator networks' views on the next state action pair. Average value estimate, This represents the dynamically adjusted target strategy to smooth out noise scales. This represents the basic standard deviation of the target policy's smoothed noise. This represents the clipped Gaussian noise. This indicates that the mean is 0 and the standard deviation is 0. Gaussian distribution, Noise clipping threshold, This represents a numerical truncation function used to limit noise or motion within a preset range. This represents the low-level instant reward for the m-th drone. This represents the target Q-value used to update the critic network. For the target actor network of the m-th drone, it is... To input the target trajectory, a noise-free target trajectory action is first generated, representing the velocity and heading adjustments of the m-th UAV in the next state. Then, clipped Gaussian-smooth noise is added, and the action is restricted to the legal action range [-1, 1] using the clip function. This results in the target action used by the target critic network to calculate the target's Q-value. .

[0035] Step S6: Update the UAV position based on the execution node and continuous trajectory control actions, including the UAV flight speed, heading angle, and position; recalculate the channel status and transmission rate between the UAV and each user according to the aforementioned formulas (1) to (3); update the local task execution queue and the UAV calculation task queue, subtask start time and completion time according to the execution node determined by the higher layer; and calculate the UAV's flight energy consumption. Computational energy consumption When the UAV has insufficient remaining energy or flies out of the service area boundary, constraints or penalties are imposed on the corresponding actions to ensure that the task execution process meets the energy causal constraints and service area constraints. The UAV position update, energy consumption calculation and energy causal constraints are shown in Equations (20) to (22), respectively, where the energy constraints are used to determine whether the trajectory actions and task execution are feasible.

[0036] (20) (twenty one) (twenty two) In the formula, and Let be the flight speed and heading angle of the m-th UAV in time slot t, respectively. The time slot length, The position of UAV m in the next time slot; The effective capacitance coefficient of the airborne processor. This represents the computing frequency of the onboard processor of the m-th UAV. This represents the total computational load performed by UAV m within time slot t. This provides the maximum available energy for the drone.

[0037] Step S7: Store the high-level unloading decision experience and the low-level trajectory control experience into their respective experience replay buffers, and update the network parameters of the high-level unloading decision agent and the low-level motion decision agent respectively. Specifically, during training, the high-level unloading decision experience... As shown in formula (23), when the high-level agent detects the DAG ready event and completes a high-level unloading decision, it forms a high-level experience sample by including the high-level state, unloading action, immediate reward, and next high-level state corresponding to the decision, and writes it into the high-level experience replay buffer; within each fixed time slot, the low-level trajectory experience of the m-th UAV is stored in the high-level experience replay buffer. Store it in the corresponding low-level experience replay buffer, as shown in formula (24).

[0038] (twenty three) (twenty four) In the formula, and These are the high-level unloading experience pool and the low-level trajectory experience pool for the m-th UAV, respectively. , , and These are the high-level state, unloading action, immediate reward, and next high-level state corresponding to a ready event; , , and These represent the low-level state, continuous trajectory control actions, instant reward, and next low-level state of the m-th UAV in time slot t, respectively.

[0039] During network updates, high-level agents sample experience samples from the high-level experience replay buffer and update the high-level unloading decision network according to the improved deep Q-network IDQN; low-level agents sample experience samples from the corresponding low-level experience replay buffer and update the low-level trajectory control network according to the improved dual-delay deep deterministic policy gradient ITD3 algorithm. To improve training stability, the target network parameters are synchronized using a soft update method, and the update relationship is shown in formula (25): (25) In the formula, This represents the current network parameters. The current network is used to update gradients based on empirical samples and learn the action value function. This represents the target network parameters. The target network has the same structure as the current network and is used to calculate the target value and provide a relatively stable training objective. The soft-update coefficients, by allowing the target network parameters to slowly follow the current network parameters in a soft-update manner, can prevent the target value from changing drastically with the current network, thereby improving training stability. Through the above high-low layered structure, the higher layers are updated according to ready events, and the lower layers are updated according to time slots. The two layers share the environment state and task completion feedback during training, but maintain independent experience pools and independent network parameter updates, thereby decoupling discrete unloading decisions and continuous trajectory control and co-optimizing them.

[0040] Step S8: The high-low layered strategy (i.e., the high-level IDQN network and the low-level ITD3 actor network of each UAV) after training convergence is deployed to the multi-UAV assisted mobile edge computing system to generate task unloading decisions and UAV trajectory control actions online, so as to maximize the total amount of computation completed by the system within the preset task time window. Specifically, during online operation, steps S1 to S3 are executed first to obtain the system status and detect ready subtasks. When a ready event occurs, the high-level IDQN outputs the task execution node (high-level unloading action) according to formula (26). In each fixed time slot, the low-level ITD3 actor network outputs the UAV trajectory control action according to formula (27) and feeds back the updated system status to the next decision cycle, so as to perform joint optimization within the preset time window T with the goal of maximizing the total amount of computation completed by the system, as shown in formula (28): (26) (27) (28) In the formula, This represents the high-level state constructed when the e-th DAG ready event is triggered, which consists of the UAV state, the current ready subtask attributes, and the corresponding user location information; This indicates that the high-level decision-making agent is being offloaded. The selected Q-value maximum unloading action belongs to the higher-level action space. ={0,1,...,M}, where a value of 0 indicates that the ready subtask is assigned to the user's local machine for execution, and a value of m indicates that the ready subtask is offloaded to the m-th drone for execution; The value function of high-level actions after training; For the low-level actor network of the m-th drone, This is the current low-level state of the drone. This outputs continuous trajectory control actions. Thus, the system can generate an unloading node when a ready subtask is detected, and synchronously generate UAV motion control actions in each time slot. By maximizing... High-level unloading decisions and low-level trajectory control are unified under the same task completion benefit objective.

[0041] In one simulation embodiment, a simulation scenario of multi-UAV-assisted mobile edge computing is built. The basic simulation parameters are: setting the number of UAVs. ground user numbers The service area has a side length of 500m, and the total number of simulation time slots is [number missing]. Single time slot length The UAV has a fixed flight altitude of H=100m, a maximum flight speed of 20m / s, an initial onboard energy of 500kJ, and an onboard computing frequency. GHz, user local computing frequency GHz, total system communication bandwidth B=10MHz, path loss exponent α=2.5, channel gain at reference distance 1m. Number of CPU cycles required to process a unit bit of task The number of DAG subtask nodes ranges from 2 to 9, and the data size of the subtasks ranges from 100 to 1000KB.

[0042] Simulation experiments were conducted to compare and verify performance across three dimensions: trajectory optimization performance under a fixed unloading strategy, unloading decision performance under a fixed UAV trajectory, and overall scheduling performance of different algorithm combinations / single-layer agents in a complete joint optimization scenario. Figure 4 , Figure 5 , Figure 6 As shown. Figure 4This experiment presents comparative curves showing how system metrics change with training rounds when a fixed task offloading and allocation scheme is used, with only the trajectory optimization algorithm being changed. The experiment uniformly uses IDQN as the high-level offloading decision algorithm, and four trajectory optimization algorithms (ITD3, TD3, DDPG, and PPO) are implemented at the lower levels. The improvement effect of different low-level algorithms on the total computational workload of the system is compared.

[0043] Depend on Figure 4 As the curves show, all algorithms exhibit exploratory oscillations in the early stages of training. With increasing training rounds, the computational load of all algorithms gradually converges and improves. The layered scheme with ITD3 consistently remains above all comparison curves, allowing it to complete more sub-task computations within the same training rounds. This result demonstrates that the ITD3 algorithm proposed in this invention, relying on a dual-critic value uncertainty adaptive noise adjustment mechanism, can effectively reduce the exploration bias in UAV trajectory control, optimize UAV flight trajectories under fixed unloading assignment conditions, improve air-to-ground communication quality, and increase the total amount of computational data the system can process per unit time.

[0044] Figure 5 This experiment compares the total computational load of the system over training rounds when only the high-level unloading decision algorithm is replaced to lock the UAV's flight trajectory. ITD3 was used uniformly as the low-level trajectory control algorithm in the experiment, while four discrete unloading decision algorithms (IDQN, DQN, PPO, and A2C) were configured for the high-level operation for comparison. Figure 5 As the curves show, traditional DQN suffers from Q-value overestimation, resulting in slow training convergence and a lower lower bound on steady-state computation. PPO and A2C algorithms are not well-suited for discrete decision-making driven by DAG ready events. In contrast, the IDQN of this invention, relying on a double-Q network structure to suppress overestimation errors and coupled with an exponentially decaying adaptive exploration rate, fully traverses various offloading schemes in the early stages of training and stably selects low-latency offloading nodes in the later stages. Overall, its steady-state computation is significantly superior to other comparative algorithms, verifying the advantages of IDQN in offloading decision-making for DAG multi-subtask dependent scenarios.

[0045] Figure 6 The system computational complexity comparison results for different combinations of reinforcement learning algorithms in scenarios with no fixed constraints and joint optimization of unloading and trajectory synchronization are divided into (a) a comparison subgraph of multiple algorithm combinations and (b) a comparison subgraph of hierarchical architecture and single-layer agent. The vertical axis represents the total computational complexity completed within the system's time window, and the horizontal axis represents the number of training iterations. Figure 6As shown in (a), the comparison combinations include five schemes: ITD3-IDQN, TD3-DQN, DDPG-PPO, DDPG-DQN, and PPO-A2C. The curves show that all algorithms gradually converge with the increase in the number of training rounds. The ITD3-IDQN hierarchical combination of this invention has the highest computational cost throughout the entire process, with an average computational cost of 55906.29KB in the last 50 training rounds, higher than all other comparison algorithm combinations. Figure 6 As shown in (b), the hierarchical architecture (ITD3-IDQN) is compared with the single-layer single-agent algorithms TD3, DDPG, and PPO. The single-layer agent simultaneously outputs two types of mixed actions: discrete unloading and continuous trajectory. The action space has a high dimension, resulting in significant training oscillations and a low upper limit for steady-state performance. In contrast, the hierarchical architecture adopted in this invention separates discrete unloading and continuous trajectory control into independent agents at high and low layers, decoupling the mixed action space. This leads to stronger training stability, and the computational cost of the system after convergence is significantly higher than that of the single-layer reinforcement learning scheme.

[0046] comprehensive Figure 6 The results from the two sets of subgraphs demonstrate that the event-driven hierarchical reinforcement learning framework of this invention, through the collaborative optimization of IDQN high-level offloading decision and ITD3 low-level trajectory control, solves the defects of high-dimensional action space, unstable training, and insufficient scheduling efficiency in complex DAG tasks, multi-UAV collaboration, and discrete-continuous hybrid decision-making scenarios. The overall joint optimization performance is superior to existing single-layer reinforcement learning and ordinary unimproved deep reinforcement learning combination schemes.

[0047] The above descriptions are merely preferred embodiments of this application, and the present invention is not limited to the above embodiments. It is understood that other improvements and variations directly derived or conceived by those skilled in the art without departing from the spirit and concept of the present invention should be considered to be included within the protection scope of the present invention.

Claims

1. A method for joint optimization of task offloading and trajectory in edge computing for multiple unmanned aerial vehicles (UAVs), characterized in that, Includes the following steps: Acquire information on the set of ground users, the set of drones, the communication environment, and the user computing tasks within the service area of ​​the multi-UAV assisted mobile edge computing system; Based on the location of ground users, the service area is spatially clustered to pre-calculate the corresponding service target points for each drone; The user computing task is modeled using a directed acyclic graph (DAG), and the state detection ready subtask is completed based on the predecessor subtask. When the ready subtask is detected, a high-level offloading decision agent is triggered. This agent evaluates the action value of the ready subtask based on the improved deep Q network IDQN, whether it is executed locally or offloaded to each UAV for execution, in order to determine the execution node of the ready subtask. A low-level motion decision agent is triggered in each fixed time slot. The agent outputs continuous trajectory control actions for each UAV based on the improved dual-delay deep deterministic policy gradient ITD3 algorithm. The continuous trajectory control actions include speed adjustment and heading angle adjustment. Based on the execution node and the continuous trajectory control action, update the UAV position, wireless channel status, task execution status, task completion time, and UAV energy consumption status; The high-level unloading decision experience and the low-level trajectory control experience are stored in the corresponding experience playback buffers, and the network parameters of the high-level unloading decision agent and the low-level motion decision agent are updated respectively. The high-low stratification strategy after training convergence is deployed to the multi-UAV assisted mobile edge computing system to generate task unloading decisions and UAV trajectory control actions online, so as to maximize the total amount of computation completed by the system within the preset task time window.

2. The task offloading and trajectory joint optimization method for multi-UAV edge computing according to claim 1, characterized in that, Obtaining the communication environment information includes: Collect the horizontal coordinates of all ground users and drones within the service area, and obtain the drone's remaining energy, current computing load, total system bandwidth, number of drone communication links, user transmit power, noise power spectral density, reference distance channel gain, and path loss index. Based on the aforementioned communication environment parameters, the three-dimensional distance between the user and the UAV, the air-to-ground channel power gain, and the uplink transmission rate are calculated and used as inputs for subsequent task offloading delay, energy consumption constraints, and high- and low-level reinforcement learning states. The three-dimensional distance, air-to-ground channel power gain, and uplink transmission rate are respectively expressed as follows: (1) (2) (3) In the formula, Let K be the horizontal coordinate of the k-th ground user. For the m-th drone in time slot t Horizontal coordinates H To fix the flight altitude of the drone, This represents the three-dimensional distance between user k and drone m; The channel power gain at the reference distance. α This refers to the path loss index. This indicates the power gain of the air-to-ground channel; B The total system bandwidth, For the first m A drone in a time slot t The number of communication links for the service For the user's transmit power, The noise power spectral density, The uplink transmission rate is [value].

3. The task offloading and trajectory joint optimization method for multi-UAV edge computing according to claim 1, characterized in that, The step of spatially clustering the service area based on the location of ground users and pre-calculating the corresponding service target points for each UAV includes: Using the set of two-dimensional location coordinates of all ground users as input, the corresponding number of cluster centers are initialized according to the number of UAVs M. Each user is assigned to the nearest cluster center, and the cluster centers are updated repeatedly until convergence. The converged cluster centers are used as the service target points of the corresponding UAVs, providing target guidance for low-level trajectory control in the early stage of training for mission-intensive areas. The objective function of the clustering process and the service target points satisfy the following relationship: (4) (5) In the formula, K represents the number of ground users. This represents the two-dimensional location coordinates of the k-th ground user. Let m be the cluster center. This represents the m-th cluster center after iterative convergence. This represents the service target point assigned to the m-th UAV.

4. The task offloading and trajectory joint optimization method for multi-UAV edge computing according to claim 1, characterized in that, The user computation task is modeled using a directed acyclic graph (DAG), and a state-checking ready subtask is completed based on the predecessor subtask, including: The computational task of the k-th ground user is modeled as a directed acyclic graph. ,in Denotes the set of subtasks for user k. It represents the data dependencies and execution order constraints between subtasks; each subtask is described by the amount of computation, the amount of input data, and the amount of output data. In each time slot, check the completion status of the predecessor subtask set of each subtask. Only when all predecessor subtasks have been completed and the necessary data transmission has been completed, add the subtask to the ready subtask set and trigger the high-level offload decision. The set of ready subtasks is determined by the completion conditions of the predecessor, denoted as: (6) In the formula, For user k, the i-th subtask Subtasks The set of precursor subtasks, This represents the set of subtasks that user k has completed up to time slot t. Let be the set of ready subtasks for user k in time slot t.

5. The task offloading and trajectory joint optimization method for multi-UAV edge computing according to claim 1, characterized in that, When the ready subtask is detected, the high-level offloading decision agent is triggered in the following ways: When a ready subtask is detected, a high-level state is constructed, which includes the positions of all UAVs, remaining energy, computational load, and attributes of the currently ready subtask. The user location that generated the subtask; setting up a high-level action space containing all candidate options for local execution and offloading to individual drones for execution; The latency of executing ready subtasks locally and the latency of offloading them to the drone for execution are calculated separately. A high-level reward function is constructed based on the difference between the two to guide IDQN to prioritize the execution node that reduces the latency of subtask completion. For a triggered ready subtask, the local execution latency, offload execution latency, and high-level reward function are respectively represented as: (7) (8) (9) In the formula, Subtasks Local execution latency, This represents the local computation frequency of user k. This represents the number of CPU cycles required to process one unit of data. For drone sub-missions The computational load, This represents the local task execution queue for user k, with the absolute value indicating the number of tasks in the queue. This indicates the total latency from unloading the subtask to execution by the drone. and These represent the input and output data volumes of the subtask, respectively. For uplink transmission rate, This represents the calculation frequency of the m-th drone. This represents the computation task queue for drone m, with the absolute value indicating the number of tasks in the queue. This is the high-level immediate reward corresponding to the e-th ready event.

6. The task offloading and trajectory joint optimization method for multi-UAV edge computing according to claim 5, characterized in that, The training process of IDQN includes: Establish a current Q-network and a target Q-network. Use the current Q-network to select the optimal action in the next state, and use the target Q-network to calculate the target Q-value to alleviate Q-value overestimation. The Huber loss function is used to update the current Q network parameters, and an epsilon-greedy mechanism that decays exponentially with training rounds is used for action exploration, achieving a strategy of sufficient exploration in the early stage of training and unloading towards stability in the later stage. The target Q-value, Huber loss function, and adaptive exploration rate are respectively represented as follows: (10) (11) (12) In the formula, This indicates that the current Q-network is in the next higher-level state. The optimal uninstallation action selected below. This indicates the next higher-level state obtained after performing the uninstallation action. Candidate actions for the next state; This represents the current high-level state corresponding to the e-th ready event being triggered. This indicates the unloading action selected and executed in the current high-level state; These are the current Q-network parameters. For the target Q network parameters, Discount factor , The target Q value; The error between the current Q estimate and the target Q value. This is the Huber loss threshold. This represents Huber's loss value. Let be the exploration rate in the t-th round of training. The initial exploration rate, To minimize the exploration rate, This is the attenuation coefficient.

7. The task offloading and trajectory joint optimization method for multi-UAV edge computing according to claim 1, characterized in that, The process of triggering the low-level motion decision agent in each fixed time slot includes: For each UAV m, an independent low-level motion decision-making agent is set up to construct the corresponding low-level state of the UAV. The low-level state includes the UAV's horizontal position, current speed, current heading angle, and normalized remaining energy. A continuous action space is set up to adjust the speed and heading of the UAV. The low-level reward function is constructed by weighted summing of the distance reward between the UAV and the service target point, the alignment reward between the heading and the target direction, the energy consumption penalty, and the boundary penalty, which guides the UAV to approach the service target point and avoids overstepping boundaries and high-energy flight. The continuous trajectory control action and low-level reward function output by the low-level motion decision agent are respectively expressed as: (13) (14) In the formula, For the continuous trajectory control actions of the m-th UAV in time slot t, Adjustment amount for drone speed, This is the heading angle adjustment amount; For the corresponding low-level instant rewards for drones, Indicates the distance to the relevant reward item. This indicates the course alignment bonus. This represents the energy consumption penalty term. This represents the boundary constraint penalty term. , and These are the corresponding weighting coefficients.

8. The task offloading and trajectory joint optimization method for multi-UAV edge computing according to claim 7, characterized in that, The training process of the ITD3 algorithm includes: For each drone, an actor network, two critic networks, and a corresponding target network are set up. The value estimation uncertainty is constructed by the output difference between the two critic networks for the same state action pair. The exploration noise intensity is adaptively adjusted according to the uncertainty, so that the drone can enhance exploration in the region of inconsistent value estimation and reduce random disturbances in the region of stable value estimation. Based on the average value estimate of target actions by the target commentator network, the target strategy smoothing noise intensity is dynamically adjusted to enhance action smoothing in low-value or uncertain states and retain better actions in high-value states. The value estimation uncertainty, adaptive exploration noise, average value of target actions, and target policy smoothing noise scale are respectively defined as follows: (15) (16) (17) In the formula, This is the current low-level state. This indicates the currently executing continuous trajectory control action. Indicates the execution of an action The next lower-level state obtained after that, This represents the target actor's trajectory and motion generated by the network. and For the two commentator networks corresponding to the m-th drone, and For the corresponding two target critic networks, This indicates the uncertainty in value estimation reflected by the difference in outputs from the two commentators; Based on the exploration of noise scale, This is the uncertainty adjustment coefficient. The adaptive exploration noise; This represents the two target commentator networks' views on the next state action pair. Average value estimate, This represents the dynamically adjusted target strategy to smooth out noise scales. The base standard deviation represents the noise smoothing of the target strategy.

9. The task offloading and trajectory joint optimization method for multi-UAV edge computing according to claim 1, characterized in that, The step of updating the UAV position, wireless channel status, task execution status, task completion time, and UAV energy consumption status based on the execution node and the continuous trajectory control action includes: The drone's flight speed, heading angle, and position are updated according to the low-level continuous trajectory control actions, and the channel status and transmission rate between the drone and each user are recalculated. According to the execution nodes determined by the higher management, update the local task execution queue, the UAV computing task queue, the start time and completion time of subtasks, and calculate the flight energy consumption and computing energy consumption of the UAV. When the drone has insufficient remaining energy or flies out of the service area boundary, constraints or penalties are imposed on the corresponding actions to ensure that the mission execution process meets energy causal constraints and service area constraints. Among them, UAV position update, computational energy consumption, and energy causal constraints are defined as follows: (18) (19) (20) In the formula, and Let be the flight speed and heading angle of the m-th UAV in time slot t, respectively. The time slot length, The position of UAV m in the next time slot; This represents the computational energy consumption of the UAV within time slot t. The effective capacitance coefficient of the airborne processor. This represents the computing frequency of the onboard processor of the m-th UAV. The total computational load performed by UAV m within time slot t. The energy consumption of the UAV flight within time slot t. This provides the maximum available energy for the drone.

10. The task offloading and trajectory joint optimization method for multi-UAV edge computing according to claim 1, characterized in that, The online generation of task unloading decisions and UAV trajectory control actions maximizes the total computational load completed by the system within the preset task time window, including: During online operation, the system status is first obtained and ready subtasks are detected. When a ready event occurs, the high-level IDQN outputs the task execution node. At each fixed time slot, the low-level ITD3 actor network outputs the drone trajectory control actions and feeds the updated system status back to the next decision cycle, thereby realizing the online generation of task unloading and trajectory joint optimization strategies. The high-level unloading actions and low-level trajectory actions during the online decision-making phase are represented as follows: (21) (22) In the formula, This represents the high-level state when the e-th ready event is triggered, which consists of the drone state, the current ready subtask attributes, and the corresponding user location information; This indicates the unloading action with the highest Q value selected by the high-level unloading decision-making agent; The value function of high-level actions after training; For the low-level actor network after training the m-th drone, To correspond to the current low-level state of the drone, For continuous trajectory control actions output; The optimization objective for the total computational workload completed by the system is expressed as: (23) In the formula, This represents the total amount of computation completed by the system within the time window T. This represents the set of subtasks that have been completed within this time window. For user k, the i-th subtask This indicates the computational cost associated with the subtask.

Images