A Deep Reinforcement Learning-Based Method for UAV Trajectory and Resource Scheduling

Abstract

This invention discloses a UAV trajectory and resource scheduling method based on deep reinforcement learning, belonging to the field of resource scheduling technology. This invention utilizes a digital twin network of the physical network to simulate and optimize the communication interaction process in real time. By minimizing the task completion latency of all user devices within the average time slot, it achieves the QoE target for task offloading services in a multi-UAV edge network, thereby improving the overall network service efficiency and user experience. To address the constructed optimization objective, a deep reinforcement learning algorithm combined with a variational autoencoder module is used to solve the mixed action space problem. This effectively handles continuous and discrete action spaces, improving the algorithm's decision-making ability. Furthermore, a graph attention network module is introduced, enabling UAVs to adaptively capture the state features of other UAVs, thereby making more accurate joint decisions. This joint decision-making mechanism can significantly improve the task offloading efficiency and service quality of a multi-UAV edge network.

Description

Technical Field

[0001] This invention relates to the field of resource scheduling technology, and in particular to a method for UAV trajectory and resource scheduling based on deep reinforcement learning. Background Technology

[0002] In recent years, many new computationally intensive and latency-sensitive network services have emerged that require substantial computing resources. Multi-access edge computing (MEC) is an innovative computing paradigm that provides computing capabilities at the network edge to support such services. However, current MEC solutions suffer from several issues, such as the susceptibility of transmission links to ground-based MEC servers being blocked by obstructions, and users far from the MEC servers often experiencing poor coverage. Deploying fixed MEC servers is particularly challenging in emergency situations such as disaster areas and hotspots. In recent years, unmanned aerial vehicles (UAVs) have leveraged their flexible deployment and wide coverage potential to assist MEC systems. UAV-assisted MEC systems hold promise for providing solutions for performing highly distributed, computationally intensive tasks.

[0003] Single-drone systems, due to their limited resources, are well-suited for scenarios with a small number of tasks, and the performance improvements they offer for any given task are limited. Furthermore, it is often difficult to meet the computational resource demands of an ever-increasing number of tasks. Conversely, a more complex system relying on the collaboration of multiple drones is worthy of investigation. It can provide flexible and abundant computational resources and a large service coverage. However, in multi-drone MEC systems, the task offloading problem presents many challenges, including multi-drone trajectory design for better user coverage, communication resource management between drones to improve transmission efficiency, and collaborative task offloading to balance the computational workload of drones. While multiple drones certainly provide more computational resources, how to coordinate the cooperative relationships between drones remains a pressing issue to be addressed.

[0004] Most existing UAV-assisted task offloading solutions focus on partial offloading, which allows tasks to be broken down and processed separately. While partial offloading has many limitations, it may be difficult to apply to indivisible computational tasks.

[0005] In the 0-1 task offloading strategy, tasks are either completely offloaded to a UAV (unmanned aerial vehicle) or an edge server for execution, or executed entirely on the local device, without being split into multiple parts processed in different locations. This offloading method is more suitable for indivisible computational tasks compared to partial offloading. However, due to the binary choice characteristic of 0-1 offloading (i.e., the task is either fully offloaded or not offloaded at all), solving the task offloading problem requires simultaneously handling continuous variables (such as resource allocation and channel selection) and discrete variables (the task's 0-1 offloading decision). This makes the solution process more complex; that is, using 0-1 offloading may turn the task offloading problem into a joint optimization problem of continuous and discrete variables, further increasing the problem's complexity.

[0006] Furthermore, user requests and resource demands are highly time-varying. In this context, long-term average optimization is necessary to improve system performance. However, solving this non-convex problem is difficult in dynamic environments and when future information is incomplete. With advancements in artificial intelligence (AI) methods, deep reinforcement learning (DRL) has shown promising results in solving long-term optimization problems. Through interaction with the environment, DRL can help make appropriate decisions to obtain optimal long-term average returns, enabling intelligent decision-making in volatile and uncertain environments, thereby solving the problem.

[0007] To address mixed-integer optimization problems, some works utilize the Deep Deterministic Policy Gradient (DDPG) algorithm for training and employ continuous variable discretization to handle the discrete components. For continuous variables, the DDPG algorithm is directly applied, overcoming the limitation of DDPG and similar algorithms that can only handle continuous action spaces. During training, the neural network parameters are updated through policy gradient optimization, and the discretized continuous actions are transformed into discrete actions through action rounding, thus enabling the handling of mixed-integer optimization problems. However, this approach typically uses rounding to discretize the continuous data output by the neural network. Specifically, even if the model outputs 4.6 and 4.9, these values ​​will be rounded to the same discrete value of 5, leading to a decrease in model performance. A major problem with this method is that it is difficult to accurately convert continuous output values ​​into discrete variables, resulting in information loss and decreased accuracy. Furthermore, the converted continuous and discrete variables are separate and lack inherent correlation, thus hindering effective joint optimization. This separation method is not conducive to capturing the dependencies between variables, thereby affecting the overall optimization effect.

[0008] To address the resource shortage and limited coverage of single UAVs, some works employ multi-UAV computation to offload user tasks and use multi-agent reinforcement learning to train the environment. Each UAV is modeled as an agent with an independent policy network and value network. Movement and task scheduling are based on the UAV's current observations of the environment, and the policy is optimized through interaction with the environment. While this approach effectively solves the resource shortage problem by utilizing multi-UAV-assisted edge computing, most current works do not fully consider UAV networking and joint decision-making after networking. Because each agent in a multi-agent system has its own independent policy and value networks, these agents typically make decisions based only on their current state information. This independent decision-making method limits effective communication and cooperation among multiple UAVs, thus failing to fully utilize collective resources and information advantages. This limitation not only affects the coordination and collaboration capabilities of the multi-UAV system but also weakens the overall system optimization effect, resulting in suboptimal system performance. Summary of the Invention

[0009] To address the shortcomings of existing technologies, this invention proposes a UAV trajectory and resource scheduling method based on deep reinforcement learning. It utilizes a digital twin network of the physical network to simulate and optimize the communication interaction process in real time, taking the QoE of task offloading services in a multi-UAV edge network as the optimization objective. This objective is achieved by minimizing the task completion latency of all user devices within the average time slot, thereby improving the service efficiency and user experience of the entire network.

[0010] The technical solution adopted in this invention is as follows:

[0011] A deep reinforcement learning-based method for UAV trajectory and resource scheduling is proposed. In an edge network system with multi-UAV assisted task offloading deployed at a ground base station, a virtual connection is established with the physical network through a digital twin synchronization link. This allows for real-time collection of system communication interaction information via the digital twin network, and the following resource scheduling steps are executed:

[0012] A 0-1 unloading strategy is used to configure the user's connection to the drone, as well as the unloading strategy for the user's unloading of computing resources. Let x be defined. m,n (t) represents the unloading strategy of user m to unload the task to drone n, if x m,n If (t) = 1, it means the task is offloaded to the human-machine interface; where m = 1, 2, ..., M, n = 1, 2, ..., N, and M and N represent the number of users and the number of drones, respectively; if x m,n If (t) = 0, it means the task will not be unloaded onto the drone; define x m,0 (t) represents the user's unloading strategy for computing resources, if x m,0If (t) = 1, it means that user m chooses local computation. If x m,0 If (t) = 0, it means that user m chooses to offload computation to the Multi-Access Edge Computing (MEC).

[0013] Define the optimization goal:

[0014]

[0015] Where X represents the task offloading decision set, Ω represents the UAV location set, P represents the device transmit power set, O represents the task priority set, t represents the time slot, and T represents the number of time slots. and These represent the latency of edge computing and the latency of local computing within the task, respectively.

[0016] The optimization objective is solved based on Markov decision process and dual value network, and the resource scheduling result is obtained based on the optimization result.

[0017] Furthermore, when solving the optimization objective based on Markov decision processes and dual value networks, the state space involved in the Markov decision process... Action space and reward function They are respectively:

[0018] state space Where I(t) represents the user task state, and ω(t) represents the drone position;

[0019] Action space Where i(t) represents the discrete action space and c(t) represents the continuous action space;

[0020] i(t) = [i1(t), i2(t), ..., i M [(t)], where i m (t) = 0 indicates local computation by the user, i m (t) = N+1 indicates that the task is offloaded to MEC computation, i m (t) = j means that the task is offloaded to drone j for calculation, j = 1, 2, ..., N;

[0021] c(t) = {Δω(t), p(t), o(t)}, where Δω(t) represents the displacement of all UAVs at time t, p(t) represents the transmit power of all users and UAVs, and o(t) represents the priority of all tasks;

[0022] reward function Set to: when preset constraints are met (which can be changed to power limits, drone flight collision limits, etc.), the reward function will be... for Otherwise, set it to the preset penalty P. u , among which, T ij (t) represents the task completion delay for user i and drone j at time t.

[0023] Furthermore, in the Markov decision process, for discrete actions, an embedding table G is used. ω Map each discrete action to a continuous latent vector space to obtain discrete action i. m Potential discrete actions Embedded table G ω Each line of G ω (i m ) represents a discrete action i m A l1 (specified value) dimensional continuous vector, where represents the potential discrete action. ω represents the trainable parameters of the embedding table;

[0024] Encoding and decoding are performed using a conditional variational autoencoder, where the encoder bases continuous actions on the current state s and potential discrete actions. Mapping to the latent space yields the latent continuous action z, and the encoder output follows a Gaussian distribution; the decoder is used to decode the latent continuous action z to obtain the predicted continuous action. And by searching With embedded table G ω The nearest neighbor is used to obtain the corresponding discrete action.

[0025] Furthermore, when training and tuning the embedding table parameters ω, encoder parameters φ, and decoder parameters ψ using the training set, the loss functions used include reconstruction error, KL divergence (relative entropy) measuring the posterior distribution of the latent space z relative to the standard Gaussian prior distribution, and prediction error.

[0026] Furthermore, the dual value network includes a first value network (serving as the target value network), a second value network, a graph attention network, and an action network. The graph attention network takes the system's current state s as input (including the user's task state and the drone's position). The graph attention network constructs an undirected graph G with drones as nodes and drone features as node features. Drone features include their position and the sum of tasks within their current coverage area. Attention coefficients between different neighboring nodes in the undirected graph G are calculated based on an attention mechanism, and then normalized using a softmax function to obtain normalized attention coefficients between different neighboring nodes. Based on these normalized attention coefficients, a multi-head attention mechanism is used to obtain enhanced drone features. The data is then used as input to the action network. The action network generates potential discrete action vectors and potential continuous action vectors. The drone features, potential discrete action vectors, potential continuous action vectors, and an immediate reward based on the reward function are then fed into the first value network to obtain the target value. The second value network takes drone features, potential discrete action vectors, potential continuous action vectors, and an immediate reward as input and outputs the current value. The corresponding value loss is calculated based on the outputs of the two value networks, and the network parameters of the action network are tuned. Finally, the optimization result of the constructed optimization objective is obtained based on the decoder in the graph attention network, action network, and conditional variational autoencoder.

[0027] Furthermore, when tuning the network parameters of the action network, a deterministic policy gradient is used for tuning.

[0028] The technical solution provided by this invention brings at least the following beneficial effects:

[0029] (1) With minimizing the sum of user task processing delays as the optimization objective, this paper addresses the trajectory planning and task scheduling problem in a multi-UAV assisted MEC scenario. The joint optimization problem of UAV trajectory planning, unloading decision, computational resource allocation, and communication resource management is formulated as a mixed integer programming problem with constraints of 0-1 unloading, transmit power, computational capability, and UAV movement range. Furthermore, this problem is transformed into a Markov Decision Process (MDP).

[0030] (2) By introducing an embedding table and a variational autoencoder, the mixed-integer programming problem is cleverly transformed into a continuous optimization problem that can be directly handled by DRL. Through the encoder, a latent space of mixed actions is constructed. By combining the latent space and the double-delay deep deterministic policy gradient algorithm (TD3), the joint optimization problem with discrete-continuous mixed action spaces is solved.

[0031] (3) Based on the networking of multiple drones, GAT is used to help the agent pay more attention to the connection between drones and the impact of drone status on each other, so as to learn better cooperative scheduling strategies. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 A schematic diagram of an edge network model for offloading multiple drone-assisted tasks;

[0034] Figure 2 This is a schematic diagram of the processing procedure of the DRUTS (Deep Reinforcement UAV Task Scheduling) method. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be described in detail and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Generally, the components of the embodiments of the present invention described and shown in the accompanying drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present invention.

[0036] This invention proposes a task scheduling and UAV trajectory planning strategy based on deep reinforcement learning for UAV-assisted edge computing (DRUTS). While addressing continuous-discrete hybrid action decision-making, it leverages the network of UAVs and considers the relationships between the state features of each UAV to make action decisions that more closely resemble real-world scenarios. The core idea is to use embedding tables and variational autoencoders (CVAEs) to transform the hybrid action space into a latent continuous action space, and to utilize graph attention networks (GATs) to generate a new state space with inter-node relationships based on the UAV network, thereby solving the trajectory planning and task scheduling problems in multi-UAV assisted MEC scenarios.

[0037] In example Figure 1 The edge network model for multi-UAV assisted task offloading shown consists of two parts: a physical network and a digital twin network. The system includes M user devices and uses a set of... This represents a set of N drones equipped with MEC servers. Represented. Considering a three-dimensional Cartesian coordinate system, the position of user m at time t can be represented by q. m (t)=[x m (t),y m (t),0] T Similarly, ω n (t)=[x n (t),y n (t),z n (t)] TThis is used to represent the position of UAV n. To achieve long-term optimization of system performance without loss of generality, this embodiment of the invention considers a multi-slot dynamic MEC network. That is, a continuous time period is divided into multiple time slots of equal length. The set of time slots is used... This indicates that the length of each time slot is δ. t .

[0038] In the initial stage of time slot t, user m generates a computationally intensive task consisting of tuple I. m (t)={d m (t),c m (t)} represents, where d m (t) represents the data transfer size in bits, c m (t) represents the number of CPU cycles required to process each bit of this task.

[0039] This invention considers a 0-1 unloading mode to optimize a hybrid discrete and continuous space, i.e., user-generated task I. m (t) Computation can be performed locally or on a drone, with the drone supporting both data transmission and computation. On one hand, drones possess more powerful computing capabilities, enabling them to assist users in performing computational tasks. On the other hand, when drone computing resources are insufficient, computational tasks can be forwarded to other drones. A digital twin system deployed at a ground base station establishes a virtual connection with the physical network through a digital twin synchronization link. This allows for monitoring of the physical network's operating parameters, model, dynamic changes in resources, and the arrival of random tasks. In other words, it collects network information in real time through the digital twin network to facilitate algorithm execution.

[0040] exist Figure 1 In the example scenario shown, this embodiment of the invention aims to optimize the long-term QoE (Quality of Experience) of task offloading services in a multi-UAV edge network. This objective is achieved by minimizing the completion latency of all user devices within the average time slot. The completion latency of all user devices within time slot t can be formulated as:

[0041]

[0042] Where, x m,n (t) represents the 0-1 unloading strategy, when x m,n When (t) = 1, it means that user m has unloaded the task to drone n, x m,n When (t) = 0, it means that user m will not unload the task to drone n, x m,0 (t) represents the user's computing resource policy, when x m,0 When (t) = 1, it indicates that user m selects local computation, x m,0 When (t) = 0, it indicates that user m chooses to unload the calculation onto MEC. and These represent the latency of task edge computation and the latency of task local computation, respectively, as shown in the following formula:

[0043]

[0044] in, These represent the transmission time during the unloading process, the processing time after unloading to the MEC server, and the waiting time from task creation to unloading, respectively. These represent the computing resources used by user m for local computation.

[0045] When Task I m (t) After unloading to drone n′, assume each task has a priority o. m (t), tasks with higher priority will be processed first. First, the tasks are sorted according to their priority. After the task priority order is arranged, the maximum computing resources of the UAV will be used to execute each task in descending order of priority.

[0046] This invention's embodiments achieve joint optimization of unloading decisions. drone trajectory Equipment transmission power and task priority To minimize the average task completion latency across all time slots, this problem can be formulated as follows:

[0047]

[0048] Where, p m (t),p n (t) represents the transmit power of user m and drone n at time t, respectively, and T represents the number of time slots.

[0049] The optimization problem given in formula (4) can be solved using the Deep Reinforcement Learning (DRL) algorithm. The DRL algorithm is based on Markov Decision Processes (MDPs) and uses a neural network to implement the environment-based output of the agent's policy in the MDP, maximizing certain rewards. This embodiment uses the TD3 algorithm (Twin Delayed Deep Deterministic Policy Gradient), which effectively reduces the bias problem in Q-value estimation by introducing a dual network (two Q-networks) and a delayed update strategy, thus improving the stability of training.

[0050] Formula (4) presents two challenges: firstly, the problem involves both continuous and discrete variables, while the traditional TD3 algorithm is only suitable for solving problems in continuous solution spaces; secondly, in the scenario of this invention, UAVs have communication links and form a network, and the UAV's decision-making is affected by the state of neighboring UAVs. To address these two challenges, this invention proposes a new algorithm based on hybrid action representation, where the UAV can capture the state characteristics of other UAVs to make joint decisions.

[0051] The training and testing of the algorithm are as follows: In the testing phase, a state s is first initialized. Then, the state is transformed into a new feature representation h that considers the relationships between nodes of the drone through the GAT (Graph Attention Network) module. This new representation h is then input into the actor network (policy network) to obtain the potential discrete action output g and continuous action output z. These are then decoded to obtain the actual discrete action i and continuous action c, which interact with the environment to obtain the next state s′. A reward and a completion flag (done) are awarded. The results of each round of testing are then input into the experience replay buffer. For use during training. The training portion is from the experience replay buffer. The algorithm retrieves data stored during the testing process and inputs it into the target critic network and the cirtic network (value network). This yields the target Q value and the current Q value, respectively. Based on a temporal difference strategy, the obtained value loss (critic loss) is used to train the critic network, which in turn trains the actor network. This algorithm, while addressing continuous-discrete mixed action decision-making, leverages the networking between UAVs and considers the relationships between the state features of each UAV, resulting in action decisions that more closely resemble real-world scenarios. (See [link to relevant documentation]). Figure 2 The specific implementation process of the new algorithm proposed in this embodiment of the invention includes:

[0052] Module 1: MDP.

[0053] In a UAV-assisted MEC system, the system's state at the next time step depends only on the current state and action. In this case, the optimization problem described by equation (4) can be represented as a Markov decision process. At time slot t, the system state is observed, and an action is chosen to maximize the long-term system reward. This process can be represented by a tuple: Where γ∈[0,1) is the discount factor. These are the state space, action space, and reward function, which are described in detail below:

[0054] (1) State space

[0055] Including the channel quality of all links in the state space increases the algorithm complexity to O(N). 2 However, considering that channel quality is related to the drone's position, the algorithm complexity of only adding the drone's position ω(t) to the state space is O(N). In addition, the user randomly generates a task at the beginning of each time slot, so the state space is represented by the state of the user's task and the position of the drone.

[0056]

[0057] Please note that the above states change at different times, which means that the drone is constantly flying and the user is generating new tasks, which is more in line with reality.

[0058] (2) Action space

[0059] If the unloading decision X is directly used as the action space, then there are M(N+2) action spaces. This increases the output dimension of the neural network and is further complicated by the 0-1 unloading rule. For each user m, there are N+2 possible unloading locations. In this embodiment, i is used. m (t)∈{1,2,…,N+1} represents the unloading position, where i m (t) = 0 indicates local computation by the user, i m (t) = N+1 indicates that the task is unloaded to MEC computation; otherwise, it indicates that the task is unloaded to the i-th node. m The calculation is performed on (t) drones, i.e., i m (t) = j means that the task is unloaded to drone j for computation, where j = 1, 2, ..., N. In this way, the output dimension of the neural network can be reduced from M(N+2) to M, while avoiding the interference of 0-1 unloading constraints on training.

[0060] In addition to unloading decisions, the decisions will also include the drone's trajectory, the user's and drone's transmission power, and the drone's resource allocation to the user. The action space is divided into a discrete action space i(t) and a continuous action space c(t):

[0061]

[0062] Wherein, the discrete action space i(t) = [i1(t), i2(t), ..., i M [(t)] is the user's uninstallation decision vector, and the continuous action space is c(t)={Δω(t),p(t),o(t)}, where Δω(t)={Δω1(t),Δω2(t),…,Δω N (t)} represents the displacement of all UAVs at time t, Δω n (t)=[Δx n(t),Δy n (t),Δz n (t)], It is the distance that the drone n moves along the X, Y, and Z axes respectively, p(t) = [p i (t),p2(t),…,p M +N(t)] is the transmit power of all users and drones, o(t)=[o m (t)], This represents the priority of all tasks. The total dimension of the action space is M + 3N + (M + N) + M = 4N + 3M.

[0063] (3) Reward function

[0064] Since the goal of the optimization problem described by formula (4) is to minimize the task completion delay by optimizing the UAV trajectory, unloading decision, launch power, and resource allocation, the reward should be negatively correlated with the task completion delay, and is set as follows:

[0065]

[0066] Where, x ij (t) represents the unloading strategy from user i to drone j at time t. ij (t) represents the task completion delay between user i and drone j at time t, i.e. When constraints are met, a negative exponential reward is given because the optimization objective is to minimize task completion latency. This design allows for smaller penalties for short completion times and larger penalties for longer delays. When constraints are not met, such as in high-risk actions like a drone flying out of bounds or two drones colliding, a P-value is given. u (Default value) penalty.

[0067] Module 2: Potential Space.

[0068] In MDP systems, traditional DRL algorithms are unsuitable due to the problem's mixed discrete-continuous action space. Therefore, this invention proposes a novel algorithm that transforms the mixed action space problem into a continuous policy learning problem by introducing an embedding table and a conditional variational autoencoder (VAE), while also considering the dependencies between discrete and continuous variables. Discrete actions are represented using an embedding table, and a latent space for continuous actions is constructed using a conditional VAE. This allows the algorithm to handle the mixed action space more accurately, avoiding the limitations of directly using either discrete or continuous actions.

[0069] For discrete actions, an embedding table is used to map each discrete action to a continuous latent vector space, where the representation of the embedding table is as follows: Each user has N+2 unloading decision variables, where l1 is the vector length of each discrete action in the latent space, and the parameters ω of this table are learnable, meaning they are optimized during the learning process. In each row of the table g... ω,im =G ω (i m ) represents a discrete action i m There are l1-dimensional continuous vectors. M users make decisions in each time period. Although each user may need their own embedding table for personalized learning, they can share a common embedding table because all users have the same action space and action meaning.

[0070] For continuous actions, a conditional variational autoencoder (CVAE) is used to encode and decode these continuous actions. CVAE consists of an encoder that maps continuous actions to a Gaussian distribution in the latent space, and a decoder that reconstructs the continuous actions from the latent space.

[0071] Encoder: The neural network is represented by q φ (z∣∣c,s,g ω,im This network maps continuous actions c to latent variables z (latent continuous actions) using parameters φ. This mapping depends on the state s and the latent discrete actions g. ω,im ,in The encoder output follows a Gaussian distribution, which can be denoted as: Γ(μ) q ,σ q The latent representation z can be obtained by sampling from this distribution, where μ q ,σ q These are the mean and variance, respectively.

[0072] Decoder: The neural network is represented as The decoder uses the parameter ψ to decode the latent space z to obtain the continuous actions. (Predicted value), this mapping also depends on the state s and the potential discrete action g. ω,im By searching g ω,im With embedded table G ω The nearest discrete action is obtained by finding the minimum value of the difference between the action and the embedded table.

[0073] In summary, this embodiment uses an encoder to encode mixed actions (including discrete and continuous parts) into a latent space. The decoder reconstructs the original hybrid action from the representation in the latent space. The encoding and decoding processes are as follows:

[0074]

[0075]

[0076] During training, the embedding table parameters ω, encoder parameters φ, and decoder parameters ψ are jointly trained by minimizing the loss function:

[0077] Reconstruction error is the predicted value of continuous parameters. The squared L2 norm error between the latent space z and the true value c encourages the decoder to accurately reconstruct the input data. The KL divergence measures the difference between the posterior distribution of the latent space z and the standard Gaussian prior distribution. This term encourages the latent space to approximate the prior distribution, helps avoid overfitting, and increases generalization ability. Its corresponding loss function is expressed as follows:

[0078]

[0079] in, Represents the mathematical expectation. Characterizing the reconstruction error, d KL (q ψ (·|c,s,g ω,im Γ(0,I) represents the corresponding KL divergence, where Γ(0,I) denotes a standard multidimensional Gaussian distribution with mean 0 and covariance matrix I.

[0080] Prediction error measures the decoder's ability to predict system state transitions, aiming to closely approximate the actual state transitions. For experimental samples (s,a,s′), the state residual is defined as δ. s,s′ = s′-s. s and s′ represent the current and next states of the system, respectively. By introducing a cascaded structure into the decoder, the corresponding prediction residuals can be obtained:

[0081]

[0082] in, δ represents the predicted state residual. s,s′ This corresponds to the actual value.

[0083] The overall training loss is the weighted sum of the two losses mentioned above:

[0084] L H (φ,ψ,ω)=L V (φ,ψ,ω)+αL D (φ,ψ,ω) (13)

[0085] Here, α is a weighting variable that depends on the importance of the loss associated with the dynamic prediction representation. The system state s is defined by dimension dim, and the continuous policy p is represented by dimension dim.

[0086] Module 3: Graph Attention Networks.

[0087] This paper addresses the task computation problem in a multi-UAV cooperative assisted MEC (Multi-UAV Collaborative Intelligence System). Since UAVs can communicate and exchange data, their actions are influenced by the states of neighboring UAVs. Given these mutual influences, the real-time states detected by the digital twin system are modeled as an undirected graph G. Because UAVs fly at high altitudes with minimal obstructions, all UAVs can communicate with each other, making graph G a complete graph. This embodiment introduces a graph attention network to adaptively capture the correlation information between UAVs.

[0088] Since the node feature vectors passed to GAT are related to the state of each UAV, let the node feature vectors be... The node characteristics of each drone are as follows: This includes the drone's location information and the total workload within the drone's coverage area. n (t)=[∑ m∈C(n) d m (t),∑ m∈C(n) c m [t] represents the total number of tasks within the drone's coverage area, and C(n) is the set of users covered by drone n. The feature vector is input into GAT, and node n and node... The attention coefficient between them can be calculated as follows: Where att(·) denotes the self-attention mechanism, and W is the corresponding learnable weight matrix. Attention coefficients e n,i This represents the importance of node i to node n. Then, the attention coefficient is normalized using the softmax function to compare different neighboring nodes.

[0089]

[0090] Among them, a n,i Represents the normalized attention coefficient. Represents the set of elements associated with the current state n.

[0091] To make the training process more stable, this embodiment employs a multi-head attention mechanism. For node n, given the normalized attention coefficients, there are L output feature representations with independent attention mechanisms. for:

[0092]

[0093] Where σ is a nonlinear function, || represents the parallel connection of multiple attention mechanism results, l represents the index of an independent attention mechanism, and W l Indicates the corresponding weight. This represents the corresponding normalized attention coefficient. By applying the graph attention mechanism, A richer and more comprehensive new feature representation that takes into account the relationships between nodes is obtained.

[0094] Module 4: Training framework based on TD3.

[0095] The generated latent space and new features are combined with the TD3 algorithm to solve the multi-UAV joint optimization problem.

[0096] Action networks incorporate new features h generated by GAT that include attention relationships between drone nodes. ′t As input, it generates a potential discrete action vector g and a potential continuous action vector z, i.e., g,z = π ζ Next, the latent action vector (g,z) is input into the decoder to generate the corresponding real mixed action. To approximate the mixed action value function Using two critic networks These networks will mix actions The data is used as input for training. During training, data stored in the experience replay buffer is used. The experience (s, a, r, s′) in the data is used to train the critic network (i.e., the value network). This embodiment uses the Clipped Double Q-Learning algorithm:

[0097]

[0098] in, and Here are the target network parameters, r represents the immediate reward, and γ is the discount factor.

[0099] Action networks update gradients through deterministic policies:

[0100]

[0101] in, It is strategy π ζ (s) is the gradient with respect to the policy parameter ζ. It is the Q-value function, which represents the action π to be performed in state s. ζ (s) the expected cumulative return. Used to weigh the impact of action choices under different conditions on overall returns.

[0102] This invention proposes a deep reinforcement learning-based method for UAV trajectory and resource scheduling, which achieves joint optimization of computational resource allocation and task offloading efficiency in multi-UAV collaboration. Utilizing the DRL algorithm, it optimizes long-term system performance through interaction with the environment. Specifically, it employs the TD3 algorithm, improving training stability through a dual-network and delayed update strategy. By introducing an embedding table and variational autoencoder, the mixed-integer programming problem is transformed into a continuous optimization problem tractable by DRL, solving the joint optimization problem with a discrete-continuous mixed action space. The method uses GAT to capture the state characteristic relationships between UAVs, improving decision accuracy and coordination, thereby increasing the collaboration efficiency between UAVs. The problems of UAV trajectory planning, task offloading decision-making, computational resource allocation, and communication resource management are formulated as a mixed-integer programming problem and transformed into a Markov decision process for solution.

[0103] In this embodiment of the invention, in the applied multi-UAV assisted task offloading edge network model (including a physical network and a digital twin network), user equipment and UAVs carrying MEC (Mobile Edge Computing) servers, as well as UAVs and user equipment in the physical network, interact wirelessly. The digital twin network is used to simulate and optimize these interactions in real time. The goal is to optimize the long-term quality of user experience (QoE) of task offloading services in the multi-UAV edge network by minimizing the task completion latency of all user equipment within the average time slot. The optimization objective is to ensure that user equipment tasks can be quickly and efficiently offloaded to suitable UAVs, thereby improving the overall network service efficiency and user experience. To address the constructed optimization objective, this embodiment proposes a novel algorithm, DRUTS, which uses a deep reinforcement learning algorithm combined with a variational autoencoder module to solve the mixed action space problem. This method can effectively handle continuous and discrete action spaces, improving the algorithm's decision-making ability. Furthermore, a graph attention network module is introduced, enabling UAVs to adaptively capture the state features of other UAVs, thereby making more accurate joint decisions. This joint decision-making mechanism can significantly improve the task offloading efficiency and service quality of the multi-UAV edge network.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0105] The above descriptions are merely some embodiments of the present invention. Those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A method for UAV trajectory and resource scheduling based on deep reinforcement learning, characterized in that, In an edge network system with a digital twin network deployed at the ground base station for offloading multiple UAV-assisted tasks, a virtual connection is established with the physical network through a digital twin synchronization link. This allows for the real-time collection of system communication interaction information via the digital twin network, and the execution of the following resource scheduling steps: A 0-1 unloading strategy is used to configure the user's connection to the drone, as well as the unloading strategy for the user's unloading of computing resources. Let x be defined. m,n (t) represents the unloading strategy of user m to unload the task to drone n, if x m,n If (t) = 1, it means the task is offloaded to the human-machine interface; where m = 1, 2, ..., M, n = 1, 2, ..., N, and M and N represent the number of users and the number of drones, respectively; if x m,n If (t) = 0, it means the task will not be unloaded onto the drone; define x m,0 (t) represents the user's unloading strategy for computing resources, if x m,0 If (t) = 1, it means that user m chooses local computation. If x m,0 If (t) = 0, it means that user m chooses to offload computation to the Multi-Access Edge Computing (MEC). Define the optimization goal: Where X represents the task offloading decision set, Ω represents the UAV location set, P represents the device transmit power set, O represents the task priority set, t represents the time slot, and T represents the number of time slots. and These represent the latency of edge computing and the latency of local computing within the task, respectively. The optimization objective is solved based on Markov decision process and dual value network, and the resource scheduling result is obtained based on the optimization result. The dual-value network comprises a first value network, a second value network, a graph attention network, and an action network. The graph attention network takes the system's current state *s* as input, which includes the user's task state and the drone's location. The graph attention network constructs an undirected graph *G* with drones as nodes and drone features as node features. Drone features include location and the sum of tasks within the current coverage area. Attention coefficients between different neighboring nodes in the undirected graph *G* are calculated based on an attention mechanism, and then normalized using a softmax function to obtain normalized attention coefficients between different neighboring nodes. Based on these normalized attention coefficients, a multi-head attention mechanism is used to obtain enhanced drone features. The data is then used as input to the action network. The action network generates potential discrete action vectors and potential continuous action vectors. The drone features, potential discrete action vectors, potential continuous action vectors, and an immediate reward based on the reward function are then fed into the first value network to obtain the target value. The second value network takes drone features, potential discrete action vectors, potential continuous action vectors, and an immediate reward as input and outputs the current value. The corresponding value loss is calculated based on the outputs of the two value networks, and the network parameters of the action network are tuned. Finally, the optimization result of the constructed optimization objective is obtained based on the decoder in the graph attention network, action network, and conditional variational autoencoder.

2. The method as described in claim 1, characterized in that, When solving the optimization objective based on Markov decision processes and dual-value networks, the state space involved in the Markov decision process is... Action space and reward function They are respectively: state space Where I(t) represents the user task state, and ω(t) represents the drone position; action space Where i(t) represents the discrete action space and c(t) represents the continuous action space; i(t) = [i1(t), i2(t), ..., i M [(t)], where i m (t) = 0 indicates local computation by the user, i m (t) = N+1 indicates that the task is offloaded to MEC computation, i m (t) = j means that the task is offloaded to drone j for calculation, j = 1, 2, ..., N; c(t) = {Δω(t), p(t), o(t)}, where Δω(t) represents the displacement of all UAVs at time t, p(t) represents the transmit power of all users and UAVs, and o(t) represents the priority of all tasks; reward function Set to: When the preset constraints are met, the reward function will be applied. Set to: Otherwise set it to Among them, P u As a pre-set penalty, T ij (t) represents the task completion delay for user i and drone j at time t.

3. The method as described in claim 2, characterized in that, In Markov decision processes, for discrete actions, an embedding table G is used. ω Map each discrete action to a continuous latent vector space to obtain discrete action i. m Potential discrete actions Embedded table G ω Each line of G ω (i m ) represents a discrete action i m l1-dimensional continuous vector, Where ω represents the trainable parameters of the embedding table, and l1 is a specified value; Encoding and decoding are performed using a conditional variational autoencoder, where the encoder bases continuous actions on the current state s and potential discrete actions. Mapping to the latent space yields the latent continuous action z, and the encoder output follows a Gaussian distribution; the decoder is used to decode the latent continuous action z to obtain the predicted continuous action. And by searching With embedded table G ω The nearest neighbor is used to obtain the corresponding discrete action.

4. The method as described in claim 3, characterized in that, When training and tuning the embedding table parameters ω, encoder parameters φ, and decoder parameters ψ using the training set, the loss functions used include reconstruction error, KL divergence (which measures the posterior distribution of the latent space z relative to the standard Gaussian prior distribution), and prediction error.

5. The method as described in claim 1, characterized in that, When tuning the network parameters of the action network, a deterministic policy gradient is used for tuning.

Images