Unmanned aerial vehicle track and resource scheduling method based on deep reinforcement learning

Abstract

The invention discloses an unmanned aerial vehicle track and resource scheduling method based on deep reinforcement learning, and belongs to the technical field of resource scheduling, and the method comprises the steps: carrying out the real-time simulation and optimization of a communication interaction process through employing a digital twin network of a physical network; by minimizing the task completion time delay of all user equipment in the average time slot, the QoE target of task offloading service in the multi-UAV edge network is realized, so that the service efficiency and user experience of the whole network are improved. And aiming at the constructed optimization target, a deep reinforcement learning algorithm is combined with a variational auto-encoder module to solve the problem of mixed action space. Continuous and discrete action spaces are effectively processed, and the decision-making ability of an algorithm is improved. Besides, a graph attention network module is also introduced, so that the unmanned aerial vehicle can adaptively capture the state characteristics of other unmanned aerial vehicles, thereby making a more accurate joint decision. The joint decision-making mechanism can significantly improve the task unloading efficiency and the service quality of the multi-unmanned aerial vehicle edge network.

Description

Technical Field

[0001] The present invention relates to the technical field of resource scheduling, and in particular to a drone trajectory and resource scheduling method based on deep reinforcement learning. Background Art

[0002] In recent years, many new computationally intensive and delay-sensitive network services have emerged that require a lot of computing resources. Multi-access edge computing (MEC) is an innovative computing paradigm that provides computing capabilities at the edge of the network to support such services. However, there are some problems with current MEC solutions, such as the transmission links of ground MEC servers are easily blocked by obstructions, and users far away from MEC servers usually do not get good coverage. Especially in emergency situations such as disaster areas and hot spots, it is difficult to deploy fixed MEC servers. In recent years, unmanned aerial vehicles (UAVs) can assist MEC systems with their flexible deployment and large coverage potential. UAV-assisted MEC systems are expected to achieve solutions for performing highly distributed computationally intensive tasks.

[0003] Single UAVs, due to their limited resources, are adapted to scenarios with a small number of tasks, and the improvement provided to the performance of any task is limited. In addition, it is often difficult to meet the increasing demand for computing resources for tasks. On the contrary, a more complex system that relies on the collaboration of multiple UAVs is worth studying. It can provide flexible and abundant computing resources and large service coverage. However, in multi-UAV MEC systems, there are many challenges in the task offloading problem, including the trajectory design of multiple UAVs to achieve better coverage of users, communication resource management between UAVs to improve transmission efficiency, and collaborative task offloading to balance the computing workload of UAVs. It is true that multiple UAVs give us more computing resources, but at the same time how to coordinate the cooperative relationship between UAVs is an urgent problem to be solved.

[0004] Most existing UAV-assisted task offloading schemes focus on partial offloading, that is, allowing tasks to be split and processed separately. Although partial offloading has many limitations, it may be difficult to apply to inseparable computing tasks.

[0005] In the task offloading strategy 0-1 offloading, the task is either completely offloaded to the UAV (drone) or edge server for execution, or completely executed on the local device, and cannot be split into multiple parts for processing at different locations. Compared with partial offloading, this offloading method is more suitable for those inseparable computing tasks. However, due to the binary selection characteristics of 0-1 offloading (that is, the task is either completely offloaded or not offloaded), when solving the task offloading problem, it is necessary to simultaneously process continuous variables (such as resource allocation, channel selection, etc.) and discrete variables (0-1 offloading decision of the task), which makes the problem-solving process more complicated. That is, if 0-1 offloading is adopted, the task offloading problem may become a joint optimization problem of continuous variables and discrete variables, which further increases the complexity of the problem.

[0006] In addition, user requests and resource requirements are highly time-varying. In this case, long-term average optimization must be performed to improve system performance. However, it is difficult to solve this non-convex problem in a dynamic environment and with incomplete future information. With the improvement of artificial intelligence (AI) methods, deep reinforcement learning (DRL) has shown good results in solving long-term optimization problems. By interacting with the environment, DRL can help take appropriate decisions to obtain the optimal long-term average return, and can make intelligent decisions in a changing and uncertain environment to solve the problem.

[0007] In order to solve the mixed integer optimization problem in the optimization problem, some works use the deep deterministic policy gradient (DDPG) algorithm for training, and use the discretization of continuous variables to process the discrete part. For continuous variables, the DDPG algorithm is directly used for processing. This solves the limitation that DDPG and other similar algorithms can only process continuous action spaces. During the training process, the parameters of the neural network are updated through policy gradient optimization, and the discretized continuous actions are converted into discrete actions by action rounding, thereby realizing the processing of mixed integer optimization problems. However, in this processing method, it usually discretizes the continuous data output by the neural network by rounding. Specifically, even if the model outputs 4.6 and 4.9, these values ​​will be rounded to the same discrete value of 5, which will lead to a decrease in model performance. There is a major problem with this method, that is, it is difficult to accurately convert continuous output values ​​into discrete variables, resulting in information loss and reduced accuracy. In addition, the converted continuous variables and discrete variables are separated from each other and lack inherent correlation, so they cannot be effectively jointly optimized. This separation method is not conducive to capturing the dependency between variables, which in turn affects the overall optimization effect.

[0008] In order to solve the problem of resource shortage and limited coverage of a single UAV, some works use multiple UAVs to offload the calculation of user tasks and use multi-agent reinforcement learning to train the environment, where each UAV is modeled as an agent with an independent policy network and value network. The movement and task scheduling are performed according to the current UAV's observation of the environment, and the strategy is optimized through interactive learning with the environment. Although this processing method effectively solves the problem of resource shortage by using multi-UAV assisted edge computing, most of the current work does not fully consider the UAV networking and joint decision-making after networking. Since each agent in the multi-agent system has its own independent policy network and value network, in the decision-making process, these agents usually make decisions only based on their current state information. This independent decision-making method limits the effective communication and cooperation between multiple UAVs, making it impossible to fully utilize the collective resource and information advantages. This limitation not only affects the coordination and collaboration capabilities of the multi-UAV system, but also weakens the optimization effect of the overall system, resulting in the system performance not being optimal. Summary of the invention

[0009] In view of the shortcomings of the existing technology, the present invention proposes a drone trajectory and resource scheduling method based on deep reinforcement learning, which utilizes the digital twin network of the physical network to simulate and optimize the communication interaction process in real time, takes the QoE of the task offloading service in the multi-drone edge network as the optimization goal, and achieves this goal by minimizing the task completion delay of all user devices in the average time slot, thereby improving the service efficiency and user experience of the entire network.

[0010] The technical solution adopted by the present invention is:

[0011] A UAV trajectory and resource scheduling method based on deep reinforcement learning is proposed. In an edge network system with multi-UAV assisted task offloading deployed with a digital twin network at the ground base station end, a virtual connection is established with the physical network through a digital twin synchronization link to collect the communication interaction information of the system in real time through the digital twin network, and perform the following resource scheduling steps:

[0012] Adopt 0-1 unloading strategy to set users to drones, and unloading strategy for users to unload computing resources, define x m,n (t) represents the offloading strategy of user m to offload the task to drone n. If x m,n (t) = 1, it means that the task is offloaded to the human-machine; where m = 1, 2, ..., M, n = 1, 2, ..., N, M and N represent the number of users and the number of drones respectively; if x m,n (t) = 0, it means that the task is not offloaded to the UAV; define x m,0 (t) represents the user's uninstallation strategy for computing resources. If x m,0(t) = 1, indicating that user m chooses local computing. m,0 (t) = 0, indicating that user m chooses to offload the calculation to the multi-access edge computing MEC;

[0013] Build optimization goals:

[0014]

[0015] Among them, X represents the task offloading decision set, Ω represents the drone location set, P represents the device transmission power set, O represents the task priority set, t represents the time slot, T represents the number of time slots, and They are the latency of task edge computing and the latency of task local computing respectively;

[0016] The optimization solution of the optimization target is realized based on the Markov decision process and the dual value network, and the resource scheduling result is obtained based on the optimization result.

[0017] Furthermore, when solving the optimization objective based on the Markov decision process and the dual value network, the state space involved in the Markov decision process is Action Space And the reward function They are:

[0018] State Space Where I(t) represents the user task status, ω(t) represents the drone position;

[0019] Action Space Among them, i(t) represents the discrete action space, and c(t) represents the continuous action space;

[0020] i(t)=[i1(t),i2(t),…,i M (t)], where i m (t) = 0 means the user performs local calculation, i m (t) = N + 1 means the task is offloaded to MEC computing, i m (t) = j means offloading the task to UAV j for calculation, j = 1, 2, ..., N;

[0021] c(t) = {Δω(t), p(t), o(t)}, where Δω(t) represents the displacement of all drones at time t, p(t) represents the transmission power of all users and drones, and o(t) represents the priority of all tasks;

[0022] Reward Function Set to: When the preset constraints are met (which can be changed to power limits, drone flight collision limits, etc.), the reward function for Otherwise, it is set to the preset penalty P u , where T ij (t) is the task completion delay between user i and drone j at time t.

[0023] Furthermore, in the Markov decision process, for discrete actions, an embedding table G is used ω Map each discrete action to a continuous latent vector space to obtain discrete action i m Potential discrete actions Embed Table G ω Each row of G ω (i m ) represents a discrete action i m l1 (specified value) dimensional continuous vector, where represents the potential discrete action ω represents the trainable parameters of the embedding table;

[0024] Conditional variational autoencoders are used for encoding and decoding, where the encoder bases the continuous action on the current state s and the potential discrete action Mapped to the latent space, the potential continuous action z is obtained, and the encoder output follows a Gaussian distribution; the decoder is used to decode the potential continuous action z to obtain the predicted continuous action And by finding With embedded table G ω The nearest neighbor gets the corresponding discrete action.

[0025] Furthermore, when the embedding table parameters ω, encoder parameters φ and decoder parameters ψ are trained and tuned through the training set, the loss functions used include reconstruction error, KL divergence (relative entropy) between the posterior distribution of the latent space z and the standard Gaussian prior distribution, and prediction error.

[0026] Furthermore, the dual value network includes a first value network (as a target value network), a second value network, a graph attention network and an action network, wherein the input of the graph attention network is the current state s of the system (which includes the user task state and the drone position), and the graph attention network uses drones as nodes and drone features as node features to construct an undirected graph G, wherein the drone features include the position and the sum of the tasks within the coverage range at the current moment; and based on the attention mechanism, the attention coefficients between different neighbor nodes of the undirected graph G are calculated, and then the softmax function is used to normalize them to obtain the normalized attention coefficients between different neighbor nodes; based on the normalized attention coefficients, a multi-head attention mechanism is used to obtain enhanced drone features And use it as the input of the action network; the action network is used to generate potential discrete action vectors and potential continuous action vectors, and then the drone features, potential discrete action vectors, potential continuous action vectors and immediate rewards based on the reward function are sent to the first value network to obtain the target value; the input of the second value network includes drone features, potential discrete action vectors, potential continuous action vectors and immediate rewards, which are used to output the current value, calculate the corresponding value loss based on the outputs of the two value networks and tune the network parameters of the action network, and then obtain the optimization result of the constructed optimization target based on the decoder in the graph attention network, action network and conditional variational autoencoder.

[0027] Furthermore, when tuning the network parameters of the action network, deterministic policy gradient is used for tuning.

[0028] The technical solution provided by the present invention brings at least the following beneficial effects:

[0029] (1) Taking minimizing the sum of user task processing delays as the optimization goal, for the trajectory planning and task scheduling problem in the multi-UAV assisted MEC scenario, the joint optimization problem of UAV trajectory planning, offloading decision, computing resource allocation and communication resource management is formulated as a mixed integer programming problem with 0-1 offloading, transmission power, computing capacity and UAV movement range as constraints. Then, the problem is transformed into a Markov decision process (MDP);

[0030] (2) By introducing embedding tables and variational autoencoders, the mixed integer programming problem is cleverly transformed into a continuous optimization problem that can be directly handled by DRL. Through the encoder, a latent space of mixed actions is constructed, and the latent space is combined with the double-delayed deep deterministic policy gradient algorithm (TD3) to solve the joint optimization problem with discrete-continuous mixed action space;

[0031] (3) Based on the networking of multiple UAVs, GAT is used to help the intelligent agent pay more attention to the connection between UAVs and the impact of UAV status on each other, so as to learn better cooperative scheduling strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 Schematic diagram of the edge network model for offloading multi-UAV auxiliary tasks;

[0034] Figure 2 Schematic diagram of the processing process of the DRUTS (Deep Reinforcement UAV Task Scheduling) method. DETAILED DESCRIPTION

[0035] To make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described in detail and completely in conjunction with the drawings in the implementation of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present invention.

[0036] The embodiment of the present invention proposes a task scheduling and UAV trajectory planning strategy for UAV-assisted edge computing based on deep reinforcement learning (i.e., Deep Reinforcement UAV Task Scheduling, DRUTS), which makes action decisions closer to the real scene by utilizing the networking between UAVs and considering the connection between the state characteristics of each UAV on the premise of solving the continuous discrete mixed action decision. The core idea is to use the embedding table and variational autoencoder (CVAE) to convert the mixed action space into a potential continuous action space, and use the graph attention network (GAT) to generate a new state space with the relationship between nodes according to the networking between UAVs, thereby solving the trajectory planning and task scheduling problems in the multi-UAV-assisted MEC scenario.

[0037] In e.g. Figure 1 In the edge network model of multi-UAV assisted task offloading shown in the figure, it consists of two parts: physical network and digital twin network. The system contains M user devices, using the collection Indicates that N drones carrying MEC servers use the set Considering the three-dimensional Cartesian coordinate system, the position of user m at time t can be represented by q m (t) = [x m (t),y m (t),0] T In other words, n (t) = [x n (t),y n (t),z n (t)] TUsed to indicate the location of drone n. In order to achieve long-term optimization of system performance, without loss of generality, the embodiment of the present invention considers a dynamic MEC network with multiple time slots. That is, a continuous time is divided into multiple time slots of equal length. The time slot set uses Indicates that the length of each time slot is δ t .

[0038] At the beginning of time slot t, user m generates a computationally intensive task consisting of a binary pair I m (t) = {d m (t),c m (t)}, where d m (t) represents the data transmission size in bits, c m (t) represents the number of CPU cycles required to process each bit of the task.

[0039] The present invention considers the 0-1 offloading mode to combine the optimization of discrete and continuous hybrid spaces, that is, the user-generated task I m (t) can be calculated locally or on drones, while drones support both data transmission and calculation. On the one hand, drones have more powerful computing capabilities and can help users perform computing tasks. On the other hand, when the computing resources of drones are insufficient, computing tasks can be forwarded to other drones. The digital twin system deployed at the ground base station establishes a virtual connection with the physical network through the digital twin synchronization link, which can monitor the working parameters, models, dynamic changes of resources and the arrival process of random tasks of the physical network, that is, collect network information in real time through the digital twin network to facilitate the operation of the algorithm.

[0040] exist Figure 1 In the example scenario shown, the embodiment of the present invention seeks to optimize the long-term QoE (Quality of Experience) of the task offloading service in the multi-UAV edge network, which is achieved by minimizing the completion delay of all user devices in the average time slot. The completion delay of all user devices in time slot t can be formulated as:

[0041]

[0042] Among them, x m,n (t) represents the 0-1 unloading strategy, when x m,n When (t) = 1, it means that user m offloads the task to drone n, x m,n When (t) = 0, it means that user m does not offload the task to drone n, x m,0 (t) represents the user computing resource strategy, when x m,0 When (t) = 1, it means that user m chooses local computing, x m,0 When (t) = 0, it means that user m chooses to offload the calculation to MEC. and They are the latency of task edge computing and the latency of task local computing, respectively. The formula is as follows:

[0043]

[0044] in, They represent the transmission time during the offloading process, the processing time after offloading to the MEC server, and the waiting time from the generation of the task to the offloading. They represent the computing resources of user m's local computing respectively.

[0045] When Task I m (t) After unloading to drone n′, each task has a priority o m (t), the tasks with high priority will be processed first. First, the tasks are sorted according to their priorities. After the task priorities are arranged, the maximum computing resources of the drone will be used to execute each task in the order from high to low priority.

[0046] The embodiment of the present invention jointly optimizes the unloading decision Drone tracks Device transmit power and task priorities To minimize the average task completion delay of all time slots, this problem can be formulated as follows:

[0047]

[0048] Among them, p m (t),p n (t) are the transmission powers of user m and drone n at time t, respectively, and T represents the number of time slots.

[0049] For the optimization problem given by formula (4), a deep reinforcement learning (DRL) algorithm can be used to solve it. The DRL algorithm is based on the Markov decision process, and uses a neural network to implement the output of the agent strategy in the Markov decision process (MDP) based on the environment to maximize certain rewards. This embodiment adopts the TD3 algorithm (Twin Delayed Deep Deterministic Policy Gradient, double delayed deep deterministic policy gradient algorithm), which effectively reduces the deviation problem of Q value estimation by introducing a dual network (two Q networks) and a delayed update strategy, which helps to improve the stability of training.

[0050] Formula (4) has two difficulties to be solved: first, the problem has both continuous variables and discrete variables, and the traditional TD3 algorithm is only suitable for solving continuous solution space problems; second, in the scenario of the embodiment of the present invention, there are communication links between drones and form a network, and the decision of the drone will be affected by the state of the adjacent drones. In order to solve these two problems, the embodiment of the present invention proposes a new algorithm, which is based on hybrid action representation and drones can capture the state characteristics of other drones to make joint decisions.

[0051] The training and testing of the algorithm are as follows: the test part first initializes a state s and then transforms the state into a new feature representation h that takes into account the relationship between the nodes of the drone through the GAT (graph attention network) module, and enters the actor network (policy network) as input to obtain the potential discrete action output g and continuous action output z, which are decoded to obtain the real discrete action i and continuous action c, and then interact with the environment to obtain the next state s′, reward reward and completion mark done, and the results of each round of testing are input into the experience playback buffer For use during training. The training part is from the experience replay buffer The data stored in the test process is taken out, and the target critic network and the cirtic network (value network) are input to obtain the target Q (targetQ) value and the current Q (currentQ) value respectively. According to the temporal difference strategy, the obtained value loss (critic loss) is used to train the critic network, and then the actor network is trained. Under the premise of solving the continuous discrete mixed action decision, the algorithm uses the networking between drones and considers the connection between the state characteristics of each drone to make action decisions that are closer to the real scene. See Figure 2 The specific implementation process of the new algorithm proposed in the embodiment of the present invention includes:

[0052] Module 1: MDP.

[0053] In the UAV-assisted MEC system, the next state of the system depends only on the current state and action. In this case, the optimization problem described by formula (4) can be expressed as a Markov decision process. At time slot t, observe the system state and select an action to maximize the long-term system reward. The process can be represented by a tuple: where γ∈[0,1) is the discount factor, They are state space, action space and reward function, which are described as follows:

[0054] (1) State space

[0055] Since adding the channel quality of all links into the state space will increase the algorithm complexity to O(N 2 ), and considering that the channel quality is related to the position of the drone, the algorithm complexity of only adding the drone position ω(t) to the state space is O(N). In addition, the user randomly generates tasks at the beginning of each time slot, so the state space is represented by the state of the user task and the position of the drone:

[0056]

[0057] Please note that the above states are changing at different time periods, which means that the drone is constantly flying and the user is generating new tasks, which is more realistic.

[0058] (2) Action Space

[0059] If the unloading decision X is directly used as the action space, there are M(N+2) action spaces, which not only increases the output dimension of the neural network, but also increases the complexity of training due to the restriction that the task obeys 0-1 unloading. For each user m, there are N+2 unloading locations that can be selected. m (t)∈{1,2,…,N+1} represents the unloading location, where i m (t) = 0 means the user performs local calculation, i m (t) = N + 1 indicates that the task is offloaded to MEC computing. Otherwise, it indicates that the task is offloaded to the i-th m (t) UAVs are used for calculation, that is, i m (t) = j means that the task is offloaded to UAV j for calculation, where j = 1, 2, ..., N. In this way, the output dimension of the neural network can be reduced from M (N + 2) to M, while avoiding the interference of 0-1 offloading constraints on training.

[0060] In addition to the offloading decision, the drone trajectory, the transmission power of the user and the drone, and the resource allocation of the drone to the user will also be determined. The action space is divided into discrete action space i(t) and continuous action space c(t):

[0061]

[0062] Among them, the discrete action space i(t) = [i1(t),i2(t),…,i M (t)] is the user’s uninstall decision vector, and the continuous action space is c(t) = {Δω(t), p(t), o(t)}, where Δω(t) = {Δω1(t), Δω2(t), …, Δω N (t)} is the displacement of all UAVs at time t, Δω n (t) = [Δx n(t),Δy n (t),Δz n (t)], is the distance that drone n moves on the X, Y, and Z axes respectively, p(t) = [p i (t),p2(t),…,p M +N(t)] is the transmission power of all users and drones, o(t)=[o m (t)], is the priority of all tasks. The total dimension of the action space is M+3N+(M+N)+M=4N+3M.

[0063] (3) Reward Function

[0064] Since the goal of the optimization problem described by formula (4) is to minimize the task completion delay by optimizing the UAV trajectory, unloading decision, transmission power and resource allocation, the reward should be negatively correlated with the task completion delay and set it as:

[0065]

[0066] Among them, x ij (t) is the unloading strategy of user i to drone j at time t, T ij (t) is the task completion delay between user i and drone j at time t, that is When the constraints are met, since the optimization goal is to minimize the task completion delay, a negative exponential reward is given. This design can impose a smaller penalty for a small completion delay and a larger penalty for a larger delay. When the constraints are not met, such as when the drone flies out of the boundary or two drones collide, a high-risk action such as P is given. u (default) penalty.

[0067] Module 2: Latent Space.

[0068] In the MDP system, since the problem has a discrete-continuous mixed action space, the traditional DRL algorithm is not applicable. Therefore, the embodiment of the present invention proposes a new algorithm that transforms the mixed action space problem into a continuous policy learning problem by introducing an embedding table and a conditional variational autoencoder, while considering the dependency between discrete variables and continuous variables. An embedding table is used to represent discrete actions, and a latent space is constructed for continuous actions through conditional VAE. In this way, the algorithm can process the mixed action space more accurately, avoiding the limitations of directly using discrete actions or continuous actions.

[0069] For discrete actions, an embedding table is used to map each discrete action into a continuous latent vector space, where the representation of the embedding table is Each user has N+2 offloading decision variables, l1 is the vector length of each discrete action in the latent space, and the parameters ω of this table are learnable, which means that they are optimized during the learning process. ω,im =G ω (i m ) represents a discrete action i m There are M users making decisions in each time period. Although each user may need his own embedding table for personalized learning, because the action space and action meaning of all users are consistent, they can share a common embedding table.

[0070] For continuous actions, a conditional variational autoencoder is used to encode and decode these continuous actions. CVAE consists of an encoder and a decoder. The encoder maps the continuous action to a Gaussian distribution in the latent space, while the decoder reconstructs the continuous action from the latent space.

[0071] Encoder: The neural network is represented as q φ (z||c,s,g ω,im ), the network maps the continuous action c to the latent variable z (latent continuous action) by using the parameter φ, which depends on the state s and the latent discrete action g. ω,im ,in The encoder output follows a Gaussian distribution, which can be expressed as: Γ(μ q ,σ q ), by sampling from this distribution, we can obtain the potential representation z, where μ q ,σ q are the mean and variance respectively.

[0072] Decoder: The neural network is represented as The decoder uses the parameter ψ to decode the latent space z to obtain continuous actions (predicted value), this mapping also depends on the state s and the potential discrete action g ω,im , by finding g ω,im With embedded table G ω The nearest discrete action is obtained by subtracting the embedded table to find the minimum value and find the nearest discrete action.

[0073] In summary, this embodiment encodes the mixed action (including discrete and continuous parts) into a latent space through an encoder Through the decoder, the original mixed action is reconstructed from the representation in the latent space. The encoding and decoding process is as follows:

[0074]

[0075]

[0076] During training, the embedding table parameters ω, encoder parameters φ, and decoder parameters ψ are jointly trained by minimizing the loss function:

[0077] The reconstruction error is the predicted value of the continuous parameter The squared L2 norm error between the latent space z and the true value c is used to encourage the decoder to accurately reconstruct the input data. The KL divergence measures the difference between the posterior distribution of the latent space z and the standard Gaussian prior distribution. This term encourages the latent space to be close to the prior distribution, which helps avoid overfitting and increase generalization ability. The corresponding loss function is expressed as follows:

[0078]

[0079] in, represents the mathematical expectation, Characterize the reconstruction error, d KL (q ψ (·|c,s,g ω,im )||Γ(0,I) is the corresponding KL divergence, Γ(0,I) represents the standard multidimensional Gaussian distribution with 0 as mean and unit matrix I as covariance matrix

[0080] The prediction error is used to measure the decoder's ability to predict the system state transition, and try to fit the actual state transition as closely as possible. For the experimental sample (s, a, s′), the state residual is defined as δ s,s′ =s′-s. s, s′ are the current state and the next state of the system respectively. By introducing the cascade structure into the decoder, the corresponding prediction residual can be obtained:

[0081]

[0082] in, represents the predicted state residual, δ s,s′ is the corresponding true value.

[0083] The comprehensive training loss is the weighted sum of the above two losses:

[0084] L H (φ,ψ,ω)=L V (φ,ψ,ω)+αL D (φ,ψ,ω) (13)

[0085] Where α is a weight variable that depends on the importance of the loss associated with the dynamic prediction representation. The dimension of the system state s is defined as dim, and the dimension of the continuous policy p is expressed as

[0086] Module 3: Graph Attention Network.

[0087] For a task calculation problem in a multi-UAV collaboratively assisted MEC, the UAVs can communicate and transmit data, so the action of the UAV will also be affected by the state of the adjacent UAVs. In view of the mutual influence between the UAVs, the real-time state detected by the digital twin system is modeled as an undirected graph G. Since the UAVs fly at a high altitude and there are very few obstructions, all UAVs can communicate with each other, so the graph G is a complete graph. This embodiment introduces a graph attention network to adaptively capture the correlation information between UAVs.

[0088] Since the node feature vector input to GAT is related to the state of each drone, let the node feature vector be The node characteristics of each drone are Including the location information of the drone and the total number of tasks within the drone’s coverage area, I n (t) = [∑ m∈C(n) d m (t),∑ m∈C(n) c m (t)] is the total number of tasks within the coverage area of ​​the drone, and C(n) is the set of users covered by drone n. The feature vector is input into GAT, node n and node The attention coefficient between can be calculated as where att(·) represents the self-attention mechanism and W is the corresponding learnable weight matrix. n,i Represents the importance of node i to node n. Then use the softmax function to normalize the attention coefficient to compare different neighbor nodes:

[0089]

[0090] Among them, a n,i represents the normalized attention coefficient, Represents the set of elements associated with the current state n.

[0091] In order to make the training process more stable, this embodiment adopts a multi-head attention mechanism. For node n, when the normalized attention coefficient is known, the output feature representation with L independent attention mechanisms is for:

[0092]

[0093] Among them, σ is a nonlinear function, || represents the parallel connection of multiple attention mechanism results, l represents the independent attention mechanism index, and W l represents the corresponding weight, represents the corresponding normalized attention coefficient. By applying the graph attention mechanism, A richer and more comprehensive new feature representation is obtained that takes into account the relationships between nodes.

[0094] Module 4: TD3-based training framework.

[0095] The generated latent space and new features are combined with the TD3 algorithm to solve the multi-UAV joint optimization problem.

[0096] The action network transforms the new feature h generated by GAT containing the attention relationship between drone nodes ′t As input, it generates a potential discrete action vector g and a potential continuous action vector z, that is, g,z = π ζ Next, the latent action vector (g, z) is input into the decoder to generate the corresponding real mixed action To approximate the mixed action-value function Using two critic networks These networks will mix actions During training, the experience replay buffer is used to store The experience (s, a, r, s′) in is used to train the critic network (i.e., the value network). This embodiment uses the Clipped Double Q-Learning algorithm:

[0097]

[0098] in, and is the target network parameter, r represents the immediate reward, and γ is the discount factor.

[0099] The action network (actor network) is updated via deterministic policy gradients:

[0100]

[0101] in, is strategy π ζ (s) is the gradient of the policy parameter ζ, is the Q-value function, which means that in state s, action π is executed ζ (s) the expected cumulative return obtained, Used to weigh the impact of action selection in different states on the overall reward.

[0102] A UAV trajectory and resource scheduling method based on deep reinforcement learning proposed in an embodiment of the present invention realizes the joint optimization of computing resource allocation and task offloading efficiency of multi-UAV collaboration, utilizes the DRL algorithm to optimize long-term system performance through interaction with the environment, and especially adopts the TD3 algorithm to improve training stability through dual networks and delayed update strategies, introduces embedding tables and variational autoencoders, converts mixed integer programming problems into continuous optimization problems that can be processed by DRL, and solves the joint optimization problem in discrete-continuous mixed action space; GAT is used to capture the state feature connection between UAVs, improve the accuracy and coordination of decision-making, and thus improve the collaboration efficiency between UAVs; the problems of UAV trajectory planning, task offloading decision, computing resource allocation and communication resource management are expressed as a mixed integer programming problem, and converted into a Markov decision process for solution.

[0103] In the embodiment of the present invention, in the edge network model (including physical network and digital twin network) of multi-UAV assisted task offloading, the user equipment and the UAV carrying MEC (mobile edge computing) server, the UAV in the physical network and the user equipment interact through wireless communication, and the digital twin network is used to simulate and optimize these interaction processes in real time; and the goal is to optimize the long-term user experience quality (QoE) of the task offloading service in the multi-UAV edge network, and this goal is achieved by minimizing the task completion delay of all user equipment in the average time slot. The optimization goal is to ensure that the tasks of the user equipment can be quickly and efficiently offloaded to the appropriate UAV, thereby improving the service efficiency and user experience of the entire network. In view of the constructed optimization goal, the embodiment of the present invention proposes a new algorithm DRUTS, which adopts a deep reinforcement learning algorithm combined with a variational autoencoder module to solve the problem of mixed action space. This method can effectively process continuous and discrete action spaces and improve the decision-making ability of the algorithm. In addition, a graph attention network module is introduced to enable the UAV to adaptively capture the state characteristics of other UAVs, so as to make more accurate joint decisions. This joint decision-making mechanism can significantly improve the task offloading efficiency and service quality of the multi-UAV edge network.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0105] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the creative concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A method for UAV trajectory and resource scheduling based on deep reinforcement learning, characterized in that: In the edge network system of multi-UAV assisted task offloading with a digital twin network deployed at the ground base station end, a virtual connection is established with the physical network through the digital twin synchronization link to collect the communication interaction information of the system in real time through the digital twin network and perform the following resource scheduling steps: Adopt 0-1 unloading strategy to set users to drones, and unloading strategy for users to unload computing resources, define x m,n (t) represents the offloading strategy of user m to offload the task to drone n. If x m，n (t) = 1, it means that the task is offloaded to the human-machine; where m = 1, 2, ..., M, n = 1, 2, ..., N, M, N represent the number of users and the number of drones respectively; if x m,n (t) = 0, it means that the task is not offloaded to the UAV; define x m，0 (t) represents the user's uninstallation strategy for computing resources. If x m，0 (t) = 1, indicating that user m chooses local computing. m，0 (t) = 0, indicating that user m chooses to offload the calculation to the multi-access edge computing MEC; Build optimization goals: Among them, X represents the task offloading decision set, Ω represents the drone location set, P represents the device transmission power set, O represents the task priority set, t represents the time slot, T represents the number of time slots, and They are the latency of task edge computing and the latency of task local computing respectively; The optimization solution of the optimization target is realized based on the Markov decision process and the dual value network, and the resource scheduling result is obtained based on the optimization result.

2. The method according to claim 1, characterized in that When solving the optimization objective based on the Markov decision process and dual value network, the state space involved in the Markov decision process Action Space And the reward function They are: State Space Where I(t) represents the user task state, ω(t) represents the drone position; Action Space Among them, i(t) represents the discrete action space, and c(t) represents the continuous action space; i(t)=[i1(t),i2(t),…,i M (t)], where i m (t) = 0 means the user performs local calculation, i m (t) = N + 1 means the task is offloaded to MEC computing, i m (t) = j means offloading the task to UAV j for calculation, j = 1, 2, ..., N; c(t) = {Δω(t), p(t), o(t)}, where Δω(t) represents the displacement of all drones at time t, p(t) represents the transmission power of all users and drones, and o(t) represents the priority of all tasks; Reward Function Set to: When the preset constraints are met (which can be changed to power limits, drone flight collision limits, etc.), the reward function for Otherwise, it is set to the preset penalty P u , where T ij (t) is the task completion delay between user i and drone j at time t.

3. The method according to claim 2, characterized in that In the Markov decision process, for discrete actions, an embedding table G is used ω Map each discrete action to a continuous latent vector space to obtain discrete action i m Potential discrete actions Embed Table G ω Each row of G ω (i m ) represents a discrete action i m l1 (specified value) dimensional continuous vector, where represents the potential discrete action ω represents the trainable parameters of the embedding table; Conditional variational autoencoders are used for encoding and decoding, where the encoder bases the continuous action on the current state s and the potential discrete action Mapped to the latent space, the potential continuous action z is obtained, and the encoder output follows a Gaussian distribution; the decoder is used to decode the potential continuous action z to obtain the predicted continuous action And by finding With embedded table G ω The nearest neighbor gets the corresponding discrete action.

4. The method according to claim 3, characterized in that When training and tuning the embedding table parameters ω, encoder parameters φ and decoder parameters ψ through the training set, the loss functions used include reconstruction error, KL divergence between the posterior distribution of the latent space z and the standard Gaussian prior distribution, and prediction error.

5. The method according to claim 3, characterized in that The dual value network includes a first value network, a second value network, a graph attention network and an action network, wherein the input of the graph attention network is the current state s of the system, which includes the user task state and the drone position. The graph attention network uses drones as nodes and drone features as node features to construct an undirected graph G, wherein the drone features include the position and the sum of the tasks within the coverage range at the current moment; and based on the attention mechanism, the attention coefficients between different neighbor nodes of the undirected graph G are calculated, and then the softmax function is used to normalize them to obtain the normalized attention coefficients between different neighbor nodes; based on the normalized attention coefficients, a multi-head attention mechanism is used to obtain enhanced drone features And use it as the input of the action network; the action network is used to generate potential discrete action vectors and potential continuous action vectors, and then the drone features, potential discrete action vectors, potential continuous action vectors and immediate rewards based on the reward function are sent to the first value network to obtain the target value; the input of the second value network includes drone features, potential discrete action vectors, potential continuous action vectors and immediate rewards, which are used to output the current value, calculate the corresponding value loss based on the outputs of the two value networks and tune the network parameters of the action network, and then obtain the optimization result of the constructed optimization target based on the decoder in the graph attention network, action network and conditional variational autoencoder.

6. The method according to claim 5, characterized in that When tuning the network parameters of the action network, deterministic policy gradient is used for tuning.

Images