Dynamic optimization method of edge network computing resources based on UAV auxiliary nodes

Abstract

The present invention claims protection for a method for dynamically optimizing edge network computing resources based on drone auxiliary nodes, and belongs to the field of communication technology. Aiming at the problem that the edge network cell suffers from the burst of local user traffic, which leads to the shortage of server computing resources and the deterioration of task offloading quality, a method for dynamically optimizing computing resources based on adaptive cruise of drone auxiliary nodes is proposed. According to the location distribution of ground users and the demand for task offloading, the deep reinforcement learning method is used to dynamically plan the drone cruise trajectory, and the server resource utilization of drone nodes and base station nodes during the cruise process is maximized through the task offloading scheduling strategy, thereby effectively reducing the task interruption rate of local users and reducing the average task offloading delay.

Description

Technical Field

[0001] The present invention belongs to the field of communication technology, and in particular relates to a method for dynamically optimizing edge network computing resources based on unmanned aerial vehicle auxiliary nodes. Background Art

[0002] With the popularization and development of mobile networks, new applications such as augmented reality, virtual reality and autonomous driving continue to emerge, greatly enriching people's daily lives. However, these applications usually have high latency requirements and consume a lot of computing resources. It is difficult for mobile terminals to process such applications quickly and effectively. Mobile edge computing can provide users with the computing resources required for task offloading at close range by sinking cloud resources to the edge network, effectively shortening the transmission latency between users and cloud servers.

[0003] However, the rapid changes in the distribution of ground users and the random business bursts of users in local areas may put tremendous pressure on the fixed server resources of the edge network, resulting in low computing resource utilization and deteriorating user service experience. Therefore, using low-altitude drones as auxiliary nodes of edge computing networks to provide flexible resource supplements for ground nodes will become an important model for future network construction and development.

[0004] Aiming at the problem that the server computing resources are tight and the task offloading quality is deteriorated due to the sudden traffic of local users in the edge network cell, the present invention proposes a dynamic optimization method of computing resources based on adaptive cruising of drone-assisted nodes, which can effectively reduce the task interruption rate of local users and reduce the average task offloading delay. Summary of the invention

[0005] The present invention aims to solve the above problems of the prior art. A method for dynamically optimizing edge network computing resources based on drone auxiliary nodes is proposed. The technical solution of the present invention is as follows:

[0006] A method for dynamically optimizing edge network computing resources based on drone auxiliary nodes, comprising the following steps:

[0007] 101. Construct a discrete time-state model based on the Markov decision process, including discretizing the UAV cruising time into time slots, setting up the time slot variable k, the ground-air network state vector s k 、UAV 3D motion vector a k , drone action reward function r k , where s k , a k , r k As the number of time slots k increases, corresponding transfers and changes are made, and the time slot variable k is initialized to 0;

[0008] 102. Take the drone controller as an intelligent agent and build a deep reinforcement learning model based on the idea of ​​double-delay deep deterministic policy gradient algorithm, including establishing a system environment collector, drone action strategy network π, drone state-action value network Q, task scheduling strategy generator, drone action reward generator, experience sample storage area E, and random sample set Mini-Batch;

[0009] 103. Let k = k + 1. If the three-dimensional coordinate position of the drone does not change within n consecutive time slots, jump to step 106. Otherwise, determine the user object set I of drone j according to the effective coverage range of drone j. j , user object set I of base station o o =II j , where I represents the set of all user objects. Through the task scheduling strategy generator, I is obtained j with I o The set of task offloading decision variables and Jump to step 104;

[0010] 104. According to and Execute the task offloading request of user i and obtain the corresponding reward value r through the drone action reward generator k , obtain the k-time slot UAV three-dimensional action vector a through the UAV action strategy network π k , the k-slot ground-to-air network state vector s k With the action vector a k Calculate to obtain k+1 , will [s k ,a k ,r k ,s k+1 ] is stored in the experience sample storage area E;

[0011] 105. Randomly sample a Mini-Batch sample set from the experience sample storage area E, and import the Mini-Batch sample set into the action strategy network π and the state-action value network Q for training, and jump to step 103;

[0012] 106. The algorithm ends.

[0013] Furthermore, in step 101, a discrete time-state model is constructed according to the Markov decision process, wherein the ground-air network state vector s of the k-time slot is k 、UAV 3D motion vector a k , drone action reward function r k , as shown in formulas (1), (2), and (3) respectively:

[0014]

[0015]

[0016]

[0017] In formula (1), represents the three-dimensional coordinate position of drone j in time slot k, represents the two-dimensional coordinate position of user i in time slot k; in formula (2), represents the horizontal motion direction of UAV j in time slot k, represents the vertical movement distance of drone j in time slot k; in formula (3), ω represents the weight factor of drone action reward function, ω∈(0,1), Δt represents the time slot size, represents the average unit task delay of user i in time slot k, as shown in formula (4), It means that the average unit task delay of user i in time slot k satisfies the average unit task tolerance delay τ i , otherwise As shown in formula (5):

[0018]

[0019]

[0020] In formula (4), represents the connection status between user i and drone j. If user i offloads the task to drone j in time slot k, then otherwise represents the connection status between user i and base station o. If user i offloads the task to base station o in time slot k, then otherwise User i can only connect to one drone or base station at most in time slot k, i.e. represents the amount of tasks that user i offloads to drone j in time slot k, represents the amount of tasks that user i offloads to base station o in time slot k, τ i Indicates the average unit task tolerance delay of user i.

[0021] Further, the step 102 constructs a deep reinforcement learning model based on the idea of ​​double-delay deep deterministic policy gradient algorithm, including establishing a system environment collector, a drone action strategy network π, a drone state-action value network Q, a task scheduling strategy generator, a drone action reward generator, an experience sample storage area E, and a random sample set Mini-Batch, specifically including:

[0022] The system environment collector is used to collect the two-dimensional coordinate position of ground users in the k-time slot ground-to-air network User task offloading request, drone 3D coordinate position and the remaining available computing resources of the drone; the drone action strategy network π generates the k-time slot ground-air network state s k The 3D motion vector of the drone is a k ; The drone state-action value network Q generates the k-time slot ground-air network state s k Execute the drone's three-dimensional motion vector a k The action evaluation value q; the task scheduling strategy generator is used to generate the k-time slot user offloading strategy and obtain the task offloading decision variable set and The drone action reward generator generates the action reward value r of drone j in time slot k after completing the unloading task in time slot k. k ; Drone performs a k After the action, the ground-air network status is changed from s k Transfer to k+1 ; Add k time slot experience samples [s k ,a k ,r k ,s k+1 ]; The random sample set Mini-Batch is composed of a fixed number of samples randomly drawn from the experience sample storage area E. Among them, the drone action strategy network π and the drone state-action value network Q are both neural networks, each containing several hidden layers, and each hidden layer contains several neurons.

[0023] Furthermore, in step 103, the task scheduling strategy generator determines the user's task offloading variable set and The method is:

[0024] 1) Add user i in the effective coverage area of ​​drone j to drone j’s service object set I j ,make Base station o service object set I o =II j ,make I j and I o According to the user Sort in descending order;

[0025] 2) Delay of unloading tasks according to user i Computation I j The amount of tasks that user i offloads to drone j

[0026] 3) Delay of unloading tasks according to user i Computation I o The amount of tasks that user i offloads to base station o

[0027] Furthermore, the amount of tasks that user i unloads to drone j in step 2) is The calculation method of is shown in formula (6) and (7):

[0028]

[0029]

[0030] in, represents the computing resources allocated by UAV j to user i in time slot k, C j represents the total amount of computing resources of UAV j, represents the uplink transmission rate from user i to drone j in time slot k, F represents the task unit size, represents the task complexity of user i.

[0031] Furthermore, the amount of tasks that user i offloads to base station o in step 3) is The calculation method of is shown in formula (8) and (9):

[0032]

[0033]

[0034] in, represents the computing resources allocated by base station o to user i in time slot k, C o represents the total amount of computing resources of base station o, represents the uplink transmission rate from user i to base station o in time slot k.

[0035] Furthermore, the task offloading delay of user i in step 2) and step 3) is shown in formula (10), and the task offloading delay constraint is shown in formula (11):

[0036]

[0037]

[0038] In formula (10), since the size of the task calculation result is much smaller than the task itself, only the uplink transmission delay of the user task offloading and the task calculation delay are considered, and the downlink transmission delay of the task calculation result is ignored. represents the total delay of task offloading for user i in time slot k, represents the transmission delay of the offloading task of user i in time slot k, as shown in formula (12); represents the computational delay of the offload task, as shown in formula (13):

[0039]

[0040]

[0041] In formula (12), They represent the uplink transmission rates from user i to drone j and base station o in time slot k, respectively, as shown in formulas (14) and (15):

[0042]

[0043]

[0044] In formulas (14) and (15), W is the user channel bandwidth, p i is the user transmission power, σ 2 is the noise power, and They represent the communication channel gains from user i to drone j and base station o in time slot k respectively.

[0045] Furthermore, in step 104, the three-dimensional motion vector a of the k-time slot drone is obtained through the drone motion strategy network π. k , the k-slot ground-to-air network state vector s k With the action vector a k Calculate to obtain k+1 , specifically including:

[0046] The k-slot ground-to-air network state vector Input the drone action strategy network π, and obtain the three-dimensional action vector of drone j through the forward propagation of each layer of neurons in the π network Calculated in, L is the horizontal moving distance of drone j in time slot k.

[0047] Furthermore, in step 105, a Mini-Batch sample data set is obtained from the experience sample storage area E by random sampling, and the method for optimizing the state-action value network and the action strategy network is:

[0048] To solve the state-action value network Q, Q includes Q(s k ,a k |θ1 Q )and Action policy network π(s k |θ π ) The network is unstable during the learning process, and Q(s k ,ak |θ1 Q ) is the target network Q′(s k ,a k |θ1 Q′ ), The target network is π(s k |θ π ) is the target network of π′(s k |θ π′ ).

[0049] Update the state-action value network by gradient descent Parameters As shown in formula (16):

[0050]

[0051] in for The learning rate, express Network structure parameters, loss function As shown in formula (17):

[0052]

[0053] Among them, a′ k+1 =a k+1 +ε,ε~clip(N(0,σ),-κ,κ), clip(·) represents the clipping function, N represents Gaussian noise with mean 0 and variance σ, κ represents the clipping parameter, γ represents the discount factor, X represents a randomly sampled sample set from E, X={x k}, x k =[s k ,a k ,r k ,s k+1 ].

[0054] Action policy network π(s k |θ π )’s network parameters θ π The update is shown in formula (18):

[0055]

[0056] where μ π is π(s k |θ π )’s learning rate, θ π represents π(s k |θ π ) network structure parameters, π(s k |θπ ) As shown in formula (19):

[0057]

[0058] Target Network and π′(s k |θ π′ ) and θ π′ The update is shown in formulas (20) and (21), where the update factor

[0059]

[0060]

[0061] The advantages and beneficial effects of the present invention are as follows:

[0062] The present invention discloses a method for dynamically optimizing edge network computing resources based on drone-assisted nodes. Most of the existing drone-assisted edge network task offloading problems focus on reducing the task offloading delay of ground users through the optimized deployment of drone resources, but ignore the situation where local area user traffic bursts may occur in actual scenarios. In order to solve the problem that the burst of local traffic in the cell leads to the shortage of edge server resources and the deterioration of task offloading quality, the present invention proposes a method for dynamically optimizing computing resources based on adaptive cruising of drone-assisted nodes. According to the location distribution of ground users and the demand for task offloading, a deep reinforcement learning method is used to dynamically plan the drone cruising trajectory, and the server resource utilization of drone nodes and base station nodes during the cruising process is maximized through the task offloading scheduling strategy, thereby effectively reducing the task interruption rate of local users and reducing the average task offloading delay. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 The present invention provides a preferred embodiment of a flow chart of a method for dynamically optimizing edge network computing resources based on drone auxiliary nodes. DETAILED DESCRIPTION

[0064] The following will describe the technical solutions in the embodiments of the present invention in detail in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention.

[0065] The technical solution of the present invention to solve the above technical problems is:

[0066] The concepts and models involved in the content of the present invention are as follows.

[0067] 1. System model:

[0068] Assuming that users in an edge network cell are randomly distributed, the edge server can provide task offloading services to users in the cell through the cell base station. A drone is configured in the cell to assist the edge node, which can provide task offloading services to users within its effective coverage area. When the traffic volume of local users in the cell bursts, the drone node can optimize the distribution of computing resources and task offloading scheduling through adaptive cruise control, reduce the task interruption rate of local users, and reduce the average task offloading delay.

[0069] 2. Other symbols involved in the present invention are explained as follows:

[0070] s k : state vector

[0071] a k : Action vector

[0072] r k : Reward function

[0073] π(s k |θ π ): Drone Action Policy Network

[0074] Q(s k ,a k |θ1 Q ), Drone State-Action Value Network

[0075] θ: Neural network parameters

[0076] Δt: time slot size

[0077] Average unit task delay of user i in time slot k

[0078] Does user i meet the average unit task tolerance delay in time slot k?

[0079] User i offloads the task volume to drone j in time slot k

[0080] User i unloads the task volume to base station o in time slot k

[0081] represents the computing resources allocated by drone j to user i in time slot k

[0082] represents the computing resources allocated by base station o to user i in time slot k

[0083] Horizontal movement direction of the drone in time slot k

[0084] The vertical movement distance of the UAV in the k-time slot

[0085] The connection status between user i and drone j in time slot k

[0086] The connection status between user i and base station o in time slot k

[0087] The transmission delay of the task offloaded by user i

[0088] D comp : The computational delay of user i’s offloading task

[0089] C j : The total computing resources of UAV j

[0090] C o : The total amount of computing resources of base station o

[0091] W: User channel bandwidth

[0092] p i : Transmitting power of user i

[0093] σ 2 : Noise power

[0094] represents the communication channel gain from user i to drone j in time slot k

[0095] represents the communication channel gain from user i to base station o in time slot k

[0096] F: Unit task size

[0097] User i's task complexity

[0098] The technical solution of the present invention is described as follows:

[0099] 1. Task offloading delay and its constraints

[0100] Task offloading delay for user i As shown in formula (1), the task delay constraint is shown in formula (2):

[0101]

[0102]

[0103] In formula (1) represents the transmission delay of the unloading task of user i in time slot k, as shown in formula (3), represents the computational delay of user i’s offloading task, as shown in formula (4):

[0104]

[0105]

[0106] In formula (3), F represents the unit task size, denote the uplink transmission rate from user i to drone j and base station o in time slot k, respectively. In formula (4), represents the task complexity of user i, represents the computing resources allocated by drone j to user i in time slot k, It represents the computing resources allocated by base station o to user i in time slot k. As shown in formulas (5) and (6),

[0107]

[0108]

[0109] In formulas (5) and (6), W is the user channel bandwidth, p i is the user transmission power, σ 2 is the noise power, and They represent the communication channel gains from user i to drone j and base station o in time slot k respectively.

[0110] 2. State vector, action vector, and reward function of the Markov decision model

[0111] The ground-air network state vector, the UAV three-dimensional action vector, and the UAV action reward function are shown in formulas (7), (8), and (9), respectively:

[0112]

[0113]

[0114]

[0115] In formula (7), represents the three-dimensional coordinate position of drone j in time slot k, represents the two-dimensional coordinate position of user i in time slot k; in formula (8), represents the horizontal motion direction of UAV j in time slot k, represents the vertical movement distance of drone j in time slot k. In formula (9), ω represents the weight factor of the drone action reward function, ω∈(0,1), represents the average unit task delay of user i in time slot k, as shown in formula (10), indicates that the average unit task delay of user i in time slot k satisfies the average unit task tolerance delay, otherwise As shown in formula (11):

[0116]

[0117]

[0118] In formula (10), represents the connection status between user i and drone j. If user i offloads the task to drone j in time slot k, then otherwise represents the connection status between user i and base station o. If user i offloads the task to base station o in time slot k, then otherwise User i can only connect to one drone or base station at most in time slot k, i.e. Δt represents the time slot size, represents the amount of tasks that user i offloads to drone j in time slot k, represents the amount of tasks that user i offloads to base station o in time slot k, τ i Indicates the average unit task tolerance delay of user i.

[0119] 3. Deep reinforcement learning model based on the idea of ​​double-delay deep deterministic policy gradient algorithm

[0120] According to the Markov decision process, the cruising time of the UAV is divided into several time slots of equal size, and in any time slot k(∈K), the relative position relationship and connection state between the UAV and the ground user remain unchanged.

[0121] The UAV controller deployed in the base station control center is used as an intelligent agent, and a deep reinforcement learning model is constructed based on the idea of ​​a double-delay deep deterministic policy gradient algorithm. The idea of ​​a double-delay deep deterministic policy gradient algorithm is derived from the literature Fujimoto S, Hoof HV, Meger D. Addressing Function Approximation Error in Actor-Critic Methods. 35th International Conference on Machine Learning, ICML2018, July 10, 2018-July 15, 2018. The deep reinforcement learning model includes a system environment collector, a UAV action policy network π(s based on a neural network), and a neural network. k |θπ ) and the drone state-action value network Q(s k ,a k |θ Q ), task scheduling strategy generator, drone action reward generator, experience sample storage area E, random sample set Mini-Batch.

[0122] The system environment collector is used to collect the two-dimensional coordinate position of ground users in the k-time slot ground-to-air network User task offloading request, drone 3D coordinate position and the remaining available computing resources of the drone; the drone action strategy network π(s k |θ π ) is used to generate the state vector s of the ground-to-air network in the k-time slot k The 3D motion vector of the drone is a k π(s k |θ π ) can use a neural network with two hidden layers, 256 neurons in each layer, and the neuron activation function can use the Relu function; the drone state-action value network Q(s k ,a k |θ Q ) is used to generate the state vector s of the ground-to-air network in the k-time slot k The three-dimensional motion vector a of the drone is executed k The action evaluation value q can be Q(s k ,a k |θ1 Q )and Dual network configuration, in which the neural network can adopt three hidden layers, respectively configured with 256, 256, and 128 neurons, and the neuron activation function can adopt the Relu function.

[0123] The task scheduling strategy generator is used to generate the k-time slot user unloading strategy, and obtain the user object set I of drone j respectively. j and user object set I of base station o o The set of task offloading decision variables and The drone action reward generator generates an action reward value r after drone j completes the unloading task in time slot k k ; Drone performs a k After the action, the ground-air network status is changed from s k Transfer to k+1 ; Add k time slot experience samples [s k ,a k ,r k ,s k+1]; the random sample set Mini-Batch is composed of a fixed number of samples randomly drawn from the experience sample storage area E; the Mini-Batch sample set is respectively imported into the action strategy network π(s k |θ π ), state-action value network Q(s k ,a k |θ1 Q )and Perform training to update the neural network parameters θ π ,θ1 Q , 4. The ground-air network state vector s k With the action vector a k Calculate to obtain k+1 Methods

[0124] The k-slot ground-to-air network state vector Input the drone action strategy network π, and obtain the three-dimensional action vector of drone j through the forward propagation of each layer of neurons in the π network Calculated in, L is the horizontal moving distance of drone j in time slot k.

[0125] 5. Calculation of the amount of tasks that users offload to drones

[0126] The calculation method of the task volume offloaded by the user to the drone is shown in formulas (12) and (13):

[0127]

[0128]

[0129] in, represents the computing resources allocated by UAV j to user i in time slot k, C j Represents the total amount of computing resources of UAV j.

[0130] 6. Calculation of the amount of tasks that users offload to the base station

[0131] The calculation method of the task volume offloaded by the user to the base station is shown in formulas (14) and (15):

[0132]

[0133]

[0134] in, represents the computing resources allocated by base station o to user i in time slot k, C o Represents the total amount of computing resources of base station o.

[0135] 7. User uninstallation task scheduling method

[0136] 1) Add user i in the effective coverage area of ​​drone j to drone j’s service object set I j ,make Base station o service object set I o =II j ,make I j and I o According to the user Sort in descending order;

[0137] 2) According to formula (12) and (13), calculate I j The amount of tasks that user i offloads to drone j

[0138] 3) According to formula (14) and (15), calculate I o The amount of tasks that user i offloads to base station o

[0139] 8. Update methods of state-action value network and action strategy network

[0140] To solve the state-action value network Q(s k ,a k |θ1 Q )and Action policy network π(s k |θ π ) The instability problem of the network during the learning process, define Q(s k ,a k |θ1 Q ) is the target network Q′(s k ,a k |θ1 Q′ ), The target network is π(s k |θ π ) is the target network of π′(s k |θ π′ ).

[0141] Update the state-action value network by gradient descent Parameters As shown in formula (16):

[0142]

[0143] in for The learning rate and loss function As shown in formula (17):

[0144]

[0145] Among them, a′ k+1 =a k+1 +ε,ε~clip(N(0,σ),-κ,κ), clip(·) represents the clipping function, N represents Gaussian noise with mean 0 and variance σ, κ represents the clipping parameter, γ represents the discount factor, X represents a randomly sampled sample set from E, X={x k}, x k =[s k ,a k ,r k ,s k+1 ].

[0146] Action policy network π(s k |θ π ) parameter θ π The update is shown in formula (18):

[0147]

[0148] where μ π is π(s k |θ π ) is the learning rate, π(s k |θ π ) As shown in formula (19):

[0149]

[0150] Target Network and π′(s k |θ π′ ) in the network parameters θ π′ , as shown in formulas (20) and (21), where the update factor

[0151]

[0152]

[0153] A method for dynamically optimizing edge network computing resources based on drone auxiliary nodes, the specific implementation method of which includes the following steps:

[0154] Step 1: Construct a discrete time-state model based on the Markov decision process, including discretizing the UAV cruising time into time slots, setting up the time slot variable k, the ground-air network state vector s k 、UAV 3D motion vector a k, drone action reward function r k , where s k , a k , r k As the number of time slots k increases, corresponding transfers and changes are made, and the time slot variable k is initialized to 0;

[0155] Step 2: Take the UAV controller as the intelligent agent, build a deep reinforcement learning model based on the double-delay deep deterministic policy gradient algorithm, and establish the UAV action strategy network π(s k |θ π ), UAV state-action value network Q(s k ,a k |θ1 Q )and

[0156] Step 3: Let k = k + 1. If the drone does not change its 3D position within n consecutive time slots, jump to step 6. Otherwise, determine the user object set I of drone j according to the effective coverage range of drone j. j , user object set I of base station o o =II j , where I represents the set of all user objects. Through the task scheduling strategy generator, I is obtained j with I o The set of task offloading decision variables and Jump to step 4;

[0157] Step 4: According to and Execute task offloading for user i and obtain the corresponding reward value r through the drone action reward generator k , obtain the k-time slot UAV three-dimensional action vector a through the UAV action strategy network π k , the k-slot ground-to-air network state vector s k With the action vector a k Calculate to obtain k+1 , will [s k ,a k ,r k ,s k+1 ] is stored in the experience sample storage area E;

[0158] Step 5: Randomly sample the Mini-Batch sample set from the storage area E, and import the Mini-Batch sample set into the action strategy network π(s k |θ π ) and the state-action value network Q(s k ,a k |θ1 Q )and Perform training and jump to step 3;

[0159] Step 6: The algorithm ends.

[0160] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0161] The above embodiments should be understood to be only used to illustrate the present invention and not to limit the protection scope of the present invention. After reading the contents of the present invention, technicians can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A method for dynamically optimizing edge network computing resources based on drone auxiliary nodes, characterized in that: The following steps are involved:

101. Construct a discrete time-state model based on the Markov decision process, including discretizing the UAV cruising time into time slots, setting up the time slot variable k, the ground-air network state vector s k 、UAV 3D motion vector a k , drone action reward function r k , where s k , a k , r k As the number of time slots k increases, corresponding transfers and changes are made, and the time slot variable k is initialized to 0; 102. Take the drone controller as an intelligent agent and build a deep reinforcement learning model based on the idea of ​​double-delay deep deterministic policy gradient algorithm, including establishing a system environment collector, drone action strategy network π, drone state-action value network Q, task scheduling strategy generator, drone action reward generator, experience sample storage area E, and random sample set Mini-Batch; 103. Let k = k + 1. If the three-dimensional coordinate position of the drone does not change within n consecutive time slots, jump to step 106. Otherwise, determine the user object set I of drone j according to the effective coverage range of drone j. j , user object set I of base station o o =II j , where I represents the set of all user objects. Through the task scheduling strategy generator, I is obtained j with I o The set of task offloading decision variables and Jump to step 104; 104. According to and Execute the task offloading request of user i and obtain the corresponding reward value r through the drone action reward generator k , obtain the k-time slot UAV three-dimensional action vector a through the UAV action strategy network π k , the k-slot ground-to-air network state vector s k With the action vector a k Calculate to obtain k+1 , will [s k ,a k ,r k ,s k+1 ] is stored in the experience sample storage area E; 105. Randomly sample a Mini-Batch sample set from the experience sample storage area E, and import the Mini-Batch sample set into the action strategy network π and the state-action value network Q for training, and jump to step 103; 106. The algorithm ends; In step 101, a discrete time-state model is constructed according to the Markov decision process, wherein the ground-air network state vector s of the k-time slot is k 、UAV 3D motion vector a k , drone action reward function r k , as shown in formulas (1), (2), and (3) respectively: In formula (1), represents the three-dimensional coordinate position of drone j in time slot k, represents the two-dimensional coordinate position of user i in time slot k; in formula (2), represents the horizontal motion direction of UAV j in time slot k, represents the vertical movement distance of drone j in time slot k; in formula (3), ω represents the weight factor of drone action reward function, ω∈(0,1), Δt represents the time slot size, represents the average unit task delay of user i in time slot k, as shown in formula (4), It means that the average unit task delay of user i in time slot k satisfies the average unit task tolerance delay τ i , otherwise As shown in formula (5): In formula (4), represents the connection status between user i and drone j. If user i offloads the task to drone j in time slot k, then otherwise represents the connection status between user i and base station o. If user i offloads the task to base station o in time slot k, then otherwise User i can only connect to one drone or base station at most in time slot k, i.e. represents the amount of tasks that user i offloads to drone j in time slot k, represents the amount of tasks that user i offloads to base station o in time slot k, τ i represents the average unit task tolerance delay of user i; In step 103, the task scheduling strategy generator determines the task offloading variable set of the user. and The method is: 1) Add user i in the effective coverage area of ​​drone j to drone j’s service object set I j ,make Base station o service object set I o =II j ,make I j and I o According to the user Sort in descending order; 2) Delay of unloading tasks according to user i Computation I j The amount of tasks that user i offloads to drone j 3) Delay of unloading tasks according to user i Computation I o The amount of tasks that user i offloads to base station o 2. According to claim 1, a method for dynamically optimizing edge network computing resources based on drone auxiliary nodes is characterized in that: The step 102 constructs a deep reinforcement learning model based on the idea of ​​a double-delay deep deterministic policy gradient algorithm, including establishing a system environment collector, a drone action strategy network π, a drone state-action value network Q, a task scheduling strategy generator, a drone action reward generator, an experience sample storage area E, and a random sample set Mini-Batch, specifically including: The system environment collector is used to collect the two-dimensional coordinate position of ground users in the k-time slot ground-to-air network User task offloading request, drone 3D coordinate position and the remaining available computing resources of the drone; the drone action strategy network π generates the k-time slot ground-air network state s k The 3D motion vector of the drone is a k ; The drone state-action value network Q generates the k-time slot ground-air network state s k Execute the drone's three-dimensional motion vector a k The action evaluation value q; the task scheduling strategy generator is used to generate the k-time slot user offloading strategy and obtain the task offloading decision variable set and The drone action reward generator generates the action reward value r of drone j in time slot k after completing the unloading task in time slot k. k ; Drone performs a k After the action, the ground-air network status is changed from s k Transfer to k+1 ; Add k time slot experience samples [s k ,a k ,r k ,s k+1 ]; The random sample set Mini-Batch is composed of a fixed number of samples randomly extracted from the experience sample storage area E; Among them, the drone action strategy network π and the drone state-action value network Q are both neural networks, each containing several hidden layers, and each hidden layer contains several neurons.

3. According to the method of claim 1, the method is characterized in that: The amount of tasks that user i offloads to drone j in step 2) The calculation method of is shown in formula (6) and (7): in, represents the computing resources allocated by UAV j to user i in time slot k, C j represents the total amount of computing resources of UAV j, represents the uplink transmission rate from user i to drone j in time slot k, F represents the task unit size, represents the task complexity of user i, τ i Indicates the average unit task tolerance delay of user i.

4. According to claim 3, a method for dynamically optimizing edge network computing resources based on drone auxiliary nodes is characterized in that: The amount of tasks that user i offloads to base station o in step 3) The calculation method of is shown in formula (8) and (9): in, represents the computing resources allocated by base station o to user i in time slot k, C o represents the total amount of computing resources of base station o, represents the uplink transmission rate from user i to base station o in time slot k.

5. The method for dynamically optimizing edge network computing resources based on drone auxiliary nodes according to claim 1 is characterized in that: The task offloading delay of user i in step 2) and step 3) is shown in formula (10), and the task offloading delay constraint is shown in formula (11): In formula (10), since the size of the task calculation result is much smaller than the task itself, only the uplink transmission delay of the user task offloading and the task calculation delay are considered, and the downlink transmission delay of the task calculation result is ignored. represents the total delay of task offloading for user i in time slot k, represents the transmission delay of the offloading task of user i in time slot k, as shown in formula (12); represents the computational delay of the offload task, as shown in formula (13): In formula (12), They represent the uplink transmission rates from user i to drone j and base station o in time slot k, respectively, as shown in formulas (14) and (15): In formulas (14) and (15), W is the user channel bandwidth, p i is the user transmission power, σ 2 is the noise power, and They represent the communication channel gains from user i to drone j and base station o in time slot k respectively.

6. The method for dynamically optimizing edge network computing resources based on drone auxiliary nodes according to claim 1 is characterized in that: In step 104, the UAV action strategy network π is used to obtain the k-time slot UAV three-dimensional action vector a k , the k-slot ground-to-air network state vector s k With the action vector a k Calculate to obtain k+1 , specifically including: The k-slot ground-to-air network state vector Input the drone action strategy network π, and obtain the three-dimensional action vector of drone j through the forward propagation of each layer of neurons in the π network Calculated in, L is the horizontal moving distance of drone j in time slot k.

7. The method for dynamically optimizing edge network computing resources based on drone auxiliary nodes according to claim 1 is characterized in that: In step 105, a Mini-Batch sample data set is obtained from the experience sample storage area E by random sampling, and the method for optimizing the state-action value network and the action strategy network is: To solve the state-action value network Q, Q includes and With the action policy network π(s k |θ π ) The network is unstable during the learning process, and the definition The target network is The target network is π(s k |θ π ) is the target network of π′(s k |θ π′ ); Update the state-action value network by gradient descent Parameters As shown in formula (16): in for The learning rate, express Network structure parameters, loss function As shown in formula (17): Among them, a′ k+1 =a k+1 +ε,ε~clip(N(0,σ),-κ,κ), clip(·) represents the clipping function, N represents Gaussian noise with mean 0 and variance σ, κ represents the clipping parameter, γ represents the discount factor, X represents a randomly sampled sample set from E, X={x k }, x k =[s k ,a k ,r k ,s k+1 ]; Action policy network π(s k |θ π )’s network parameters θ π The update is shown in formula (18): where μ π is π(s k |θ π )’s learning rate, θ π represents π(s k |θ π ) network structure parameters, π(s k |θ π ) As shown in formula (19): Target Network and π′(s k |θ π′ ) and θ π′ The update is shown in formulas (20) and (21), where the update factor

Images