UAV-assisted edge computing task offloading method and system based on deep reinforcement learning

Abstract

The present invention discloses a method for unloading unmanned aerial vehicle-assisted edge computing tasks based on deep reinforcement learning, which performs joint optimization of online decision-making, computational offloading and path optimization for computationally intensive tasks of mobile user devices in unmanned aerial vehicle-assisted edge computing. At the edge of the network, multiple user devices move randomly and generate computationally intensive tasks in real time. Each task can be arbitrarily divided into two parts, which are calculated in parallel on the user device and the unmanned aerial vehicle server. At the same time, the unmanned aerial vehicle moves in three dimensions, reducing the average transmission overhead and improving the unloading efficiency. On the basis of reducing the response delay and energy consumption overhead, the resource utilization efficiency of the edge computing system is improved, and the sustainability, high quality and high robustness of the edge computing system are guaranteed.

Description

Technical Field

[0001] The present invention belongs to the field of edge computing technology, and more specifically, relates to a method and system for unloading drone-assisted edge computing tasks based on deep reinforcement learning. Background Art

[0002] With the rapid development of Internet of Things technology, the number of mobile devices continues to surge, and it is driving the rapid development of new mobile applications, such as automatic navigation, online interactive games, AR / VR virtual technology, etc. The introduction of mobile edge computing provides an economical and efficient solution to the limited computing power and battery capacity of mobile devices. However, how to provide reliable computing services in areas with limited coverage of communication infrastructure remains a challenging problem. Drones have the characteristics of high flexibility and maneuverability, and have become an important means of assisting wireless communications. Drones equipped with small servers can be used as aerial computing platforms to provide flexible and elastic computing services for mobile devices, with the advantages of efficient and convenient deployment and wide coverage. Drone-assisted mobile edge computing technology has become one of the key technologies of 5G networks and has received widespread attention from academia and industry.

[0003] At present, in the process of edge computing, optimizing edge computing tasks has become a very important research topic. In order to reduce the power consumption and processing time of the user end, the commonly used optimization method is to offload the computing tasks of the user device terminal to the edge server.

[0004] However, the current drone-assisted edge computing task offloading methods all have some defects that cannot be ignored: First, the current drone-assisted edge computing task offloading methods are mostly binary offloading, that is, the computing tasks are either all calculated locally or all offloaded to the drone server. This method may lead to a waste of local computing resources of user devices and remote computing resources of drones; Second, most of the current drone-assisted edge computing task offloading methods assume that the drone starts from a fixed starting point and goes to a fixed end point and does not consider the drone's ability to move in the vertical direction. Therefore, on the one hand, ignoring the flexible deployment capability of drones will lead to being out of touch with reality. For example, in emergencies, drones are deployed at any location to offload edge computing tasks. Therefore, such methods cannot make optimal path planning for random starting points; on the other hand, ignoring the vertical mobility of drones cannot stimulate drones to adjust their flight trajectories to the maximum extent to optimize communication conditions and shared computing resources; third, most of the current drone-assisted edge computing task offloading scenarios only consider the a priori known situation of user equipment fixed or mobile paths, but ignore the possibility of random movement of user equipment in reality, and make non-optimal allocation of drone flight trajectories and system computing resources, which will lead to low overall work efficiency of the edge computing system and inability to make online offloading decisions based on the real-time changing system environment; fourth, the edge computing system composed of the three major elements of joint online decision-making, partial offloading and path optimization will face highly complex non-convex optimization problems. The existing drone-assisted edge computing task offloading methods cannot find the global optimal solution in a large solution space, which will lead to the technical problem that the edge computing system cannot make the optimal decision. Summary of the invention

[0005] In response to the above defects or improvement needs of the prior art, the present invention provides a UAV-assisted edge computing task offloading method and system based on deep reinforcement learning, which aims to solve the technical problem that the existing UAV-assisted edge computing task offloading method adopts binary offloading, resulting in waste of local computing resources of user equipment and remote computing resources of UAV, and the problem that the existing UAV-assisted edge computing task offloading method ignores the flexible deployment capability and vertical mobility capability of UAV, resulting in being out of touch with reality and unable to make optimal path planning for random starting points, and the problem that the existing UAV-assisted edge computing task offloading method ignores the possibility of random movement of user equipment in practice, resulting in the edge computing system being unable to make online offloading decisions according to the real-time changing system environment, thereby resulting in low overall work efficiency, and the technical problem that the existing UAV-assisted edge computing task offloading method is difficult to solve the highly complex non-convex optimization problem composed of the three major elements of joint online decision-making, partial offloading and path optimization, resulting in the edge computing system being unable to complete the continuous optimal offloading process.

[0006] To achieve the above object, according to one aspect of the present invention, a method for unloading UAV-assisted edge computing tasks based on deep reinforcement learning is provided, comprising the following steps:

[0007] (1) Set the time slot number t = 1;

[0008] (2) Determine whether t is greater than the total number of time slots T in the edge computing system. If so, proceed to step (8); otherwise, proceed to step (3);

[0009] (3) The server-equipped drone obtains the parameter W of the task to be processed by the nth terminal user in the tth time slot from the nth terminal user in the edge computing system. n (t) and coordinate parameter c n (t), obtain the coordinate parameter c of the drone in the t-1th time slot from itself u (t-1), and according to the parameters W of the task to be processed n (t) and coordinate parameter c n (t) Obtain the full local processing delay of the nth end user in the tth time slot Full local processing energy consumption Where n represents the serial number of the terminal users with pending tasks in the edge computing system, and n∈[1, the total number of terminal users with pending tasks in the edge computing system N].

[0010] (4) The server-mounted drone obtains the parameter W of the task to be processed for each terminal user in the tth time slot according to step (3). n (t) and coordinate parameter c n (t), and the coordinate parameter c of the drone at the t-1th time slot u (t-1), obtain the normalized parameters W of the pending tasks for each terminal user in the tth time slot n ′(t) and the normalized coordinate parameter c n ′(t), and the normalized coordinate parameter c of the drone at the t-1th time slot u ′(t-1).

[0011] (5) The server-mounted drone takes the normalized parameters W of the task to be processed for each terminal user in the tth time slot obtained in step (4) n ′(t) and the normalized coordinate parameter c n ′(t), and the normalized coordinate parameter c of the drone at the t-1th time slot u ′(t-1) inputs the pre-trained deep reinforcement learning model w to obtain the optimal horizontal movement angle ω(t) and optimal horizontal movement distance Δd of the drone at the tth time slot u(t), optimal vertical movement distance Δz u (t) and the optimal task offloading amount for each end user

[0012] (6) The optimal horizontal movement angle ω(t) and optimal horizontal movement distance Δd of the drone equipped with the server at the tth time slot obtained in step (5) u (t), optimal vertical movement distance Δz u (t) and the optimal task offloading amount for each end user Get the local processing delay of the nth terminal user in the tth time slot Task uplink delay Local processing energy consumption And the task uplink energy consumption And the remote processing latency of the drone carrying the server and remote processing power consumption

[0013] (7) The local processing delay of each end user in the tth time slot obtained by the server-carrying drone according to step (6) Task uplink delay Local processing energy consumption Task uplink energy consumption And the remote processing latency of the drone carrying the server Remote processing power consumption Obtain the task response delay T(t) and task energy consumption E(t) of the edge computing system in the tth time slot (by this time, the edge computing system has completed the unloading process of the tth time slot), set t=t+1, and return to step (2);

[0014] (8) The server-carrying drone obtains the full local processing delay of each end user in the tth time slot according to step (3) Full local processing energy consumption And the task response delay T(t) and task energy consumption E(t) of the edge computing system in each time slot obtained in step (7) are used to obtain the system utility U of the edge computing system.

[0015] (9) The drone equipped with the server feeds back the system utility U of the edge computing system obtained in step (8) as a service result to each end user, and the process ends.

[0016] Preferably, the parameter W of the task to be processed by the nth terminal user in the tth time slot is n (t) includes the task size D of the tasks to be processed n (t), and the number of CPU cycles C required to process 1 bit of the task to be processed n(t); coordinate parameter c of the nth terminal user in the tth time slot n (t) includes the x-axis coordinate x n (t), y-axis coordinate y n (t), and the z-axis coordinate z n (t).

[0017] Full local processing delay of the nth end user in the tth time slot It is calculated according to the following formula:

[0018]

[0019] Among them, F N The frequency of the local CPU chip used by the nth end user.

[0020] The total local processing energy consumption of the nth end user in the tth time slot It is calculated according to the following formula:

[0021]

[0022] Wherein κ is the capacitance value of the effective switch capacitor of the local CPU chip used by the nth terminal user.

[0023] Preferably, step (4) comprises the following sub-steps:

[0024] (4-1) The parameter W of the task to be processed by each terminal user in the tth time slot obtained according to step (3) n (t) Obtain the normalized task parameter W to be processed for each terminal user in the tth time slot n ′(t);

[0025] (4-2) The coordinate parameter c of each terminal user in the tth time slot obtained according to step (3) n (t), obtain the normalized coordinate parameter c of each terminal user in the tth time slot n ′(t);

[0026] (4-3) According to the coordinate parameter c of the drone in the t-1th time slot u (t-1), obtain the normalized coordinate parameter c of the drone at the t-1th time slot u ′(t-1).

[0027] Preferably, the normalized parameter W of the task to be processed by each terminal user in the tth time slot is n ′(t) includes the normalized task size D of the task to be processed n ′(t), and the normalized number of CPU cycles required to process 1 bit of the task to be processed Cn ′(t);

[0028] The normalized task size D of each terminal user to be processed in the tth time slot n ′(t) is calculated according to the following formula:

[0029] D n ′(t)=D n (t) / ξ D

[0030] where ξ D is a normalization factor, and the value range of the normalization factor is determined by the task size to be processed. The larger the value is, the larger the value is, and vice versa.

[0031] The number of CPU cycles C required for each terminal user to process 1 bit of the tasks to be processed in the tth time slot after normalization n ′(t) is calculated according to the following formula:

[0032] C n ′(t)=C n (t) / ξ C

[0033] where ξ C is the normalization factor.

[0034] The normalized coordinate parameter c of each terminal user in the tth time slot is n ′(t) includes the normalized x-axis coordinate x n ′(t), normalized y-axis coordinate y n ′(t), and the normalized z-axis coordinate z n ′(t);

[0035] The normalized x-axis coordinate x of each terminal user in the tth time slot is n ′(t) is calculated according to the following formula:

[0036] x n ′(t)=x n (t) / ξ X

[0037] where ξ X is a normalization factor, and its value range is determined by the size of the x-axis coordinate. The larger the x-axis coordinate is, the larger the value is, and vice versa.

[0038] The normalized y-axis coordinate y of each terminal user in the tth time slot is n ′(t) is calculated according to the following formula:

[0039] yn ′(t)=y n (t) / ξ Y

[0040] where ξ Y is a normalization factor, and its value range is determined by the size of the y-axis coordinate. The larger the y-axis coordinate is, the larger the value is, and vice versa.

[0041] The normalized z-axis coordinate z of each end user in the tth time slot is n ′(t) is calculated according to the following formula:

[0042] z n ′(t)=z n (t) / ξ Z

[0043] where ξ Z is the normalization factor.

[0044] The normalized coordinate parameter c of the drone at the t-1th time slot u ′(t-1) includes the normalized x-axis coordinate x u ′(t-1), the normalized y-axis coordinate y u ′(t-1), and the normalized z-axis coordinate z u ′(t-1).

[0045] The normalized x-axis coordinate x of the drone at the t-1th time slot u ′(t-1) is calculated according to the following formula:

[0046] x u ′(t-1)=x u (t-1) / ξ X

[0047] The normalized y-axis coordinate y of the drone at the t-1th time slot u ′(t-1) is calculated according to the following formula:

[0048] y u ′(t-1)=y u (t-1) / ξ Y

[0049] The normalized z-axis coordinate z of the drone at the t-1th time slot u ′(t-1) is calculated according to the following formula:

[0050] z u ′(t-1)=z u (t-1) / ξ Z .

[0051] Preferably, step (6) comprises the following sub-steps:

[0052] (6-1) The optimal horizontal movement angle ω(t) and optimal horizontal movement distance Δd of the drone at the tth time slot obtained according to step (5) u (t), optimal vertical movement distance Δz u (t), obtain the coordinate parameters c of the drone at the tth time slot u (t).

[0053] (6-2) The coordinate parameter c of the drone carrying the server at the tth time slot obtained according to step (6-1) u (t) and the task offloading amount of each terminal user that the drone should execute in the tth time slot obtained according to step (5) Remotely offload a portion of the pending tasks of each terminal user in the tth time slot, and obtain the uplink delay of the task of the nth terminal user in the tth time slot Task uplink energy consumption And the remote processing latency of the drone carrying the server Remote processing power consumption

[0054] (6-3) The task offloading amount of each terminal user that the drone should execute in the tth time slot obtained in step (5) The remaining part of each terminal user's pending task in the tth time slot is processed locally, and the local processing delay of the nth terminal user in the tth time slot is obtained. and local processing energy consumption

[0055] Preferably, the x-axis coordinate x of the drone carrying the server at the tth time slot is u (t) is calculated according to the following formula:

[0056] x u (t) = x u (t-1)+Δd u (t)cos(ω(t))

[0057] The y-axis coordinate y of the drone carrying the server at the tth time slot u (t) is calculated according to the following formula:

[0058] y u (t) = y u (t-1)+Δd u (t)sin(ω(t))

[0059] The z-axis coordinate z of the drone carrying the server at the tth time slot u (t) is calculated according to the following formula:

[0060] z u (t) = z u (t-1)+Δz u (t)

[0061] The uplink delay of the task of the nth terminal user in the tth time slot It is calculated according to the following formula:

[0062]

[0063] in is the wireless communication rate at which the nth terminal user sends its pending task to the UAV carrying the server in the tth time slot, which can be expressed by Shannon’s theorem as Among them B u is the wireless channel bandwidth of the UAV; P n is the transmission power of the nth terminal user; σ 2 is the additive white Gaussian noise power; P NLOS is the ambient noise power; is the channel gain of the nth terminal user in the tth time slot, which can be expressed by the free space path loss model as where g 0 is the channel gain when the reference distance is 1m; is the straight-line distance between the nth terminal user and the drone carrying the server at the tth time slot, which can be expressed by the Euclidean distance as

[0064] The uplink energy consumption of the task of the nth terminal user in the tth time slot It is calculated according to the following formula:

[0065]

[0066] Where P u is the received power of the drone carrying the server.

[0067] The remote processing delay of the task of the nth end user in the tth time slot on the drone equipped with the server It is calculated according to the following formula:

[0068]

[0069] in The frequency of the server CPU chip allocated to the nth end user by the server-carrying drone.

[0070] The energy consumption of the task of the nth end user in the tth time slot is processed remotely by the drone equipped with the server. It is calculated according to the following formula:

[0071]

[0072] Local processing delay of the nth end user in the tth time slot It is calculated according to the following formula:

[0073]

[0074] The local processing energy consumption of the nth end user in the tth time slot It is calculated according to the following formula:

[0075]

[0076] Preferably, the task response delay T(t) of the edge computing system in the tth time slot is calculated according to the following formula:

[0077]

[0078] The task energy consumption E(t) of the edge computing system in the tth time slot is calculated according to the following formula:

[0079]

[0080] Preferably, step (8) comprises the following sub-steps:

[0081] (8-1) The server-carrying drone obtains the full local processing delay of each end user in the tth time slot according to step (3) Full local processing energy consumption Get the total local processing latency of the edge computing system and total local processing energy consumption

[0082] Specifically, the total local processing latency of the edge computing system It is calculated according to the following formula:

[0083]

[0084] Total local processing energy consumption of edge computing systems It is calculated according to the following formula:

[0085]

[0086] (8-2) The drone equipped with the server obtains the task response delay T(t) and task energy consumption E(t) of the edge computing system in each time slot according to step (7), and obtains the total task response delay T of the edge computing system system and the total energy consumption of the task E system .

[0087] Specifically, the total task response delay T of the edge computing system is system It is calculated according to the following formula:

[0088]

[0089] The total energy consumption of edge computing system tasks E system It is calculated according to the following formula:

[0090]

[0091] (8-3) The total local processing delay of the edge computing system obtained according to step (8-1) and total local processing energy consumption And the total task response delay T of the edge computing system obtained according to step (8-2) system and the total energy consumption of the task E system , get the system utility U of the edge computing system:

[0092]

[0093] Preferably, the deep reinforcement learning model w adopts a deep reinforcement learning model of the TensorFlow architecture, which is trained according to the following steps:

[0094] (A1) A user trajectory and task file dataset consisting of a plurality of user trajectory and task files is obtained, and the user trajectory and task file dataset is randomly divided into a training set and a test set according to a ratio of 100:2500.

[0095] (A2) Use the training set obtained in step (A1) to train the deep reinforcement learning model w to obtain a preliminarily trained deep reinforcement learning model.

[0096] (A3) using the test set obtained in step (A1) to verify the deep reinforcement learning model w preliminarily trained in step (A2) until the optimization target P1 of the obtained edge computing system reaches the optimal value, thereby obtaining the finally trained deep reinforcement learning model w;

[0097] Wherein step (A2) comprises the following sub-steps:

[0098] (A2-1) Set iteration counter i=0;

[0099] (A2-2) Determine whether i is equal to the preset model iteration number I, if yes, proceed to step (A2-15), otherwise proceed to step (A2-3);

[0100] (A2-3) Randomly select a user trajectory and task file from the training set obtained in step (A1), and randomly initialize the initial (when time slot k = 0) coordinate parameters c of the drone carrying the server u (0), and set the iteration counter k = 1;

[0101] (A2-4) Determine whether k is greater than the preset model simulation time slot number K, if so, proceed to step (A2-12), otherwise proceed to step (A2-5);

[0102] (A2-5) obtaining the coordinate parameters of the drone carrying the server in the k-1th time slot, and extracting the parameters and coordinate parameters of the task to be processed of each user in the kth time slot from the user trajectory and task file obtained in step (A2-3);

[0103] (A2-6) Obtain the full local processing delay of each user in the kth time slot according to the parameters and coordinate parameters of the task to be processed of each user in the kth time slot obtained in step (A2-5) and full local processing energy consumption

[0104] (A2-7) normalizing the coordinate parameters of the drone carrying the server at the k-1th time slot, and the parameters and coordinate parameters of the task to be processed of each user at the kth time slot obtained in step (A2-5), so as to obtain the normalized coordinate parameters of the drone carrying the server at the k-1th time slot, and the normalized parameters and coordinate parameters of the task to be processed of each user at the kth time slot;

[0105] (A2-8) inputting the normalized coordinate parameters of the drone carrying the server at the k-1th time slot obtained in step (A2-7), and the normalized parameters of the task to be processed and the normalized coordinate parameters of each user at the kth time slot into the deep reinforcement learning model w, so as to obtain the action a(k) of the drone at the kth time slot through the ε-greedy strategy, where ε is the greediness value;

[0106] The deep reinforcement learning model w adopts the ε-greedy strategy for action selection, that is, there is a probability of ε to take random actions, and a probability of 1-ε to take the network output value of the deep reinforcement learning model w and the action noise The value range of ε is [0,1], preferably 0.5. is Gaussian noise with mean equal to 0 and standard deviation equal to 0.05.

[0107] (A2-9) according to the action a(k) of the drone in the kth time slot obtained in step (A2-8), obtain the task response delay T(k) and task energy consumption E(k) of the edge computing system in the kth time slot, as well as the coordinate parameters of the drone in the kth time slot;

[0108] (A2-10) updating the greediness value ε and the deep reinforcement learning model w in step (A2-8) to obtain updated greediness value ε and deep reinforcement learning model w respectively;

[0109] The greedy value ε is updated according to the following formula:

[0110]

[0111] The deep reinforcement learning model w adopts the reward function R, and has:

[0112] R(k)=α 1 T(k)+α 2 E(k)

[0113] Among them, α 1 and α 2 The value range of is [0,1], and the preferred value is 0.5;

[0114] The deep reinforcement learning model w adopts a soft update method and has:

[0115] θ w′ ←τθ w +(1-τ)θ w′

[0116] Wherein, τ is the update rate, and its value range is [0,1], preferably 0.05; θ w ,θ w′ They are the network parameters before and after the update of the deep reinforcement learning model w.

[0117] (A2-11) Set the iteration counter k=k+1, and return to step (A2-4);

[0118] (A2-12) The total local processing delay of each user in the kth time slot obtained according to step (A2-6) and full local processing energy consumption And the task response delay T(k) and task energy consumption E(k) of the edge computing system in the kth time slot obtained in step (A2-9) to obtain the system utility U of the edge computing system;

[0119] (A2-13) According to the system utility U of the edge computing system obtained in step (A2-12), the optimization target P1 of the edge computing system is obtained:

[0120] (P1:)maxU

[0121] stC1:Δd u (k)≤ΔD max

[0122] C2:|Δz u (k)|≤ΔZ max

[0123] The constraint C1 indicates that the horizontal movement distance of the drone cannot exceed the maximum horizontal movement distance ΔD max ; Constraint C2 means that the vertical movement distance of the drone cannot exceed the maximum vertical movement distance ΔZ max ;

[0124] (A2-14) Set the iteration counter i=i+1, and return to step (A2-2);

[0125] (A2-15) The model iteration is completed, and a preliminarily trained deep reinforcement learning model is obtained.

[0126] According to another aspect of the present invention, a drone-assisted edge computing task offloading system based on deep reinforcement learning is provided, comprising:

[0127] The first module is used to set the time slot number t=1;

[0128] The second module is used to determine whether t is greater than the total number T of time slots in the edge computing system, if so, enter the eighth module, otherwise enter the third module;

[0129] The third module is set in the drone equipped with the server, and is used to obtain the parameter W of the task to be processed by the nth terminal user in the tth time slot from the nth terminal user in the edge computing system. n (t) and coordinate parameter c n (t), obtain the coordinate parameter c of the drone in the t-1th time slot from itself u (t-1), and according to the parameters W of the task to be processed n (t) and coordinate parameter c n (t) Obtain the full local processing delay of the nth end user in the tth time slot Full local processing energy consumption Where n represents the serial number of the terminal users with pending tasks in the edge computing system, and n∈[1, the total number of terminal users with pending tasks in the edge computing system N].

[0130] The fourth module is set in the drone equipped with the server, and is used to calculate the parameter W of the task to be processed for each terminal user in the tth time slot according to the third module. n (t) and coordinate parameter c n (t), and the coordinate parameter c of the drone at the t-1th time slot u (t-1), obtain the normalized parameters W of the pending tasks for each terminal user in the tth time slot n ′(t) and the normalized coordinate parameter c n ′(t), and the normalized coordinate parameter c of the drone at the t-1th time slot u ′(t-1).

[0131] The fifth module is set in the drone equipped with the server, and is used to convert the normalized parameters W of the task to be processed of each terminal user in the tth time slot obtained in the fourth module into n ′(t) and the normalized coordinate parameter c n ′(t), and the normalized coordinate parameter c of the drone at the t-1th time slot u ′(t-1) inputs the pre-trained deep reinforcement learning model w to obtain the optimal horizontal movement angle ω(t) and optimal horizontal movement distance Δd of the drone at the tth time slot u (t), optimal vertical movement distance Δz u (t) and the optimal task offloading amount for each end user

[0132] The sixth module is set in the drone equipped with the server, and is used to calculate the optimal horizontal movement angle ω(t) and the optimal horizontal movement distance Δd of the drone at the tth time slot obtained in the fifth module. u (t), optimal vertical movement distance Δz u (t) and the optimal task offloading amount for each end user Get the local processing delay of the nth terminal user in the tth time slot Task uplink delay Local processing energy consumption And the task uplink energy consumption And the remote processing latency of the drone carrying the server and remote processing power consumption

[0133] The seventh module is set in the drone equipped with the server, and is used to calculate the local processing delay of each terminal user in the tth time slot according to the sixth module. Task uplink delay Local processing energy consumption Task uplink energy consumption And the remote processing latency of the drone carrying the server Remote processing power consumption Obtain the task response delay T(t) and task energy consumption E(t) of the edge computing system in the tth time slot (by this time, the edge computing system has completed the unloading process of the tth time slot), set t=t+1, and return to the second module;

[0134] The eighth module is set in the drone equipped with the server, and is used to calculate the full local processing delay of each terminal user in the tth time slot according to the third module. Full local processing energy consumption And the seventh module obtains the task response delay T(t) and task energy consumption E(t) of the edge computing system in each time slot, and obtains the system utility U of the edge computing system.

[0135] The ninth module is set on the drone equipped with the server and is used to feed back the system utility U of the edge computing system obtained by the eighth module as a service result to each end user, and the process ends.

[0136] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0137] (1) Since the present invention adopts steps (2) to (7), it fully considers the physical location and task situation of the user device in the actual scene, and updates the state according to the real-time situation. At the same time, it unifies the input parameters of different dimensions, different ranges and different dimensions, simplifies the complexity of the state space in the dynamic environment, enhances the correlation between the various input parameters, and improves the state analysis and feature extraction capabilities of the deep reinforcement learning model for complex dynamic environments. It solves the problem that the existing UAV-assisted edge computing task offloading method ignores the possibility of random movement of user devices in reality, resulting in the edge computing system being unable to make online offloading decisions according to the real-time changing system environment, thereby resulting in low overall work efficiency;

[0138] (2) Since the present invention adopts steps (1) and (6), it fully considers the actual situation that the UAV has three-dimensional mobility and will be deployed at random locations, and solves the problem that the existing UAV-assisted edge computing task offloading method ignores the UAV's flexible deployment capability and vertical mobility, resulting in being out of touch with reality and unable to make optimal path planning for random starting points;

[0139] (3) Since the present invention adopts steps (5) to (7), the drone can quickly analyze the position and resource status of the user device and itself through the deep reinforcement learning model, so as to quickly make the drone movement path and targeted unloading decisions for each user. Therefore, it can solve the technical problem that the existing drone-assisted edge computing task unloading method adopts binary unloading, resulting in waste of local computing resources of the user device and remote computing resources of the drone;

[0140] (4) Since the present invention adopts steps (1) to (9), it models the continuous time slot interaction process between the user equipment and the drone in a dynamic environment, so that the drone can continuously and optimally unload the user equipment, and at the same time obtain the system utility of the edge computing system in the entire unloading process. Therefore, it can solve the technical problem that the existing algorithms are difficult to solve the highly complex non-convex optimization problem of the three major elements of comprehensive online decision-making, partial unloading and path optimization, resulting in the edge computing system being unable to complete the continuous optimal unloading process;

[0141] (5) The implementation process of the present invention is simple and has high computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0142] Figure 1 It is a flow chart of the method for unloading UAV-assisted edge computing tasks based on deep reinforcement learning of the present invention;

[0143] Figure 2 It is an environmental architecture diagram of the UAV-assisted edge computing task offloading method based on deep reinforcement learning of the present invention;

[0144] Figure 3 It is a schematic diagram of performance comparison between the method of the present invention and other existing methods;

[0145] Figure 4 It is a schematic diagram of performance comparison between the method of the present invention and other existing methods at different time slots. DETAILED DESCRIPTION

[0146] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0147] The main technical concept of the present invention is to jointly optimize online decision-making, computation offloading and path optimization for computationally intensive tasks of mobile user devices in drone-assisted edge computing. At the edge of the network, multiple user devices move randomly and generate computationally intensive tasks in real time. Each task can be arbitrarily divided into two parts, which are calculated in parallel on the local user device and the drone server. At the same time, the drone moves in three dimensions to reduce the average transmission overhead and improve the offloading efficiency. On the basis of reducing response delay and energy consumption overhead, the resource utilization efficiency of the edge computing system is improved, ensuring the sustainability, high quality and high robustness of the edge computing system. The environmental architecture diagram of the present invention is shown in the figure. Figure 2 shown.

[0148] like Figure 1 As shown, the present invention discloses a method for unloading UAV-assisted edge computing tasks based on deep reinforcement learning, comprising the following steps:

[0149] (1) Set the time slot number t = 1;

[0150] (2) Determine whether t is greater than the total number of time slots T in the edge computing system. If so, proceed to step (8); otherwise, proceed to step (3);

[0151] (3) The server-equipped drone obtains the parameter W of the task to be processed by the nth terminal user in the tth time slot from the nth terminal user in the edge computing system. n (t) and coordinate parameter c n (t), obtain the coordinate parameter c of the drone in the t-1th time slot from itself u (t-1), and according to the parameters W of the task to be processed n (t) and coordinate parameter c n (t) Obtain the full local processing delay of the nth end user in the tth time slot Full local processing energy consumption Where n represents the serial number of the terminal users with pending tasks in the edge computing system, and n∈[1, the total number of terminal users with pending tasks in the edge computing system N].

[0152] Specifically, the parameter W of the task to be processed by the nth terminal user in the tth time slot is n (t) includes the task size D of the tasks to be processed n (t), and the number of CPU cycles C required to process 1 bit of the task to be processed n (t); coordinate parameter c of the nth terminal user in the tth time slot n (t) includes the x-axis coordinate x n (t), y-axis coordinate y n (t), and the z-axis coordinate zn (t).

[0153] Full local processing delay of the nth end user in the tth time slot It is calculated according to the following formula:

[0154]

[0155] Among them, F N The frequency of the local CPU chip used by the nth end user.

[0156] The total local processing energy consumption of the nth end user in the tth time slot It is calculated according to the following formula:

[0157]

[0158] Wherein κ is the capacitance value of the effective switch capacitor of the local CPU chip used by the nth terminal user.

[0159] (4) The server-mounted drone obtains the parameter W of the task to be processed for each terminal user in the tth time slot according to step (3). n (t) and coordinate parameter c n (t), and the coordinate parameter c of the drone at the t-1th time slot u (t-1), obtain the normalized parameters W of the pending tasks for each terminal user in the tth time slot n ′(t) and the normalized coordinate parameter c n ′(t), and the normalized coordinate parameter c of the drone at the t-1th time slot u ′(t-1).

[0160] Specifically, step (4) includes the following sub-steps:

[0161] (4-1) The parameter W of the task to be processed by each terminal user in the tth time slot obtained according to step (3) n (t) Obtain the normalized task parameter W to be processed for each terminal user in the tth time slot n ′(t);

[0162] Specifically, the normalized parameter W of the task to be processed by each end user in the tth time slot is n ′(t) includes the normalized task size D of the task to be processed n ′(t), and the normalized number of CPU cycles required to process 1 bit of the task to be processed C n ′(t);

[0163] More specifically, the normalized task size D of each end user's pending task in the tth time slot is n ′(t) is calculated according to the following formula:

[0164] D n ′(t)=D n (t) / ξ D

[0165] where ξ D is a normalization factor, and the value range of the normalization factor is determined by the task size to be processed. The larger the value is, the larger the value is, and vice versa.

[0166] The number of CPU cycles C required for each terminal user to process 1 bit of the tasks to be processed in the tth time slot after normalization n ′(t) is calculated according to the following formula:

[0167] C n ′(t)=C n (t) / ξ C

[0168] where ξ C is a normalization factor. The value range of the normalization factor is determined by the number of CPU cycles required to process 1-bit tasks in the tasks to be processed. The larger the value, the larger the value, and vice versa.

[0169] (4-2) The coordinate parameter c of each terminal user in the tth time slot obtained according to step (3) n (t), obtain the normalized coordinate parameter c of each terminal user in the tth time slot n ′(t);

[0170] Specifically, the normalized coordinate parameter c of each terminal user in the tth time slot is n ′(t) includes the normalized x-axis coordinate x n ′(t), normalized y-axis coordinate y n ′(t), and the normalized z-axis coordinate z n ′(t);

[0171] The normalized x-axis coordinate x of each terminal user in the tth time slot is n ′(t) is calculated according to the following formula:

[0172] x n ′(t)=x n (t) / ξ X

[0173] where ξ Xis a normalization factor, and its value range is determined by the size of the x-axis coordinate. The larger the x-axis coordinate is, the larger the value is, and vice versa.

[0174] The normalized y-axis coordinate y of each terminal user in the tth time slot is n ′(t) is calculated according to the following formula:

[0175] y n ′(t)=y n (t) / ξ Y

[0176] where ξ Y is a normalization factor, and its value range is determined by the size of the y-axis coordinate. The larger the y-axis coordinate is, the larger the value is, and vice versa.

[0177] The normalized z-axis coordinate z of each end user in the tth time slot is n ′(t) is calculated according to the following formula:

[0178] z n ′(t)=z n (t) / ξ Z

[0179] where ξ Z is a normalization factor. The value range of the normalization factor is determined by the size of the z-axis coordinate. The larger the z-axis coordinate is, the larger the value is, and vice versa.

[0180] (4-3) According to the coordinate parameter c of the drone in the t-1th time slot u (t-1), obtain the normalized coordinate parameter c of the drone at the t-1th time slot u ′(t-1).

[0181] Specifically, the normalized coordinate parameter c of the drone at the t-1th time slot is u ′(t-1) includes the normalized x-axis coordinate x u ′(t-1), the normalized y-axis coordinate y u ′(t-1), and the normalized z-axis coordinate z u ′(t-1).

[0182] The normalized x-axis coordinate x of the drone at the t-1th time slot u ′(t-1) is calculated according to the following formula:

[0183] x u ′(t-1)=x u (t-1) / ξ X

[0184] The normalized y-axis coordinate y of the drone at the t-1th time slot u ′(t-1) is calculated according to the following formula:

[0185] y u ′(t-1)=y u (t-1) / ξ Y

[0186] The normalized z-axis coordinate z of the drone at the t-1th time slot u ′(t-1) is calculated according to the following formula:

[0187] z u ′(t-1)=z u (t-1) / ξ Z

[0188] (5) The server-mounted drone takes the normalized parameters W of the task to be processed for each terminal user in the tth time slot obtained in step (4) n ′(t) and the normalized coordinate parameter c n ′(t), and the normalized coordinate parameter c of the drone at the t-1th time slot u ′(t-1) inputs the pre-trained deep reinforcement learning model w to obtain the optimal horizontal movement angle ω(t) and optimal horizontal movement distance Δd of the drone at the tth time slot u (t), optimal vertical movement distance Δz u (t) and the optimal task offloading amount for each end user

[0189] (6) The optimal horizontal movement angle ω(t) and optimal horizontal movement distance Δd of the drone equipped with the server at the tth time slot obtained in step (5) u (t), optimal vertical movement distance Δz u (t) and the optimal task offloading amount for each end user Get the local processing delay of the nth terminal user in the tth time slot Task uplink delay Local processing energy consumption And the task uplink energy consumption And the remote processing latency of the drone carrying the server and remote processing power consumption

[0190] Specifically, step (6) includes the following sub-steps:

[0191] (6-1) The optimal horizontal movement angle ω(t) and optimal horizontal movement distance Δd of the drone at the tth time slot obtained according to step (5)u (t), optimal vertical movement distance Δz u (t), obtain the coordinate parameters c of the drone at the tth time slot u (t).

[0192] Specifically, the x-axis coordinate x of the drone carrying the server at the tth time slot is u (t) is calculated according to the following formula:

[0193] x u (t) = x u (t-1)+Δd u (t)cos(ω(t))

[0194] The y-axis coordinate y of the drone carrying the server at the tth time slot u (t) is calculated according to the following formula:

[0195] y u (t) = y u (t-1)+Δd u (t)sin(ω(t))

[0196] The z-axis coordinate z of the drone carrying the server at the tth time slot u (t) is calculated according to the following formula:

[0197] z u (t) = z u (t-1)+Δz u (t)

[0198] The advantage of this sub-step (6-1) is that it takes into account the three-dimensional mobility of the UAV, realizes the optimal movement of the UAV in the current time slot, and then completes the optimal flight path planning during the continuous unloading process.

[0199] (6-2) The coordinate parameter c of the drone carrying the server at the tth time slot obtained according to step (6-1) u (t) and the task offloading amount of each terminal user that the drone should execute in the tth time slot obtained according to step (5) Remotely offload a portion of the pending tasks of each terminal user in the tth time slot, and obtain the uplink delay of the task of the nth terminal user in the tth time slot Task uplink energy consumption And the remote processing latency of the drone carrying the server Remote processing power consumption

[0200] Specifically, the uplink delay of the task of the nth terminal user in the tth time slot is It is calculated according to the following formula:

[0201]

[0202] in is the wireless communication rate at which the nth terminal user sends its pending task to the UAV carrying the server in the tth time slot, which can be expressed by Shannon’s theorem as Among them B u is the wireless channel bandwidth of the UAV; P n is the transmission power of the nth terminal user; σ 2 is the additive white Gaussian noise power; P NLOS is the ambient noise power; is the channel gain of the nth terminal user in the tth time slot, which can be expressed by the free space path loss model as where g 0 is the channel gain when the reference distance is 1m; is the straight-line distance between the nth terminal user and the drone carrying the server at the tth time slot, which can be expressed by the Euclidean distance as

[0203] The uplink energy consumption of the task of the nth terminal user in the tth time slot It is calculated according to the following formula:

[0204]

[0205] Where P u is the received power of the drone carrying the server.

[0206] The remote processing delay of the task of the nth end user in the tth time slot on the drone equipped with the server It is calculated according to the following formula:

[0207]

[0208] in The frequency of the server CPU chip allocated to the nth end user by the server-carrying drone.

[0209] The energy consumption of the task of the nth end user in the tth time slot is processed remotely by the drone equipped with the server. It is calculated according to the following formula:

[0210]

[0211] (6-3) The task offloading amount of each terminal user that the drone should execute in the tth time slot obtained in step (5) The remaining part of each terminal user's pending task in the tth time slot is processed locally, and the local processing delay of the nth terminal user in the tth time slot is obtained. and local processing energy consumption

[0212] Specifically, the local processing delay of the nth end user in the tth time slot is It is calculated according to the following formula:

[0213]

[0214] The local processing energy consumption of the nth end user in the tth time slot It is calculated according to the following formula:

[0215]

[0216] The advantage of this step is that by analyzing the state of the edge computing system, the terminal user's pending tasks in the current time slot are divided and calculated locally and offloaded to the drone for calculation, thereby achieving efficient utilization of the computing resources of the edge computing system.

[0217] (7) The local processing delay of each end user in the tth time slot obtained by the server-carrying drone according to step (6) Task uplink delay Local processing energy consumption Task uplink energy consumption And the remote processing latency of the drone carrying the server Remote processing power consumption Obtain the task response delay T(t) and task energy consumption E(t) of the edge computing system in the tth time slot (by this time, the edge computing system has completed the unloading process of the tth time slot), set t=t+1, and return to step (2);

[0218] Specifically, the task response delay T(t) of the edge computing system in the tth time slot is calculated according to the following formula:

[0219]

[0220] The task energy consumption E(t) of the edge computing system in the tth time slot is calculated according to the following formula:

[0221]

[0222] The advantage of the above steps (1) to (7) is that by discretizing the time dimension of the edge computing system, the function of continuous unloading of the drone-assisted edge computing system is established, and the continuous process of unloading tasks of each user device in each time slot is realized.

[0223] (8) The server-carrying drone obtains the full local processing delay of each end user in the tth time slot according to step (3) Full local processing energy consumption And the task response delay T(t) and task energy consumption E(t) of the edge computing system in each time slot obtained in step (7) are used to obtain the system utility U of the edge computing system.

[0224] Step (8) includes the following sub-steps:

[0225] (8-1) The server-carrying drone obtains the full local processing delay of each end user in the tth time slot according to step (3) Full local processing energy consumption Get the total local processing latency of the edge computing system and total local processing energy consumption

[0226] Specifically, the total local processing latency of the edge computing system It is calculated according to the following formula:

[0227] Total local processing energy consumption of edge computing systems It is calculated according to the following formula:

[0228]

[0229] (8-2) The drone equipped with the server obtains the task response delay T(t) and task energy consumption E(t) of the edge computing system in each time slot according to step (7), and obtains the total task response delay T of the edge computing system system and the total energy consumption of the task E system .

[0230] Specifically, the total task response delay T of the edge computing system is system It is calculated according to the following formula:

[0231]

[0232] The total energy consumption of edge computing system tasks E system It is calculated according to the following formula:

[0233]

[0234] (8-3) The total local processing delay of the edge computing system obtained according to step (8-1) and total local processing energy consumption And the total task response delay T of the edge computing system obtained according to step (8-2) system and the total energy consumption of the task E system , get the system utility U of the edge computing system:

[0235]

[0236] The advantage of this step is that by counting the delay and energy consumption consumed during the continuous unloading process, the effectiveness of the continuous unloading process of the edge computing system can be evaluated.

[0237] (9) The drone equipped with the server feeds back the system utility U of the edge computing system obtained in step (8) as a service result to each end user, and the process ends.

[0238] Specifically, the deep reinforcement learning model w in the present invention adopts a deep reinforcement learning model of the TensorFlow architecture, which is trained according to the following steps:

[0239] (A1) A user trajectory and task file dataset is obtained, which is composed of multiple user trajectory and task files (2600 in this example, each user trajectory and task file records the parameters and coordinate parameters of the tasks to be processed of 5 users in 20 time slots), and the user trajectory and task file dataset is randomly divided into a training set and a test set in a ratio of 100:2500.

[0240] (A2) Use the training set obtained in step (A1) to train the deep reinforcement learning model w to obtain a preliminarily trained deep reinforcement learning model.

[0241] This step includes the following sub-steps:

[0242] (A2-1) Set iteration counter i=0;

[0243] (A2-2) Determine whether i is equal to the preset model iteration number I, if yes, proceed to step (A2-15), otherwise proceed to step (A2-3);

[0244] Specifically, the preset number of model iterations I in the present invention has a value range of (1, +∞), preferably 1000.

[0245] (A2-3) Randomly select a user trajectory and task file from the training set obtained in step (A1), and randomly initialize the initial (when time slot k = 0) coordinate parameters c of the drone carrying the server u(0), and set the iteration counter k = 1;

[0246] The advantage of this sub-step (A2-3) is that it takes into account the flexible deployment capability of the UAV and achieves high robustness of the edge computing system to random deployment locations.

[0247] (A2-4) Determine whether k is greater than the preset model simulation time slot number K, if so, proceed to step (A2-12), otherwise proceed to step (A2-5);

[0248] Specifically, the value range of the preset model simulation time slot number K in the present invention is (1, +∞), preferably 20.

[0249] (A2-5) obtaining the coordinate parameters of the drone carrying the server in the k-1th time slot, and extracting the parameters and coordinate parameters of the task to be processed of each user in the kth time slot from the user trajectory and task file obtained in step (A2-3);

[0250] (A2-6) Obtain the full local processing delay of each user in the kth time slot according to the parameters and coordinate parameters of the task to be processed of each user in the kth time slot obtained in step (A2-5) and full local processing energy consumption

[0251] Specifically, the acquisition process of this step has been disclosed in step (3) above and will not be repeated here.

[0252] (A2-7) normalizing the coordinate parameters of the drone carrying the server at the k-1th time slot, and the parameters and coordinate parameters of the task to be processed of each user at the kth time slot obtained in step (A2-5), so as to obtain the normalized coordinate parameters of the drone carrying the server at the k-1th time slot, and the normalized parameters and coordinate parameters of the task to be processed of each user at the kth time slot;

[0253] The advantage of this step is that it unifies parameters with different value ranges and dimensions, enhances the correlation between different parameters, and improves the training speed of the deep reinforcement learning model.

[0254] (A2-8) inputting the normalized coordinate parameters of the drone carrying the server at the k-1th time slot obtained in step (A2-7), and the normalized parameters of the task to be processed and the normalized coordinate parameters of each user at the kth time slot into the deep reinforcement learning model w, so as to obtain the action a(k) of the drone at the kth time slot through the ε-greedy strategy, where ε is the greediness value;

[0255] The deep reinforcement learning model w adopts the ε-greedy strategy for action selection, that is, there is a probability of ε to take random actions, and a probability of 1-ε to take the network output value of the deep reinforcement learning model w and the action noise The value range of ε is [0,1], preferably 0.5. is Gaussian noise with mean equal to 0 and standard deviation equal to 0.05.

[0256] The advantage of this step is that the ε-greedy strategy is introduced to enhance the exploration ability of the deep reinforcement learning model w, and improve the model's ability to solve complex non-convex optimization problems that integrate dynamic environments and continuous action spaces.

[0257] (A2-9) according to the action a(k) of the drone in the kth time slot obtained in step (A2-8), obtain the task response delay T(k) and task energy consumption E(k) of the edge computing system in the kth time slot, as well as the coordinate parameters of the drone in the kth time slot;

[0258] Specifically, the acquisition process of this step has been disclosed in the above steps (6) and (7) and will not be repeated here.

[0259] (A2-10) updating the greediness value ε and the deep reinforcement learning model w in step (A2-8) to obtain updated greediness value ε and deep reinforcement learning model w respectively;

[0260] The greedy value ε is updated according to the following formula:

[0261]

[0262] The deep reinforcement learning model w adopts the reward function R, and has:

[0263] R(k)=α 1 T(k)+α 2 E(k)

[0264] Among them, α 1 and α 2 The value range of is [0,1], and the preferred value is 0.5;

[0265] The deep reinforcement learning model w adopts a soft update method and has:

[0266] θ w′ ←τθ w +(1-τ)θ w′

[0267] Wherein, τ is the update rate, and its value range is [0,1], preferably 0.05; θ w ,θ w′They are the network parameters before and after the update of the deep reinforcement learning model w.

[0268] The advantage of this step is that the greedy value ε is updated to ensure the balance of the deep reinforcement learning model w from the exploration tendency in the early stage of training to the utilization tendency in the later stage of training; the reward function R is used to provide the model with the correct training direction to ensure the simultaneous optimization of the dual objectives of task response delay and task energy consumption; and the stability of model training is ensured by soft update.

[0269] (A2-11) Set the iteration counter k=k+1, and return to step (A2-4);

[0270] (A2-12) The total local processing delay of each user in the kth time slot obtained according to step (A2-6) and full local processing energy consumption And the task response delay T(k) and task energy consumption E(k) of the edge computing system in the kth time slot obtained in step (A2-9) to obtain the system utility U of the edge computing system;

[0271] Specifically, the acquisition process of this step has been disclosed in step (8) above and will not be repeated here.

[0272] (A2-13) According to the system utility U of the edge computing system obtained in step (A2-12), the optimization target P1 of the edge computing system is obtained:

[0273] (P1:)maxU

[0274] stC1:Δd u (k)≤ΔD max

[0275] C2:Δz u (k)≤ΔZ max

[0276] The constraint C1 indicates that the horizontal movement distance of the drone cannot exceed the maximum horizontal movement distance ΔD max ; Constraint C2 means that the vertical movement distance of the drone cannot exceed the maximum vertical movement distance ΔZ max ;

[0277] (A2-14) Set the iteration counter i=i+1, and return to step (A2-2);

[0278] (A2-15) The model iteration is completed, and a preliminarily trained deep reinforcement learning model is obtained;

[0279] (A3) Use the test set obtained in step (A1) to verify the deep reinforcement learning model w preliminarily trained in step (A2) until the optimization target P1 of the obtained edge computing system reaches the optimal value, thereby obtaining the final trained deep reinforcement learning model w.

[0280] The three performance comparison results of the method of the present invention compared with other methods and the three performance comparison results under different time slots are as follows Figure 3 and Figure 4 shown. Figure 3 (a) with Figure 4 (a) is the average response delay performance indicator, Figure 3 (b) with Figure 4 (b) is the average system energy consumption performance index, and Figure 3 (c) with Figure 4 (c) is the average system utility performance index. ODEGD is the method of the present invention, DDPG, TD3, AC, DDQN, and DQN are traditional deep reinforcement learning methods, and All_edge is a method for remote calculation of all tasks. Figure 3 and Figure 4 It can be fully demonstrated that the edge computing offloading strategy of the present invention has the above significant advantages.

[0281] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0282] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for unloading UAV-assisted edge computing tasks based on deep reinforcement learning, characterized in that: The following steps are involved: (1) Set the time slot number t = 1; (2) Determine whether t is greater than the total number of time slots T in the edge computing system. If so, proceed to step (8); otherwise, proceed to step (3); (3) The server-equipped drone obtains the parameter W of the task to be processed by the nth terminal user in the tth time slot from the nth terminal user in the edge computing system. n (t) and coordinate parameter c n (t), obtain the coordinate parameter c of the drone in the t-1th time slot from itself u (t-1), and according to the parameters W of the task to be processed n (t) and coordinate parameter c n (t) Obtain the full local processing delay of the nth end user in the tth time slot Full local processing energy consumption Where n represents the serial number of the terminal user with pending tasks in the edge computing system, and n∈[1, the total number of terminal users with pending tasks in the edge computing system N]; the parameter W of the pending task of the nth terminal user in the tth time slot is n (t) includes the task size D of the tasks to be processed n (t), and the number of CPU cycles C required to process 1 bit of the tasks to be processed n (t); coordinate parameter c of the nth terminal user in the tth time slot n (t) includes the x-axis coordinate x n (t), y-axis coordinate y n (t), and the z-axis coordinate z n (t); (4) The server-mounted drone obtains the parameter W of the task to be processed for each terminal user in the tth time slot according to step (3). n (t) and coordinate parameter c n (t), and the coordinate parameter c of the drone at the t-1th time slot u (t-1), obtain the normalized parameters W of the pending tasks for each terminal user in the tth time slot n ′(t) and the normalized coordinate parameter c n ′(t), and the normalized coordinate parameter c of the drone at the t-1th time slot u ′(t-1); (5) The server-mounted drone takes the normalized parameters W of the task to be processed for each terminal user in the tth time slot obtained in step (4) n ′(t) and the normalized coordinate parameter c n ′(t), and the normalized coordinate parameter c of the drone at the t-1th time slot u ′(t-1) inputs the pre-trained deep reinforcement learning model w to obtain the optimal horizontal movement angle ω(t) and optimal horizontal movement distance Δd of the drone at the tth time slot u (t), optimal vertical movement distance Δz u (t) and the optimal task offloading amount for each end user (6) The optimal horizontal movement angle ω(t) and optimal horizontal movement distance Δd of the drone equipped with the server at the tth time slot obtained in step (5) u (t), optimal vertical movement distance Δz u (t) and the optimal task offloading amount for each end user Get the local processing delay of the nth terminal user in the tth time slot Task uplink delay Local processing energy consumption And the task uplink energy consumption And the remote processing latency of the drone carrying the server and remote processing power consumption Full local processing delay of the nth end user in the tth time slot It is calculated according to the following formula: where F N The frequency of the local CPU chip used by the nth end user; The total local processing energy consumption of the nth end user in the tth time slot It is calculated according to the following formula: Where κ is the capacitance value of the effective switch capacitor of the local CPU chip used by the nth terminal user; (7) The local processing delay of each end user in the tth time slot obtained by the server-carrying drone according to step (6) Task uplink delay Local processing energy consumption Task uplink energy consumption And the remote processing latency of the drone carrying the server Remote processing power consumption Obtain the task response delay T(t) and task energy consumption E(t) of the edge computing system in the tth time slot, set t=t+1, and return to step (2); (8) The server-carrying drone obtains the full local processing delay of each end user in the tth time slot according to step (3) Full local processing energy consumption And the task response delay T(t) and task energy consumption E(t) of the edge computing system in each time slot obtained in step (7) are used to obtain the system utility U of the edge computing system; (9) The drone equipped with the server feeds back the system utility U of the edge computing system obtained in step (8) as a service result to each end user, and the process ends.

2. The method for unloading UAV-assisted edge computing tasks based on deep reinforcement learning according to claim 1 is characterized in that: Step (4) includes the following sub-steps: (4-1) The parameter W of the task to be processed by each terminal user in the tth time slot obtained according to step (3) n (t) Obtain the normalized task parameter W to be processed for each terminal user in the tth time slot n ′(t); (4-2) The coordinate parameter c of each terminal user in the tth time slot obtained according to step (3) n (t), obtain the normalized coordinate parameter c of each terminal user in the tth time slot n ′(t); (4-3) According to the coordinate parameter c of the drone in the t-1th time slot u (t-1), obtain the normalized coordinate parameter c of the drone at the t-1th time slot u ′(t-1).

3. The method for unloading UAV-assisted edge computing tasks based on deep reinforcement learning according to claim 2 is characterized in that: The normalized parameter W of the task to be processed for each terminal user in the tth time slot n ′(t) includes the normalized task size D of the task to be processed n ′(t), and the normalized number of CPU cycles required to process 1 bit of the task to be processed C n ′(t); The normalized task size D of each terminal user to be processed in the tth time slot n ′(t) is calculated according to the following formula: D n ′(t)=D n (t) / ξ D where ξ D is a normalization factor, the value range of which is determined by the task size of the task to be processed; The number of CPU cycles C required for each terminal user to process 1 bit of the tasks to be processed in the tth time slot after normalization n ′(t) is calculated according to the following formula: C n ′(t)=C n (t) / ξ C where ξ C is the normalization factor; The normalized coordinate parameter c of each terminal user in the tth time slot is n ′(t) includes the normalized x-axis coordinate x n ′(t), normalized y-axis coordinate y n ′(t), and the normalized z-axis coordinate z n ′(t); The normalized x-axis coordinate x of each terminal user in the tth time slot is n ′(t) is calculated according to the following formula: x n ′(t)=x n (t) / ξ X where ξ X is the normalization factor, and the value range of the normalization factor is determined by the size of the x-axis coordinate; The normalized y-axis coordinate y of each terminal user in the tth time slot is n ′(t) is calculated according to the following formula: y n ′(t)=y n (t) / ξ Y where ξ Y is a normalization factor, the value range of which is determined by the size of the y-axis coordinate; The normalized z-axis coordinate z of each end user in the tth time slot is n ′(t) is calculated according to the following formula: z n ′(t)=z n (t) / ξ Z where ξ Z is the normalization factor; The normalized coordinate parameter c of the drone at the t-1th time slot u ′(t-1) includes the normalized x-axis coordinate x u ′(t-1), the normalized y-axis coordinate y u ′(t-1), and the normalized z-axis coordinate z u ′(t-1); The normalized x-axis coordinate x of the drone at the t-1th time slot u ′(t-1) is calculated according to the following formula: x u ′(t-1)=x u (t-1) / ξ X The normalized y-axis coordinate y of the drone at the t-1th time slot u ′(t-1) is calculated according to the following formula: y u ′(t-1)=y u (t-1) / ξ Y The normalized z-axis coordinate z of the drone at the t-1th time slot u ′(t-1) is calculated according to the following formula: z u ′(t-1)=z u (t-1) / ξ Z 。 4. The method for unloading UAV-assisted edge computing tasks based on deep reinforcement learning according to claim 3 is characterized in that: Step (6) includes the following sub-steps: (6-1) The optimal horizontal movement angle ω(t) and optimal horizontal movement distance Δd of the drone at the tth time slot obtained according to step (5) u (t), optimal vertical movement distance Δz u (t), obtain the coordinate parameters c of the drone at the tth time slot u (t); (6-2) The coordinate parameter c of the drone carrying the server at the tth time slot obtained according to step (6-1) u (t) and the task offloading amount of each terminal user of the drone in the tth time slot obtained according to step (5) Remotely offload a portion of the pending tasks of each terminal user in the tth time slot, and obtain the uplink delay of the task of the nth terminal user in the tth time slot Task uplink energy consumption And the remote processing latency of the drone carrying the server Remote processing power consumption (6-3) The task offloading amount of each terminal user of the drone in the tth time slot obtained according to step (5) The remaining part of each terminal user's pending task in the tth time slot is processed locally, and the local processing delay of the nth terminal user in the tth time slot is obtained. and local processing energy consumption 5. The method for unloading UAV-assisted edge computing tasks based on deep reinforcement learning according to claim 4 is characterized in that: The x-axis coordinate x of the drone carrying the server at the tth time slot u (t) is calculated according to the following formula: x u (t)=x u (t-1)+Δd u (t)cos(ω(t)) The y-axis coordinate y of the drone carrying the server at the tth time slot u (t) is calculated according to the following formula: y u (t)=y u (t-1)+Δd u (t)sin(ω(t)) The z-axis coordinate z of the drone carrying the server at the tth time slot u (t) is calculated according to the following formula: z u (t)=z u (t-1)+Δz u (t) The uplink delay of the task of the nth terminal user in the tth time slot It is calculated according to the following formula: in is the wireless communication rate at which the nth terminal user sends its pending task to the UAV carrying the server in the tth time slot, which can be expressed by Shannon’s theorem as Among them B u is the wireless channel bandwidth of the UAV; P n is the transmission power of the nth terminal user; σ 2 is the additive white Gaussian noise power; P NLOS is the ambient noise power; is the channel gain of the nth terminal user in the tth time slot, which can be expressed by the free space path loss model as Where g0 is the channel gain when the reference distance is 1m; is the straight-line distance between the nth terminal user and the drone carrying the server at the tth time slot, which can be expressed by the Euclidean distance as The uplink energy consumption of the task of the nth terminal user in the tth time slot It is calculated according to the following formula: Where P u is the received power of the drone carrying the server; The remote processing delay of the task of the nth end user in the tth time slot on the drone equipped with the server It is calculated according to the following formula: in The frequency of the server CPU chip allocated to the nth end user by the drone carrying the server; The energy consumption of the task of the nth end user in the tth time slot is processed remotely by the drone equipped with the server. It is calculated according to the following formula: Local processing delay of the nth end user in the tth time slot It is calculated according to the following formula: The local processing energy consumption of the nth end user in the tth time slot It is calculated according to the following formula:

6. The method for unloading UAV-assisted edge computing tasks based on deep reinforcement learning according to claim 5 is characterized in that: The task response delay T(t) of the edge computing system in the tth time slot is calculated according to the following formula: The task energy consumption E(t) of the edge computing system in the tth time slot is calculated according to the following formula:

7. The method for unloading UAV-assisted edge computing tasks based on deep reinforcement learning according to claim 6 is characterized in that: Step (8) includes the following sub-steps: (8-1) The server-carrying drone obtains the full local processing delay of each end user in the tth time slot according to step (3) Full local processing energy consumption Get the total local processing latency of the edge computing system and total local processing energy consumption Specifically, the total local processing latency of the edge computing system It is calculated according to the following formula: Total local processing energy consumption of edge computing systems It is calculated according to the following formula: (8-2) The drone equipped with the server obtains the task response delay T(t) and task energy consumption E(t) of the edge computing system in each time slot according to step (7), and obtains the total task response delay T of the edge computing system system and the total energy consumption of the task E system ; Specifically, the total task response delay T of the edge computing system is system It is calculated according to the following formula: The total energy consumption of edge computing system tasks E system It is calculated according to the following formula: (8-3) The total local processing delay of the edge computing system obtained according to step (8-1) and total local processing energy consumption And the total task response delay T of the edge computing system obtained according to step (8-2) system and the total energy consumption of the task E system , get the system utility U of the edge computing system:

8. The method for unloading UAV-assisted edge computing tasks based on deep reinforcement learning according to claim 7 is characterized in that: The deep reinforcement learning model w uses a deep reinforcement learning model based on the TensorFlow architecture, which is trained according to the following steps: (A1) obtaining a user trajectory and task file dataset consisting of a plurality of user trajectory and task files, and randomly dividing the user trajectory and task file dataset into a training set and a test set in a ratio of 100:2500; (A2) using the training set obtained in step (A1) to train the deep reinforcement learning model w to obtain a preliminarily trained deep reinforcement learning model; (A3) using the test set obtained in step (A1) to verify the deep reinforcement learning model w preliminarily trained in step (A2) until the optimization target P1 of the obtained edge computing system reaches the optimal value, thereby obtaining the finally trained deep reinforcement learning model w; Wherein step (A2) comprises the following sub-steps: (A2-1) Set iteration counter i=0; (A2-2) Determine whether i is equal to the preset model iteration number I, if yes, proceed to step (A2-15), otherwise proceed to step (A2-3); (A2-3) Randomly select a user trajectory and task file from the training set obtained in step (A1), and randomly initialize the initial coordinate parameters c of the drone carrying the server u (0), and set the iteration counter k = 1; (A2-4) Determine whether k is greater than the preset model simulation time slot number K, if so, proceed to step (A2-12), otherwise proceed to step (A2-5); (A2-5) obtaining the coordinate parameters of the drone carrying the server in the k-1th time slot, and extracting the parameters and coordinate parameters of the task to be processed of each user in the kth time slot from the user trajectory and task file obtained in step (A2-3); (A2-6) Obtain the full local processing delay of each user in the kth time slot according to the parameters and coordinate parameters of the task to be processed of each user in the kth time slot obtained in step (A2-5) and full local processing energy consumption (A2-7) normalizing the coordinate parameters of the drone carrying the server at the k-1th time slot, and the parameters and coordinate parameters of the task to be processed of each user at the kth time slot obtained in step (A2-5), so as to obtain the normalized coordinate parameters of the drone carrying the server at the k-1th time slot, and the normalized parameters and coordinate parameters of the task to be processed of each user at the kth time slot; (A2-8) inputting the normalized coordinate parameters of the drone carrying the server at the k-1th time slot obtained in step (A2-7), and the normalized parameters of the task to be processed and the normalized coordinate parameters of each user at the kth time slot into the deep reinforcement learning model w, so as to obtain the action a(k) of the drone at the kth time slot through the ε-greedy strategy, where ε is the greediness value; The deep reinforcement learning model w adopts the ε-greedy strategy for action selection, that is, there is a probability of ε to take random actions, and a probability of 1-ε to take the network output value of the deep reinforcement learning model w and the action noise The sum of ε is in the range of [0,1]. is Gaussian noise with mean equal to 0 and standard deviation equal to 0.05; (A2-9) according to the action a(k) of the drone in the kth time slot obtained in step (A2-8), obtain the task response delay T(k) and task energy consumption E(k) of the edge computing system in the kth time slot, as well as the coordinate parameters of the drone in the kth time slot; (A2-10) updating the greediness value ε and the deep reinforcement learning model w in step (A2-8) to obtain updated greediness value ε and deep reinforcement learning model w respectively; The greedy value ε is updated according to the following formula: The deep reinforcement learning model w adopts the reward function R, and has: R(k)=α1T(k)+α2E(k) Among them, the value range of α1 and α2 is [0,1]; The deep reinforcement learning model w adopts a soft update method and has: i w' ←tth w +(1-τ)θ w' Among them, τ is the update rate, and its value range is [0,1]; θ w ,θ w′ They are the network parameters before and after the update of the deep reinforcement learning model w; (A2-11) Set the iteration counter k=k+1, and return to step (A2-4); (A2-12) The total local processing delay of each user in the kth time slot obtained according to step (A2-6) and full local processing energy consumption And the task response delay T(k) and task energy consumption E(k) of the edge computing system in the kth time slot obtained in step (A2-9) to obtain the system utility U of the edge computing system; (A2-13) According to the system utility U of the edge computing system obtained in step (A2-12), the optimization target P1 of the edge computing system is obtained; P1:max U s.t.C1:Δd u (k)≤ΔD max C2:|Δz u (k)|≤ΔZ max The constraint C1 indicates that the horizontal movement distance of the drone cannot exceed the maximum horizontal movement distance ΔD max ; Constraint C2 means that the vertical movement distance of the drone cannot exceed the maximum vertical movement distance ΔZ max ; (A2-14) Set the iteration counter i=i+1, and return to step (A2-2); (A2-15) The model iteration is completed, and a preliminarily trained deep reinforcement learning model is obtained.

9. A UAV-assisted edge computing task offloading system based on deep reinforcement learning, characterized in that: include: The first module is used to set the time slot number t=1; The second module is used to determine whether t is greater than the total number T of time slots in the edge computing system, and if so, enter the eighth module, otherwise enter the third module; The third module is set in the drone equipped with the server, and is used to obtain the parameter W of the task to be processed by the nth terminal user in the tth time slot from the nth terminal user in the edge computing system. n (t) and coordinate parameter c n (t), obtain the coordinate parameter c of the drone in the t-1th time slot from itself u (t-1), and according to the parameters W of the task to be processed n (t) and coordinate parameter c n (t) Obtain the full local processing delay of the nth end user in the tth time slot Full local processing energy consumption Where n represents the serial number of the terminal user with pending tasks in the edge computing system, and n∈[1, the total number of terminal users with pending tasks in the edge computing system N]; the parameter W of the pending task of the nth terminal user in the tth time slot is n (t) includes the task size D of the tasks to be processed n (t), and the number of CPU cycles C required to process 1 bit of the tasks to be processed n (t); coordinate parameter c of the nth terminal user in the tth time slot n (t) includes the x-axis coordinate x n (t), y-axis coordinate y n (t), and the z-axis coordinate z n (t); The fourth module is set in the drone equipped with the server, and is used to calculate the parameter W of the task to be processed for each terminal user in the tth time slot according to the third module. n (t) and coordinate parameter c n (t), and the coordinate parameter c of the drone at the t-1th time slot u (t-1), obtain the normalized parameters W of the pending tasks for each terminal user in the tth time slot n ′(t) and the normalized coordinate parameter c n ′(t), and the normalized coordinate parameter c of the drone at the t-1th time slot u ′(t-1); The fifth module is set in the drone equipped with the server, and is used to convert the normalized parameters W of the task to be processed of each terminal user in the tth time slot obtained in the fourth module into n ′(t) and the normalized coordinate parameter c n ′(t), and the normalized coordinate parameter c of the drone at the t-1th time slot u ′(t-1) inputs the pre-trained deep reinforcement learning model w to obtain the optimal horizontal movement angle ω(t) and optimal horizontal movement distance Δd of the drone at the tth time slot u (t), optimal vertical movement distance Δz u (t) and the optimal task offloading amount for each end user The sixth module is set in the drone equipped with the server, and is used to calculate the optimal horizontal movement angle ω(t) and the optimal horizontal movement distance Δd of the drone at the tth time slot obtained in the fifth module. u (t), optimal vertical movement distance Δz u (t) and the optimal task offloading amount for each end user Get the local processing delay of the nth terminal user in the tth time slot Task uplink delay Local processing energy consumption And the task uplink energy consumption And the remote processing latency of the drone carrying the server and remote processing power consumption Full local processing delay of the nth end user in the tth time slot It is calculated according to the following formula: where F N The frequency of the local CPU chip used by the nth end user; The total local processing energy consumption of the nth end user in the tth time slot It is calculated according to the following formula: Where κ is the capacitance value of the effective switch capacitor of the local CPU chip used by the nth terminal user; The seventh module is set in the drone equipped with the server, and is used to calculate the local processing delay of each terminal user in the tth time slot according to the sixth module. Task uplink delay Local processing energy consumption Task uplink energy consumption And the remote processing latency of the drone carrying the server Remote processing power consumption Obtain the task response delay T(t) and task energy consumption E(t) of the edge computing system in the tth time slot, set t=t+1, and return to the second module; The eighth module is set in the drone equipped with the server, and is used to calculate the full local processing delay of each terminal user in the tth time slot according to the third module. Full local processing energy consumption And the task response delay T(t) and task energy consumption E(t) of the edge computing system in each time slot obtained in the seventh module, and the system utility U of the edge computing system is obtained; The ninth module is set on the drone equipped with the server and is used to feed back the system utility U of the edge computing system obtained by the eighth module as a service result to each end user, and the process ends.

Images