A method and device for unloading tasks of drone-assisted mobile edge computing

Abstract

The present application discloses a method and device for unloading tasks of drone-assisted mobile edge computing, the method comprising: constructing a communication model, transmitting its computing tasks to a drone, obtaining the energy consumption value and data transmission rate between the drone and the terminal device when the drone approaches a terminal device, and calculating the transmission time and transmission energy consumption according to the energy consumption value and the data transmission rate; constructing an unloading calculation model, performing unloading calculation and local calculation, obtaining the waiting time of the terminal device in the virtual queue, determining the percentage of tasks to be unloaded and controlled at the next location of the drone, and unloading the first task to the drone according to the task percentage. This method aims at the continuous problem in uncertainty, proposes an optimized drone trajectory design and user unloading task ratio scheduling, improves the ability of drone-assisted MEC to unload tasks, and maximizes system stability.

Description

Technical Field

[0001] The present application relates to the field of drone technology, and more particularly to a method and device for unloading tasks of drone-assisted mobile edge computing. Background Art

[0002] The increasing popularity of the Internet of Things (IoT) provides a promising platform for complex mobile applications such as autonomous driving, augmented reality, and various cognitive applications. With the rapid development of various mobile applications, it leads to network traffic congestion and long delays. The demand for wireless communication has reached an unprecedented level. At the same time, most mobile applications are designed with latency, computational intensiveness, and energy intensiveness in mind. Therefore, designers expect to provide a low-latency and energy-controllable computing mechanism to achieve the purpose of callability. In order to improve computing speed, alleviate communication congestion, and save energy consumption for mobile users, Mobile Edge Computing (MEC) is seen as a potential method to enable mobile users to offload intensive computing to nearby mobile edge servers, thereby providing high-quality services to mobile users.

[0003] Since MEC not only improves computing power but also improves energy saving of mobile devices, it has been studied intensively in recent years as a key technology of 5G. The definition, computing and communication modeling of MEC, its advantages and applications are discussed. For example, some task offloading problems are treated as mixed integer nonlinear programs to reduce computing delays and save battery life. Or the computational offloading mechanism of MC in 5G networks is explored to minimize energy consumption.

[0004] Despite extensive research on MEC, it cannot avoid the static MEC limitation because it cannot guarantee that MEC can be deployed anytime and anywhere. There is a high probability that the infrastructure will be destroyed sometimes, such as after a natural disaster. In addition, it is not practical to install infrastructure for temporary purposes or in rural areas such as hotspots and mountainous areas. In the above cases, the installed MEC devices cannot be well utilized. Due to the flexibility of Unmanned Aerial Vehicle (UAV), drone-assisted MEC is introduced, in which drones can carry MEC servers as computing servers for mobile users in flexible locations. Due to limited payload and energy, the computing power and flight time of drones are limited. Therefore, in drone-assisted MEC systems, minimizing task completion time and energy consumption are two basic optimization goals. Summary of the invention

[0005] This application provides a task offloading method, which improves the efficiency of task offloading through drone-assisted mobile edge computing, thereby saving time and energy. In addition, this application also provides a drone-assisted MEC system, which enables multiple mobile users to partially offload their computing tasks to drones, jointly optimize the drone's ratio and trajectory, minimize the time and energy consumption of the entire system, and maximize the stability of the system.

[0006] Since each computing task has a maximum tolerance time, if the computing time (including waiting time) is longer than the tolerance, the task is considered expired, which means that MEC cannot respond to the user. If the system cannot complete most tasks before the expiration, the system is considered unstable. Therefore, it is necessary to maximize stability and balance the calculation of system workload, extend the system running time, and maximize the number of completed tasks. The specific technical solution is as follows:

[0007] In a first aspect, the present application provides a method for unloading tasks of drone-assisted mobile edge computing, the method comprising:

[0008] A communication model is constructed, and its computing task is transmitted to the UAV, and time is divided into N time slots, N≥1, and each time slot corresponds to at least one task; the flight trajectory of the UAV at the first height is indicated according to the discrete position of the UAV in each time slot; when the UAV approaches the terminal device, the energy consumption value and the data transmission rate between the UAV and the terminal device are obtained, and the transmission time and the transmission energy consumption are calculated according to the energy consumption value and the data transmission rate;

[0009] Constructing an offload calculation model, using the offload calculation model to perform offload calculation and local calculation of the UAV and the terminal device, obtaining the waiting time of the terminal device in the virtual queue, and calculating the total time cost according to the waiting time, wherein the total time cost is equal to the maximum value of the transmission time summary, the offload time, and the local calculation time;

[0010] Calculate the total energy consumption of the UAV auxiliary system according to the total time cost, wherein the total energy consumption includes: signal energy consumption, offloading computing energy consumption, local computing energy consumption and UAV propulsion energy consumption;

[0011] The percentage of the task of offloading and controlling the drone at the next position is determined according to the total energy consumption of the drone auxiliary system, and the first task is offloaded to the drone according to the task percentage.

[0012] Optionally, in a possible implementation manner, the method further includes: establishing a local computing model, and determining the local computing time and local computing energy consumption of the terminal device according to the local computing model;

[0013] The calculating the total time cost according to the waiting time includes: determining the total time cost according to the waiting time and the local calculation time.

[0014] Optionally, in another possible implementation, the percentage of tasks of the drone at the next location is the percentage of tasks unloaded by the drone according to time.

[0015] In a second aspect, the present application provides another task offloading method, which is applied to a task control system, and the method includes:

[0016] Sending a configuration file of the offloading task to a task control system, which provides an environment for the offloading task;

[0017] When the task control system receives the configuration file, obtaining a current state of the mobile edge computing network;

[0018] The agent module takes corresponding action measures according to the observation information in the environment, and the agent module is located on the drone;

[0019] The task control system feeds back the next state of the mobile edge computing network to the agent module, wherein the feedback includes an evaluation of the action measures; wherein the agent module includes a local network, and the local network generates an action to provide an environment for the offloading task;

[0020] Execute the action, determine the proportion of the task and offload it to the drone, and drive the drone to the target location.

[0021] In a third aspect, the present application also provides a task offloading device for drone-assisted mobile edge computing, the device comprising:

[0022] A building block for building a communication model;

[0023] The processing unit is used to transmit its computing task to the drone, divide the time into N time slots, N≥1, and each time slot corresponds to at least one task; indicate the flight trajectory of the drone at a first altitude according to the discrete position of the drone in each time slot; when the drone approaches the terminal device, obtain the energy consumption value and the data transmission rate between the drone and the terminal device, and calculate the transmission time and transmission energy consumption according to the energy consumption value and the data transmission rate;

[0024] The construction unit is also used to construct an unloading calculation model.

[0025] The processing unit is further used to perform offload calculation and local calculation of the drone and the terminal device using the offload calculation model, obtain the waiting time of the terminal device in the virtual queue, and calculate the total time cost according to the waiting time, wherein the total time cost is equal to the maximum value of the transmission time summary, the offload time, and the local calculation time;

[0026] A calculation unit, used to calculate the total energy consumption of the UAV auxiliary system according to the total time cost, wherein the total energy consumption includes: signal energy consumption, unloading calculation energy consumption, local calculation energy consumption and UAV propulsion energy consumption;

[0027] The processing unit is further used to determine the percentage of the task of unloading and controlling the drone at the next position according to the total energy consumption of the drone auxiliary system, and unload the first task to the drone according to the task percentage.

[0028] Optionally, in a possible implementation, the construction unit is further used to establish a local computing model, and determine the local computing time and local computing energy consumption of the terminal device according to the local computing model;

[0029] The processing unit is further configured to determine the total time cost according to the waiting time and the local calculation time.

[0030] Optionally, in another possible implementation, the percentage of tasks of the drone at the next location is the percentage of tasks unloaded by the drone according to time.

[0031] In a fourth aspect, the present application further provides a task offloading device, which is applied to a task control system, and the device includes:

[0032] An offloading unit, used for sending a configuration file of an offloading task to a task control system, which provides an environment for the offloading task;

[0033] an acquisition unit, configured to acquire a current state of a mobile edge computing network when the task control system receives the configuration file;

[0034] The acquisition unit is further used to acquire the corresponding action measures taken by the agent module according to the observation information in the environment, and the agent module is located on the drone;

[0035] A processing unit, configured to feed back the next state of the mobile edge computing network to the agent module, wherein the feedback includes an evaluation of the action measures; execute the action, determine the proportion of tasks and offload them to the drone, and drive the drone to a target location;

[0036] The proxy module includes a local network, and the local network generates an action to provide an environment for the offloading task.

[0037] In a fifth aspect, the present application provides a communication device, comprising a processor and a memory, wherein the processor and the memory are coupled, and the memory is used to store instructions; the processor is used to execute the instructions in the memory, so that the communication device executes the methods described in the first and second aspects above.

[0038] In a sixth aspect, the present application further provides a computer-readable storage medium, wherein instructions are stored in the storage medium. When the instructions are executed, the methods described in the first and second aspects above are implemented.

[0039] In the technical solution of the present application, the operating life of mobile devices can be extended and the computing speed can be accelerated by providing additional computing resources on the MEC server. In addition, offloading tasks to neighboring MEC servers can avoid mobile users from frequently using the cloud to communicate or upload their tasks to the cloud, thereby alleviating communication congestion. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A structural diagram of a MEC system with a single UAV and a mobile user provided by the present invention;

[0041] Figure 2 A flowchart of a task offloading method provided by the present invention;

[0042] Figure 3 A schematic diagram of a DRL agent provided by the present invention trained in a MEC network of a drone;

[0043] Figure 4 A structural schematic diagram of a task offloading device provided by the present invention;

[0044] Figure 5 A schematic structural diagram of a communication device provided by the present invention. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0046] Mobile edge computing (MEC) has become popular as a key technology for communication, computing, and caching. By harvesting the advantages of easy deployment and flexibility of unmanned aerial vehicles (UAVs), one of the main functions of MEC is to use drones equipped with MEC servers to provide computing support for offloading tasks for mobile users in temporary hot spots or certain emergency situations (such as sports competition areas or areas destroyed by natural disasters). Although drones carried with MEC servers have many advantages, they are limited by their limited computing resources and sensitive energy consumption. However, due to the complexity of drone-assisted MEC systems, traditional optimization methods cannot achieve their computing resource optimization and energy consumption optimization well. In addition, with the growth of MEC servers and mobile users, the computational cost of MEC system optimization usually increases exponentially. Therefore, it is extremely challenging to control the location of drones and schedule the task offloading ratio. In this paper, a DRL method is proposed to optimize the drone trajectory design and user offloading task ratio scheduling to improve the performance of drone-assisted MEC systems for continuous problems in uncertainty. The system stability is maximized and the energy consumption is minimized.

[0047] At present, although there are a lot of literatures exploring the capabilities of drone-assisted MEC, most of the research is conducted based on research objectives. For example, the research objectives can be divided into three types. The first type is to minimize the energy consumption of the entire system or the energy consumption of mobile users. For example, a drone is designed to fly from a predefined initial position to a final position, so that it has the ability to offload computing. The drone not only provides offloading services for mobile users, but also explores the drone-assisted mobile edge computing (MEC) system. At the same time, the drone continues to fly while providing offloading computing services.

[0048] In addition, in addition to considering the energy consumption of the MEC system, the computing rate of the drone-assisted MEC system is also considerable. There are two types of offloading calculations in MEC, including binary offloading and partial offloading. In the binary offloading mode, the user chooses to perform all computing tasks locally only, or chooses to offload all computing tasks to the MEC server. For the partial offloading mode, the user can partially perform computing tasks locally and in the MEC server, where the local computing and the offloading computing are performed in parallel. The two-stage substitution algorithm and the three-stage substitution algorithm are used to solve the partial offloading and binary offloading problems respectively. For example, the partial offloading of the drone-assisted MEC system is studied based on the penalty double decomposition and L0 standard algorithm, thereby minimizing the total processing time, including the transmission time, the computing time, and the local computing time. Most interestingly, the simulation results show that better performance can be achieved when the drone remains stationary within a set of time intervals to collect data. Due to the limited onboard energy of the drone, the energy consumption and computing rate are too heavy. Therefore, the energy minimization without time limit is first studied, followed by the task completion time minimization problem. After that, it jointly optimizes the drone energy and completion time through the Pareto optimal solution.

[0049] It is worth noting that, as mentioned above, existing studies have studied drone-assisted MEC systems with the goal of minimizing energy consumption and task completion time separately without considering the balance between the two aspects. In addition, although Zhan et al. considered the trade-off between energy consumption and task completion time, they did not consider the long-term stability of the entire system. Deep reinforcement learning, as a learning method, has good performance in long-term decision-making, and its agents can make decisions on sequential problems in uncertain environments. Chen et al. used deep reinforcement learning technology to arrange offloading and improve mobile users' satisfaction with latency and energy consumption. However, they did not consider partial offloading because the computing tasks can only be processed on local devices or drones, so it has much less freedom to control and optimize offloaded tasks.

[0050] This application places a MEC system with a single drone and mobile users in three-dimensional Cartesian coordinates, such as Figure 1 As shown in the figure, the UAV flies at a fixed altitude to provide computing services to mobile users. Since the MEC server has greater computing power than mobile users, mobile users can offload their computing-intensive and delay-sensitive tasks to the UAV so that mobile users can reduce energy costs and speed up computing. The set of mobile users is described by. Specifically, the user can offload 1 of the current task to the UAV and process 1 of the task on the local device. The control agent can plan the trajectory of the UAV and the offloading ratio of the task. In order to minimize the total energy consumption of the system and maximize the number of completed tasks in time, it is necessary to define models in the MEC system, including the communication model, the offloading computing model, and the local computing model.

[0051] Table 1

[0052]

[0053]

[0054] Specifically, an embodiment of the present application provides a method for unloading tasks of drone-assisted mobile edge computing, such as Figure 2 As shown, including the following:

[0055] 101: construct a communication model, transmit its computing task to the drone, divide time into N time slots, N≥1, and each time slot corresponds to at least one task; indicate the flight trajectory of the drone at a first height according to the discrete position of the drone in each time slot; when the drone approaches the terminal device, obtain the energy consumption value and the data transmission rate between the drone and the terminal device, and calculate the transmission time and transmission energy consumption according to the energy consumption value and the data transmission rate;

[0056] 102: construct an offload calculation model, use the offload calculation model to perform offload calculation and local calculation of the drone and the terminal device, obtain the waiting time of the terminal device in the virtual queue, and calculate the total time cost according to the waiting time, wherein the total time cost is equal to the maximum value of the transmission time summary, the offload time, and the local calculation time;

[0057] 103: Calculate the total energy consumption of the UAV auxiliary system according to the total time cost, where the total energy consumption includes: signal energy consumption, offloading computing energy consumption, local computing energy consumption, and UAV propulsion energy consumption;

[0058] 104: Determine a percentage of the task of offloading and controlling the drone at the next position according to the total energy consumption of the drone auxiliary system, and offload the first task to the drone according to the task percentage.

[0059] Optionally, the method also includes: establishing a local computing model, and determining the local computing time and local computing energy consumption of the terminal device according to the local computing model; and calculating the total time cost according to the waiting time includes: determining the total time cost according to the waiting time and the local computing time.

[0060] The percentage of the tasks of the drone at the next location is the percentage of the tasks unloaded by the drone according to time.

[0061] The above steps 101 to 104 are described in detail below.

[0062] Step 101 specifically includes: constructing a communication model:

[0063] In the communication model, users transfer their computational tasks to drones. Time T is divided into N time slots, where N>K and τ th A time slot is defined as τ∈T={1,2,...,N}, and the length of the time slot is small enough. Therefore, at most one task is generated in each time slot. The location of mobile user k is given by uk=[xk,yk,0],k∈κ. The trajectory of the drone on the horizontal plane at height H can be indicated by the discrete position of the drone in each time slot, which is defined as h τ =[x τ ,y τ ,H],τ∈T. It is assumed that the UAV can return to its initial position after completing the mission. Therefore, the flight of the UAV has the following constraints as relation (1).

[0064]

[0065] Equation (1) indicates that the speed of the UAV must satisfy the maximum speed constraint υ max .

[0066] Since the drone flies around when the mobile user approaches, there are Line of Sight (LoS) and Non-Line of Sight (NoS) links between the drone and the mobile user. LoS The LoS link probability between the UAV and the mobile user can be retrieved from, given as relation (2), or obtained from the α and β environment-related variables.

[0067]

[0068] The transmission loss between the UAV and the mobile user k is expressed as equation (3).

[0069]

[0070] Where c is the speed of light, f c is the carrier frequency, and the parameters ηLoS and ηNLoS represent the environment-related losses of the LoS and NLoS links. Therefore, the data transmission rate from the mobile user to the drone is expressed by equation (4).

[0071]

[0072] Where Β represents bandwidth, p τ,k Denotes mobile user k in τ th The transmission power of the time slot, σ 2 Represents the noise power.

[0073] Assumption Dτ,k bit data needs to be th The time slot is calculated for mobile user k, d τ,k To indicate the computational load offloaded by user k and the UAV at τ th Therefore, in the communication model, the transmission time and transmission energy consumption are calculated by equation (5) and equation (6), respectively.

[0074]

[0075] Step 102 specifically includes: constructing an offloading calculation model:

[0076] After transmitting the offloaded computing tasks, drones and mobile users perform offloaded computing and local computing, C τ,k It is expressed as the CPU cycles required to calculate each bit of data, so the calculation time and Closure can be calculated:

[0077]

[0078] Among them, f UAV Indicates the CPU frequency of the MEC server installed on the drone, k = 10 -26 It is a hardware-related constant. The drone has an idle state to save energy and a running state to process tasks.

[0079] Since multiple tasks are transmitted to the drone by many mobile users, there will be a waiting time for the transmitted tasks. Assume that there is a virtual queue Λ in the drone, which will be provided on a first-come, first-served basis. An algorithm determines the unloading ratio and drone position of the next time period of the queue head task. Assume that there is an a-1 task waiting for this virtual queue, the current task will be a th The element is added to the virtual queue, so the waiting time of user k can be calculated as shown in equation (9):

[0080]

[0081] In the offloading computation model, the total time cost can be given as equation (10):

[0082]

[0083] Local calculation model:

[0084] Similar to the calculation process of the offload calculation model, given the CPU frequency f of mobile user k k , we can obtain the local computing time and local computing energy consumption in equations (11) and (12).

[0085]

[0086] Total time and effort:

[0087] The total time cost is equal to the maximum value of the transmission time, offloading time, and local computation time, as shown in equation (13).

[0088]

[0089] It should be noted that for each task, Δ τ,k Defined as expiration time, if If the total time exceeds the expiration time, the algorithm is considered to have failed to complete the task because the task has expired.

[0090] The above step 103 includes:

[0091] The total energy consumption of the drone-assisted MEC system includes: communication energy consumption, offload computing energy consumption, local computing energy consumption and drone propulsion energy consumption. The above embodiment describes the first three types of energy consumption, and equation (15) shows that their calculation is as follows:

[0092]

[0093] Where ξ=0.5MT / N, M represents the mass of the UAV, including the payload, so for the usage period τ th The total energy consumption is:

[0094]

[0095] Step 104 specifically includes:

[0096] In this UAV-assisted MEC system, in order to minimize the total energy consumption and time consumption, and at the same time, minimize the number of tasks completed before the task expiration time, the objective problem proposed is as follows:

[0097]

[0098] where λ1 and λ2 are normalization factors and ω∈[0,1] is a scaling term that balances the tasks completed and the energy consumed; to clarify this, E τ,k and Define. τ,k and The total is defined as th Cost, and F τ,k is a flag value describing whether the task is processed before the expiration time. It can be assigned the value 1, if the total time cost is less than the maximum tolerance, and 0, otherwise, given by the following relation:

[0099]

[0100] in, Total is the total consumption shown in equation (7).

[0101] It should be understood that in this example, if we only care about E τ,k The maximum tolerance and response speed of E can be deleted if the task has been processed before. τ,k However, one may also want to extend the life of the drone by sacrificing response time, so, alternatively, it makes more sense to minimize time cost and energy consumption.

[0102] Although the objective optimization problem is intuitive, obtaining the optimal solution is quite challenging. First, due to the complexity of the MEC nature, the coordinator cannot fully observe the MEC network environment and its status. In addition, the variables and constraints of the UAV-MEC network are too complex to be formalized as a standard optimization problem and solved using traditional optimization methods. Therefore, the optimization problem is formulated as a Markov Decision Process (MDP) so that it can be adapted to reinforcement learning. Furthermore, the optimal solution to the objective problem can be obtained using the Deep Reinforcement Learning (DRL) method.

[0103] In this work, since the MDP is considered to have a finite number of states, and the terminal state is defined as the MEC server being overloaded and unable to process newly arrived tasks. In fact, an episode can be considered terminated when the task waiting time on the MEC server exceeds a threshold. For simplicity, this embodiment assumes that the drone still has enough energy to return to its initial state after completing all tasks.

[0104] As mentioned above, in order to solve the target problem of the DRL method, it is necessary to form the problem using the MDP framework and specify the state, action, reward, and transition probability. In the system, as mentioned earlier, each episode contains a set of states. From the system description, it can be known that the state is quite complex because it contains user device states, mission profiles, network channel distribution, and various parameters of the drone. If we first define that the MDP in each episode is described by a set of states, it is defined as:

[0105]

[0106] where s τ ∈S is the time slot τ and s τ The general state in is equivalent to {f k ,f UAV ,Ω τ,k ,τ τ,k ,h τ,Λ τ}, where f k and f UAV They are k th CPU frequency of user and UAV;Ω τ,k Defined as the information of the first task on mobile user k, Ω τ,k ={D τ,k ,C τ,k ,Δ τ,k}; r τ,k represents the current transmission speed from mobile user k to the UAV; h τ is the position of the UAV at time step τ; Λ τ is the task queue state of the UAV at time step τ.

[0107] The state is transferred according to the executed operations and the internal transition probability of the MEC network. Each operation consists of two parts: determining the percentage of tasks to be unloaded and controlling the next position of the drone. It can generally be defined as:

[0108] a τ ={d τ , k, h τ+1}, (19)

[0109] where d τ,k is the percentage of tasks that are offloaded at time step τ. In other words, it is the percentage of tasks that are offloaded at time step τ. τ,k Offload to the drone and 1-d on the local device τ,k Process;h τ+1 Decide the next position of the drone. Note that the MDP environment state changes every time an action is performed conditional on the current state.

[0110] When considering the environment transition, it is assumed that the channel gain does not change during the time period, so when the UAV moves to a new location, the data transmission speed r i,k will change because r i,k Depends on the distance and the loss. Assuming that the channel gain does not change when the drone is flying, the transition probability of the MEC network can be given as follows, see equation (20):

[0111]

[0112] Among them, s and s′ are the current state and the next state, and the sum of the probabilities is equal to 1.

[0113] ∑p(s′ τ , R τ |s τ , a τ )=1. (21)

[0114] In addition, when an action is performed on a set of offloaded tasks (usually referred to as a reward in DRL), feedback from the MEC network to the DRL model can be calculated. Specifically, the feedback is the one-step reward of the agent obtained by a one-step action, which is described as the number of tasks completed before expiration minus the corresponding energy and time consumption, expressed as:

[0115]

[0116] This relation indicates that when the offload task is processed before the expiration date, F Τ , the agent is granted, and the agent is penalized for energy and time consumption. Energy E Τ and time The total consumption value has been smoothed using a logarithmic function because if the raw value is used, the learning model may be affected by energy fluctuations and time-consuming feedback. In addition, C is a small constant value to encourage the model to continue running and accumulate rewards over time. The interpretation of other parameters is defined in equation (16).

[0117] In order to maximize the long-term cumulative reward of the proposed model, each action is evaluated and the expected long-term reward is given by:

[0118]

[0119] Where R τ is the immediate reward, and γG τ+1 is the discounted long-term reward that can be calculated by equation (24). γ represents the discount of future rewards, γ∈[0,1].

[0120]

[0121] When k = 0, the feedback is R τ The immediate reward is ; although γ = 1 and k > 0, future rewards are not discounted. Since the environment only provides immediate rewards to the learning agent during interaction and learning, the expected future rewards are usually generated by a policy π, which is a sequence of actions corresponding to a set of states.

[0122] The expected value of the state to perform the operation is called the operation value function Q(s,a), and the maximum value Q(s,a) is called the optimal operation value function Q * (s,a), is given by the following formula (25):

[0123]

[0124] Among them, the target problem is equivalent to finding the best strategy π *, which maximizes the expected long-term return. In practice, there is more than one optimal strategy, and this example includes but is not limited to finding all the optimal strategies.

[0125]

[0126] Suggested approach

[0127] In this embodiment, deep reinforcement learning (DRL) is used to find the optimal strategy to maximize the expected long-term return. Generally, when the state space and dynamic space are discrete, reinforcement learning uses Q-table to retrieve the optimal strategy to solve the decision-making problem, which has good performance. However, the state space of the above-mentioned drone auxiliary system is continuous.

[0128] One possible approach is to use discrete values ​​to represent the continuous state space through discretization methods. However, this will lead to a huge state space or unreasonable solutions, which is extremely challenging to balance these two factors. In addition, when in a discrete state space, if the resolution is increased to improve the discretization accuracy, the search space will grow exponentially. On the other hand, it has a high probability of causing the model to diverge.

[0129] First, reinforcement learning

[0130] In order to introduce the DRL algorithm in this work, it is necessary to explain how to apply the reinforcement learning (RL) framework in the current drone-based MEC network. First, a drone-assisted MEC network is formulated as a Markov decision process (MDP), which means that the current state of the environment is only affected by the last state but not by the previous steps. Therefore, the agent can make decisions based on the current state because only the current state can affect the next state of the environment. After that, the agent can explore the MDP environment to accumulate experience. At the same time, based on its experience, the agent makes the best decision, which is called exploitation (maximizing reward by using existing information). Ideally, the agent can explore all possible states in the above MDP and search for all possible actions for each state. In addition, the optimal strategy can be obtained through the iterative method of temporal difference learning. Specifically, the goal of the optimization problem can be considered to find the optimal action value for the model, and the action value can be defined as:

[0131]

[0132] Optimal value Q * (s,a) can be deduced that the number of iterations approaches infinity, Qτ→Q asτ→∞, and Q * (s,a) assignment:

[0133]

[0134] However, it is almost impossible to explore all possible states and find the optimal operation values ​​for all states because these states contain continuous components. In addition, it is not possible to iterate approximately infinite time steps because it will take a lot of computing resources and time to train the model. Therefore, it is necessary to take an approximate function to replace the state feature map and roughly estimate the Q-table in reinforcement learning to reduce the research space. In addition, the model can be trained with reasonable computing resources and time to achieve the optimal point. Considering the complexity of the calculation of the mobile edge offloading of drones, the introduced artificial neural network can be used as an estimator to replace the simple linear estimator.

[0135] Through this estimator, the optimal policy can be obtained using general policy iteration (GPI). Specifically, the following two steps are iterated:

[0136]

[0137] Until |Q′(s, a)-Q(s, a)|<ξ, where δ is the learning rate and ξ is the convergence threshold. Let Q(s τ , a τ ;θ) becomes the Q function parameterized by θ. The loss function can be defined as:

[0138]

[0139] Then perform gradient descent to minimize the loss and update the parameters of the deep learning model.

[0140]

[0141] Here, δ is the learning rate.

[0142] 2. Deep Reinforcement Learning

[0143] Deep reinforcement learning introduces the use of deep neural networks instead of Q-learning networks (DQN) as a shorthand. Although it seems intuitive to replace the Q table in the RL algorithm with a deep neural network, and supervised learning has the correct answer (label), RL must learn the correct answer from continuous, evaluable, and continuous feedback. In other words, unlike supervised learning, the label data of RL comes from RL iterative updates, so the label data changes in each iteration.

[0144] The label data is a score that evaluates the actions taken based on the current state. Due to the feedback of environmental noise, the model will oscillate during development and exploration. To address this challenge, DQN uses a target network to back up the deep Q network and fix the weights in certain episodes (i.e., the fixing method).

[0145] The general process of the uninstallation system and the DRL agent learning process is as follows Figure 3 shown.

[0146] First, the user sends the configuration file of the prepared offloading task to the offloading control system, which is represented as the environment in the offloading system.

[0147] Second, whenever the environment receives a task profile, it collects the current state of the mobile edge computing network.

[0148] Third, a DRL agent also resides on the UAV, which takes actions based on observation information from the environment.

[0149] Fourth, the environment feeds back the next state of the MEC network to the DRL agent for corresponding actions. This feedback can be considered as an evaluation of the action, and the feedback is also known as a reward.

[0150] There are two deep neural networks in the DRL agent, called the local network and the target network. The local network generates actions for the environment.

[0151] Finally, the control agent (environment) performs actions, decides the proportion of the task that can be offloaded to the drone and drives it to a new location.

[0152] C. Training Process

[0153] In the embodiment, it is described in detail how the DRL agent is trained in the drone-based MEC network. Figure 3 As shown, the training algorithm is shown.

[0154] First, an experience replay buffer is created to store the collected data. Figure 3 As shown in Figure 2, each time the agent takes an action and interacts with the environment, the system generates a record. Each record contains the current state S τ , action a has been executed τ , and the reward r τ and the next state S τ +1, formed into a tuple <S τ ,a τ ,r τ ,S τ +1>.

[0155] The experience response buffer is a queue-like buffer with a fixed length; new records are stored into the buffer, and when the buffer is full, it discards the oldest record and keeps the latest record. The experience response buffer is crucial for learning DRL and converging to a robust policy, as it is wasteful to use a sample only once in traditional RL. In addition, it is common to train a deep learning model with multiple deep learning epochs; the number of epochs in deep learning defines how many times the model is trained with the same training samples. The model can converge faster and learn from rare samples that are important for a robust policy.

[0156] In addition, although the MDP framework is used to formulate the MEC network environment, it is desirable to decouple the sequential dependencies in the learning and interaction processes. Finally, the noise of the training samples can be reduced by extracting batches of samples from the experience buffer instead of using a single training sample at one time.

[0157] Secondly, a deep neural network (deep Q-Network) representing the q-value function is defined. The input and output sizes of the network are set equal to the state space and action space, since the input will be the state and the output will be the probability of the available actions. The target network is created by copying the DQN. The two copies of the network have different training purposes. The first copy, often called the local network, is responsible for interacting with the environment and generating training data samples. The target network is crucial to training because it prevents the learned model from oscillating from the feedback noise of the environment. During training, the model tries to minimize the loss between the Q-values ​​of the target network and the local network. Let is the best value from the target network, the output can be given by:

[0158]

[0159] In other words, the model uses the value of the target network to monitor the local network to prevent oscillation, so the actual loss is obtained according to the DQN function:

[0160]

[0161] A score window has been internalized to smooth the reward scores. Due to the complexity and noisy feedback of the mobile edge computing network system, its rewards are still quite noisy; therefore, it is more reliable to evaluate the model with the average score of all rewards in the current window. The score window is a queue that is updated as the training progresses; when the queue is full, the score window will discard the old scores and keep the latest scores.

[0162] The algorithm trains the model by starting an episode. It is defined that an episode ends when the UAV server is full, which means that the waiting time exceeds the threshold. Whenever a new episode starts, the MEC network environment must be reset. As mentioned in the previous section, it is hoped that the model will keep the UAV server running without explosion. The state features are pre-occupied before they are input into the model. The large difference in the scale of the feature values ​​will cause the model to be biased towards features with significant values. However, less critical features ignore the key features represented by small values. Therefore, the components of the state have been normalized before connecting the state and the input together.

[0163] The agent starts interacting with the mobile edge computing environment and the local network by merging the ∈-greedy algorithm. Each interaction produces an experience tuple, including the current state, action, reward, and next state, denoted as (S, A, R, S′); these experience tuples are collected and stored in the experience buffer for training the local network. Note that the learning agent chooses the best action with probability 1-∈ given the current state and policy, and takes random actions with probability. In order to balance exploration and exploitation, it decays over time because it is hoped that the model will spend more time exploring the environment at the beginning of training than in later events. As the model has more knowledge about the environment, it is hoped that the learning agent will exploit experience more instead of exploration.

[0164] Finally, the agent extracts sample batches of experiences, tuples the response buffer, and trains the local network. As described above, the learning agent attempts to minimize the loss between the output of the local network and the target network. At each step of gradient descent, the weights of the local network are updated. In the original DQN method, the algorithm updates the target network every N steps, overwriting the weights of the target network with the weights of the local network. In this embodiment, a smooth update of the target network by lillicrap et al. is used instead of a fixed method, with a meaningful update every N steps. Specifically, the target network is updated using a small change in the local weight discount ρ. The Adam algorithm is used to optimize the loss function and update the local weights.

[0165] Among them, the generation of training data blocks in the above steps and training steps can be performed first, and then other steps can be performed, so the server time training and single time data generation steps can be run. In addition, they can also be run simultaneously.

[0166] In summary, the algorithm starts by initializing the replay buffer, local network, target network, and score window size. The algorithm resets the MEC network in each episode, and the learning agent interacts with the environment to generate training data and store it in the experience buffer. The DRL agent can extract sample data from the experience buffer and train the local network. After every N steps, the target network is overwritten by the local network.

[0167] The present application also provides another task offloading method, which is applied to a task control system. The method includes:

[0168] Sending a configuration file of the offloading task to a task control system, which provides an environment for the offloading task;

[0169] When the task control system receives the configuration file, obtaining a current state of the mobile edge computing network;

[0170] The agent module takes corresponding action measures according to the observation information in the environment, and the agent module is located on the drone;

[0171] The task control system feeds back the next state of the mobile edge computing network to the agent module, wherein the feedback includes an evaluation of the action measures; wherein the agent module includes a local network, and the local network generates an action to provide an environment for the offloading task;

[0172] Execute the action, determine the proportion of the task and offload it to the drone, and drive the drone to the target location.

[0173] This application maximizes system stability and minimizes energy consumption and computational delay of drone-assisted MEC systems. Simulation results show that this method is superior to existing work and has good scalability. Specifically, it includes the following beneficial effects:

[0174] By determining the weight and energy consumption of the drone-assisted MEC system, a centralized problem is considered, and the information of mobile users is partially collected on the virtual control panel; based on the information collected on the control panel, the control center (DRL agent) decides the task transfer ratio and drone position in the next step. Then, this system environment is formulated as a Markov decision process (MDP).

[0175] After formulating the UAV-assisted MEC system as a centralized decision-making problem, the control center needs to determine the task transfer ratio of each task from the mobile users. Under uncertainty, it can be regarded as a sequential decision problem. Therefore, a deep Q-learning network (DQN) is adopted to jointly optimize the UAV trajectory and the task transfer ratio, while maximizing the stability of the system and minimizing the total computing time and energy consumption of the UAV-assisted MEC system.

[0176] Furthermore, unlike previous studies on drone-assisted MEC systems, this approach simultaneously optimizes the overall system energy consumption, computation time, completed task count, and system workload balance. The provided model is an end-to-end learning model and does not require further optimization based on the output.

[0177] Finally, simulation results show that the proposed method can significantly reduce the computation time and energy consumption of the entire system compared with the greedy algorithm. In addition, the parameters of DQN are explored under a given UAV-assisted MEC network.

[0178] In addition, the present application also provides a task offloading device for drone-assisted mobile edge computing, such as Figure 4 As shown, the device includes: a construction unit 401, a processing unit 402 and a calculation unit 403. In addition, the device may also include other units or modules, such as a storage unit, a transceiver unit, etc., which is not limited in this embodiment.

[0179] Among them, the construction unit 401 is used to construct a communication model; the processing unit 402 is used to transmit its computing task to the drone, divide the time into N time slots, N≥1, and each time slot corresponds to at least one task; indicate the flight trajectory of the drone at a first altitude according to the discrete position of the drone in each time slot; when the drone approaches the terminal device, obtain the energy consumption value and the data transmission rate between the drone and the terminal device, and calculate the transmission time and transmission energy consumption according to the energy consumption value and the data transmission rate.

[0180] The construction unit 401 is further used to construct an offloading calculation model; the processing unit 402 is further used to use the offloading calculation model to perform offloading calculation and local calculation of the drone and the terminal device, obtain the waiting time of the terminal device in the virtual queue, and calculate the total time cost according to the waiting time, wherein the total time cost is equal to the maximum value of the transmission time summary, the offloading time, and the local calculation time;

[0181] The calculation unit 403 is used to calculate the total energy consumption of the UAV auxiliary system according to the total time cost, and the total energy consumption includes: signal energy consumption, unloading calculation energy consumption, local calculation energy consumption and UAV propulsion energy consumption; the processing unit 402 is also used to determine the percentage of the task of unloading and controlling the UAV at the next position according to the total energy consumption of the UAV auxiliary system, and unload the first task to the UAV according to the task percentage.

[0182] Optionally, the construction unit 401 is further used to establish a local computing model, and determine the local computing time and local computing energy consumption of the terminal device according to the local computing model;

[0183] The processing unit 402 is further configured to determine the total time cost according to the waiting time and the local calculation time.

[0184] The percentage of the tasks of the drone at the next location is the percentage of the tasks unloaded by the drone according to time.

[0185] In addition, in another embodiment, a task offloading device is provided, which is applied to a task control system, and the device includes:

[0186] The offloading unit is used to send a configuration file of the offloading task to a task control system, which provides an environment for the offloading task.

[0187] An acquisition unit is used to acquire a current state of the mobile edge computing network when the task control system receives the configuration file.

[0188] The acquisition unit is further used to acquire the agent module to take corresponding action measures according to the observation information in the environment, and the agent module is located on the drone.

[0189] A processing unit is used to feed back the next state of the mobile edge computing network to the agent module, wherein the feedback includes an evaluation of the action measures; execute the action, determine the proportion of the task and offload it to the drone, and drive the drone to the target location.

[0190] The proxy module includes a local network, and the local network generates an action to provide an environment for the offloading task.

[0191] In addition, an embodiment of the present application further provides a communication device, which may be the task offloading device involved in the above embodiment, or a part of the task offloading device, and is used to implement the method described in the above embodiment.

[0192] like Figure 5 As shown, the communication device includes a processor 110, a transceiver 120 and a memory 130. In addition, other devices or modules may also be included, such as a camera, a sensor, at least one interface, etc. The memory 130 is coupled to the processor 110, and the memory 130 stores a computer program necessary for the communication device.

[0193] The processor 110 may be composed of an integrated circuit (IC), for example, a single packaged IC, or a plurality of packaged ICs having the same or different functions.

[0194] Optionally, the processor 110 is a processing chip or a chip circuit.

[0195] The transceiver 120 may include components such as a receiver, a transmitter, and an antenna, or may also include a transceiver module, such as a communication module such as a Bluetooth module, a baseband module, a radio frequency module, etc., and may support direct memory access.

[0196] The memory 130 may include a volatile memory, such as a random access memory (RAM); it may also include a non-volatile memory, such as a flash memory, a hard disk (HDD) or a solid-state drive (SSD); the memory may also include a combination of the above-mentioned types of memory. The memory may store a program or code, and the processor 110 may implement the functions of the communication device by executing the program or code.

[0197] In the above embodiments, all or part of the above embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the above embodiments may be implemented in the form of a computer program product, which is not limited in this embodiment. Figure 4 The functions of the construction unit 401 , the processing unit 402 and the calculation unit 403 in the illustrated device embodiment may be implemented by the processor 110 , or by the transceiver 120 controlled by the processor 110 ; in addition, the function of the storage unit may be implemented by the memory 130 .

[0198] The present application also provides a computer program product, which includes one or more computer program instructions. When a computer loads and executes the computer program instructions, the process or function described in the above embodiments of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0199] In addition, the present application also provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program may include some or all of the steps in the drone-assisted computing offloading method provided in the present application. The storage medium may be a disk, an optical disk, a read-only storage memory ROM, or a random access memory RAM, etc.

[0200] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention or some parts of the embodiments.

[0201] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for unmanned aerial vehicle-assisted mobile edge computing task offloading, characterized in that: The method comprises: A communication model is constructed, and its computing task is transmitted to the UAV, and time is divided into N time slots, N≥1, and each time slot corresponds to at least one task; the flight trajectory of the UAV at the first height is indicated according to the discrete position of the UAV in each time slot; when the UAV approaches the terminal device, the energy consumption value and the data transmission rate between the UAV and the terminal device are obtained, and the transmission time and the transmission energy consumption are calculated according to the energy consumption value and the data transmission rate; Constructing an offload calculation model, using the offload calculation model to perform offload calculation and local calculation of the UAV and the terminal device, obtaining the waiting time of the terminal device in the virtual queue, and calculating the total time cost according to the waiting time, wherein the total time cost is equal to the maximum value of the transmission time summary, the offload time, and the local calculation time; Calculate the total energy consumption of the UAV auxiliary system according to the total time cost, wherein the total energy consumption includes: communication energy consumption, offloading computing energy consumption, local computing energy consumption and UAV propulsion energy consumption; Determining a percentage of tasks for offloading and controlling the drone at a next location according to the total energy consumption of the drone auxiliary system, and offloading the first task to the drone according to the percentage of the tasks; Wherein, the method further comprises: Establishing a local computing model, and determining the local computing time and local computing energy consumption of the terminal device according to the local computing model; The calculating the total time cost according to the waiting time comprises: The total time cost is determined according to the waiting time and the local calculation time.

2. The method according to claim 1, characterized in that The percentage of tasks of the drone at the next location is the percentage of tasks unloaded by the drone according to time.

3. A task offloading device for drone-assisted mobile edge computing, characterized in that: The device comprises: A building block for building a communication model; The processing unit is used to transmit its computing task to the drone, divide the time into N time slots, N≥1, and each time slot corresponds to at least one task; indicate the flight trajectory of the drone at a first altitude according to the discrete position of the drone in each time slot; when the drone approaches the terminal device, obtain the energy consumption value and the data transmission rate between the drone and the terminal device, and calculate the transmission time and transmission energy consumption according to the energy consumption value and the data transmission rate; The construction unit is also used to construct an unloading calculation model. The processing unit is further used to perform offload calculation and local calculation of the UAV and the terminal device using the offload calculation model, obtain the waiting time of the terminal device in the virtual queue, and calculate the total time cost according to the waiting time, wherein the total time cost is equal to the maximum value of the transmission time summary, the offload time, and the local calculation time; A calculation unit, used to calculate the total energy consumption of the UAV auxiliary system according to the total time cost, wherein the total energy consumption includes: communication energy consumption, unloading calculation energy consumption, local calculation energy consumption and UAV propulsion energy consumption; The processing unit is further used to determine a percentage of tasks for offloading and controlling the drone at a next position according to the total energy consumption of the drone auxiliary system, and offload the first task to the drone according to the percentage of the tasks; The construction unit is further used to establish a local computing model, and determine the local computing time and local computing energy consumption of the terminal device according to the local computing model; The processing unit is further configured to determine the total time cost according to the waiting time and the local calculation time.

4. The device according to claim 3, characterized in that The percentage of tasks of the drone at the next location is the percentage of tasks unloaded by the drone according to time.

5. A communication device, characterized in that: The system comprises a processor and a memory, wherein the processor and the memory are coupled, and the memory is used to store instructions; The processor is used to execute instructions in the memory so that the communication device performs the task offloading method of drone-assisted mobile edge computing as described in any one of claims 1 to 2.

6. A computer-readable storage medium, wherein instructions are stored in the storage medium, characterized in that: When the instruction is executed, the task offloading method of the drone-assisted mobile edge computing as described in any one of claims 1 to 2 is implemented.

Images