Unmanned aerial vehicle task offloading and trajectory planning method and system

Abstract

The present application relates to the field of mobile edge computing, and more particularly to a method and system for unmanned aerial vehicle task offloading and trajectory planning. The present application constructs a system model of a target area, a mobile model of an unmanned aerial vehicle, a task model, and a power consumption model of the unmanned aerial vehicle. Based on the system model of the target area, the mobile model of the unmanned aerial vehicle, the task model, and the power consumption model of the unmanned aerial vehicle, an optimization function and constraint conditions of an unmanned aerial vehicle task offloading and path planning strategy are constructed. The process of solving the unmanned aerial vehicle task offloading and path planning strategy is described as a Markov decision process, a state space, an action space, and a reward function are established, and a deep reinforcement learning framework is obtained. A double deep Q network is used to solve the Markov decision process, and the optimal unmanned aerial vehicle task offloading and path planning strategy that satisfies the constraint conditions is obtained. The present application significantly improves the task completion rate of the system and guarantees the stability and effectiveness of the unmanned aerial vehicle in a complex environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of mobile edge computing, and particularly relates to a method and system for unmanned aerial vehicle task offloading and trajectory planning. BACKGROUND

[0002] In recent years, with the popularity of mobile edge computing (MEC) technology, unmanned aerial vehicles (UAVs) have gradually become a research hotspot as a flexible computing resource providing tool. In the mobile edge computing scenario, the unmanned aerial vehicle can serve as a mobile edge server to provide computing services for local devices on the ground. For example, in some remote areas or temporary activity sites where the ground infrastructure is not perfect, the unmanned aerial vehicle can quickly fly to the designated location to set up a temporary edge computing environment to assist the ground devices in data processing, thereby expanding the coverage of mobile edge computing and enhancing the flexibility and adaptability of the system.

[0003] Currently, in the task offloading and trajectory planning of the existing unmanned aerial vehicle assisted mobile edge computing system, for task offloading, factors such as data volume, transmission rate, server processing capacity, etc. are usually considered, and a mathematical model is established according to the computing resources of the unmanned aerial vehicle and the ground base station, the communication link status and other information to decide the position of task offloading. This can consider the allocation of all tasks and computing resources as a whole, and can achieve good results for relatively simple task relationships and sufficient computing resources. For trajectory planning, most of the methods focus on finding the shortest path or planning based on static strategies. The method of finding the shortest path determines the shortest path step by step by constantly updating the distance value of the node, and the planning based on static strategies usually formulates the task offloading and trajectory planning of the unmanned aerial vehicle according to the pre-set rules and conditions.

[0004] However, at present, there are still some technical problems to be solved for the application of unmanned aerial vehicles in task assistance. First, for task offloading, the existing technology focuses on independent tasks, ignoring the complex dependency relationship between many tasks in actual scenarios, which may lead to unreasonable scheduling, such as starting to execute subsequent tasks before some dependent pre-task is completed, thereby causing low efficiency of task execution and increasing the overall task processing delay, which cannot meet the needs of some application scenarios with high real-time requirements. And for unmanned aerial vehicle energy consumption modeling, although the existing method considers some energy consumption factors, the energy consumption modeling is relatively simple and cannot fully reflect the dynamic changes of energy consumption of unmanned aerial vehicles in flight, task offloading and data transmission process. Such assumptions may be effective in experimental environment, but in real application, it is easy to cause decision-making errors, so that the unmanned aerial vehicle cannot complete the scheduled task or cannot efficiently utilize energy. In addition, for the trajectory planning problem, the existing method focuses on finding the shortest path or planning based on static strategy, and does not fully combine the task offloading demand and energy consumption limit, so that the unmanned aerial vehicle cannot achieve comprehensive balance of tasks and energy during movement, especially in complex task scenarios. Such strategy is difficult to adapt to the dynamic change of environmental demand, affecting the overall performance of the unmanned aerial vehicle. SUMMARY

[0005] Therefore, the technical problem to be solved by the present application is to overcome the defects that the existing method ignores the dependency relationship between tasks in actual scenarios, the energy consumption modeling cannot dynamically reflect the energy consumption changes, and the trajectory planning does not fully combine the task offloading demand and energy consumption limit, resulting in the decline of the scheduling efficiency of the computing resources of the unmanned aerial vehicle.

[0006] To solve the above technical problems, the present application provides a method for unmanned aerial vehicle task offloading and trajectory planning, comprising the following steps:

[0007] Modeling the target area as an MxM hexagonal grid, numbering each region in turn according to the order of the unmanned aerial vehicle passing through each region to obtain a system model of the target area; wherein M is the set number of rows and columns;

[0008] Based on the system model of the target area, a unmanned aerial vehicle movement model is constructed, which includes the flight speed of the unmanned aerial vehicle, the flight distance and flight time of the unmanned aerial vehicle between adjacent two regions, the position representation of the unmanned aerial vehicle in each region, and the region decision variable;

[0009] A task model is constructed, which includes:

[0010] obtaining the time consumed by each task node in the current region based on the offloading decision variable of each task node in the current region, the time calculated by each node in the current region on the local device, the time calculated by each node in the current region on the UAV, and the transmission time of each node from the local device to the UAV;

[0011] obtaining the execution start time of each task node in the current region based on the directed acyclic graph of the current region and the transmission time between nodes in the current region due to the dependency relationship;

[0012] obtaining the completion time of each task node in the current region based on the time consumed by each task node in the current region and the execution start time of each task node in the current region;

[0013] obtaining the time for completing the directed acyclic graph of the current region based on the completion time of each task node in the current region and the execution start time of the first task node in the current region;

[0014] obtaining the data amount assisted by the UAV in the current region based on the offloading decision variable of each task node in the current region and the data amount size calculated by each task node in the current region;

[0015] constructing a UAV energy consumption model, wherein the UAV energy consumption model comprises:

[0016] obtaining the total energy consumption of the UAV in the current region based on the region decision variable of the current region, the energy consumption of the UAV for calculating the directed acyclic graph of the current region, the energy consumption of the UAV hovering in the current region, and the flight energy consumption of the UAV flying from the previous region to the next region;

[0017] obtaining the energy of the UAV entering the previous region based on the total energy consumption of the UAV in the previous region and the energy of the UAV entering the previous region,

[0018] obtaining the energy of the UAV entering the current region;

[0019] constructing an optimization function and constraint condition of the UAV task offloading and path planning strategy based on the system model of the target region, the UAV movement model, the task model and the UAV energy consumption model, describing the process of solving the UAV task offloading and path planning strategy as a Markov decision process, establishing a state space, an action space and a reward function, and obtaining a deep reinforcement learning framework;

[0020] solving the Markov decision process by using a double deep Q network, and obtaining an optimal UAV task offloading and path planning strategy satisfying the constraint condition.

[0021] Preferably, the position of the UAV in each region indicates that the region number of the last time the UAV enters in the flight process is N, the region number set of the target region is {1, 2, …, n, …, N}, the coordinate of the start position q s of the UAV is (x1, y1, 0), the coordinate of the end position q f of the UAV is (x N , y N , 0), and the position q[n] of the nth region is (x n , y n , H); wherein x n is the horizontal coordinate of the center point of the nth region, y n is the vertical coordinate of the center point of the nth region, and H is the height of the UAV flight.

[0022] Preferably, the directed acyclic graph of the current region is constructed, comprising:

[0023] each task in the current region is taken as a task node, each task node includes the data size required for transmission and the data size at the time of calculation when the task node is offloaded from the local device to the UAV, a task node set of the current region is constructed;

[0024] if the data d ji generated by the jth task node is required when the ith task node of the current region is calculated, a directed edge E ji (d ji ) is constructed from the jth task node to the ith task node, a directed edge set of the current region is constructed based on all directed edges of the current region;

[0025] the directed acyclic graph of the current region is constructed based on the task node set and the directed edge set of the current region.

[0026] Preferably, after the UAV enters the nth region, whether to hover to assist in calculating the tasks in the current region is determined according to the region decision variable of the nth region, when is 0, the UAV jumps to the adjacent region, and when is 1, the UAV hovers to assist in calculating the tasks in the current region.

[0027] when the offload decision variable S n,i of the ith task node of the nth region is 0, the ith task node of the nth region is offloaded to the local device, and when the offload decision variable S n,i of the ith task node of the nth region is 1, the ith task node of the nth region is offloaded to the UAV, and S n,i ∈{0, 1}.

[0028] Preferably, the calculation formula of the time consumed by each task node of the current region is:

[0029]

[0030] wherein, is the time consumed for executing the i-th task node of the n-th region, is the time calculated by the i-th task node of the n-th region on the local device, n,i is the data size needed for calculating the i-th task node of the n-th region, is the time calculated by the i-th task node of the n-th region on the UAV, uav is the computing capability of the UAV, is the transmission time of the i-th task node of the n-th region from the local device to the UAV, n,i is the data size needed for transmission of the i-th task node of the n-th region from the local device to the UAV, trans is the rate of receiving and sending data by the UAV within the current region;

[0031] The calculation formula of the execution start time of each task node of the current region is:

[0032]

[0033] wherein, is the execution start time of the i-th task node of the n-th region, max(.) is the maximum value, is the set of pre-task of the i-th task node of the n-th region, is the completion time of the j-th task node of the n-th region, is the transmission time caused by the j-th task node of the n-th region for the pre-task of the i-th task node, ji is the data size needed for the j-th task node to generate for calculating the i-th task node of the n-th region, n,j is the offloading decision variable of the j-th task node of the n-th region, n,i is the offloading decision variable of the i-th task node of the n-th region;

[0034] The calculation formula of the completion time of each task node of the current region is:

[0035]

[0036] wherein, is the completion time of the i th task node of the n th region;

[0037] The time calculation formula of the directed acyclic graph of the current region is:

[0038]

[0039] wherein t total [n] is the time of completing the directed acyclic graph of the n th region, is the completion time of the L[n] th task node of the n th region, L[n] is the total number of task nodes of the n th region, is the execution start time of the first task node of the n th region;

[0040] The data amount calculation formula of the unmanned aerial vehicle assisting in calculation in the current region is:

[0041] C[n] =∑ i∈L[n] S n,i c n,i ,

[0042] wherein C[n] is the data amount of the unmanned aerial vehicle assisting in calculation in the n th region.

[0043] Preferably, the formula of the total energy consumption of the unmanned aerial vehicle in the current region is:

[0044]

[0045] wherein, is the total energy consumption of the unmanned aerial vehicle in the n th region, is the region decision variable of the n th region, is the energy consumption of the unmanned aerial vehicle hovering in the n th region, is the energy consumption of the unmanned aerial vehicle calculating the directed acyclic graph of the n th region, is the flight energy consumption of the unmanned aerial vehicle flying from the previous region to the next region;

[0046] The formula of the energy of the unmanned aerial vehicle entering the current region is:

[0047]

[0048] wherein E[n] is the energy of the unmanned aerial vehicle entering the n th region, E[n-1] is the energy of the unmanned aerial vehicle entering the n-1 th region, is the total energy consumption of the unmanned aerial vehicle in the n-1 th region;

[0049] The energy consumption of the unmanned aerial vehicle hovering in the n th region is:

[0050]

[0051] wherein p stay is the power of the UAV hovering;

[0052] the energy consumption of the UAV computing the directed acyclic graph of the nth region is:

[0053]

[0054] wherein, is the energy consumption of the UAV processing the ith task node of the nth region, p exec is the power of the UAV computing the task, is the energy consumption of the UAV receiving the transmission of the ith task node of the nth region, p rece is the power of the UAV receiving the transmission, is the energy consumption of the UAV caused by the jth task node of the nth region triggering the ith task node as a preceding task

[0055] p send is the power of the UAV transmitting;

[0056] the flight energy consumption of the UAV flying from the previous region to the next region is:

[0057]

[0058]

[0059] wherein p fly is the flight power of the UAV, t fly is the flight time of the UAV between two adjacent regions, P0 is the power of the UAV blade, P1 is the induced power of the UAV hovering, v0 is the average speed of the UAV rotor, U tip is the tip speed of the UAV blade, d0 is the fuselage drag ratio of the UAV, s is the solidity of the UAV rotor, A is the area of the UAV rotor, p is the air density, v fly is the flight speed of the UAV.

[0060] Preferably, the optimization function and the constraint condition of the UAV task offloading and path planning strategy include:

[0061] The optimization function of the UAV task offloading and path planning strategy is:

[0062]

[0063] The constraint condition includes:

[0064]

[0065] wherein, a1 is a first weight coefficient, a1 is 1 if the UAV finally reaches the end position, a1 is 0 if the UAV does not finally reach the end position, is a reward for the UAV finally reaching the end position, a is a second weight coefficient, R p is a punishment for the UAV crossing the boundary, R p is a negative value, a2 is 1 if the UAV exceeds the boundary of the region during flight, a2 is 0 if the UAV does not exceed the boundary of the region during flight, N is the region number of the last time the UAV enters during flight, a3 is a third weight coefficient, R t [n] is a reward for the UAV calculating the unloading task in the nth region, R t [n] = ξ × C[n], ξ is a first coefficient, R c [n] is a reward for the UAV assisting the region calculation in the nth region, q U [1] is the first region entered by the UAV, q S is the starting position of the UAV, q U [N] is the position of the last region entered by the UAV, N is the region number of the last time the UAV enters during flight, q F is the end position of the UAV, E[1] is the initial energy of the UAV in the first region.

[0066] Preferably, the process of solving the UAV task unloading and path planning strategy is described as a Markov decision process, a state space, an action space and a reward function are established, and a deep reinforcement learning framework is obtained, including:

[0067] The state space includes: coordinate representation of the current position coordinates of the UAV in the hexagonal grid, energy and directed acyclic graph task features of the current region wherein, the directed acyclic graph task features of the current region wherein, is the normalized adjacency matrix of the directed acyclic graph of the current region, is the normalized task node feature matrix of the directed acyclic graph of the current region, and I is the unit matrix;

[0068] The action space includes: hovering in the current region, moving to the upper adjacent region of the current region, moving to the upper right adjacent region of the current region, moving to the lower right adjacent region of the current region, moving to the lower adjacent region of the current region, moving to the lower left adjacent region of the current region, and moving to the upper left adjacent region of the current region.

[0069] The reward function includes: a reward for the UAV finally reaching the end position Penalty R for UAV crossing the boundary p , negative reward for UAV consuming energy in the nth region Reward R for UAV assisting region calculation in the nth region c [n], reward R for UAV calculating offloading task in the nth region t [n] = ξ × C[n], UAV distance change reward R d = η × Δ d ;

[0070] Wherein, R t [n] and R c [n] coefficients include E[n] / E[1], R d coefficients include (E[1]-E[n]) / E[1], ξ is the first coefficient, ∈ is the second coefficient, η is the third coefficient, Δ d is the change of the distance between the UAV and the end position.

[0071] Preferably, the double deep Q network is used to solve the Markov decision process, comprising:

[0072] Step S601: initialize the parameters θ of the Q network, the parameters θ ′ of the target network, the noise network parameters, the capacity I of the experience replay pool, the discount factor γ ∈ [0,1), the priority intensity parameter α, the weight correction parameter β, the constant ∈, the preset batch size m, the training step number T, the current training number k, the maximum training number K, and the initial state s0 of the UAV;

[0073] Step S602: initialize the time step index t as 1;

[0074] Step S603: input the current state s t of the UAV into the Q network to obtain the Q values corresponding to all actions, select the action with the maximum Q value as the preliminary action, generate noise using the noise network, and obtain the execution action a t based on the preliminary action and the noise;

[0075] Step S604: the UAV executes the selected action a t , state transition is performed according to the system model of the target region, the UAV movement model, the task model, the UAV energy consumption model and the state transition probability, to obtain the reward function value r t and the next state s t+1 of the UAV, and store the generated experience pool sample (s t , a t , r t , s t+1 ) into the experience pool;

[0076] Step S605: Determine whether the number of experience pool samples is less than I, if less than I, t=t+1, return to execute step S603 until the number of experience pool samples is greater than or equal to I;

[0077] Step S606: When the number of experience pool samples is greater than or equal to I, based on the reward function value of each experience pool sample, the discount factor γ, the Q value corresponding to the execution action, the maximum Q value in all actions obtained by the target network in the next state of the unmanned aerial vehicle, the TD error of each experience pool sample is calculated;

[0078] Step S607: Based on the TD error of each experience pool sample, the priority intensity parameter α, the constant ∈, the priority of each experience pool sample is calculated, according to the priority of each experience pool sample, the experience pool samples of the preset batch size m are taken out, and the sampling frequency of each experience pool sample is calculated;

[0079] Step S608: Based on the sampling frequency of each experience pool sample taken out, the weight correction parameter β, the capacity I of the experience replay pool, the importance sampling weight of each experience pool sample taken out is calculated; according to the importance sampling weight, the TD error of each experience pool sample is corrected;

[0080] Step S609: Based on the TD error of each experience pool sample after correction, the gradient descent method is used to update the parameter θ of the Q network;

[0081] Step S610: Update the parameter θ of the target network every T steps ′ ;

[0082] Step S611: Determine whether k is less than K, if yes, let k=k+1, return to execute step S602, if not, the optimal unmanned aerial vehicle task offloading and path planning strategy meeting the constraint condition is obtained.

[0083] The application also provides an unmanned aerial vehicle task offloading and trajectory planning system, characterized by comprising:

[0084] A system model construction module is configured to model the target area as a hexagonal grid of MxM, sequentially number each region according to the order of the unmanned aerial vehicle passing through each region, and obtain a system model of the target area; wherein M is the set number of rows and columns.

[0085] An unmanned aerial vehicle movement model construction module is configured to construct an unmanned aerial vehicle movement model based on the system model of the target area, wherein the unmanned aerial vehicle movement model includes the flight speed of the unmanned aerial vehicle, the flight distance and flight time of the unmanned aerial vehicle between adjacent two regions, the position representation of the unmanned aerial vehicle in each region, and the region decision variable.

[0086] a task model construction module, configured to construct a task model, the task model comprising:

[0087] based on the offloading decision variable of each task node in the current region, the time of each node in the current region calculated locally by the device, the time of each node in the current region calculated by the UAV, and the transmission time of each node from the local device to the UAV, obtain the time consumed by each task node in the current region to execute;

[0088] based on the directed acyclic graph of the current region and the transmission time between nodes in the current region due to the dependency relationship, obtain the execution start time of each task node in the current region;

[0089] based on the time consumed by each task node in the current region to execute and the execution start time of each task node in the current region, obtain the completion time of each task node in the current region;

[0090] based on the completion time of each task node in the current region and the execution start time of the first task node in the current region, obtain the time to complete the directed acyclic graph of the current region;

[0091] based on the offloading decision variable of each task node in the current region and the data size calculated by each task node in the current region, obtain the data size of the UAV to assist in calculation in the current region;

[0092] a UAV energy consumption model construction module, configured to construct a UAV energy consumption model, the UAV energy consumption model comprising:

[0093] based on the region decision variable of the current region, the energy consumption of the UAV to calculate the directed acyclic graph of the current region, the energy consumption of the UAV hovering in the current region, and the flight energy consumption of the UAV flying from the previous region to the next region, obtain the total energy consumption of the UAV in the current region;

[0094] based on the total energy consumption of the UAV in the previous region and the energy of the UAV when entering the previous region,

[0095] obtain the energy of the UAV when entering the current region;

[0096] a deep reinforcement learning framework construction module, configured to construct an optimization function and a constraint condition of a UAV task offloading and path planning strategy based on the system model of the target region, the UAV movement model, the task model, and the UAV energy consumption model, and describe the process of solving the UAV task offloading and path planning strategy as a Markov decision process, establish a state space, an action space, and a reward function, and obtain a deep reinforcement learning framework;

[0097] The solving module is used for solving the Markov decision process by using a double deep Q network, and an optimal unmanned aerial vehicle task offloading and path planning strategy satisfying a constraint condition is obtained.

[0098] The unmanned aerial vehicle task offloading and trajectory planning method and system model the target region as an MxM region, construct a directed acyclic graph task model for each region, wherein each task node covers the data size required to be transmitted from the local device to the unmanned aerial vehicle and the data size at the time of calculation, and each directed edge contains the data size required to be transmitted by two task nodes having a dependency relationship, thereby completely describing the task nodes and their relationship, accurately depicting the sequence and data flow direction between the task nodes, accurately expressing the complex task topology, effectively avoiding scheduling confusion caused by improper handling of task dependencies, and improving the scheduling efficiency of the unmanned aerial vehicle computing resources.

[0099] By constructing an accurate energy consumption model, the energy consumption factors of the unmanned aerial vehicle in each link such as flight, task offloading, and data transmission are comprehensively and meticulously analyzed, the energy consumption of the unmanned aerial vehicle under different operations is quantified, the energy distribution strategy is optimized, the task completion rate is ensured, and the operation time of the unmanned aerial vehicle is prolonged.

[0100] The present application can dynamically adjust the behavior of the unmanned aerial vehicle in a complex environment by unified decision of the offloading scheme and path planning through deep reinforcement learning, thereby improving the task completion capability of the system under energy limited conditions. Specifically, the present application comprehensively considers the time delay requirement of the task and the remaining energy of the unmanned aerial vehicle through deep reinforcement learning, optimizes the offloading decision and flight path, reasonably allocates energy for task processing and movement under the premise of meeting the task time delay constraint, ensures that the unmanned aerial vehicle can complete more auxiliary computing tasks, and successfully reaches the target position. Compared with the prior art, the present application avoids the low efficiency problem caused by the separation of offloading decision and path planning in the traditional method, and overcomes the defect that uneven energy distribution may cause task failure or energy depletion halfway. Through this comprehensive optimization strategy, the present application significantly improves the task completion rate of the system, and guarantees the stability and effectiveness of the unmanned aerial vehicle in a complex environment. BRIEF DESCRIPTION OF DRAWINGS

[0101] In order to make the content of the present application more easily understood, the present application will be further described in detail below according to specific embodiments of the present application and in conjunction with the drawings, wherein:

[0102] Figure 1 is a flowchart of the unmanned aerial vehicle task offloading and trajectory planning method of the present application.

[0103] Figure 2is a flow chart for solving a Markov decision process by using a double deep Q network. DETAILED DESCRIPTION

[0104] The present application will be further described below in conjunction with the drawings and specific embodiments so that those skilled in the art can better understand and implement the present application, but the embodiments are not intended to limit the present application.

[0105] As shown in Figure 1 , Figure 1 is a flow chart of a UAV task offloading and trajectory planning method of the present application.

[0106] The first embodiment provides a UAV task offloading and trajectory planning method, which comprises the following steps:

[0107] Step S1: model the target area as an MxM hexagonal grid, number each region in turn according to the order in which the UAV passes through each region to obtain a system model of the target area; wherein M is the set number of rows and columns;

[0108] Wherein, different numbers of users are distributed in each region, and at the same time, the UAV can only assist in calculating one hexagonal region, and after the calculation is completed, it jumps to the adjacent region; the order in which the UAV passes through each region is numbered in turn, n represents the nth region passed through by the UAV, and the region where the UAV starts is recorded as 1.

[0109] Step S2: based on the system model of the target area, a UAV movement model is constructed, which comprises: the flight speed of the UAV, the flight distance and flight time of the UAV between adjacent two regions, the position representation of the UAV in each region, and the region decision variable;

[0110] In this embodiment, preferably, the position representation of the UAV in each region comprises: the last region number entered by the UAV in the flight process is N, the region number set of the target area is {1, 2, …, n, …, N}, and the coordinates of the starting position q s of the UAV are (x1, y1, 0), the coordinates of the ending position q f are (x N , y N , 0), and the position q[n] in the nth region is (x n , y n , H); wherein x n is the horizontal coordinate of the center point of the nth region, y n is the vertical coordinate of the center point of the nth region, and H is the flight height of the UAV.

[0111] In this embodiment, specifically, it is assumed that the drone moves at a constant speed. Since each area is modeled as a hexagon, the distance the drone jumps each time is fixed. The flight distance between two adjacent areas is D, and the flight speed of the drone is v. fly , the flight time of the UAV between two adjacent areas is t fly .

[0112] Step S3: Constructing a task model, the task model includes:

[0113] Based on the offloading decision variables of each task node in the current area, the calculation time of each node in the current area on the local device, the calculation time of each node in the current area on the drone, and the transmission time of each node offloading from the local device to the drone, the time consumed by executing each task node in the current area is obtained;

[0114] In this embodiment, specifically, the time consumed by executing each task node in the current area is calculated as follows:

[0115]

[0116] in, The time consumed to execute the i-th task node in the n-th region, The time for the i-th task node in the n-th region to be calculated on the local device, c n,i The amount of data when calculating the i-th task node in the n-th region, is the time for the i-th task node in the n-th area to be calculated by the drone, f uav For the computing power of drones, is the transmission time of the i-th task node in the n-th region from the local device to the drone, w n,i is the amount of data that needs to be transferred from the local device to the UAV when the i-th task node in the n-th region is unloaded, R trans The rate at which the drone receives and sends data in the current area;

[0117] Based on the directed acyclic graph of the current area and the transmission time caused by the dependency relationship between the nodes in the current area, the execution start time of each task node in the current area is obtained;

[0118] In this embodiment, specifically, the calculation formula for the execution start time of each task node in the current area is:

[0119]

[0120] in, is the execution start time of the i-th task node of the n-th region, max() is the maximum value, is the set of pre-task of the i-th task node of the n-th region, is the completion time of the j-th task node of the n-th region, is the transmission time caused by the j-th task node being the pre-task of the i-th task node of the n-th region, d ji is the data amount needed by the j-th task node for calculating the i-th task node of the n-th region, S n,j is the offloading decision variable of the j-th task node of the n-th region, S n,i is the offloading decision variable of the i-th task node of the n-th region;

[0121] Based on the time consumed by executing each task node of the current region and the execution start time of each task node of the current region, the completion time of each task node of the current region is obtained.

[0122] In this embodiment, specifically, the calculation formula of the completion time of each task node of the current region is:

[0123]

[0124] wherein, is the completion time of the i-th task node of the n-th region;

[0125] Based on the completion time of each task node of the current region and the execution start time of the first task node of the current region, the time for completing the directed acyclic graph of the current region is obtained.

[0126] In this embodiment, specifically, the calculation formula of the time for completing the directed acyclic graph of the current region is:

[0127]

[0128] wherein, t total is the time for completing the directed acyclic graph of the n-th region, is the completion time of the L[n]-th task node of the n-th region, L[n] is the total number of task nodes of the n-th region, is the execution start time of the first task node of the n-th region;

[0129] Based on the offloading decision variable of each task node of the current region and the data amount size calculated by each task node of the current region, the data amount of the unmanned aerial vehicle assisting in calculation in the current region is obtained.

[0130] In this embodiment, specifically, the calculation formula for the amount of data assisted by the drone in the current area is:

[0131] C[n]=∑ i∈L[n] S n,i c n,i ,

[0132] Among them, C[n] is the amount of data assisted by the UAV in the nth area.

[0133] In this embodiment, preferably, constructing a directed acyclic graph (DAG) of the current area includes:

[0134] Each task in the current area is taken as a task node. Each task node includes the amount of data required to be transmitted from the local device to the drone and the amount of data required for calculation, and the task node set of the current area is constructed.

[0135] Among them, the task node set of the nth area is:

[0136]

[0137] Among them, V j (w j ,c j ) is the jth task node in the nth region, V i (w i ,c i ) is the i-th task node in the n-th region, is the Lth region of the nth region [n] task nodes, w i is the amount of data that needs to be transferred from the local device to the UAV when the i-th task node in the n-th region is unloaded, c i The amount of data to be calculated for the i-th task node in the n-th region.

[0138] If the calculation of the i-th task node in the current area requires the amount of data d generated by the j-th task node ji , then construct a directed edge E from the jth task node to the ith task node ji (d ji ), based on all directed edges in the current region, build the directed edge set of the current region;

[0139] Based on the task node set and directed edge set of the current area, a directed acyclic graph of the current area is constructed.

[0140] Step S4: Constructing a UAV energy consumption model, which includes:

[0141] The total energy consumption of the UAV in the current region is obtained based on the regional decision variable of the current region, energy consumption of the UAV in calculating the directed acyclic graph of the current region, energy consumption of the UAV hovering in the current region, and flight energy consumption of the UAV flying from the previous region to the next region;

[0142] In this embodiment, specifically, the formula of the total energy consumption of the UAV in the current region is:

[0143]

[0144] wherein, E[n] is the total energy consumption of the UAV in the nth region, is the regional decision variable of the nth region, is the energy consumption of the UAV hovering in the nth region, is the energy consumption of the UAV in calculating the directed acyclic graph of the nth region, is the flight energy consumption of the UAV flying from the previous region to the next region;

[0145] The energy of the UAV entering the current region is obtained based on the total energy consumption of the UAV in the previous region and the energy of the UAV entering the previous region;

[0146] In this embodiment, specifically, the formula of the energy of the UAV entering the current region is:

[0147]

[0148] wherein, E[n] is the energy of the UAV entering the nth region, E[n-1] is the energy of the UAV entering the (n-1)th region, E[n-1] is the total energy consumption of the UAV in the (n-1)th region;

[0149] The UAV calculates the energy consumption of the directed acyclic graph of the nth region is:

[0150]

[0151] wherein, is the energy consumption of the UAV processing the ith task node of the nth region, p exec is the power of the UAV calculating the task, is the energy consumption of the UAV receiving the transmission of the ith task node of the nth region, p rece is the power of the UAV receiving the transmission, is the consumption of the UAV caused by the jth task node of the nth region being the preceding task of the ith task node

[0152] p send is the flight power of the UAV;

[0153] flight energy consumption of the UAV from a previous region to a next region is:

[0154]

[0155] wherein p fly is the flight power of the UAV, t fly is the flight time of the UAV between two adjacent regions, P0 is the power of the UAV blade, P1 is the induced power of the UAV hovering, v0 is the average speed of the UAV rotor, U tip is the tip speed of the UAV blade, d0 is the fuselage drag ratio of the UAV, s is the solidity of the UAV rotor, A is the area of the UAV rotor, p is the air density, v fly is the flight speed of the UAV.

[0156] In this embodiment, specifically, after the UAV enters the nth region, whether to hover to assist in calculating the task in the current region is determined according to the regional decision variable of the nth region, when is 0, the UAV jumps to an adjacent region, and when is 1, the UAV hovers to assist in calculating the task in the current region.

[0157] When the offloading decision variable S n,i of the ith task node of the nth region is 0, the ith task node of the nth region is offloaded to a local device, and when the offloading decision variable S n,i of the ith task node of the nth region is 1, the ith task node of the nth region is offloaded to the UAV, S n,i e {0, 1}.

[0158] Step S5: based on the system model of the target region, the UAV movement model, the task model and the UAV energy consumption model, an optimization function and a constraint condition of the UAV task offloading and path planning strategy are constructed, and a process of solving the UAV task offloading and path planning strategy is described as a Markov decision process, a state space, an action space and a reward function are established, and a deep reinforcement learning framework is obtained;

[0159] In this embodiment, the UAV uses limited energy to assist in calculation, and the UAV starts to make a decision as soon as it reaches a region. Only when the UAV decides to stay in the current region, the current region is assisted in calculation. Once the energy of the UAV is exhausted, the process optimization problem is stopped.

[0160] In this embodiment, preferably, the optimization function and constraint condition of the unmanned aerial vehicle task offloading and path planning strategy include:

[0161] The optimization function of the unmanned aerial vehicle task offloading and path planning strategy is:

[0162]

[0163] The constraint condition includes:

[0164]

[0165] wherein, α1 is a first weight coefficient, if the unmanned aerial vehicle finally reaches the end position, then α1 is 1, if the unmanned aerial vehicle does not finally reach the end position, then α1 is 0, is a reward for the unmanned aerial vehicle finally reaching the end position, α is a second weight coefficient, R p is a punishment for the unmanned aerial vehicle crossing the boundary, R p is a negative value, if the unmanned aerial vehicle exceeds the boundary during flight, then α2 is 1, if the unmanned aerial vehicle does not exceed the boundary during flight, then α2 is 0, N is the region number of the last time the unmanned aerial vehicle enters during flight, α3 is a third weight coefficient, R t [n] is a reward for the unmanned aerial vehicle calculating the offloading task in the nth region, R t [n] = ξ × C[n], ξ is a first coefficient, R c [n] is a reward for the unmanned aerial vehicle calculating in the nth region, q U [1] is the first region entered by the unmanned aerial vehicle, q S is the starting position of the unmanned aerial vehicle, q U [N] is the position of the last region entered by the unmanned aerial vehicle, N is the region number of the last time the unmanned aerial vehicle enters during flight, q F is the end position of the unmanned aerial vehicle, E[1] is the initial energy of the unmanned aerial vehicle in the first region.

[0166] In reinforcement learning, an agent continuously learns how to choose actions to maximize cumulative rewards by interacting with an environment. A reinforcement learning problem is usually described by a Markov Decision Process (MDP), which is usually composed of the following five tuples:

[0167] S: state space, representing the set of all possible states

[0168] A: action space, representing the set of all possible actions

[0169] P(s ′ |s,a): state transition probability, representing the probability of transitioning to state s after taking action a in state s ′ s

[0170] R(s,a): Reward function, which represents the reward obtained after taking action a in state s

[0171] γ∈[0,1): Discount factor used to weigh the importance of current rewards against future rewards.

[0172] like Figure 2 As shown, Figure 2 This is a flowchart for solving a Markov decision process using a dual deep Q-network. This embodiment treats the drone as an agent interacting with the environment, allowing the agent to make decisions about drone trajectory planning and task offloading. Each time the drone enters an area, it makes a decision based on the current state of the environment.

[0173] In this embodiment, the process of solving the drone task offloading and path planning strategy is specifically described as a Markov decision process. The state space, action space, and reward function are established to obtain a deep reinforcement learning framework, including:

[0174] The state space includes: the coordinate representation of the current position of the drone in the hexagonal grid, the energy and the characteristics of the directed acyclic graph task in the current area Among them, the characteristics of the directed acyclic graph task in the current area in, is the normalized adjacency matrix of the directed acyclic graph of the current region, is the normalized feature matrix of the task nodes in the directed acyclic graph of the current region, and I is the identity matrix;

[0175] The coordinates of the drone's current position in the hexagonal grid are represented as a two-dimensional vector. For example, (0, 0) represents the grid area in the upper left corner of the hexagonal grid. In order to address the problem of different matrix dimensions caused by different numbers of directed acyclic graph (DAG) nodes, this embodiment sets a maximum number of nodes, and low-dimensional matrices are padded with zeros.

[0176] The action space includes: hovering in the current area, moving to the upper adjacent area of ​​the current area, moving to the upper right adjacent area of ​​the current area, moving to the lower right adjacent area of ​​the current area, moving to the lower adjacent area of ​​the current area, moving to the lower left adjacent area of ​​the current area, and moving to the upper left adjacent area of ​​the current area.

[0177] The reward function includes: the reward for the drone to finally reach the destination Penalty for drones crossing the border p , the negative reward for the drone consuming energy in the nth area The reward R calculated by the drone in the nth auxiliary area c[n], the UAV calculates the reward R of unloading tasks in the nth region t [n] = ξ * C[n], the UAV distance change reward R d = η * Δ d ;

[0178] In order to assist the calculation when the UAV has sufficient energy and approach the destination when the UAV is insufficient, a coefficient is added when designing the reward function, wherein R t [n], R c [n] includes E[n] / E[1], R d includes (E[1]-E[n]) / E[1], ξ is a first coefficient, ∈ is a second coefficient, μ is a third coefficient, and Δ d is the distance change between the UAV and the destination position.

[0179] The UAV calculates the reward R of assisting the region in the nth region c [n], specifically: if the nth region is successfully assisted in calculation, a fixed positive reward is obtained, otherwise, if the time is not saved, it indicates that the decision is improper, and a fixed negative reward is returned.

[0180] Step S6: a double deep Q network is used to solve the Markov decision process, and an optimal UAV task unloading and path planning strategy satisfying the constraint condition is obtained.

[0181] In the task unloading and trajectory planning decision, the UAV generates an optimal UAV task unloading and path planning strategy satisfying the constraint condition based on a deep reinforcement learning framework by real-time perception of the grid environment, and decides the moving direction and the calculation method (local calculation or unloading to the UAV) of the task node in the next step. In the action execution phase, the UAV moves and completes the corresponding task according to the decision result, and simultaneously updates the state information in real time, including the position, the remaining energy and the remaining task information in the environment, etc. These feedbacks are taken as the inputs of the reinforcement learning algorithm, and in the network training, the parameters of the strategy network are continuously optimized through priority experience sampling and loss calculation, so as to gradually improve the quality of the decision. In addition, by setting the punishment for the out-of-bound of the UAV, the reward for the UAV finally reaching the destination position, the UAV distance change reward, etc., it is ensured that the UAV can efficiently complete the auxiliary calculation and reach the target position.

[0182] The standard deep Q network uses the same network for action selection and Q value estimation, which is easy to cause overestimation bias, because each time the max operation is estimated, the maximum Q value in the current estimation is selected, however, since the network prediction has errors, these errors may sometimes cause the Q value of a non-optimal action to be overestimated, therefore, the double deep Q network (Double DQN) is adopted in the present application, and a new neural network is introduced as a target network relative to the deep Q network, specifically as follows:

[0183] Preferably, the solving the Markov decision process by using the double deep Q network comprises:

[0184] Step S601: initializing parameters θ of the Q network, parameters θ ′ of the target network, the noise network parameter, the capacity I of the experience replay pool, the discount factor γ ∈ [0, 1), the priority intensity parameter α, the weight correction parameter β, the constant ∈, the preset batch size m, the training step number T, the current training number k, the maximum training number K, and the initial state s0 of the UAV;

[0185] Step S602: initializing the time step index t as 1;

[0186] Step S603: obtaining the current state s t of the UAV; t inputting the Q network to obtain Q values corresponding to all actions, selecting an action with a maximum Q value as a preliminary action, generating noise by using the noise network, and obtaining an execution action a t based on the preliminary action and the noise;

[0187] The traditional ∈-greedy strategy controls the exploration degree by a preset exploration value. Due to random selection of actions, the direction of policy learning may fluctuate greatly, especially in some key states, random exploration may cause the agent to deviate from the learning goal, and controllable noise exploration of the noise network can better balance the stability and exploration demand of learning, and the noise network introduces noise in the neural network weight, so that the network dynamically adapts to the exploration behavior in different environments. A learnable noise term is added to the weight w and the bias b:

[0188]

[0189] where w μ is the mean part of the weight w, w σ is the standard deviation part of the weight w, ⊙ is element-level multiplication, ∈ w is a noise sample applied to the weight w, b μ is the mean part of the bias b, b σ is the standard deviation part of the bias b, and ∈ b is a noise sample applied to the bias b.

[0190] Step S604: the UAV executes the selected action a t , state transition is performed according to the system model of the target area, the UAV movement model, the task model, the UAV energy consumption model and the state transition probability, to obtain a reward function value r t and the next state s t+1 of the UAV, and an experience pool sample (s t , at ,r t ,s t+1 ) into the experience pool;

[0191] Step S605: determining whether the number of experience pool samples is less than I, if less than I, t=t+1, returning to execute step S603 until the number of experience pool samples is greater than or equal to I;

[0192] Step S606: when the number of experience pool samples is greater than or equal to I, calculating the TD error of each experience pool sample based on the reward function value of each experience pool sample, the discount factor γ, the Q value corresponding to the executed action, and the maximum Q value among all actions obtained by the target network in the next state of the UAV;

[0193] The formula for calculating the TD error of each experience pool sample is:

[0194]

[0195] Wherein, δ t is the TD error of the experience pool sample corresponding to the time step t, Q(s t ,a t ; θ) is the predicted Q value of the experience pool sample corresponding to the time step t, is the target Q value of the experience pool sample corresponding to the time step t, is the maximum Q value among all actions obtained by the target network in the next state of the UAV.

[0196] Step S607: based on the TD error of each experience pool sample, the priority intensity parameter α, and the constant ∈, calculating the priority of each experience pool sample, according to the priority of each experience pool sample, taking out the experience pool samples of a preset batch size m, and calculating the sampling frequency of each taken-out experience pool sample;

[0197] The priority p t of each experience pool sample is related to its TD error δ t , and the priority p t of each experience pool sample is calculated by the formula: p t = (δ t + ∈) α The calculation formula of the sampling frequency P(t) of each taken-out experience pool sample is: P(t) = p t / ∑ e p e , p t is the experience pool sample corresponding to the time step t, and pe is the experience pool sample corresponding to the time step e;

[0198] Step S608: based on the sampling frequency of each experience pool sample taken out, the weight correction parameter β, the capacity I of the experience replay pool, the importance sampling weight of each experience pool sample taken out is calculated, and the importance sampling weight formula is ω t is the importance sampling weight of the experience pool sample corresponding to the time step t; according to the importance sampling weight, the TD error of each experience pool sample is corrected, and the formula is: Loss t =ω t (δ t ) 2 , Loss t is the TD error of the experience pool sample corresponding to the time step t after correction;

[0199] Since the priority experience replay is non-uniform sampling, the experience pool samples with large TD errors will be sampled more frequently, resulting in that the contribution of these experience pool samples to Q value update is excessively enlarged. In order to correct the sampling bias and ensure that the algorithm can still unbiasedly estimate the performance of the strategy, the present application introduces importance sampling correction. The core idea is to adjust the weight of each experience pool sample according to the sampling probability of the sample, so as to rebalance the contribution of different samples to learning.

[0200] Step S609: based on the TD error of each experience pool sample after correction, the gradient descent method is used to update the parameters θ of the Q network; the updating mode is: wherein, θ new is the updated parameter of the Q network, is the gradient of Q (s t , a t ; θ) with respect to θ, and σ is the learning rate;

[0201] Step S610: update the parameters θ' of the target network every T steps; the updating mode is: θ' new ←τ·θ new +(1-τ)·θ', wherein θ' new is the updated parameter of the target network, and τ is a coefficient for controlling the updating degree of the parameters of the target network;

[0202] Step S611: determine whether k is less than K, if yes, set k=k+1, return to execute step S602, and if not, the optimal unmanned aerial vehicle task offloading and path planning strategy meeting the constraint condition is obtained.

[0203] In this embodiment, in actual application, based on Example 1, a method for unmanned aerial vehicle task offloading and trajectory planning proposed by the present invention can be used to construct a deep reinforcement learning model, and a lightweight network structure can be used to adapt to scenarios with limited computing resources. By carrying an environmental perception module (such as a camera and a computing module), the unmanned aerial vehicle can perceive the task environment in real time. During the actual flight process, the environmental perception module captures task information (such as task volume and distribution) in real time, combines the energy and location of the unmanned aerial vehicle itself, and inputs the reinforcement learning model to generate a decision. According to the action instructions output by the model, the unmanned aerial vehicle adjusts the flight path and performs task offloading.

[0204] In this paper, a deep reinforcement learning framework, ETOU-DQN, is proposed for path planning and task offloading optimization for drones in energy-constrained scenarios. This framework uses a combination of dual deep Q networks (DQN), prioritized experience replay (PER), and noisy networks to optimize the drone's flight trajectory through reinforcement learning, while dynamically deciding whether to offload computing tasks to the drone or execute them locally. By sensing the drone's remaining energy and the distribution of surrounding computing tasks, the framework plans the optimal flight path and makes the best task offloading decisions while ensuring reasonable energy consumption.

[0205] The deep reinforcement learning framework ETOU-DQN (Energy-aware Trajectory and Offloading with DQN) is designed to maximize the amount of tasks and data processing that drones can complete under limited energy conditions. It not only optimizes the energy utilization of drones, but also improves the execution efficiency of complex task scheduling. It is suitable for dynamic task scheduling in drone-assisted edge computing and IoT scenarios. Specifically, at each decision moment, the state of the drone, including the coordinate representation of the drone's current position coordinates in the hexagonal grid, the energy and the characteristics of the directed acyclic graph task in the current area, is calculated. The input is fed into the policy network with the noisy network. In this state, the drone selects an action based on the output of the deep reinforcement learning framework: moving to a nearby grid or hovering in the current grid to execute a directed acyclic graph (DAG) task. If the drone chooses to fly, it can move to an adjacent grid, update its state, and restart the next round of decision-making. If the drone chooses to hover, it will execute the user-submitted directed acyclic graph (DAG) task within the current grid.

[0206] Specifically, the unmanned aerial vehicle determines whether each node in a directed acyclic graph (DAG) task is executed by the unmanned aerial vehicle or locally executed by a user according to a deep reinforcement learning framework. Through such dynamic task scheduling, the unmanned aerial vehicle can optimize the efficiency of task execution under limited energy. After the task is executed, the environment returns a reward function value according to the task completion, energy consumption and other factors, and the unmanned aerial vehicle stores the current experience as experience into an experience replay pool. Prioritized experience replay improves sampling efficiency by introducing a priority mechanism, that is, different priority weights are given according to the influence of the experience pool samples on the policy network, so that the experience with greater influence on the policy is preferentially sampled, thereby accelerating the update of the policy. At the same time, in order to avoid sampling bias, importance sampling correction is used to reweight the experience, thereby ensuring the convergence of the algorithm. During the training process, the unmanned aerial vehicle samples experience from the prioritized experience replay pool for updating network parameters. The target network in the double deep Q network is periodically updated to calculate stable target Q values and is used to update the parameters θ of the Q network. Through this process, the unmanned aerial vehicle optimizes its flight path and task offloading strategy while performing tasks by continuously learning from experience.

[0207] The present application combines double DQN, noise network and prioritized experience replay, so that the unmanned aerial vehicle can simultaneously optimize task scheduling and trajectory planning in a complex directed acyclic graph (DAG) task environment, and effectively balance exploration and utilization. The present application effectively solves the deficiencies of the prior art in task scheduling, path planning and energy management, has good task completion rate and end point arrival rate, and at the same time enhances the adaptability of the system in a complex environment.

[0208] The second embodiment provides an unmanned aerial vehicle task offloading and trajectory planning system, comprising:

[0209] A system model construction module is configured to model the target area as an MxM hexagonal grid, sequentially number each region according to the order in which the unmanned aerial vehicle passes through each region, and obtain a system model of the target area; wherein M is the set number of rows and columns.

[0210] An unmanned aerial vehicle movement model construction module is configured to construct an unmanned aerial vehicle movement model based on the system model of the target area, wherein the unmanned aerial vehicle movement model includes the flight speed of the unmanned aerial vehicle, the flight distance and flight time of the unmanned aerial vehicle between adjacent two regions, the position representation of the unmanned aerial vehicle in each region, and the region decision variable.

[0211] A task model construction module is configured to construct a task model, wherein the task model includes:

[0212] obtaining the time consumed by each task node in the current region based on the offloading decision variable of each task node in the current region, the time calculated by each node in the current region on the local device, the time calculated by each node in the current region on the UAV, and the transmission time of each node from the local device to the UAV;

[0213] obtaining the execution start time of each task node in the current region based on the directed acyclic graph of the current region and the transmission time between nodes in the current region due to the dependency relationship;

[0214] obtaining the completion time of each task node in the current region based on the time consumed by each task node in the current region and the execution start time of each task node in the current region;

[0215] obtaining the time for completing the directed acyclic graph of the current region based on the completion time of each task node in the current region and the execution start time of the first task node in the current region;

[0216] obtaining the data amount of the UAV for assisting in the calculation in the current region based on the offloading decision variable of each task node in the current region and the data amount size when each task node in the current region is calculated;

[0217] a UAV energy consumption model construction module, configured to construct a UAV energy consumption model, wherein the UAV energy consumption model comprises:

[0218] obtaining the total energy consumption of the UAV in the current region based on the region decision variable of the current region, the energy consumption of the UAV for calculating the directed acyclic graph of the current region, the energy consumption of the UAV for hovering in the current region, and the flight energy consumption of the UAV for flying from the previous region to the next region;

[0219] obtaining the energy of the UAV when entering the current region based on the total energy consumption of the UAV in the previous region and the energy of the UAV when entering the previous region,

[0220] obtaining the energy of the UAV when entering the current region based on the total energy consumption of the UAV in the previous region and the energy of the UAV when entering the previous region,

[0221] a deep reinforcement learning framework construction module, configured to construct an optimization function and a constraint condition of a UAV task offloading and path planning strategy based on a system model of a target region, a UAV movement model, a task model and a UAV energy consumption model, and describe the process of solving the UAV task offloading and path planning strategy as a Markov decision process, establish a state space, an action space and a reward function, and obtain a deep reinforcement learning framework;

[0222] a solving module, configured to solve the Markov decision process by using a double deep Q network, and obtain an optimal UAV task offloading and path planning strategy satisfying the constraint condition.

[0223] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0224] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0225] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0226] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0227] Obviously, the above-described embodiments are only examples and are not intended to limit the present application. Based on the above description, those skilled in the art can make other variations and modifications of the present application without departing from the present application. Thus, the present application is not limited to the embodiments described above, but rather only by the claims that follow.

Claims

1. A method for unmanned aerial vehicle task offloading and trajectory planning, characterized in that, The method comprises the following steps: Modeling the target area as an MxM hexagonal grid, numbering each area in turn according to the order in which the unmanned aerial vehicle passes through each area to obtain a system model of the target area; wherein M is the number of rows and columns set; Based on the system model of the target area, an unmanned aerial vehicle movement model is constructed, which includes the flight speed of the unmanned aerial vehicle, the flight distance and flight time of the unmanned aerial vehicle between adjacent two areas, the position representation of the unmanned aerial vehicle in each area, and the area decision variable; A task model is constructed, which includes: Based on the unloading decision variable of each task node of the current area, the time for each node of the current area to calculate locally on the local device, the time for each node of the current area to calculate on the unmanned aerial vehicle, and the transmission time of each node from the local device to the unmanned aerial vehicle, the time consumed for executing each task node of the current area is obtained; Based on the directed acyclic graph of the current area and the transmission time between nodes of the current area due to the dependency relationship, the execution start time of each task node of the current area is obtained; Based on the time consumed for executing each task node of the current area and the execution start time of each task node of the current area, the completion time of each task node of the current area is obtained; Based on the completion time of each task node of the current area and the execution start time of the first task node of the current area, the time for completing the directed acyclic graph of the current area is obtained; Based on the unloading decision variable of each task node of the current area and the data size when each task node of the current area calculates, the data size of the unmanned aerial vehicle for assisting calculation in the current area is obtained; a model of the energy consumption of the unmanned aerial vehicle is constructed, which includes: Based on the area decision variable of the current area, the energy consumption of the unmanned aerial vehicle for calculating the directed acyclic graph of the current area, the energy consumption of the unmanned aerial vehicle hovering in the current area, and the flight energy consumption of the unmanned aerial vehicle flying from the previous area to the next area, the total energy consumption of the unmanned aerial vehicle in the current area is obtained; Based on the total energy consumption of the unmanned aerial vehicle in the previous area and the energy of the unmanned aerial vehicle when entering the previous area, the energy of the unmanned aerial vehicle when entering the current area is obtained; Based on the system model of the target area, the unmanned aerial vehicle movement model, the task model and the energy consumption model of the unmanned aerial vehicle, an optimization function and constraint condition of the unmanned aerial vehicle task offloading and path planning strategy are constructed, and the process of solving the unmanned aerial vehicle task offloading and path planning strategy is described as a Markov decision process, a state space, an action space and a reward function are established, and a deep reinforcement learning framework is obtained; The Markov decision process is solved by using a double deep Q network, and the optimal unmanned aerial vehicle task offloading and path planning strategy satisfying the constraint condition is obtained. 2.The method of claim 1, wherein, The position of the unmanned aerial vehicle in each region is represented as follows: the region number of the last time the unmanned aerial vehicle enters in the flight process is N, the region number set of the target region is {1, 2, …, n, …, N}, the coordinate of the starting position q s of the unmanned aerial vehicle is (x1, y1, 0), the coordinate of the ending position q f of the unmanned aerial vehicle is (x N ,y N ,0), and the position q[n] of the unmanned aerial vehicle in the nth region is (x n ,y n ,H); wherein x n is the horizontal coordinate of the center point of the nth region, y n is the vertical coordinate of the center point of the nth region, and H is the height of the flight of the unmanned aerial vehicle. 3.The method of claim 1, wherein, The directed acyclic graph of the current area is constructed, which includes: Each task in the current area is taken as a task node, each task node includes the data size required for transmission of the task node from the local device to the unmanned aerial vehicle and the data size when calculating, and a task node set of the current area is constructed; If the data amount d generated by the jth task node is needed when the ith task node in the current region is calculated ji , a directed edge E ji (d ji ) from the jth task node to the ith task node is constructed, and a directed edge set of the current region is constructed based on all directed edges of the current region. Based on the task node set and the directed edge set of the current area, a directed acyclic graph of the current area is constructed.

4. The method of claim 1, wherein, After the UAV enters the nth region, according to the region decision variable of the nth region decides whether to hover to assist in calculating the task in the current region, when is 0, the UAV jumps to an adjacent region, when is 1, the UAV hovers to assist in calculating the task in the current region; When the unloading decision variable S of the i-th task node in the n-th region n,i When it is 0, the i-th task node in the n-th region is offloaded to the local device. When the offloading decision variable S of the i-th task node in the n-th region is n,i = 1, the i-th task node in the n-th area is unloaded to the drone, S n,i ∈{0,1}.

5. The method of claim 4, wherein, The calculation formula of the time consumed for executing each task node of the current area is: wherein, is the time consumed for executing the ith task node of the nth region, is the time calculated by the local device for the ith task node of the nth region, c n,i is the data size for the ith task node of the nth region to calculate, is the time calculated by the UAV for the ith task node of the nth region, f uav is the computing power of the UAV, is the transmission time for the ith task node of the nth region to unload from the local device to the UAV, w n,i is the data size required to be transmitted for the ith task node of the nth region to unload from the local device to the UAV, R trans is the rate at which the UAV receives and transmits data within the current region; The calculation formula of the execution start time of each task node in the current region is: wherein, is the start time of the i-th task node of the n-th region, max(.) is the maximum value, is the set of pre-task of the i-th task node of the n-th region, is the completion time of the j-th task node of the n-th region, is the transmission time of the j-th task node of the n-th region as the pre-task of the i-th task node, d ji is the data amount needed by the j-th task node to calculate the i-th task node of the n-th region, S n,j is the offloading decision variable of the j-th task node of the n-th region, S n,i is the offloading decision variable of the i-th task node of the n-th region; The calculation formula of the completion time of each task node in the current region is: wherein, is the completion time of the ith task node of the nth region; The time calculation formula of the directed acyclic graph for completing the current region is: wherein t total [n] is the time to complete the directed acyclic graph for the nth region, L[n] is the total number of task nodes for the nth region, is the execution start time of the first task node for the nth region; The data amount calculation formula of the UAV for assisting the calculation in the current region is: C[n] =∑ i∈L[n] S n,i c n,i , Wherein, C[n] is the data amount of the UAV for assisting the calculation in the nth region.

6. The method of claim 5, wherein, The formula of the total energy consumption of the UAV in the current region is: wherein, is the total energy consumption of the UAV in the nth region, is the region decision variable of the nth region, is the energy consumption of the UAV hovering in the nth region, is the energy consumption of the UAV computing the directed acyclic graph of the nth region, is the flight energy consumption of the UAV flying from the previous region to the next region; The formula of the energy of the UAV when entering the current region is: Wherein, E[n] is the energy of the unmanned aerial vehicle entering the nth region, E[n-1] is the energy of the unmanned aerial vehicle entering the n-1th region, is the total energy consumption of the unmanned aerial vehicle in the n-1th region; Energy consumption of the drone hovering in the nth region is: wherein p stay is the power for the drone to hover; The unmanned aerial vehicle calculates energy consumption of a directed acyclic graph of the nth region is: wherein, Pn,i is the energy consumption of the UAV for processing the i-th task node of the n-th region, exec Pn is the power of the UAV for computing the task, Pn,i is the energy consumption of the UAV for receiving the transmission from the i-th task node of the n-th region, rece Pn is the power of the UAV for receiving the transmission, Pn,j is the energy consumption of the UAV for the i-th task node caused by the j-th task node of the n-th region for the preceding task of the i-th task node Pn is the power of the UAV for transmitting, send Pn is the power of the UAV for transmitting, The flight energy consumption of the UAV flying from a previous area to a next area is: wherein p fly is the flight power of the UAV, t fly is the flight time of the UAV between two adjacent regions, P0 is the power of the UAV blade, P1 is the induced power of the UAV hovering, v0 is the average speed of the UAV rotor, U tip is the tip speed of the UAV blade, d0 is the fuselage drag ratio of the UAV, s is the solidity of the UAV rotor, A is the rotor area of the UAV, p is the air density, v fly is the flight speed of the UAV.

7. The method of claim 6, wherein, The optimization function and constraint condition of the constructed UAV task offloading and path planning strategy include: The constraint condition includes: wherein, a1 is a first weight coefficient, a1 is 1 if the UAV finally reaches the terminal position, and a1 is 0 if the UAV does not finally reach the terminal position, R qF is a reward for the UAV finally reaching the terminal position, a is a second weight coefficient, R p is a punishment for the UAV crossing the boundary, R p is a negative value, a2 is 1 if the UAV exceeds the boundary of the region during flight, and a2 is 0 if the UAV does not exceed the boundary of the region during flight, N is the region number last entered by the UAV during flight, a3 is a third weight coefficient, R t [n] is a reward for the UAV calculating the unloading task in the nth region, R t [n] = ξ × C[n], ξ is a first coefficient, R c [n] is a reward for the UAV assisting the region calculation in the nth region, q U [1] is the first region entered by the UAV, q S is the starting position of the UAV, q U [N] is the position of the last region entered by the UAV, N is the region number last entered by the UAV during flight, q F is the terminal position of the UAV, E[1] is the initial energy of the UAV in the first region.

8. The method of claim 7, wherein, The process of solving the UAV task offloading and path planning strategy is described as a Markov decision process, the state space, action space and reward function are established, and a deep reinforcement learning framework is obtained, including: The state space includes: coordinate representation of current position coordinates of the unmanned aerial vehicle in a hexagonal grid, energy and directed acyclic graph task features of the current region wherein the directed acyclic graph task features of the current region wherein, is a normalized adjacency matrix of the directed acyclic graph of the current region, is a normalized task node feature matrix of the directed acyclic graph of the current region, and I is a unit matrix; The action space includes: hovering in the current region, moving to the upper adjacent region of the current region, moving to the upper right adjacent region of the current region, moving to the lower right adjacent region of the current region, moving to the lower adjacent region of the current region, moving to the lower left adjacent region of the current region, and moving to the upper left adjacent region of the current region; The reward function includes: the reward of the UAV finally reaching the end position The punishment R of the UAV crossing the boundary p The negative reward of the UAV consuming energy in the nth region The reward R of the UAV in the nth region assisting the region calculation c [n], the reward R of the UAV calculating the unloading task in the nth region t [n]=ξ×C[n], the UAV distance change reward R d =η×Δ d ; wherein R t [n], R c The coefficient of R d The coefficient of R d is a first coefficient, ∈ is a second coefficient, η is a third coefficient, and Δ 9. The method of claim 8, wherein, The double deep Q network is used to solve the Markov decision process, including: Step S601: initialize the parameters θ of the Q network, the parameters θ' of the target network θ'=θ, the noise network parameters, the capacity I of the experience replay pool, the discount factor γ∈[0,1), the priority intensity parameter α, the weight correction parameter β, the constant ∈, the preset batch size m, the training step number T, the current training number k, the maximum training number K, and the initial state s0 of the UAV; Step S602: initialize the time step index t as 1; Step S603: obtaining the current state s of the UAV t The input Q network obtains the Q values corresponding to all actions, selects an action with the maximum Q value as a preliminary action, generates noise by using the noise network, and obtains an execution action a based on the preliminary action and the noise t ; Step S604: the UAV executes the selected action a t , according to the system model of the target area, the UAV movement model, the task model, the UAV energy consumption model and the state transition probability, state transition is performed to obtain a reward function value r t and the next state s t+1 of the UAV t , a t , r t , s t+1 ) into the experience pool; Step S605: determine whether the number of experience pool samples is less than I, if less than I, t=t+1, and return to execute step S603 until the number of experience pool samples is greater than or equal to I; Step S606: when the number of experience pool samples is greater than or equal to I, based on the reward function value of each experience pool sample, the discount factor γ, the Q value corresponding to the executed action, and the maximum Q value of all actions obtained by the target network in the next state of the UAV, the TD error of each experience pool sample is calculated; Step S607: based on the TD error of each experience pool sample, the priority intensity parameter α, and the constant ∈, the priority of each experience pool sample is calculated, and according to the priority of each experience pool sample, the experience pool samples of the preset batch size m are taken out, and the sampling frequency of each experience pool sample is calculated; Step S608: based on the sampling frequency of each taken-out experience pool sample, the weight correction parameter β, and the capacity I of the experience replay pool, the importance sampling weight of each taken-out experience pool sample is calculated; and according to the importance sampling weight, the TD error of each experience pool sample is corrected; Step S609: based on the corrected TD error of each experience pool sample, the gradient descent method is used to update the parameters θ of the Q network. Step S610: update the parameters θ of the target network every T steps ′ ; Step S611: judging whether k is less than K, if yes, setting k=k+1, returning to step S602, if not, obtaining the optimal unmanned aerial vehicle task offloading and path planning strategy satisfying the constraint condition.

10. An unmanned aerial vehicle task offloading and trajectory planning system, comprising: Comprise: The system model construction module is used for modeling the target area as an M*M hexagonal grid, numbering each region in turn according to the sequence of the unmanned aerial vehicle passing through each region, and obtaining a system model of the target area; wherein M is the set number of rows and columns; The unmanned aerial vehicle movement model construction module is used for constructing an unmanned aerial vehicle movement model based on the system model of the target area, wherein the unmanned aerial vehicle movement model comprises: the flight speed of the unmanned aerial vehicle, the flight distance and flight time of the unmanned aerial vehicle between adjacent two regions, the position representation of the unmanned aerial vehicle in each region, and the region decision variable; The task model construction module is used for constructing a task model, wherein the task model comprises: Based on the offloading decision variable of each task node of the current region, the time of each node of the current region calculated on the local device, the time of each node of the current region calculated on the unmanned aerial vehicle, and the transmission time of each node unloaded from the local device to the unmanned aerial vehicle, the time consumed for executing each task node of the current region is obtained; Based on the directed acyclic graph of the current region and the transmission time between the nodes of the current region due to the dependency relationship, the execution start time of each task node of the current region is obtained; Based on the time consumed for executing each task node of the current region and the execution start time of each task node of the current region, the completion time of each task node of the current region is obtained; Based on the completion time of each task node of the current region and the execution start time of the first task node of the current region, the time for completing the directed acyclic graph of the current region is obtained; Based on the offloading decision variable of each task node of the current region and the data size when each task node of the current region is calculated, the data size of the unmanned aerial vehicle for assisting calculation in the current region is obtained; the unmanned aerial vehicle energy consumption model construction module is used for constructing an unmanned aerial vehicle energy consumption model, wherein the unmanned aerial vehicle energy consumption model comprises: Based on the region decision variable of the current region, the energy consumption of the unmanned aerial vehicle for calculating the directed acyclic graph of the current region, the energy consumption of the unmanned aerial vehicle hovering in the current region, and the flight energy consumption of the unmanned aerial vehicle flying from the previous region to the next region, the total energy consumption of the unmanned aerial vehicle in the current region is obtained; Based on the total energy consumption of the unmanned aerial vehicle in the previous region and the energy of the unmanned aerial vehicle when entering the previous region, the energy of the unmanned aerial vehicle when entering the current region is obtained; The deep reinforcement learning framework construction module is used for constructing the optimization function and the constraint condition of the unmanned aerial vehicle task offloading and path planning strategy based on the system model of the target area, the unmanned aerial vehicle movement model, the task model and the unmanned aerial vehicle energy consumption model, describing the process of solving the unmanned aerial vehicle task offloading and path planning strategy as a Markov decision process, establishing a state space, an action space and a reward function, and obtaining a deep reinforcement learning framework; The solving module is used for solving the Markov decision process by using a double deep Q network, and obtaining the optimal unmanned aerial vehicle task offloading and path planning strategy satisfying the constraint condition.

Images