A UAV-assisted edge offloading decision-making method based on deep reinforcement learning

Abstract

A drone-assisted edge offloading decision method based on deep reinforcement learning includes the following steps: setting S static edge servers and N drones carrying storage devices; the tasks on the server are averaged with λ i,t Poisson distribution arrives with a certain probability p i,t Offload tasks to the server for processing, while the remaining tasks are handled by drones. An optimization problem model is established with the goal of minimizing task completion time and computational cost. Deep reinforcement learning methods are used to solve this problem, resulting in the optimal drone allocation scheme and the offloading scheme between servers and drones. This method is suitable for mobile edge computing system scenarios where server computing tasks are unbalanced. It can rationally deploy drones based on the computational load of edge servers and make appropriate offloading decisions.

Description

Technical Field

[0001] The present invention belongs to the field of 5G network edge computing offloading technology, and in particular relates to a drone-assisted edge offloading decision-making method. Background Art

[0002] Mobile edge computing deploys computing resources near IoT devices, bringing cloud computing capabilities to the edge of IoT devices, effectively reducing computing latency and conserving energy consumption. Traditional edge servers are statically deployed. Regional and time-varying fluctuations in offloading workloads can overload some servers in certain areas and at certain times. Overly densely deployed edge servers can leave servers idle for large portions of the time, resulting in wasteful use. Dynamically deploying edge servers based on workload volume is a promising solution. Small drones are a practical option for commercial applications due to their ease of deployment, low acquisition and maintenance costs, high maneuverability, and hovering capabilities. Deploying computing resources on drones, which possess a certain level of computing power to process tasks, can collaborate with edge servers to serve IoT devices, achieving efficient offloading services. Therefore, utilizing drones to assist with edge offloading decisions—specifically, how to dynamically deploy drones and balance offloading decisions between drones and servers to reduce task completion time and drone usage costs—is a crucial research issue. Summary of the Invention

[0003] To minimize task completion time and drone usage costs, this paper introduces computing-capable drones into edge computing networks and designs a drone-assisted computation offloading method within a mobile edge computing environment. First, IoT devices offload all tasks to the nearest server. Second, computing resources are deployed on drones, scheduling the number of drones and assisting edge servers with excessive workloads. Finally, the task offloading ratio between drones and the supporting servers is adjusted to minimize task completion time and drone usage costs for the entire network system.

[0004] In order to achieve the above process, the present invention provides the following technical solutions:

[0005] A UAV-assisted edge offloading decision-making method based on deep reinforcement learning includes the following steps:

[0006] Step 1: Deploy S edge servers (referred to as servers) in the traffic network. Represents a collection of servers. The number of drones is N. The unit mission cost of the drone is cost d , the cost of the server computing unit task is cost sEach drone is deployed within the coverage of a server to assist the server in completing the offloading task. The deployment situation is expressed as G(t) = [a 1,t ,a 2,t ,a 3,t ,…,a S,t ], a i,t represents the number of drones assisting server i at time t;

[0007] Step 2: The task arrival of server i in each time slot t is subject to the mean value λ i,t Poisson distribution. We let the tasks arriving at server i be p i,t The probability of selecting server i to process is λ, then the arrival rate of server-side tasks is λ i,t p i,t , the arrival rate of the UAV-side tasks is λ i,t (1-p i,t ), note p t =[p 1,t ,p 2,t ,…,p S,t ];

[0008] Step 3: Use queuing theory to calculate the time the task spends on the server (queuing time + processing time) and the time the task spends on the drone, and establish an optimization problem P1 with the goal of minimizing the task completion time and calculating the task cost.

[0009] Step 4: Use the deep reinforcement learning algorithm to solve the mathematical problem P1 summarized in step 3, so as to obtain the drone assistance solution G(t) and the computing task offloading solution p in a single time slot. t .

[0010] Furthermore, in step 3, the expression for calculating the task completion time and the task cost is:

[0011]

[0012] Where λ is the arrival rate of computing tasks in all time slots, G is the deployment of drones in all time slots, p is the probability of unloading computing tasks in all time slots, ω1, ω2 and ω3 are weight coefficients, T i,t,s represents the time that the task stays on server i, T i,t,d,s Indicates the dwell time when the task is offloaded to the drone.

[0013] special where μ s The processing rate of the server. in is the length of the sending queue of the computing task on the server side, is the sending strength of the server, is the uplink transmission rate between the server and the UAV, w is the bandwidth between the UAV and the server, P i is the transmission power of server i, g i is the channel gain between server i and UAV, σ 2 is the noise between server i and the drone, is the length of the computing task queue on the drone side, where μ d is the processing rate of the drone.

[0014] Furthermore, in step 3, the expression of the restriction condition is:

[0015] ω1+ω2+ω3=1 (1)

[0016]

[0017] p i,t ≤1 (3)

[0018] λ i,t (1-p i,t )<μ tr (4)

[0019] λ i,t (1-P i,t )<μ d ×a i,t (5)

[0020] λ i ,tp i,t <μ s (6)

[0021] Among them, formula (1) is the constraint of the weight factor, formula (2) is the constraint of the number of auxiliary drones, formula (3) is the probability constraint of unloading, and formulas (4)(5)(6) are the constraints between the task arrival speed and processing speed in queuing theory.

[0022] Furthermore, in step 4, the steps of using the deep reinforcement learning algorithm to solve the optimization problem P1 in step 3 are as follows:

[0023] Step 4.1: Based on the independence between time slots and the independence between servers, transform the optimization problem P1 into problem P2;

[0024] P2:

[0025]

[0026] st(1)~(6)

[0027] Step 4.2: Initialize the network parameters θ of the two DNNs by random methods a,1 and θ p,1 , used to make deployment and uninstallation decisions. Initialize two empty memories to store historical experience gained after training.

[0028] Step 4.3: Select the SoftPlus function as the activation function of the hidden layer of the first DNN network and the SoftMax function as its output layer function. The arrival rate of tasks in each time slot is λ t As its input, its output is in Represented by the network parameters θ a,t In the first DNN network, for λ t Mapping function of state;

[0029] Step 4.4: Select ReLu as the activation function of the hidden layer of the second DNN network and the sigmoid function as the output layer function. As input, the output is in Represented by the network parameters θ p,t In the second DNN network, Mapping function of state;

[0030] Step 4.5: The above steps and is the relaxation vector, which is not necessarily the optimal decision for the corresponding state, so the λ at this moment is t and the relaxation vector and Arrange the elements in ascending order to get and Will Each element in and Traverse the elements in and calculate the minimum value of P2 Get the number of drones on each server G(t) * and uninstall decisions

[0031] Step 4.6: Replace the And the obtained G(t) * Add to the memory of the first DNN network, G(t) * and Add the second DNN to the Memory. If the Memory is full, use the latest G(t) * and Replace the oldest data in Memory. Whenever the number of time slots reaches the training interval δ, a batch of data is taken from Memory as a training sample set. and is a set of time indices;

[0032] Step 4.7: Use and Train two DNN networks separately and use the Adam algorithm to update and optimize the training parameters θ a,t and θ p,t ;

[0033] Step 4.8: Repeat steps 4.3 to 4.7 until the number of iterations reaches the limit value M;

[0034] Step 4.9: Use the trained deep reinforcement learning network to find the drone deployment and unloading plan for each time slot.

[0035] The present invention is applicable to mobile edge computing system scenarios with unbalanced server computing tasks. It can rationally deploy drones and make appropriate offloading decisions based on the computing task load of the edge servers. The beneficial effects of the present invention are mainly manifested in: it is applicable to mobile edge computing system scenarios with unbalanced server computing tasks. It can rationally deploy drones and make appropriate offloading decisions based on the computing task load of the edge servers and the computing resource situation, effectively reducing the task completion delay and computing task cost of the edge computing system. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a model diagram of task selection server calculation in the present invention.

[0037] Figure 2 It is a model diagram of the task selection UAV calculation of the present invention.

[0038] Figure 3 It is a flow chart of a specific embodiment of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described below with reference to the accompanying drawings.

[0040] Combine Figure 1 、 Figure 2 and Figure 3 ,A UAV-assisted edge offloading decision method based on deep reinforcement learning includes the following steps:

[0041] Step 1: If Figure 1 As shown, S edge servers (referred to as servers for short) are deployed. Represents the set of servers. The number of drones is N. The unit mission cost of the drone is cost d , the cost of the server computing unit task is cost s Each drone is deployed within the coverage of a server to assist the server in completing the offloading task. The deployment situation is expressed as G(t) = [a 1,t ,a 2,t ,a 3, t,…,a S,t ], a i,t represents the number of drones assisting server i at time t;

[0042] Step 2: The task arrival of server i in each time slot t is subject to the mean value λ i,t Poisson distribution. We let the tasks arriving at server i be p i,t The probability of selecting server i to process is λ, then the arrival rate of server-side tasks is λ i,t p i,t , the arrival rate of the UAV-side tasks is λ i,t (1-p i,t ), note p t =[p 1,t ,p 2,t ,…,p S,t ].

[0043] Step 3: When the task selection is processed on the drone side, a waiting queue will be formed, such as Figure 2 As shown in Figure 1, queuing theory is used to calculate the time a task spends on the server (queuing time + processing time) and the time a task spends on the drone. This allows us to establish an optimization problem P1 with the goal of minimizing the task completion time and calculating the task cost.

[0044] Furthermore, in step 3, the constraint expression is set as follows:

[0045] 1) Weight factor constraints. The weight factors ω1, ω2, and ω3 respectively represent the emphasis on task completion time, the cost of the drone computing task, and the cost of the server computing task in P1, and must satisfy ω1+ω2+ω3=0.

[0046] 2) UAV quantity constraint. A UAV can serve at most one server in a single time slot, so

[0047] 3) The task arrival rate is constrained by the server processing and sending rate and the drone processing rate. In order to ensure the steady state of the queue, the processing speed cannot be greater than the task arrival rate, so λ i,tp i,t <μ s ,λ i,t (1-p i,t )<μ tr and λ i,t (1-p i,t )<μ d ×a i,y .

[0048] Furthermore, in step 3, the calculation method for the task's stay time on the server or drone side is as follows:

[0049] 1) Average queue length of computing tasks on the server side. Using the M / M / 1 model of queuing theory, we can get the service intensity of the server side. μ s is the processing rate of the server, then the average queue length in the server is

[0050] 2) Average queue length of sending tasks on the server side. is the uplink transmission rate between the server and the UAV, w is the bandwidth between the UAV and the server, P i is the transmission power of server i, g i is the channel gain between server i and UAV, σ 2 is the noise between server i and the drone. is the intensity of the task sent by the server. Using the M / M / 1 model of queuing theory, the average queue length of the task sent by the server is

[0051] 3) Average queue length of computing tasks on the drone side. Using the M / M / C model of queuing theory, is the length of the computing task queue on the drone side, where μ d is the processing rate of the drone.

[0052] 4) Calculate the total cost of the task. The cost of the drone's unit task is cost d , the cost of the server computing unit task is cost s , the computation costs of the computing tasks on the server side and the drone side are and

[0053] Furthermore, in step 3, the expressions for task completion time and task cost are:

[0054] P1:

[0055]

[0056] st.

[0057] ω1+ω2+ω3=1 (1)

[0058]

[0059] p i,t ≤1 (3)

[0060] λ i,t (1-p i,t )<μ tr (4)

[0061] λ i,t (1-p i,t )<μ d ×a i,t (5)

[0062] λ i,t p i,t <μ s (6)

[0063] T i,t,s represents the time the task stays on server i, T i,t,d,s Indicates the dwell time when the task is offloaded to the drone, Among them, formula (1) is the constraint of the weight factor, formula (2) is the constraint of the number of auxiliary drones, formula (3) is the probability constraint of unloading, and formulas (4)(5)(6) are the constraints between the task arrival speed and processing speed in queuing theory.

[0064] Step 4: Use the deep reinforcement learning algorithm to solve the optimization problem P1 summarized in step 3, so as to obtain the drone assistance solution G(t) and the computing task offloading solution p for each time slot. t .

[0065] Furthermore, in step 4, the steps of using the deep reinforcement learning algorithm to solve the P1 optimization problem in step 3 are as follows:

[0066] Step 4.1: According to the independence between time slots and the independence between servers, the optimization problem P1 is transformed into problem P2.

[0067] P2:

[0068]

[0069] st(1)~(6)

[0070] Step 4.2: Initialize the network parameters θ of the two DNNs by random methods a,1 and θ p,1 , using network parameters to generate deployment and uninstallation decisions. Initialize two empty memories to store historical experience gained after training.

[0071] Step 4.3: Select the SoftPlus function as the activation function of the hidden layer of the first DNN network and the SoftMax function as its output layer function. The arrival rate of tasks in each time slot is λ t As its input, its output is in Represented by the network parameters θ a,t In the first DNN network, for λ t Mapping function of state;

[0072] Step 4.4: Select ReLu as the activation function of the hidden layer of the second DNN network and the sigmoid function as the output layer function. As input, the output is in Represented by the network parameters θ p,t In the second DNN network, Mapping function of state;

[0073] Step 4.5: The above steps and is the relaxation vector, which is not necessarily the optimal decision for the corresponding state, so the λ at this moment is t and the relaxation vector and Arrange the elements in ascending order to get and Will Each element in and Traverse the elements in and calculate the minimum value of P2 Get the number of drones on each server G(t) * and uninstall decisions

[0074] Step 4.6: Replace the And the obtained G(t) * Add to the memory of the first DNN network, G(t) * and Add the second DNN to the Memory. If the Memory is full, use the latest G(t)* and Replace the oldest data in Memory. Whenever the number of time slots reaches the training interval δ, a batch of data is taken from Memory as a training sample set. and is a set of time indices;

[0075] Step 4.7: Use and Train two DNNs separately and use the Adam algorithm to update and optimize the training parameters θ a,t and θ p,t ;

[0076] Step 4.8: Repeat steps 4.3 to 4.7 until the number of iterations reaches the limit value M;

[0077] Step 4.9: Use the trained deep reinforcement learning network to find the drone deployment and unloading plan for each time slot.

[0078] For example Figure 1 The scenario of S edge servers and N drones shown in FIG is used to illustrate a specific embodiment of the present invention, as shown in FIG. Figure 3 shown.

[0079] First, there are S edge servers and N drones in the scenario. All IoT devices in the scenario offload all the computing tasks generated to the nearest server.

[0080] Secondly, the task on the server is executed with a certain probability p i,t The server processes the data, and the rest is offloaded to the drone.

[0081] Then, based on queuing theory, the task's residence time when processed by the server (queuing time + processing time) and the task's residence time when processed by the drone are calculated, and an optimization problem P1 is established with the optimization goal of minimizing the task completion time and the cost consumed by the task calculation.

[0082] Then, the deep reinforcement learning method is used to solve the optimization problem P1, and the optimal drone number allocation plan G(t) and the unloading plan p between the server and the drone are obtained. t ;

[0083] Finally, the obtained UAV quantity allocation scheme G(t) and the unloading scheme p between the server and the UAV are t Deployed in the network.

[0084] The embodiments of this specification are merely examples of implementations of the invention and are provided for illustrative purposes only. The scope of protection of the present invention should not be considered limited to the specific embodiments described in these embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by a person of ordinary skill in the art based on the invention.

Claims

1. A UAV-assisted edge offloading decision-making method based on deep reinforcement learning, characterized in that: The steps include: Step 1: Deploy S edge servers (referred to as servers) in the traffic network. Represents a collection of servers; The number of drones is N; the unit mission cost of the drone is cost d , the cost of the server computing unit task is cost s Each drone is deployed within the coverage of a server to assist the server in completing the offloading task. The deployment situation is expressed as G(t) = [a 1,t ,a 2,t ,a 3,t ,…,a S,t ], a i,t represents the number of drones assisting server i at time t; Step 2: The task arrival of server i in each time slot t is subject to the mean value λ i,t Poisson distribution; let the tasks arriving at server i be p i,t The probability of selecting server i to process is λ, then the arrival rate of server-side tasks is λ i,t p i,t , the arrival rate of the UAV-side tasks is λ i,t (1-p i,t ), note p t =[p 1,t ,p 2,t ,…,p S,t ]; Step 3: Use queuing theory to calculate the time a task spends on the server and the time a task spends on the drone. The time spent on the server includes the queuing time plus the processing time. Optimization problem P1 is established to minimize the task completion time and calculate the task cost. The calculation method for the time a task spends on the server or drone is as follows: 1) Average queue length of computing tasks on the server side; using the M / M / 1 model of queuing theory, the service intensity on the server side can be obtained μ s is the processing rate of the server, then the average queue length in the server is 2) Average queue length of sending tasks on the server side; is the uplink transmission rate between the server and the UAV, w is the bandwidth between the UAV and the server, P i is the transmission power of server i, g i is the channel gain between server i and UAV, σ 2 is the noise between server i and the drone; is the intensity of the task sent by the server. Using the M / M / 1 model of queuing theory, the average queue length of the task sent by the server is 3) Average queue length of computing tasks on the drone side; using the M / M / C model of queuing theory, is the length of the computing task queue on the drone side, where μ d is the processing rate of the drone; 4) Calculate the total cost of the task; the cost of the drone's unit task is cost d , the cost of the server computing unit task is cost s , the computation costs of the computing tasks on the server side and the drone side are and Furthermore, in step 3, the expressions for task completion time and task cost are: T i,t,s represents the time the task stays on server i, T i,t,d,s Indicates the dwell time when the task is offloaded to the drone, Among them, formula (1) is the constraint of the weight factor, formula (2) is the constraint of the number of auxiliary drones, formula (3) is the probability constraint of unloading, and formulas (4)(5)(6) are the constraints between the arrival speed and processing speed of tasks in queuing theory; Step 4: Use the deep reinforcement learning algorithm to solve the mathematical problem P1 summarized in step 3, so as to obtain the drone assistance solution G(t) and the computing task offloading solution p in a single time slot. t The steps of using the deep reinforcement learning algorithm to solve the mathematical problem P1 summarized in step 3 are: Step 4.1: Based on the independence between time slots and the independence between servers, transform the optimization problem P1 into problem P2; Step 4.2: Initialize the network parameters θ of the two DNNs by random methods a,1 and θ p,1 , use network parameters to generate deployment and uninstallation decisions; initialize two empty memories to store historical experience gained after training; Step 4.3: Select the SoftPlus function as the activation function of the hidden layer of the first DNN network and the SoftMax function as its output layer function; the task arrival rate λ in each time slot t As its input, its output is in Represented by the network parameters θ a,t In the first DNN network, for λ t Mapping function of the state; Step 4.4: Select ReLu as the activation function of the hidden layer of the second DNN network and the sigmoid function as the output layer function; output the first DNN network As input, the output is in Represented by the network parameters θ p,t In the second DNN network, Mapping function of the state; Step 4.5: The above steps and is the relaxation vector, which is not necessarily the optimal decision for the corresponding state, so the λ at this moment is t and the relaxation vector and Arrange the elements in ascending order to get and Will Each element in and Traverse the elements in and calculate the minimum value of P2 Get the number of drones on each server G(t) * and uninstall decisions Step 4.6: Replace the And the obtained G(t) * Add to the memory of the first DNN network, G(t) * and Add the second DNN to the Memory; if the Memory is full, use the latest one G(t) * and Replace the oldest data in Memory; whenever the number of time slots reaches the training interval δ, a batch of data is taken from Memory as a training sample set and is a set of time indices; Step 4.7: Use and Train two DNNs separately and use the Adam algorithm to update and optimize the training parameters θ a,t and θ p,t ; Step 4.8: Repeat steps 4.3 to 4.7 until the number of iterations reaches the limit value M; Step 4.9: Use the trained deep reinforcement learning network to find the drone deployment and unloading plan for each time slot.

Images