Unmanned aerial vehicle assisted MEC system mobile edge computing unloading time delay optimization method

Abstract

The invention provides an unmanned aerial vehicle auxiliary MEC system mobile edge computing unloading time delay optimization method. An unmanned aerial vehicle auxiliary mobile edge computing task unloading system model is established; establishing an optimization problem of minimizing the maximum time delay of a user unit UE under the system model, and minimizing the calculation time delay of all tasks by jointly optimizing user scheduling, a task unloading proportion, an unmanned aerial vehicle flight angle and an unmanned aerial vehicle operation memory as constraints; modeling the optimization problem into a Markov decision process problem, and solving the process problem by adopting a deep reinforcement learning mode; two AC networks are used in the algorithm, soft updating and delay updating strategies are adopted during parameter updating, and a behavior cloning algorithm and a softmax function are introduced to form an improved algorithm; and training the algorithm improved in the step S1 by using the system model made in the step S1 to obtain an experimental result. According to the method, the calculation unloading process can be effectively optimized, the time delay of the system is reduced, and the stability and convergence of the system are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer wireless communication technology, and in particular to a method for optimizing the mobile edge computing offloading delay of a drone-assisted MEC system. Background Art

[0002] In recent years, with the vigorous development of 5G technology and the popularization of computing-intensive applications running on user devices, people's lives have become more convenient. These mobile applications usually require a lot of computing resources and generate high energy consumption. Although the current user equipment is equipped with powerful processors, due to the volume limitations of the user unit UE, only small processors can be used, which leads to limited computing power and battery capacity, and cannot ensure that the computing power and experimental requirements of mobile applications are met, thus failing to provide satisfactory service quality. In order to overcome the above shortcomings, mobile edge computing is a new idea proposed in recent years. Mobile edge computing is an emerging technology that deploys MEC servers on a small cloud computing platform at the edge of the mobile network to support task-intensive applications, and offloads the UE's computing tasks to the cloud through the mobile network, which can greatly improve the UE's ability to perform computing tasks and greatly reduce the UE's energy consumption.

[0003] However, cloud servers are often far away from user units UE in space, which will result in high transmission delay and have a negative impact on user experience. In order to reduce the transmission delay of the backhaul link, the MEC server can be deployed closer to the user unit UE, thereby effectively reducing communication resources and energy consumption. However, in recent years, with the development of drone technology, drones have begun to be used as an auxiliary tool for mobile edge computing. With their flexible mobility and strong computing resources, drones can reduce communication delays while efficiently covering a wide area of ​​users. Drone-assisted mobile edge computing systems usually use drones to carry MEC servers to provide computing services to ground users. The mobility of drones allows them to dynamically adjust service areas and computing resources according to the user's location.

[0004] In traditional drone-assisted MEC systems, the offloading of computing tasks and the scheduling of resources mainly rely on preset algorithms for decision-making. With the development of intelligent technologies such as deep reinforcement learning, the human-machine collaboration model has begun to be introduced into the MEC system, that is, by combining user operations and machine computing power, more flexible and efficient computing resource management can be achieved. In this context, drone-assisted mobile edge computing systems can not only make optimization decisions in highly dynamic environments. Summary of the invention

[0005] The purpose of the present invention is to provide a method for optimizing the mobile edge computing offloading delay of a drone-assisted MEC system. By constructing a system model and state space and combining the flight and task offloading strategies of the drone, the method can effectively optimize the computing offloading process, reduce the system delay, and improve the stability and convergence of the system.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is: a method for optimizing the mobile edge computing offloading delay of a drone-assisted MEC system, comprising the following steps: S1: Establish a UAV-assisted mobile edge computing task offloading system model; S2: Establish an optimization problem of minimizing the maximum delay of the user unit UE under the system model, and minimize the computing delay of all tasks by jointly optimizing user scheduling, task offloading ratio, drone flight angle and drone running memory as constraints; S3: Model the optimization problem as a Markov decision process problem, and solve the process problem by deep reinforcement learning; use two sets of AC networks to replace the single AC network in the algorithm, and adopt soft update and delayed update strategies when updating parameters, introduce behavioral cloning algorithm and softmax function to form an improved algorithm; S4. Use the system model produced in S1 to train the improved algorithm in S1 to obtain experimental results.

[0007] Preferably, the system model in S1 includes K user units UE and a drone equipped with a MEC server, wherein the user units and the drone are initially randomly distributed in a rectangular area, and the drone and the user units move randomly.

[0008] Preferably, the optimization problem in S2 is expressed as: Among them, α k (i) is whether the kth user unit is unloaded in the i-th time slot, R k (i) is the unloading ratio of the kth user in the i-th time slot, f k (i) is whether there is a congestion between the kth user unit and the drone at the i-th time slot, X, Y, X k , Y k are the horizontal and vertical coordinates of the user unit and the UAV, respectively. H is the flight altitude of the UAV. min and H max are the minimum and maximum flight altitudes of the UAV, respectively. edge,k (i) and E battery are the energy consumed by the kth user unit during interaction and the total power of the drone at the i-th time slot, respectively. UAV (i) and are the running memory used by the drone in the i-th time slot and the maximum running memory of the drone, respectively.

[0009] Preferably, S3 comprises the following steps: S31. Two sets of AC networks with the same architecture are introduced into the algorithm. When taking the state-action value of the next state-action pair, the smaller value of the two Critic networks is used to estimate the state-action value of the next state-action pair. S32, when updating the target network, a soft update method is adopted, which gradually adjusts the parameters of the target network to smoothly approach the latest main network parameters; S33, introduce BC algorithm to directly learn the optimal strategy by imitating the expert's demonstration data; S34. Use the softmax function to convert the vector output by the policy network into a probability distribution.

[0010] Preferably, the S31 comprises the following steps: S311, use CriticNetwork1 and CriticNetwork2 to estimate the state-action value of all actions a' in the next state s', recorded as Q1(s',a') and Q2(s',a'); S312, take the smaller value of Q1(s',a') and Q2(s',a') as the final target value Q target (s',a'), the mathematical expression is as follows: Q target (s',a')=min(Q1(s',a'),Q2(s',a')) When a smaller Q value is selected through the dual critic network, the overestimation error that may be caused by a single critic network is avoided.

[0011] Preferably, the core idea of ​​the soft update in S32 is to partially update the parameters of the target network instead of completely replacing them every time. The update formula is: θ target ←τθ main +(1-τ)θ target where θ target represents the parameters of the target network, θ main Represents the parameters of the current main network. τ is a very small constant, and the value of τ is usually 0.005.

[0012] Preferably, in S33, a behavior cloning regularization term [π(s)-a) is added to the updating step of the standard strategy of the TD3 algorithm. 2 ], the original strategy of TD3 algorithm is: π old = argmaxΕ(s,a) ~ D [Q(s,π(s))]; Among them, (s,a)～D represents sampling of a state-action pair (s, a) from the experience replay pool or dataset D, where D is the set of states and actions experienced by the experience replay pool or strategy; Q(s, π(s)) is the Q-value function, which estimates the cumulative reward that can be obtained by executing strategy π(s) in state s; The strategy of TD3-BC-S with the behavior cloning regularization term is: new = argmaxΕ (s,a) ~ D [λQ(s,π(s))-(π(s)-a) 2 ]; where λ is a hyperparameter, and the solution formula for λ is as follows: in, is a constant or adjustable parameter, usually representing a certain ratio or weight. When its value is 2.5, the effect is more obvious; N represents the number of samples, that is, the total number of state-action pairs (s, a) used to calculate the average value.

[0013] Preferably, the softmax function in S34 is an activation function for multi-classification problems, which converts a vector containing unnormalized values ​​into a probability distribution, which encourages the policy network to explore different actions; the softmax function is as follows: Among them, x represents the vector output by the policy network, softmax(x) is a vector of the same dimension as x, and each element represents the probability of selecting the corresponding action.

[0014] The beneficial effects of the present invention are:

[0015] In order to prevent the problem of overestimation of Q value, this scheme uses a dual AC network structure in the algorithm, and adopts a soft update method when updating the AC network parameters. In order to improve the stability of the algorithm and enhance the overall performance of the algorithm, the BC algorithm is introduced. The vector output by the policy network is directly converted into a probability distribution using the softmax function, and the policy network is encouraged to explore different actions. Experiments show that the proposed algorithm can effectively reduce the user's latency, which not only improves the system performance and user experience, but also provides a solid technical foundation for large-scale applications and energy conservation and environmental protection. It has the characteristics of high accuracy, strong robustness, and low computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a workflow diagram in an embodiment of the present invention;

[0018] Figure 2 is a system model diagram proposed in an embodiment of the present invention;

[0019] Figure 3 is a model structure diagram of the TD3-BC-S algorithm in an embodiment of the present invention;

[0020] Figure 4 is a comparison diagram of the effects of the improved algorithm and the benchmark algorithm in an embodiment of the present invention;

[0021] Figure 5 This is a comparison chart of the effects of various algorithms under different task loads in an embodiment of the present invention;

[0022] Figure 6 is a comparison diagram of the effects of whether or not the input state is normalized in an embodiment of the present invention;

[0023] Figure 7 This is a comparison diagram of the effects of different learning rates in an embodiment of the present invention;

[0024] Figure 8 is a comparison diagram of effects under different discount factors in an embodiment of the present invention;

[0025] Fig. 9 It is a comparison diagram of effects under different delayed update times in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The present invention is further described in detail below in conjunction with embodiments and with reference to the accompanying drawings.

[0027] The present invention discloses a method for optimizing the time delay of mobile edge computing offloading in a drone-assisted MEC system. Figure 1 The embodiment includes the following steps: S1: Establish a UAV-assisted mobile edge computing task offloading system model; S2: Establish an optimization problem of minimizing the maximum delay of the user unit UE under the system model, and minimize the computing delay of all tasks by jointly optimizing user scheduling, task offloading ratio, drone flight angle and drone running memory as constraints; S3: Model the optimization problem as a Markov decision process problem, and solve the process problem by deep reinforcement learning; use two sets of AC networks to replace the single AC network in the algorithm, and adopt soft update and delayed update strategies when updating parameters, introduce behavioral cloning algorithm and softmax function to form an improved algorithm; S4. Use the system model produced in S1 to train the improved algorithm in S1 to obtain experimental results.

[0028] like Figure 2 As shown, the system model in S1 includes K user units UE and a drone equipped with a MEC server, wherein the user units and the drone are initially randomly allocated in a rectangular area with a side length of 100 m, and the drone and the user units move randomly.

[0029] The optimization problem in S2 is expressed as: Among them, α k (i) is whether the kth user unit is unloaded in the i-th time slot, R k (i) is the unloading ratio of the kth user in the i-th time slot, f k (i) is whether there is a congestion between the kth user unit and the drone at the i-th time slot, X, Y, X k , Y k are the horizontal and vertical coordinates of the user unit and the UAV, respectively. H is the flight altitude of the UAV. min and H max are the minimum and maximum flight altitudes of the UAV, respectively. edge,k (i) and E battery are the energy consumed by the kth user unit during interaction and the total power of the drone at the i-th time slot, respectively. UAV (i) and are the running memory used by the drone in the i-th time slot and the maximum running memory of the drone, respectively.

[0030] Constraints (1) and (2) restrict that only one UE performs computation offloading in time slot i. Constraint (3) indicates that the computation task offloading ratio is between 0 and 1. Constraint (4) indicates whether there is congestion between any UE and the UAV in any time slot. Constraints (5) and (6) restrict the geographic locations of the UAV and UE to be within the specified range. Constraint (7) restricts the flight altitude of the UAV to be within the specified range. Constraint (7) restricts the total power consumption of the UAV to be less than the power of the UAV itself. Constraint (9) restricts the size of the UAV's computation task in time slot i to be less than the maximum computation capacity of the UAV.

[0031] like Figure 3As shown, S3 includes the following steps: S31. Two sets of AC networks with the same architecture are introduced into the algorithm. When taking the state-action value of the next state-action pair, the smaller value of the two Critic networks is used to estimate the state-action value of the next state-action pair. S32. When updating the target network, a soft update method is adopted. The soft update gradually adjusts the parameters of the target network to smoothly approach the latest main network parameters, thereby improving the stability of the algorithm; S33, introduce the BC algorithm, which is a reinforcement learning method based on supervised learning, which directly learns the optimal strategy by imitating the expert's demonstration data; S34. Use the softmax function to convert the vector output by the policy network into a probability distribution.

[0032] The S31 comprises the following steps: S311, use CriticNetwork1 and CriticNetwork2 to estimate the state-action value of all actions a' in the next state s', recorded as Q1(s',a') and Q2(s',a'); S312, take the smaller value of Q1(s',a') and Q2(s',a') as the final target value Q target (s',a'), the mathematical expression is as follows: Q target (s',a')=min(Q1(s',a'),Q2(s',a')) When a smaller Q value is selected through the dual critic network, the overestimation error that may be caused by a single critic network is avoided.

[0033] The core idea of ​​the soft update in S32 is to partially update the parameters of the target network instead of completely replacing them every time. The update formula is: target ←τθ main +(1-τ)θ target where θ target represents the parameters of the target network, θ main Represents the parameters of the current main network. τ is a very small constant, and the value of τ is usually 0.005.

[0034] In S33, a behavior cloning regularization term [π(s)-a) is added to the update step of the standard strategy of the TD3 algorithm. 2 ], the original strategy of TD3 algorithm is: π old = argmaxΕ (s,a) ~ D [Q(s,π(s))]; Among them, (s,a)～D represents sampling of a state-action pair (s, a) from the experience replay pool or dataset D, where D is the set of states and actions experienced by the experience replay pool or strategy; Q(s, π(s)) is the Q-value function, which estimates the cumulative reward that can be obtained by executing strategy π(s) in state s; The strategy of TD3-BC-S with the behavior cloning regularization term is: new = argmaxΕ (s,a)～D [λQ(s,π(s))-(π(s)-a) 2 ]; where λ is a hyperparameter, and the solution formula for λ is as follows: in, is a constant or adjustable parameter, usually representing a certain ratio or weight. When its value is 2.5, the effect is more obvious; N represents the number of samples, that is, the total number of state-action pairs (s, a) used to calculate the average value.

[0035] The softmax function in S34 is an activation function for multi-classification problems, which converts a vector containing unnormalized values ​​into a probability distribution, which encourages the policy network to explore different actions; the softmax function is as follows: Among them, x represents the vector output by the policy network, softmax(x) is a vector of the same dimension as x, and each element represents the probability of selecting the corresponding action.

[0036] Step S4, using the system model produced in step S1 to train the improved algorithm to obtain experimental results, includes the following steps: S41. Use Python language to write drawing code and obtain the effect comparison chart between the improved algorithm and the benchmark algorithm, such as Figure 4 As shown. Figure 4 It can be seen that the TD3-BC-S algorithm is second only to the DDPG algorithm in terms of convergence speed, but is superior to the other algorithms in terms of convergence performance; S42, the comparison of the effects of various algorithms under different task volumes is shown in the figure Figure 5 As shown by Figure 5 It can be seen that when the amount of tasks that the user unit needs to calculate is different, the performance of the TD3-BC-S algorithm is better than the other benchmark algorithms. In addition, when the task scale gradually increases, the increase of the TD3-BC-S algorithm is the smallest, which is also the advantage of the algorithm; S43, the effect comparison of whether the input state is normalized is as follows Figure 6 As shown by Figure 6It can be seen that the performance when the input state is not normalized is lower than that when it is normalized. The reason is that the difference between the input values ​​without state normalization is too large, which makes it difficult for the agent to learn the relationship between the states, reducing the learning efficiency, and thus making the output value after DNN network training too high; S44, the effect comparison chart under different learning rates is as follows Figure 7 As shown by Figure 7 It can be seen that when the learning rate of the algorithm is 0.01, its convergence and stability are better than the other parameters. This is because a learning rate that is too high will cause the parameter update amplitude to be too large, making the training process unstable; a learning rate that is too low will make it difficult for the model to jump out of the local minimum during the training process, which indicates that the model has not converged stably during the training process; S45, the effect comparison chart under different discount factors is as follows Figure 8 As shown by Figure 8 It can be seen that when the discount factor of the algorithm is 0.1, its convergence and stability are better than the other parameters. The reason is that the lower the discount factor γ, the lower the value of future rewards, which makes the UAV more courageous to explore the environment, thus making the UAV try new actions and states; S46, the effect comparison chart under different delay update times is as follows Fig. 9 As shown by Fig. 9 It can be seen that when the number of delayed updates of the algorithm is 4, its convergence and stability are better than the other parameters. The reason is that the Critic network may not be fully stable yet, and frequent updates of the Actor network may lead to unstable strategies, which may easily cause oscillations during training and even fall into local optimal solutions.

[0037] It should be noted that the parts not described in detail in the above embodiments are all prior art.

[0038] The above description is only a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any technician familiar with the profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for optimizing the mobile edge computing offloading latency of a drone-assisted MEC system, characterized in that: The following steps are involved: S1: Establish a UAV-assisted mobile edge computing task offloading system model; S2: Establish an optimization problem of minimizing the maximum delay of the user unit UE under the system model, and minimize the computing delay of all tasks by jointly optimizing user scheduling, task offloading ratio, drone flight angle and drone running memory as constraints; S3: Model the optimization problem as a Markov decision process problem, and solve the process problem by deep reinforcement learning; Two sets of AC networks are used to replace the single AC network in the algorithm, and soft update and delayed update strategies are adopted when updating parameters. The behavioral cloning algorithm and softmax function are introduced to form an improved algorithm. S4. Use the system model produced in S1 to train the improved algorithm in S1 to obtain experimental results.

2. According to claim 1, a method for optimizing the mobile edge computing offloading delay of a drone-assisted MEC system is characterized by: The system model in S1 includes K user units UE and a drone equipped with a MEC server, wherein the user units and the drone are initially randomly allocated in a rectangular area, and the drone and the user units move randomly.

3. According to claim 1, a method for optimizing the mobile edge computing offloading delay of a drone-assisted MEC system is characterized in that: The optimization problem in S2 is expressed as: X,Y∈(0,L), H min ≤H≤H max , Among them, α k (i) is whether the kth user unit is unloaded in the i-th time slot, R k (i) is the unloading ratio of the kth user in the i-th time slot, f k (i) is whether there is a congestion between the kth user unit and the drone at the i-th time slot, X, Y, X k , Y k are the horizontal and vertical coordinates of the user unit and the UAV, respectively. H is the flight altitude of the UAV. min and H max are the minimum and maximum flight altitudes of the UAV, respectively. edge,k (i) and E battery are the energy consumed by the kth user unit during interaction and the total power of the drone at the i-th time slot, respectively. UAV (i) and are the running memory used by the drone in the i-th time slot and the maximum running memory of the drone, respectively.

4. According to claim 1, a method for optimizing the mobile edge computing offloading delay of a drone-assisted MEC system is characterized by: The S3 comprises the following steps: S31. Two sets of AC networks with the same architecture are introduced into the algorithm. When taking the state-action value of the next state-action pair, the smaller value of the two Critic networks is used to estimate the state-action value of the next state-action pair. S32, when updating the target network, a soft update method is adopted, which gradually adjusts the parameters of the target network to smoothly approach the latest main network parameters; S33, introduce BC algorithm to directly learn the optimal strategy by imitating the expert's demonstration data; S34. Use the softmax function to convert the vector output by the policy network into a probability distribution.

5. According to claim 1, a method for optimizing the mobile edge computing offloading delay of a drone-assisted MEC system is characterized by: The S31 comprises the following steps: S311, use CriticNetwork1 and CriticNetwork2 to estimate the state-action value of all actions a' in the next state s', recorded as Q1(s',a') and Q2(s',a'); S312, take the smaller value of Q1(s',a') and Q2(s',a') as the final target value Q target (s',a'), the mathematical expression is as follows: Q target (s',a')=min(Q1(s',a'),Q2(s',a')) When a smaller Q value is selected through the dual critic network, the overestimation error that may be caused by a single critic network is avoided.

6. The method for optimizing the mobile edge computing offloading delay of a drone-assisted MEC system according to claim 1 is characterized in that: The core idea of ​​the soft update in S32 is to partially update the parameters of the target network instead of completely replacing them every time. The update formula is: target ←τθ main +(1-τ)θ target where θ target represents the parameters of the target network, θ main Represents the parameters of the current main network. τ is a very small constant, and the value of τ is usually 0.

005.

7. The method for optimizing the mobile edge computing offloading delay of a drone-assisted MEC system according to claim 1 is characterized in that: In S33, a behavior cloning regularization term [π(s)-a) is added to the update step of the standard strategy of the TD3 algorithm. 2 ], the original strategy of TD3 algorithm is: π old = argmaxΕ (s,a) ~ D [Q(s,π(s))]; Among them, (s,a) ~ D represents sampling of a state-action pair (s, a) from the experience replay pool or dataset D, where D is the set of states and actions experienced by the experience replay pool or strategy; Q(s, π(s)) is the Q-value function, which estimates the cumulative reward that can be obtained by executing strategy π(s) in state s; The strategy of TD3-BC-S with the behavior cloning regularization term is: new = arg max E (s,a) ~ D [λQ(s,π(s))-(π(s)-a) 2 ]; where λ is a hyperparameter, and the solution formula for λ is as follows: in, is a constant or adjustable parameter, usually representing a certain ratio or weight. When its value is 2.5, the effect is more obvious; N represents the number of samples, that is, the total number of state-action pairs (s, a) used to calculate the average value.

8. The method for optimizing the mobile edge computing offloading delay of a drone-assisted MEC system according to claim 1 is characterized in that: The softmax function in S34 is an activation function for multi-classification problems, which converts a vector containing unnormalized values ​​into a probability distribution, which encourages the policy network to explore different actions; the softmax function is as follows: Among them, x represents the vector output by the policy network, softmax(x) is a vector of the same dimension as x, and each element represents the probability of selecting the corresponding action.

Images