Task offloading and safe navigation method for UAV-assisted MEC system based on SafeRL
By introducing adaptive Lagrangian multiplier and SAC algorithms in the drone assisted edge computing system, the user scheduling and drone mobility are optimized, and the balance between system utility and navigation safety is solved, achieving more efficient mission unloading and safe navigation.
Patent Information
- Application Number
- CN202311068460.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-23
- Publication Date
- 2025-01-10
- Estimated Expiration
- 2043-08-23
Smart Images
Figure CN117131317B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of edge computing, and relates to a task offloading and safe navigation method of a drone-assisted edge computing system, and specifically to a task offloading and safe navigation method of a drone-assisted MEC system based on SafeRL. Background Art
[0002] Traditional task offloading strategies often have a disadvantage, that is, once the parameters of the drone network change dynamically, these mathematical optimization algorithms have to be re-iterated, which is often inefficient when facing actual MEC systems with constantly changing user locations, propagation environment characteristics, etc. The intelligent offloading and navigation strategy through deep reinforcement learning (DRL) is very suitable for complex and dynamically changing drone network environments. This method first needs to transform the specific problem into an optimization problem based on a Markov process (MDP), aiming to find an optimal strategy that can maximize the expected cumulative return. After that, the agent is controlled by the reinforcement learning algorithm to interact with the drone-assisted edge computing system environment and optimize the task offloading decision, plan the flight trajectory, and reasonably allocate computing resources through previous experience, so as to minimize the system transmission delay.
[0003] However, for drone-assisted MEC systems, there are often some risk areas (such as nature reserves) within the flight range of drones, and drones are not recommended to fly in these areas for reasons of safety and privacy. Therefore, both the system utility and the safety of drones should be considered at the same time, so that drones can provide excellent unloading services while ensuring safe navigation as much as possible.
[0004] To solve this problem, a very direct approach is to introduce a penalty term in the reward function to penalize the drone's flight time in the risk area. This penalty term can be adjusted according to the severity of the risk area to reflect the risk area that the agent should avoid entering. In this way, the agent will receive negative rewards during training, which will encourage it to avoid entering these areas, ensuring the safety of the drone's navigation to a certain extent.
[0005] However, when considering risky areas within the drone's service range, even if the penalty term is introduced into the reward function, classic reinforcement learning algorithms often have difficulty in balancing system utility and navigation safety. A loose penalty term may not lead to an excellent risk-averse strategy, while a strict penalty may make the agent too conservative and unable to explore the environment. Summary of the invention
[0006] To solve the above problems, the present invention discloses a task offloading and safe navigation method for a drone-assisted MEC system based on SafeRL. On the basis of the SAC algorithm, an adaptive Lagrange multiplier is introduced to convert the constrained optimization problem into an unconstrained optimization problem, and then jointly optimize user scheduling, drone mobility and resource allocation strategies to maximize the system utility while minimizing the drone's task execution time in the risk area.
[0007] To achieve the above object, the technical solution of the present invention is as follows:
[0008] The task offloading and safe navigation method of the UAV-assisted MEC system based on SafeRL includes the following steps:
[0009] First, design the drone-assisted MEC system.
[0010] There is a UAV and M mobile terminal devices in the system. The system has the following settings: the flight altitude of the UAV is always on a plane with a height of H, and there are O restricted areas in the plane. Each time step t of the UAV is divided into flight time t fly (t) and the transmission calculation time t hover (t); The mobile terminal device can move randomly on the ground; The ground-to-air channel model between the UAV and the terminal device is provided by the 3GPP protocol R15. The entire system model can be divided into three parts: communication model, calculation model, and risk assessment model.
[0011] The communication model defines the following parameters:
[0012] (x(t), y(t), H) is the initial coordinate of the UAV at time step t; (x n (t),y n (t),0) is the initial coordinate of user n at time step t; B is the communication bandwidth; P up is the uplink transmission power; is the probability of NLoS between the UAV and user n at time step t; σ represents the noise power generated by the complex white noise channel; L represents the transmission loss, d n (t) represents the distance between the UAV and user n at time step t; g n (t) represents the channel gain between the UAV and user n at time step t; r n (t) is the wireless transmission rate between the UAV and user n at time step t,
[0013] Each variable is calculated by the following formula:
[0014]
[0015]
[0016]
[0017] The calculation model defines the following parameters:
[0018] D n (t) represents the computational data size of drone n at time step t; β n (t) represents the proportion of tasks that user n unloads from the drone server at time step t, s represents the number of CPU cycles required to process a unit byte; f MU represents the computing power of the user; v(t), alpha(t) represent the flight speed and direction of the drone at the time step; t fly Indicates the time used for flight in each time step; M UAV represents the payload of the drone; represents the local execution delay of user n at time step t; represents the transmission delay of user n writing part of the task to the drone at time step t; represents the execution time of the user offloading task on the drone at time step t; E fly (t) represents the flight energy consumption of the UAV at time step t; Represents the computing power of the server onboard the drone; Represents the computational energy consumption of the drone at time step t.
[0019] Each variable is calculated by the following formula:
[0020]
[0021]
[0022]
[0023]
[0024]
[0025]
[0026] The risk assessment model defines the following parameters. The model uses the mission execution time of the drone in the risk area as the risk measurement indicator. O represents the number of risk areas within the drone's flight range, and t fly_out (t) represents the flight time of the UAV outside the risk area at time step t, t fly_in (t) represents the flight time of the UAV in the risk area at time step t, and C(t) is used as an indicator to determine whether the UAV’s terminal position (x(t+1), y(t+1), H) at time step t is located in any risk area o (0 means outside and 1 means inside).
[0027] The mission time of the UAV in the risk area at time step t is expressed as
[0028] t risk (t) = t fly_in (t)+C(t)t hover (t)
[0029] In combination with the analysis of the above-mentioned drone-assisted MEC system model, the present invention constructs a constrained optimization problem involving user scheduling, drone mobility and resource allocation, aiming to minimize the maximum processing delay of all user devices while limiting the mission time of drones in the risk area, thereby achieving a balance between maximizing performance and minimizing cumulative risk. The constrained optimization problem is formalized as follows, where λ is the adaptive Lagrange multiplier and ∈ is a constant constraint threshold.
[0030]
[0031]
[0032]
[0033]
[0034]
[0035]
[0036]
[0037]
[0038] C1 and C2 ensure that only one user is scheduled for computation per time step. C3 ensures that users perform computations by partial offloading, with the offloading ratio of computational tasks between 0 and 1. C4 and C5 ensure that drones and users can only move within a given area. C6 ensures that the total consumption of drone flight and computation cannot exceed the maximum battery capacity. C7 states the number of tasks that must be completed during the entire time period.
[0039] The present invention obtains an intelligent strategy to solve this problem based on the secure reinforcement learning algorithm SAC-Lagrangian algorithm. For the four elements of secure reinforcement learning, namely state s(t), action a(t), reward r(t), and cost c(t), the following designs are performed respectively, where n(t) is the user selected by the drone for service at this time step, and α(t) is the task offloading ratio:
[0040]
[0041] a(t)=(n(t),α(t),v(t),β(t))
[0042]
[0043] c(t)=t risk (t).
[0044] For the secure reinforcement learning framework, at time step t, the agent makes a decision based on the current observed state through the strategy π and obtains the action a(t), which can be defined as follows, where Learn parameters for the policy:
[0045]
[0046] In order to adapt to the constrained optimization problem, the present invention not only defines the long-term discounted return R containing the entropy regularization term under the strategy π in the SAC algorithm π , and the long-term discounted cost is also defined, each expressed as follows, where γ is the discount factor and α is the temperature parameter used to control the importance of the entropy term:
[0047] According to the Bellman equation, the state value function and action value function can be expressed as:
[0048]
[0049]
[0050] Correspondingly, the state cost function and action cost function can be expressed as:
[0051]
[0052]
[0053] The CMDP problem in the solution used in the present invention can be interpreted as finding the optimal strategy π*, maximizing the long-term return and minimizing the long-term cost within the constraint boundary. Therefore, the constrained optimization problem can be expressed as:
[0054]
[0055]
[0056] The above problem can be solved by Lagrangian relaxation. The Lagrangian multiplier is introduced to transform it into an equivalent unconstrained problem. In each iteration, the strategy π and the Lagrangian multiplier λ are updated alternately:
[0057]
[0058] SAC-Lagrangian deep neural network learning strategy used in the present invention and two action-value functions Q θ1 and Q θ2 , and an action cost value function Q φ The training method of the two action value functions is similar to that of the SAC algorithm, using clipped double Q learning for network training. In addition, the SAC-Lagrangian algorithm also uses a replay buffer like other offline policy algorithms. The experience transitions of each round are stored in the replay buffer D, from which a batch of experience transitions are randomly sampled when updating the network parameters. The action value function and the action cost function can be approximately expressed as:
[0059]
[0060]
[0061] All Q functions (Q θ1 , Q θ2 and Q φ ) can be updated in advance by minimizing the Bellman residual, as follows, and is the parameter of the target network, which is trained by soft updating. It is worth mentioning that in order to ensure the process of sampling action is derivable, the present invention uses the reparameterization technique:
[0062]
[0063]
[0064] For policy network parameters The policy gradient update is as follows
[0065]
[0066] In addition, the Lagrange multiplier λ is updated at a slower rate. If the agent violates fewer constraints, the penalty coefficient will gradually decrease; otherwise, it will gradually increase. The update process is as follows:
[0067]
[0068] where η is the single Lagrangian multiplication of the update size, [χ] + Used to ensure that λ>0. Through continuous iterative updates, the optimal strategy π can be obtained * , thereby maximizing long-term returns while minimizing long-term costs.
[0069] The beneficial effects of the present invention are:
[0070] The present invention reduces the task execution time of drones in risk areas as much as possible by considering the notification of minimizing the total delay of system task execution, and constructs the problem as a constrained optimization problem. Using a new safety reinforcement learning algorithm, an adaptive penalty coefficient is introduced to transform the constrained optimization problem into an unconstrained optimization problem, and then jointly optimize user scheduling, drone mobility and resource allocation strategies. According to various comparative experiments, compared with existing solutions, the present invention is more outstanding in maximizing system utility and ensuring flight safety. BRIEF DESCRIPTION OF THE DRAWINGS
[0071] (In order to ensure objectivity in the experiment, the shape of the risk area is uniformly set to a circle)
[0072] Figure 1 Schematic diagram of the processing delay convergence performance of the four algorithms, SAC, SAC-penalty, SAC-Lagrangian and full local execution, when the time step is 1000, O=4, and R=65.
[0073] Figure 2 Schematic diagram of the convergence performance of drone risk area mission execution time of four algorithms, SAC, SAC-penalty, SAC-Lagrangian and full local execution, when the total time step is 1000, O=4, and R=65.
[0074] Figure 3 The flight trajectory of the UAV under the guidance of the optimal strategy when O=4 and R=65.
[0075] Figure 4 Comparison of processing delays of SAC, SAC-penalty, and SAC-Lagrangian at different risk zone radii when O=4.
[0076] Figure 5 Comparison of the UAV risk area mission execution time of SAC, SAC-penalty, and SAC-Lagrangian under different risk area radii when O=4. DETAILED DESCRIPTION
[0077] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.
[0078] As shown in the figure, the task offloading and safe navigation method of the drone-assisted MEC system based on SafeRL of the present invention includes the following steps:
[0079] S1: Design a drone-assisted MEC system.
[0080] There is a UAV and M mobile terminal devices in the system. The system has the following settings: the flight altitude of the UAV is always on a plane with a height of H, and there are O restricted areas in the plane. Each time step t of the UAV is divided into flight time and transmission calculation time; the mobile terminal device can move randomly on the ground; the ground-to-air channel model between the UAV and the terminal device is provided by the 3GPP protocol R15.
[0081] S2: Define the basic elements
[0082] (x(t), y(t), H) is the initial coordinate of the UAV at time step t; (x n (t),y n (t),0) is the initial coordinate of user n at time step t; B is the communication bandwidth; P up is the uplink transmission power; is the probability of NLoS between the UAV and user n at time step t; σ represents the noise power generated by the complex white noise channel; L represents the transmission loss. n (t) represents the distance between the UAV and user n at time step t; g n (t) represents the channel gain between the UAV and user n at time step t; r n (t) is the wireless transmission rate between the UAV and user n at time step t. n (t) represents the computational data size of drone n at time step t; β n (t) represents the proportion of tasks that user n unloads from the drone server at time step t, s represents the number of CPU cycles required to process a unit byte; f MU represents the computing power of the user; v(t), alpha(t) represent the flight speed and direction of the drone at the time step; t fly Indicates the time used for flight in each time step; M UAV represents the payload of the drone; represents the local execution delay of user n at time step t; represents the transmission delay of user n writing part of the task to the drone at time step t; represents the execution time of the user offloading task on the drone at time step t; E fly (t) represents the flight energy consumption of the UAV at time step t; Represents the computing power of the server onboard the drone; Represents the computational energy consumption of the drone at time step t.
[0083] S3: Define how to calculate important performance indicators
[0084]
[0085]
[0086]
[0087]
[0088]
[0089]
[0090]
[0091] t risk (t) = t fly_in (t)+C(t)t hover (t)
[0092] S4: This problem is summarized as a constrained optimization problem based on CMDP, which aims to maximize the long-term cumulative utility of the system while minimizing the mission execution time of the drone in the risk area. The problem is formalized as follows:
[0093]
[0094]
[0095]
[0096]
[0097]
[0098]
[0099]
[0100]
[0101] C1 and C2 ensure that only one user is scheduled for computation per time step. C3 ensures that users perform computations by partial offloading, with the offloading ratio of computational tasks between 0 and 1. C4 and C5 ensure that drones and users can only move within a given area. C6 ensures that the total consumption of drone flight and computation cannot exceed the maximum battery capacity. C7 states the number of tasks that must be completed during the entire time period.
[0102] S5: Based on the secure reinforcement learning framework, design the four elements of secure reinforcement learning, namely state, action, reward, and cost. The specific design is as follows:
[0103]
[0104] a(t)=(n(t),α(t),v(t),β(t))
[0105]
[0106] c(t)=t risk (t).
[0107] Where n(t) is the user that the drone chooses to serve at this time step, and α(t) is the task offloading ratio.
[0108] S6: The constrained problem is transformed into an unconstrained problem through the Lagrangian relaxation algorithm.
[0109]
[0110] S7: Use SAC-Lagrangian to solve this problem, deep neural network learning strategy and two action-value functions Q θ1 and Q θ2 , and an action cost value function Q φ The training method of the two action value functions is similar to the SAC algorithm, using double Q learning for network training. In addition, the SAC-Lagrangian algorithm also uses a replay buffer like other offline policy algorithms. The experience transitions of each round are stored in the replay buffer D, and a batch of experience transitions are randomly sampled from it when updating the network parameters. The action value function and the action cost function are approximately expressed as:
[0111]
[0112]
[0113] All Q functions (Q θ1 , Q θ2 and Q φ ) can be updated in advance by minimizing the Bellman residual, as follows, and is the parameter of the target network, which is trained by soft updating. It is worth mentioning that in order to ensure the process of sampling action is derivable, the present invention uses the reparameterization technique:
[0114]
[0115]
[0116] For policy network parameters The policy gradient update is as follows
[0117]
[0118] In addition, the Lagrange multiplier λ is updated at a slower rate. If the agent violates fewer constraints, the penalty coefficient will gradually decrease; otherwise, it will gradually increase. The update process is as follows:
[0119]
[0120] where η is the single Lagrangian multiplication of the update size, [χ] + Used to ensure that λ>0. Through continuous iterative updates, the optimal strategy π can be obtained * , thereby maximizing long-term returns while minimizing long-term costs.
[0121] It should be noted that the above content only illustrates the technical idea of the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.
Claims
1. Task offloading and safe navigation method of drone-assisted MEC system based on SafeRL, characterized by: The following steps are involved: S1: Design a drone-assisted MEC system, which contains one drone and M mobile terminal devices; S2: Calculate performance indicators, including the wireless transmission rate between the drone and user n at time step t, the local execution delay of user n at time step t, the flight energy consumption of the drone at time step t, the transmission delay of user n to offload part of the task to the drone at time step t, the execution time of the task offloaded by the user on the drone at time step t, the computing power of the server carried by the drone, the computing energy consumption of the drone at time step t, and the task time of the drone in the risk area at time step t; The risk assessment model defines the following parameters: The model uses the mission execution time of the drone in the risk area as the risk measurement indicator, t fly_in (t) represents the flight time of the UAV in the risk area at time step t, and C(t) is used as an indicator to determine whether the UAV’s terminal position (x(t+1), y(t+1), H) at time step t is located in any risk area; Time step t: UAV mission time in risk area t risk (t) is expressed as: t risk (t)=t fly_in (t)+C(t)t hover (t); S3: Constrained Optimization Formalization; The formalization described in S3 is to reduce the problem to a constrained optimization problem based on CMDP, as follows: C1 and C2 ensure that only one user is scheduled for computation at each time step; C3 ensures that the user performs computation by partial offloading, and the offloading ratio of the computation task is between 0 and 1; C4 and C5 ensure that the drone and the user can only move within a given area; C6 ensures that the total consumption of drone flight and computation cannot exceed the maximum battery capacity; C7 specifies the number of tasks that must be completed during the entire time period; S4: Based on the secure reinforcement learning framework, the four elements of secure reinforcement learning are designed, namely, state s(t), action a(t), reward r(t), and cost c(t). The specific design is as follows: a(t)=(n(t),α(t),v(t),β(t)) c(t)=t risk (t) Where n(t) is the user selected by the drone for service at time step t, and α(t) is the task offloading ratio; For the secure reinforcement learning framework, at time step t, the agent makes a decision through strategy π based on the current observed state and obtains action a(t), which is defined as follows: in Learn parameters for the policy; S5: The constrained problem is transformed into an unconstrained problem through the Lagrangian relaxation algorithm: S6: Use the SAC-Lagrangian algorithm to solve the optimal strategy, including: Using SAC-Lagrangian, deep neural network learning policy and two action-value functions Q θ1 and Q θ2 , and an action cost value function Q φ , the two action value functions use double Q learning for network training; in addition, the SAC-Lagrangian algorithm uses a replay buffer; the experience transformation of each round is stored in the replay buffer, and a batch of experience transformations are randomly sampled from it when updating the network parameters. The action value function and the action cost function are expressed as: Q θ1 , Q θ2 and Q φ The function is updated in advance by minimizing the Bellman residual as follows: and are the parameters of the target network, which are trained by soft updating; in order to ensure that the sampling action process is derivable, the reparameterization technique is used: For policy network parameters The policy gradient update is as follows: Among them, λ is the Lagrange multiplier, and the update process of λ is as follows: Where η is the single Lagrange multiplier update size, [ ] + Used to ensure λ>0; through continuous iterative updates, the optimal strategy π is finally obtained * , thereby maximizing long-term returns while minimizing long-term costs.
2. The task offloading and safe navigation method of the UAV-assisted MEC system based on SafeRL according to claim 1 is characterized by: S1 describes the design of a drone-assisted MEC system, as follows: The system has the following settings: The flight altitude of the drone is always on a plane with a height of H, and there are O restricted areas in the plane. Each time step t of the drone is divided into flight time and transmission calculation time; the mobile terminal device moves randomly on the ground; the ground-to-air channel model between the drone and the terminal device is provided by the 3GPP protocol R15; the entire system model is divided into three parts: communication model, calculation model, and risk assessment model.
3. The task offloading and safe navigation method of the UAV-assisted MEC system based on SafeRL according to claim 2 is characterized by: The communication model defines the following parameters: (x(t), y(t), H) is the initial coordinate of the UAV at time step t; (x n (t),y n (t),0) is the initial coordinate of user n at time step t; B is the communication bandwidth; P up is the uplink transmission power; P n NLoS (t) is the probability of NLoS between the UAV and user n at time step t; σ represents the noise power generated by the complex white noise channel; L represents the transmission loss; d n (t) represents the distance between the UAV and user n at time step t; g n (t) represents the channel gain between the UAV and user n at time step t; r n (t) is the wireless transmission rate between the UAV and user n at time step t; Each variable is calculated by the following formula:
4. The task offloading and safe navigation method of the UAV-assisted MEC system based on SafeRL according to claim 2 is characterized by: The computational model defines the following parameters: D n (t) represents the computational data size of the drone at time step t; β n (t) represents the proportion of tasks that user n unloads from the drone server at time step t, s represents the number of CPU cycles required to process a unit byte; f MU represents the computing power of the user; v(t) represents the flight speed of the drone at time step t; t fly Indicates the time used for flight in each time step; M UAV represents the payload of the drone; represents the local execution delay of user n at time step t; represents the transmission delay of user n offloading part of the task to the UAV at time step t; represents the execution time of the user offloading task on the drone at time step t; E fly (t) represents the flight energy consumption of the UAV at time step t; Represents the computing power of the server onboard the drone; represents the computational energy consumption of the drone at time step t; Each variable is calculated by the following formula: