An edge computing offloading method based on deep reinforcement learning

Abstract

The present application belongs to the field of mobile edge network computing offloading, and particularly relates to an edge computing offloading method based on deep reinforcement learning. The present application firstly models the computing offloading problem in the mobile edge network as a multi-objective optimization problem, comprehensively considers the time delay and energy consumption, wherein the parameters to be optimized are offloading decisions, CPU frequency of devices and transmission power. The MOP is decomposed into a series of scalar optimization sub-problems by using the decomposition idea, then each sub-problem is modeled as a Markov decision process, and the model parameters of all sub-problems are cooperatively optimized according to the parameter transmission strategy based on the neighborhood and the DRL training algorithm. Further, the Pareto optimal solution of the problem is obtained.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of mobile edge network computing offloading, and particularly relates to an edge computing offloading method based on deep reinforcement learning. BACKGROUND

[0002] The application generated by the current user equipment is increasingly complex, and the requirement for computing power is also higher and higher, and mobile edge computing (MEC) is regarded as a reliable way to process such application requests. The computing offloading of tasks in MEC has always been a key problem, and perfect computing offloading decision and the corresponding optimal resource allocation scheme can greatly improve the service performance of MEC and user experience. In addition, in the problem solving process, how to effectively improve the computing efficiency to meet the low delay characteristics of MEC also has important research significance.

[0003] Multi-objective optimization problem (MOP) is a problem that exists in various fields in reality, and the concept of multi-objective optimization is that when multiple objectives need to be achieved in a certain situation, due to the internal conflict between the objectives, the optimization of each objective is at the expense of the degradation of other objectives, and each objective cannot be optimized at the same time, and each objective must have a weight. However, how to allocate such weights has become a hot research topic. The typical multi-objective problem solving idea is to model the multi-objective problem as an optimization problem of a numerical function.

[0004] As a powerful analysis tool capable of processing massive data, deep learning can extract features from complex environments, so it is very feasible to apply deep learning to the MEC computing offloading problem model. In addition, deep reinforcement learning, which is composed of deep learning and reinforcement learning, has the powerful analysis ability of deep learning and the complex exploration interaction ability of reinforcement learning, and can be used to solve problems in a relatively complex environment. In this paper, deep learning and deep reinforcement learning are applied to the field of MEC computing offloading, and according to the neighbor-based parameter transfer strategy and the DRL training algorithm, the Pareto front (PF) can be directly obtained through the trained neural network model. SUMMARY

[0005] The application proposes a computing offloading solving method (DDPG-OLA) based on a deep deterministic policy gradient algorithm and a neighbor-based parameter transfer strategy, which can effectively exert the advantages of deep reinforcement learning and quickly obtain the Pareto optimal solution. Compared with the existing parallel method, the method has better model performance and shorter solving time.

[0006] The solution of the application is: first, the computing offloading problem in the mobile edge network is modeled as a multi-objective optimization problem, considering the delay and energy consumption comprehensively, wherein the parameters to be optimized are offloading decision, CPU frequency of the device and transmission power. The MOP is decomposed into a series of scalar optimization sub-problems by using the decomposition idea, then each sub-problem is modeled as a Markov decision process, and the model parameters of all sub-problems are collaboratively optimized according to the neighborhood-based parameter transfer strategy and DRL training algorithm. Further, the Pareto optimal solution of the problem is obtained.

[0007] The specific steps of the application are:

[0008] Step 1: model the computing offloading problem as a multi-objective optimization problem, and the optimization objectives are delay and energy consumption:

[0009]

[0010]

[0011]

[0012] C3: 0≤p i ≤p max i∈N

[0013] C4: s i ∈{0,1}i∈N

[0014] wherein s i is the offloading decision, is the delay when the task is offloaded to the MEC server, is the delay of the local execution of the mobile device i, is the energy consumption when the task is offloaded to the MEC server, is the energy consumption of the local execution of the mobile device i; the constraint condition C1 indicates that the execution time of each task cannot exceed the allowed maximum delay C2 indicates that the local CPU working frequency of the device can only be dynamically adjusted within a limited range, C3 indicates that the transmission power of the device is p max , and C4 indicates that the offloading of the task is binary offloading.

[0015] Step 2: the problem is decomposed into multiple sub-problems by using the linear decomposition strategy;

[0016] Step 3: each sub-problem is modeled as a Markov decision process, and the DDPG algorithm in DRL is used to train the model of one sub-problem;

[0017] Step 4: further collaboratively optimize the model parameters of all sub-problems according to the neighborhood-based parameter transfer strategy;

[0018] Step 5: repeat step 3 and step 4 to train each device's task to obtain a trained model;

[0019] Step 6: use the trained model to solve the multi-objective optimization problem to obtain the Pareto front of the problem.

[0020] The beneficial effects of the present application are that the present application is an edge computing offloading method based on deep reinforcement learning, which uses a linear decomposition strategy to decompose a multi-objective optimization problem into a set of scalar optimization sub-problems, and solves the model parameters in a cooperative manner. Solving each scalar optimization problem can obtain a Pareto optimal solution, so when all sub-problems are solved, the expected PF can be obtained. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 The present application is a top-level structure diagram of the method;

[0022] Figure 2 The present application is a top-level structure diagram of the method; DETAILED DESCRIPTION

[0023] The following simulation is combined to prove the effectiveness and progress of the present application:

[0024] As shown in the flowchart of the present application, it specifically includes: Figure 1

[0025] Step 1: model the computing offloading problem as a multi-objective optimization problem, where the offloading decision considers binary offloading.

[0026] Step 1-1: consider the delay of local execution of mobile device i as The energy consumption is

[0027] Where c i is the number of CPU working cycles required for computing task execution, is the working frequency of the mobile device CPU, k is the hardware related coefficient, and its specific value is associated with the specific chip structure. Here, it is set to a constant value of 10 -26 .

[0028] Step 1-2: consider the delay of offloading tasks to the MEC server as The energy consumption is

[0029] Where d i is the data size of the computing task, f C is the working frequency of the server CPU, r i is the data transmission rate, according to Shannon's theorem, r i ​may be expressed as:

[0030]

[0031] where w is the bandwidth of the channel, p i denotes the transmission power, h i denotes the channel coefficient between mobile device i and small base station (SBS), σ 2 is the noise power during transmission.

[0032] Step 1-3: Modeling the problem as a multi-objective optimization problem and considering certain constraints, we can get:

[0033]

[0034] where s i is the offloading decision.

[0035] Step 1-4: Considering the constraints and the parameters to be optimized,

[0036]

[0037]

[0038] C3: 0 ≤ p i ≤ p max i∈N

[0039] C4: s i ∈{0,1} i∈N

[0040] where C1 represents that the execution time of each task cannot exceed its allowed maximum delay. C2 represents that the local CPU working frequency of the device can only be dynamically adjusted within a limited range, C3 limits the transmission power of the device, and C4 represents that the task offloading is binary offloading.

[0041] Step 2: Decompose the subtasks into a series of scalar subproblems through the method of linear weighted sum. Give a set of uniformly distributed weight vectors λ 1 ,…, λ N , such as (1, 0), (0.9, 0.1), …, (0, 1), where where M represents the number of optimization objectives, and in this method, the optimization objectives are energy consumption and delay, M takes 2. The objective function of the jth subproblem after decomposition is as follows:

[0042]

[0043] Step 3: In order to solve each subproblem through DRL, model the subproblem as a Markov decision process, and solve a subproblem through the DDPG algorithm.

[0044] Step 3-1: The state space, action space and reward value are set, the state space and action space are set to be consistent, that is, {unloading decision, CPU working frequency, transmission power}, and the reward value is set to the target function value of the current state minus the target function value of the next state.

[0045] Step 3-2: The model is trained using the DDPG algorithm.

[0046] Step 3-3: A simple forward propagation is performed to obtain the Pareto optimal solution of the sub-problem.

[0047] Step 4: The N scalar optimization sub-problems are solved in a collaborative manner through a neighborhood-based parameter transfer strategy.

[0048] Step 4-1: On the basis of step three, the trained model of the i-th sub-problem is taken out, and a small amount of training is added to obtain the model of the i-th sub-problem, because the weight vectors of the two problems are adjacent, therefore, it is feasible for the sub-problem to assist in solving through the knowledge of its adjacent sub-problem.

[0049] Step 5: The above steps are repeated to train and solve the task of each device.

[0050] Step 6: The trained model of each device is used for simple forward propagation to obtain the final approximate PF.

[0051] The edge computing offloading method based on deep reinforcement learning proposed in the application is applied to a single cell, that is, only one edge server is contained, it is assumed that the number of channels is not considered, each mobile device accesses the edge server in a single cell through OFDMA, therefore, there is no interference between devices.

[0052] In order to verify the correctness of the solution of the method, the application performs a simulation experiment. First, the weight is fixed to 0.8, the training batch is set to 1000, and the step is 100. As can be seen from the figure, for all the tested methods, the deep reinforcement learning DDPG algorithm can obtain a smaller calculation cost, therefore, it can be known that the method is feasible for solving the problem.

[0053] At the same time, in order to verify the feasibility of the method for solving the PF, the parameter transfer between models is considered, when the first model is trained, the weight is modified, the parameters of the previous model are read, and a small batch of training is performed, which is repeatedly performed. The finally obtained PF is shown in the figure, and it can be seen that the method is feasible for obtaining the PF.

Claims

1. An edge computing offloading method based on deep reinforcement learning, characterized in that, Includes the following steps: Step 1: Model the computational unloading problem as a multi-objective optimization problem, where the unloading decision considers binary unloading; Define mobile devices The latency for local execution is Energy consumption is ,in To calculate the number of CPU cycles required to execute a task, The operating frequency of the mobile device's CPU. This refers to the hardware correlation coefficient. Define the latency for offloading tasks to the MEC server as: Energy consumption is ,in To calculate the data size of the task, The operating frequency of the server CPU. For data transmission rate, according to Shannon's theorem, Represented as: ， in For the bandwidth of the channel, Indicates transmission power. Indicates mobile device Channel coefficients with small base stations (SBS), This refers to the noise power during transmission. The optimization objectives are set as latency and energy consumption, and the optimization problem is defined as follows: ， ， ， ， ， in , , To facilitate the offloading decision, constraint C1 states that the execution time of each task cannot exceed its maximum allowable latency. C2 indicates that the device's local CPU operating frequency can only be dynamically adjusted within a limited range; C3 indicates that the device's maximum transmission power is... C4 indicates that the task is unloaded as a binary unload; Step 2: Decompose the problem into multiple subproblems using a linear decomposition strategy; specifically, provide a set of uniformly distributed weight vectors. , Where M represents the number of optimization objectives; the decomposed th The objective functions for the subproblems are as follows: ， Step 3: Model each subproblem as a Markov decision process and train the model for one subproblem using the DDPG algorithm in DRL; including: Step 31: Set up the state space, action space, and reward value. The state space and action space are set up the same way, which is {unload decision, CPU operating frequency, transmit power}. The reward value is set as the objective function value of the current state minus the objective function value of the next state. Step 32: Train the model using the DDPG algorithm; Step 33: Perform forward propagation to obtain the Pareto optimal solution to the subproblem; Step 4: Further collaboratively optimize the model parameters of all subproblems based on a neighborhood-based parameter transfer strategy; specifically: Take out the first A model trained on the first sub-problem, after a small amount of further training, yields the second sub-problem. The model for each sub-problem is designed so that, since the weight vectors of the two problems are adjacent, the sub-problems are solved with the help of knowledge from their adjacent sub-problems. Step 5: Repeat steps 3 and 4 to train and solve the task for each device to obtain the trained model; Step 6: Use the trained model to solve the multi-objective optimization problem and obtain the Pareto front of the problem.

Images