An edge computing task offloading method based on deep reinforcement learning

Abstract

The application belongs to the field of edge computing, and discloses an edge computing task offloading method based on deep reinforcement learning, which comprises the following steps: A, establishing a system model; B, in a local computing mode, tasks are calculated on a device and energy is obtained at the same time; C, in an edge computing mode, a device offloads tasks to an edge node for calculation; D, modeling an offloading decision problem; and E, selecting an optimal offloading decision by adding a long short-term memory network (LSTM) layer in a DDPG algorithm model. The application adds a long short-term memory network (LSTM) layer in a DDPG main network. Due to the long short-term memory function of the LSTM network, the environment state of a period of time before the current environment state can be recorded, and then the recorded environment state and the detected current environment state are integrated, so that the current environment state can be more accurately obtained, and thus the system can make more accurate actions, generate the optimization strategy required by us, and improve the overall performance of the system.

Description

Technical Field

[0001] This invention belongs to the field of edge computing, and in particular relates to an edge computing task offloading method based on deep reinforcement learning. Background Technology

[0002] With the development of smart devices, the number of smart devices in our lives is increasing, leading to a rise in computing tasks. To reduce the computing pressure on smart devices and handle tasks that they cannot directly compute, edge computing (MEC) has emerged. An MEC architecture typically includes a cloud service center, multiple edge computing servers, and multiple smart devices. MEC technology usually deploys edge servers close to smart devices, referring to physical or network distance. Therefore, when a smart device has a task requiring computation, and its own computing power cannot meet the demand, the computation task can be offloaded to a nearby edge server. Edge servers can respond quickly to user service requests and have a significant advantage in reducing latency, making them ideal for applications with high latency requirements, such as live streaming and VR. As more researchers join edge computing research, the task offloading problem in edge computing has also seen significant progress. Previous research mostly focused on optimizing the allocation of computing and network resources. When the number of offloaded tasks is small, the system can simply select a nearby edge computing server to offload the task. When the computation task is too large and the edge server cannot meet the demand, the task is then offloaded to the cloud computing center, at which point the system essentially functions as cloud computing. When there are too many smart devices, the computing tasks also increase. At this time, some edge servers near multiple smart devices may face too many tasks being unloaded at the same time, resulting in excessive pressure and increased latency in task computing. Consequently, the overall system performance may decline due to improper allocation of resources.

[0003] In the current development scenario, the existence of most real-world applications means that edge servers also face a lot of offloading tasks, which can impact the performance of our entire system. Therefore, how to select the appropriate edge server for offloading different tasks in multi-scenario, multi-tasking environments has become a problem we need to consider. Solving this problem can improve the overall performance of the system. Summary of the Invention

[0004] The purpose of this invention is to provide an edge computing task offloading method based on deep reinforcement learning, so as to solve the technical problem of system performance degradation when the pressure of offloading tasks on edge servers is too high.

[0005] To address the aforementioned technical problems, the present invention provides a specific technical solution for an edge computing task offloading method based on deep reinforcement learning, as follows:

[0006] An edge computing task offloading method based on deep reinforcement learning sets the working environment to a wireless charging environment, where the device can obtain energy from an access point via a wireless receiver, and includes the following steps:

[0007] Step A: Establish a system model;

[0008] Step B: In local computing mode, tasks are computed on the device and acquire energy simultaneously;

[0009] Step C: In edge computing mode, the device offloads the task to the edge node for computation;

[0010] Step D: Unload decision problem modeling;

[0011] Step E: Select the optimal offloading decision by adding a Long Short-Term Memory (LSTM) layer to the DDPG algorithm model.

[0012] Furthermore, step A includes the following specific steps:

[0013] Step A1: An edge computing network (MEC) consisting of one access point and N fixed wireless devices is represented as a set N = {1, 2, ..., N}.

[0014] Step A2: Each device uses a time-division multiplexing circuit to avoid interference between communications;

[0015] Step A3: The system time is divided into consecutive time frames T of equal length;

[0016] Step A4: At each mark time, the energy stored by the wireless devices and the communication speed between them are related to the wireless channel gain;

[0017] Step A5: At the beginning of a time frame, the time aT is used for device charging, where a∈[0,1]; Step A6: The i-th device harvests energy: E i =μPh i aΤ, μ∈(0,1) represents the energy harvesting efficiency, P represents the energy transfer efficiency, h i It is the channel gain between the access point and device i;

[0018] Step A7: Each device uses the acquired energy to complete a computational task before the end of a time frame;

[0019] Step A8: w i w represents the weight assigned to device i; i The larger the value, the greater the computing power allocated to device i;

[0020] Step A9: Using a binary offloading strategy, this task is either computed locally or offloaded to an AccessPoint for computation. i =1 indicates user i Offload the computational task to Access Point, x i =0 indicates that the task is performed locally.

[0021] Furthermore, step B includes the following specific steps:

[0022] Step B1: Number of bits processed by the device: f i t i / φ,

[0023] f i Indicates the device's CPU computing speed.

[0024] t i This represents the calculation time for device i, where 0 ≤ t i ≤Τ,

[0025] φ represents the number of cycles required to process 1 bit of task data, φ>0;

[0026] Step B2: Energy consumption due to calculation is affected by k i f i 3 t i ≤E i The constraints, k i This indicates the calculation of the energy efficiency coefficient;

[0027] Step B3: Obtain the local computing speed from the above:

[0028]

[0029] Fixed parameters: Furthermore, step C includes the following specific steps:

[0030] Step C1: When the device has power, it offloads the task to the edge node for computation;

[0031] Step C2: Assuming the calculation and transmission rates of the Access Point are more than three orders of magnitude larger than those of the resource-constrained device, then ignoring the time spent by the Access Point on task calculation and download, we get:

[0032] Step C3: The device must exhaust its harvested energy during task unloading to achieve maximum computing speed.

[0033] at this time: Therefore, the calculation rate is:

[0034] B is the communication bandwidth, N0 is the receiver noise power, and τ is the signal strength. i T is assumed to be the unloading time of device i.

[0035] Furthermore, step D includes the following specific steps:

[0036] Step D1: Assume only the wireless channel gain h = {h i |i∈N} is time-varying. This parameter is used as the system environment variable input, while other parameters are fixed. Within a marked time frame, the system's computational rate is obtained:

[0037]

[0038] x={x i |i∈N}

[0039] τ={τ i |i∈N}

[0040] Step D2: For the channel gain h of each time frame, obtain the maximum rate-weighted sum:

[0041]

[0042]

[0043]

[0044] x i ∈{0,1}

[0045] If device i calculates locally, i.e., x i =0, from which we can deduce: τ i =0,

[0046] After obtaining the unloading decision x using the algorithm, the problem is represented as follows:

[0047]

[0048]

[0049]

[0050] Furthermore, step E includes the following specific steps:

[0051] Step E1: First, the environment outputs the environment state quantity to the actor at the t-th time frame;

[0052] Step E2: The environmental state is processed through the LSTM network layer to obtain the true environmental state, which is then input into the main network of the actor. The actor then performs action a in its current state.

[0053] Step E3: The actor network inputs the generated action 'a' into the critic, who gives a score and feeds it back to the actor. At the same time, the environment also receives action 'a' and provides feedback to bring the environment to the next state 'h'. t+1 ;

[0054] Step E4: After a period of training, the actor network and the critic network send a period of data to their respective target networks. Due to the setting of the target networks, the target values ​​of the actor network and the critic network will be more stable, so that the target values ​​can produce better optimization effects for the main network.

[0055] Step E5: Due to the setting of the experience replay pool, the actor network will use the data obtained from each training iteration (s) t ,a t ,r t ,s t+1 Input into the experience replay pool;

[0056] Step E6: Every so often, the system will randomly retrieve some data N*(s) from the experience replay pool. i ,a i ,r i ,s i+1 The inputs are fed into the actor network and the critic network respectively to update the parameters of the main network;

[0057] Step E7: After repeated iterations, obtain the optimal unloading action and generate the optimal unloading strategy π.

[0058] The edge computing task offloading method based on deep reinforcement learning proposed in this invention has the following advantages: This invention learns from past offloading experience to improve the offloading actions generated by DDPG through reinforcement learning. Compared to traditional methods, DDPG can solve the problem of reinforcement learning for continuous actions and continuous state spaces. However, unlike other DDPG methods, this invention adds a Long Short-Term Memory (LSTM) layer to the main DDPG network. Due to the LSTM network's long short-term memory function, it can record the environmental state a period of time prior to the current environmental state and integrate it with the currently detected environmental state to obtain a more accurate current environmental state. This allows the system to make more accurate actions, generate the desired optimization strategy, and improve the overall system performance. This method can provide excellent research results for the application of deep reinforcement learning in edge computing task offloading. Attached Figure Description

[0059] Figure 1 This is a diagram of the edge computing task offloading system architecture for deep reinforcement learning according to the present invention.

[0060] Figure 2 This is a flowchart of the edge computing task offloading method for deep reinforcement learning according to the present invention. Detailed Implementation

[0061] To better understand the purpose, structure, and function of this invention, the following detailed description of an edge computing task offloading method based on deep reinforcement learning is provided in conjunction with the accompanying drawings.

[0062] like Figure 1 Figure 2 As shown, the present invention provides an edge computing task offloading method based on deep reinforcement learning, setting the working environment to a wireless charging environment, wherein the device can obtain energy from an access point via a wireless receiver, and includes the following steps:

[0063] Step A: Establish a system model

[0064] a. An edge computing network (MEC network) consisting of one access point and N fixed wireless devices is represented as a set N = {1, 2, ..., N};

[0065] b. Each device uses time-division-multiplexing circuitry to avoid interference between communications;

[0066] c. The system time is divided into consecutive time frames T of equal length;

[0067] d. For each tag time, the energy stored by the wireless devices and the communication speed between them are both related to the wireless channel gain;

[0068] e. At the beginning of a time frame, the time aT is used for device charging, where a∈[0,1];

[0069] f. Energy harvested by the i-th device: E i =μPh i aΤ, μ∈(0,1) represents the energy harvesting efficiency, P represents the energy transfer efficiency, h i It is the channel gain between the access point and device i;

[0070] g. Each device must complete a computational task before the end of a time frame using the energy it receives;

[0071] hw i This represents the weight assigned to device i; the larger it is, the greater the computational speed assigned to device i.

[0072] j. This invention considers a binary offloading strategy, where the task is either computed locally or offloaded to an Access Point for computation, x i =1 indicates user i Offload the computational task to Access Point, x i =0 indicates that the task is performed locally.

[0073] Step B. Local computing mode, in which tasks are computed on the device and require energy to be acquired simultaneously;

[0074] a. Number of bits processed by the device: f i t i / φ,

[0075] f i This indicates the device's iCPU computing speed (cycles / second).

[0076] t i This represents the calculation time for device i, where 0 ≤ t i ≤Τ,

[0077] φ represents the number of cycles required to process 1 bit of task data, φ>0;

[0078] b. Energy consumption due to calculation is affected by k i f i 3 t i ≤E i Constraints

[0079] k i This indicates the calculation of the energy efficiency coefficient;

[0080] c. From the above, the local computing rate (bits / second) can be obtained:

[0081]

[0082] Fixed parameters:

[0083] Step C. Edge computing mode, in which the device offloads the task to edge nodes for computation;

[0084] a. Our devices can offload computational tasks to edge nodes when they have power;

[0085] b. This invention assumes that the computation and transmission rates of the Access Point are much greater than those of the size and resource-constrained device (by more than three orders of magnitude), and that the time for the results downloaded to the device is much shorter than the time for the data to be unloaded to the edge server. Therefore, we ignore the time spent by the Access Point on task computation and download, and we can obtain:

[0086] c. Our equipment exhausts its harvested energy during task offload to achieve maximum computing speed, at which point: Therefore, the computation speed can be obtained as:

[0087] B is the communication bandwidth, N0 is the receiver noise power, and τ is the signal strength. i T is assumed to be the unloading time of device i.

[0088] Step D. Problem Modeling;

[0089] a. In our system, we assume that only the wireless channel gain h = {h i |i∈N} is time-varying; this parameter is input as a system environment variable, while other parameters are fixed. Within a marked time frame, we can obtain the system's computational rate:

[0090]

[0091] x={x i |i∈N}

[0092] τ={τ i |i∈N}

[0093] b. For the channel gain h of each time frame, we can obtain the maximum rate-weighted sum:

[0094]

[0095]

[0096]

[0097] x i ∈{0,1}

[0098] (If device i calculates locally, i.e., x) i =0, from which we can deduce: τ i =0)

[0099] After obtaining the unloading decision x through the algorithm, the problem can be represented as follows:

[0100]

[0101]

[0102]

[0103] This invention addresses the unloading decision problem. Traditional algorithms, when making unloading decisions based on environmental states, may fail to accurately capture the environmental state in complex situations. Therefore, my designed algorithm incorporates a Long Short-Term Memory (LSTM) layer into the DDPG algorithm model to better capture the environmental state.

[0104] Step E. Select the optimal unloading decision by adding a Long Short-Term Memory (LSTM) layer to the DDPG algorithm model.

[0105] We optimize the algorithm by using the DDPG algorithm from reinforcement learning. Compared to the DQN algorithm, which can only be used for discrete action acquisition, the DDPG algorithm has a significant advantage in acquiring continuous actions. For the environmental state h of each time frame... t =(s t ,r t ,s t+1In this model, actions are input into the actor policy network. DDPG is based on an actor-critic model with four neural networks: a policy network and a target policy network in the actor network, and a Q-network and a target Q-network in the critic network. The two target networks are built to stabilize the target value, thus improving the data optimization performance of the main network. When the policy network receives environmental variables, it formulates a corresponding policy, generating an action. This action is then output to the critic's Q-network to generate a Q-value. The actor's policy network then continuously optimizes its parameters based on the generated Q-value, bringing subsequent Q-values ​​closer to the optimal value. In this process, the actor is like a performer, and the critic is like a judge. The critic gives a score based on the actor's performance, causing the actor to gradually perform actions with higher scores. The environment is like the audience, and the judge adjusts its scoring strategy based on the audience's reactions, making the scoring strategy increasingly perfect. The generation of unloading actions relies on deep neural networks, characterized by their embedded parameters θ, such as the weights connecting hidden neurons. Due to the complexity of environmental states, it is usually difficult for the system to directly perceive the true and complete environmental state. Therefore, we add a Long Short-Term Memory (LSTM) layer to the actor network. The LSTM network structure has three gate structures: forget gate, input gate, and output gate. Through these three gate structures, the system can integrate long-term historical data, thus better combining the current known state information to deduce the actual environmental state.

[0106] a. In our method, we first output the environment state quantity from the environment to the actor at the t-th time frame;

[0107] b. The environmental state is processed through the LSTM network layer to obtain the true environmental state, which is then input into the main network of the actor. The actor then performs action a based on its current state.

[0108] c. The actor network inputs the generated action 'a' into the critic, who then scores it and feeds it back to the actor. Simultaneously, the environment also receives action 'a' and provides feedback to advance to the next state 'h'. t+1 ;

[0109] d. After a period of training, the actor network and the critic network send a period of data to their respective target networks. Due to the setting of the target networks, the target values ​​of the actor network and the critic network will be more stable, so that the given target values ​​can produce better optimization effects for the main network.

[0110] e. Due to the setting of the experience replay pool, the actor network will use the data obtained from each training session (s) t ,a t ,r t ,s t+1 Input into the experience replay pool;

[0111] f. Every so often, the system will randomly retrieve some data N*(s) from the experience replay pool. i ,a i ,r i ,s i+1 The parameters are fed into the actor and critic networks respectively to update the main network parameters, improve the utilization of experience, and accelerate convergence.

[0112] g. Through repeated iterations, we gradually obtain the optimal unloading action and generate the optimal unloading strategy π.

[0113] This invention learns from past unloading experience to improve the unloading actions generated by DDPG through reinforcement learning. While DDPG can address reinforcement learning problems for continuous actions and continuous state spaces compared to traditional methods, it differs from other DDPG methods by incorporating a Long Short-Term Memory (LSTM) layer into the main DDPG network. Due to the LSTM network's long short-term memory function, it can record the environment state a period of time prior to the current state and integrate it with the currently detected environment state to obtain a more accurate understanding of the current environment state. This allows the system to make more accurate actions and generate the desired optimization strategy. This method can yield excellent research results for the application of deep reinforcement learning in edge computing task unloading.

[0114] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A method for offloading edge computing tasks based on deep reinforcement learning, wherein the working environment is set as a wireless charging environment, and the device obtains energy from an access point through a wireless receiver, characterized in that, Includes the following steps: Step A: Establish a system model; Step A1: An edge computing network (MEC) consisting of one access point and N fixed wireless devices is represented as a set. ; Step A2: Each device uses a time-division multiplexing circuit to avoid interference between communications; Step A3: The system time is divided into consecutive time frames of equal length. ; Step A4: At each mark time, the energy stored by the wireless devices and the communication speed between them are related to the wireless channel gain; Step A5: At the beginning of a time frame, The amount of time is used for device charging, here ; Step A6: Each device harvests energy: , Indicates energy harvesting efficiency. Represents energy transfer efficiency. Access points and devices Channel gain between; Step A7: Each device uses the acquired energy to complete a computational task before the end of a time frame; Step A8: Indicates allocation to device The weights; The larger the size, the more it is allocated to the device. The greater the computational speed, the better; Step A9: Use a binary offloading strategy to either compute the task locally or offload it to an AccessPoint for computation. express Offload the computing task to Access Point. This indicates that the task is performed locally. Step B: In local computing mode, tasks are computed on the device and acquire energy simultaneously; Step B1: Number of bits processed by the device: , Indicates equipment CPU computing speed Indicates equipment Calculation time, , This represents the number of cycles required to process 1 bit of task data. >0; Step B2: Energy consumption due to calculations is affected Constraints This indicates the calculation of the energy efficiency coefficient; Step B3: Obtain the local computing speed from the above: Fixed parameters: ; Step C: In edge computing mode, the device offloads the task to the edge node for computation; Step C1: When the device has power, it offloads the task to the edge node for computation; Step C2: Assuming the calculation and transmission rates of the Access Point are more than three orders of magnitude larger than those of the resource-constrained device, then ignoring the time spent by the Access Point on task calculation and download, we get: , ; Step C3: The device exhausts its harvested energy during task unloading to achieve maximum computing speed. At this point: Therefore, the computation rate is: , For communication bandwidth, For receiver noise power, Assuming it is a device Uninstallation time; Step D: Unload decision problem modeling; Step D1: Assume only wireless channel gain It is time-varying. This parameter is input as a system environment variable, while other parameters are fixed. Within a marked time frame, the system's computational rate is obtained. Step D2: Channel gain for each time frame The maximum rate-weighted sum is obtained: If the device Local computing, i.e. ,infer: , Unloading decisions are obtained through algorithms. The problem is presented as follows: ； Step E: Select the optimal offloading decision by adding a Long Short-Term Memory (LSTM) network layer to the DDPG algorithm model; Step E1: First, in the... Each time frame outputs environmental state quantities from the environment to the actor; Step E2: The environmental state is processed through the LSTM network layer to obtain the true environmental state, which is then input into the main network of the actor. The actor then performs action a in its current state. Step E3: The actor network inputs the generated action 'a' into the critic, who provides a score and feeds it back to the actor. Simultaneously, the environment also receives action 'a' and provides feedback to advance to the next state. ; Step E4: After a period of training, the actor network and the critic network send a period of data to their respective target networks. Due to the setting of the target networks, the target values ​​of the actor network and the critic network will be more stable, so that the target values ​​can produce better optimization effects for the main network. Step E5: Due to the setting of the experience replay pool, the actor network will use the data obtained from each training session. Input into the experience replay pool; Step E6: Every so often, the system will randomly retrieve some data from the experience replay pool. The parameters are fed into the actor network and the critic network respectively to update the main network parameters; Step E7: Through repeated iterations, obtain the optimal unloading action and generate the optimal unloading strategy. .

Images