Edge computing unloading strategy based on deep reinforcement learning

Abstract

The invention relates to the technical field of mobile edge computing, and particularly discloses an edge computing unloading strategy based on deep reinforcement learning, comprising the following steps: S1, establishing a computing framework composed of a plurality of terminal devices, a plurality of edge servers and a plurality of cloud servers; s2, modeling a cloud edge resource scheduling system; s3, calculating the system cost of the task; and S4, a calculation task unloading method. The method has the advantages that the task priority queue is established according to the emergency degree of the tasks, the high-priority tasks are processed preferentially, the system model is established based on the calculation unloading optimization target, the edge calculation unloading method based on the residual network and the D3QN algorithm is provided, the optimal task unloading strategy is obtained, and the task unloading efficiency is improved. According to the method, the burden of the cloud server is relieved by using a deep reinforcement learning method, data can be processed more quickly and reliably, and the energy consumption and the calculation delay are reduced while the task is unloaded.

Description

Technical Field

[0001] The present invention relates to the field of mobile edge computing technology, and in particular to an edge computing offloading strategy based on deep reinforcement learning. Background Art

[0002] With the advancement of Internet of Things (IoT) technology, the amount of data generated by different applications has increased dramatically. Traditional cloud computing has powerful computing, storage, network and other resources, but it still produces uncontrollable service delays in the face of surging data traffic, and the centralized processing model is difficult to meet the high performance requirements of users. Mobile Edge Computing (MEC) is regarded as an effective solution. It deploys computing and storage resources on edge devices closest to the data source, reduces the burden on cloud servers, can process data more quickly and reliably, and can greatly reduce network transmission delays. MEC enables interconnection between mobile smart devices by supporting heterogeneous device connections and providing computing power. The development of edge computing has promoted the development of a series of intelligent applications, such as intelligent manufacturing. These scenarios require real-time processing of large amounts of data and rapid response. Edge computing is an emerging technology that meets this demand.

[0003] In mobile edge computing, computing offloading refers to the technology that the terminal device transfers part or all of the computing tasks to the edge computing environment to solve the deficiencies of mobile devices in terms of resource storage, computing performance and energy efficiency. One of the benefits of doing so is that it reduces the network delay from the terminal to the cloud and improves the response speed of the application. In addition, by utilizing the computing power of edge devices, the load on the cloud server can be reduced and energy consumption can be reduced. The key to task offloading is to choose a suitable task allocation strategy. When allocating tasks, factors such as the computing power, storage capacity and network bandwidth of edge devices need to be considered. At the same time, it is also necessary to reasonably allocate tasks based on the characteristics of the tasks and the requirements for latency. Task offloading also needs to consider the collaboration between devices. By allocating tasks to multiple edge devices for simultaneous execution, latency can be further reduced and system performance can be improved. In addition, collaborative execution can also increase the reliability and fault tolerance of the system. When an edge device fails, other devices can take over its tasks to ensure the continuous operation of the application.

[0004] During the task offloading process, edge devices usually have lower computing power and energy supply. Task offloading may lead to high energy consumption and high latency problems in edge devices. Therefore, for mobile edge computing systems, how to reduce energy consumption and computing latency while offloading tasks is an important challenge. Currently, most computing offloading strategies are carried out under the condition of overall task offloading, and only consider a single indicator of latency or energy consumption, without combining the two for optimization. To this end, the present invention takes the weighted sum of latency and energy consumption of task processing as the optimization target, and proposes an edge computing offloading strategy based on deep reinforcement learning. Summary of the invention

[0005] The present invention proposes an edge computing offloading strategy based on deep reinforcement learning to solve the above problems. By optimizing the computing resource allocation of local, edge servers and cloud servers, the optimal task offloading strategy is formulated for the computing task offloading system in each time slot. The optimization goal is to minimize the task processing delay and energy consumption. The deep reinforcement learning method is used to establish a computing offloading model based on the residual network and the D3QN algorithm, and the optimal task offloading strategy is found to minimize the delay and energy consumption caused by the offloading task.

[0006] The technical solution of the present invention is achieved in this way:

[0007] An edge computing offloading strategy based on deep reinforcement learning includes the following steps:

[0008] S1. Establish a computing architecture consisting of several terminal devices, several edge servers, and several cloud servers. The set of user terminal devices is represented by N = {N1, N2, N3...N n} indicates that N i represents the terminal device of the i-th user, and the edge server is represented by M = {M1, M2, M3...M m} indicates that M j Denotes the jth edge server, and the cloud server is represented by C = {C1, C2, C3...C c} means that C k represents the kth cloud server. The terminal device connects to the base station through a wireless network and offloads its tasks to the edge server. The edge server and the cloud server are connected through a wireless network;

[0009] S2. Modeling of cloud-edge resource scheduling system. In each time slot, the task vector generated by all terminal devices is U = {U1, U2, U3...U n}, each terminal device N i The computational task to be performed is modeled as a triple U i =(D i ,C i ,S i), i∈[1,n],

[0010] Where D i Indicates the task input data size, C i It indicates the task computation amount of the user terminal device, that is, the number of CPU cycles required to calculate the task, and the number of basic CPU operations required to complete the task. i Indicates the task priority. The task priority is calculated according to the remaining time of the task, and urgent tasks are given priority for calculation;

[0011] S3, calculate the system cost of the task, the unloading decision of the terminal device task in time slot t is Indicates that the user offloads the task to the edge server. Indicates that the user offloads the task to the cloud server. This means that the task is processed locally on the terminal device, and the latency and energy consumption of the task on the terminal device, edge server, and cloud server are calculated respectively. The system cost is calculated by the weighted sum of latency and energy consumption. The calculation formula is:

[0012] Z(t)=α·T(t)+β·E(t),

[0013] (α,β≥0)∧(α+β=1)

[0014] Among them, α and β represent hyperparameters set according to the offloading task types with different requirements, T(t) represents the total processing delay of the task, E(t) represents the total energy consumption of processing the task, and the minimum Z(t) offloading strategy is taken;

[0015] S4, computing task offloading method,

[0016] S4.1. Convert the task computation offloading problem into a Markov decision process and define the Markov decision process (MDP) as: MDP = (S, A, P, R, γ)

[0017] Among them, S is the state space; A is the action space; P is the state transfer function, P(s t+1 |s t ,a t ) indicates the state is s t Execute action a t Then it changes to state s t+1 The probability of; R is the reward function; γ is the discount factor, γ = 1 means that the future reward is the same as the current reward, and γ < 1 means that the future reward will be discounted over time;

[0018] S4.2. Establish a computational offloading model based on the residual network and D3QN algorithm, and continuously update the model parameters through the gradient descent method to gradually reduce the loss function, thereby finding a strategy with the minimum loss function.

[0019] Furthermore, when calculating the system cost of the task in step S3, each user terminal N i The transmission rate is R, and the calculation formula is:

[0020]

[0021] where p i Indicates the transmission power between the user terminal and the server, which cannot exceed the maximum transmission power p max , W represents the channel bandwidth, lb is the logarithm with base 2, N0 represents the channel noise power, the interference between users is negligible, h i It represents the channel gain of the uplink between the user terminal and the server, and is calculated as follows:

[0022]

[0023] Where l represents the link loss constant, d0 represents the distance parameter, and d i represents the link transmission distance, and δ represents the link loss index;

[0024] when When the computing task is executed on the local terminal device, the computing delay of the task is The calculation formula is:

[0025]

[0026] Among them, C i Indicates the task computation amount of the user terminal, that is, the number of CPU cycles required to compute the task. Indicates the CPU frequency of the user terminal;

[0027] The energy consumption generated by executing the task on the local terminal device is:

[0028]

[0029] Where κ is the effective capacitance coefficient, which depends on the device structure;

[0030] when When the task is U, the computational delay includes the transmission delay of offloading the task to the edge server and the execution time of the task on the edge server. i Offload to edge server M j The transmission delay is:

[0031] The user terminal transmission rate R is expressed as:

[0032] Among them, p i,j represents the transmission power between the user terminal and the edge server, W i,j represents the channel bandwidth between the user terminal and the edge server, h i,j represents the channel gain of the uplink between the user terminal and the edge server;

[0033] Without considering the delay and energy consumption of the edge server back to the terminal, the transmission task U i The transmission energy consumption to the edge server is:

[0034]

[0035] Execution time of edge server processing tasks for:

[0036]

[0037] Among them, f i,j ∈[0,f max ] represents the frequency of the edge server CPU, f max is the maximum CPU frequency that the edge server can achieve.

[0038] The total execution time of the task offloaded to the edge server is:

[0039]

[0040] when When , the computation delay of the task only includes the transmission delay of the task from the terminal device to the cloud server, and the transmission delay of the task from the edge server to the edge server is:

[0041]

[0042] The energy consumption of transmitting tasks to the cloud server is:

[0043]

[0044] Among them, p i,k represents the transmission power between the user terminal and the cloud server, W i,k represents the channel bandwidth between the user terminal and the cloud server, h i,k Indicates the channel gain of the uplink between the user terminal and the cloud server.

[0045] Furthermore, the CPU frequency allocated by the server for processing tasks does not exceed the maximum CPU frequency, and when the terminal device offloads the task to the server for processing, its transmission power does not exceed the maximum transmission power.

[0046] Furthermore, the calculation of the unloading model in step S4.2 includes the following steps:

[0047] a. The state space and action space are used as the input of the model, and the input state first enters the residual network for processing;

[0048] b. The residual network extracts key features from the input state by stacking multiple convolutional layers and pooling layers;

[0049] c. After the residual network extracts features, the feature information enters the fully connected layer;

[0050] d. The fully connected layer is responsible for learning the mapping relationship from state to action, and finally outputs the Q value corresponding to each action;

[0051] e. The agent selects the optimal action based on the Q value;

[0052] f. Update the Q value through the D3QN algorithm based on the selected action and the reward obtained.

[0053] The operation method of finding the strategy with the minimum loss function in step S4 comprises the following steps:

[0054] Initialize the current Q network parameters θ, initialize the target network Q′ parameters θ′, and assign the Q network parameters to the Q′ network, θ→θ′, initialize the learning rate ζ, the discount factor γ,

[0055] Input: state space S, action space A,

[0056] Output: Task offloading decision.

[0057] Start the loop:

[0058] Step 1. Observe the current state s t , perform action a t ;

[0059] Step 2. According to different uninstall strategies Calculate the reward value R;

[0060] Step 3. Input the state space and action space, and use the residual network to extract key features and input them into the fully connected layer;

[0061] Step 4. The fully connected layer outputs the Q value corresponding to each action;

[0062] Step 5. Calculate the target Q value:

[0063] Step 6. Calculate the loss function:

[0064] Step 7. Update network parameters by gradient descent:

[0065] Step 8. Select the unloading strategy with the smallest loss function;

[0066] End the loop,

[0067] Step 9. The algorithm ends.

[0068] By adopting the above technical scheme, the beneficial effects of the present invention are as follows: by establishing a task priority queue according to the urgency of the task, high-priority tasks are processed first, and a system model is established based on the calculation offloading optimization target, an edge computing offloading method based on the residual network and the D3QN algorithm is proposed, and the optimal task offloading strategy is obtained, so that the delay and energy consumption generated by the offloaded task are minimized. The present invention reduces the burden on the cloud server by using a deep reinforcement learning method, and can process data more quickly and reliably, reducing energy consumption and reducing computing delays while offloading tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0070] Figure 1 It is a system architecture diagram of the present invention;

[0071] Figure 2 It is a model flow chart of the present invention. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] An edge computing offloading strategy based on deep reinforcement learning includes the following steps:

[0074] S1. Establish a computing architecture consisting of several terminal devices, several edge servers, and several cloud servers (such as Figure 1As shown in FIG. 1 , it is composed of multiple terminal devices and servers. The terminal devices are covered by densely deployed signal base stations. The set of user terminal devices is represented by N = {N1, N2, N3 ... N n} indicates that N i represents the terminal device of the i-th user, and the edge server is represented by M = {M1, M2, M3...M m} indicates that M j Denotes the jth edge server, and the cloud server is represented by C = {C1, C2, C3...C c} means that C k represents the kth cloud server. The terminal device connects to the base station through a wireless network and offloads its tasks to the edge server. The edge server and the cloud server are connected through a wireless network;

[0075] S2. Modeling of cloud-edge resource scheduling system. In each time slot, the task vector generated by all terminal devices is U = {U1, U2, U3...U n}, each terminal device N i The computational task to be performed is modeled as a triple U i =(D i ,C i ,S i ), i∈[1,n],

[0076] Where D i Indicates the task input data size, C i It indicates the task computation amount of the user terminal device, that is, the number of CPU cycles required to calculate the task, and the number of basic CPU operations required to complete the task. i Indicates the task priority. The task priority is calculated according to the remaining time of the task, and urgent tasks are given priority for calculation;

[0077] S3, calculate the system cost of the task, the unloading decision of the terminal device task in time slot t is Indicates that the user offloads the task to the edge server. Indicates that the user offloads the task to the cloud server. This means that the task is processed locally on the terminal device.

[0078] Each user terminal N i The transmission rate is R, and the calculation formula is:

[0079]

[0080] where p i Indicates the transmission power between the user terminal and the server, which cannot exceed the maximum transmission power p max, W represents the channel bandwidth, lb is the logarithm with base 2, N0 represents the channel noise power, the interference between users is negligible, h i represents the channel gain of the uplink between the user terminal and the server, which is an independent random variable and is calculated as follows:

[0081]

[0082] Where l represents the link loss constant, d0 represents the distance parameter, and d i represents the link transmission distance, and δ represents the link loss index;

[0083] when When the computing task is executed on the local terminal device, the computing delay of the task is The calculation formula is:

[0084]

[0085] Among them, C i Indicates the task computation amount of the user terminal, that is, the number of CPU cycles required to compute the task. Indicates the CPU frequency of the user terminal;

[0086] The energy consumption generated by executing the task on the local terminal device is:

[0087]

[0088] Where κ is the effective capacitance coefficient, which depends on the device structure;

[0089] when When the user's computing task is selected to be processed by the edge server, the computing task is first transmitted to the base station, and then the edge server executes the computing task. According to formula (1), the task U i Offload to edge server M j The transmission delay is:

[0090] The user terminal transmission rate R is expressed as:

[0091] Among them, p i,j represents the transmission power between the user terminal and the edge server, W i,j represents the channel bandwidth between the user terminal and the edge server, h i,j represents the channel gain of the uplink between the user terminal and the edge server, which is an independent random variable;

[0092] Without considering the delay and energy consumption of the edge server back to the terminal, the transmission task U iThe transmission energy consumption to the edge server is:

[0093]

[0094] Execution time of edge server processing tasks for:

[0095]

[0096] Among them, f i,j ∈[0,f max ] represents the frequency of the edge server CPU, f max is the maximum CPU frequency that the edge server can achieve.

[0097] The total execution time of the task offloaded to the edge server is:

[0098]

[0099] when When t is reached, the task is offloaded to the cloud server for processing. Since the computing power of the cloud server is much greater than that of the edge server and the terminal, if the terminal decides to offload the task to the cloud server for execution within time slot t, the computing delay of the task only includes the transmission delay of the task from the terminal device to the cloud server. According to formula (1), the transmission delay of the task offloaded to the edge server is:

[0100]

[0101] The energy consumption of transmitting tasks to the cloud server is:

[0102]

[0103] Among them, p i,k represents the transmission power between the user terminal and the cloud server, W i,k represents the channel bandwidth between the user terminal and the cloud server, h i,k Represents the channel gain of the uplink between the user terminal and the cloud server, which is an independent random variable.

[0104] The present invention is based on a mobile edge computing scenario with multiple users and multiple servers. Under the constraints of limited server computing resources and communication overload, by optimizing the computing resource allocation of local terminal devices, edge servers and cloud servers, an optimal task offloading strategy is formulated for the computing task offloading system in each time slot. The optimization goal is to minimize the task processing delay and energy consumption. The delay and energy consumption of task processing on the terminal device, edge server and cloud server are calculated respectively, and the system cost is calculated as the weighted sum of the delay and energy consumption.

[0105] Combining formulas (3), (8), and (9), the total processing delay of the task is:

[0106]

[0107] Combining formulas (4), (6), and (10), the total energy consumption of the processing task is:

[0108]

[0109] The computational delay of a task includes the transmission delay of offloading the task to the edge server and the execution time of the task on the edge server. The calculation formula is:

[0110] Z(t)=α·T(t)+β·E(t) (13)

[0111] (α,β≥0)∧(α+β=1)

[0112] Among them, α and β represent hyperparameters set according to the offloading task types with different requirements, T(t) represents the total processing delay of the task, E(t) represents the total energy consumption of processing the task, and the minimum Z(t) offloading strategy is taken;

[0113] The optimization target of the present invention is defined as:

[0114]

[0115] S4, computing task offloading method,

[0116] S4.1. Convert the task computation offloading problem into a Markov decision process and define the Markov decision process (MDP) as: MDP = (S, A, P, R, γ)

[0117] Among them, S is the state space; A is the action space; P is the state transfer function, P(s t+1 |s t ,a t ) indicates the state is s t Execute action a t Then it changes to state s t+1 The probability of a reward function; R is the reward function; γ is the discount factor, which is used to calculate the cumulative return of future rewards. γ = 1 means that the future reward is the same as the current reward, while γ < 1 means that the future reward will be discounted over time;

[0118] The detailed definitions are as follows:

[0119] State space: During task offloading, the state includes the channel gain between the terminal device and each server, the link transmission distance between the terminal and the server, and the task information.

[0120] Action space: Based on the observed environment state, all possible actions form the action space vector A. The action is the offloading operation of the task, expressed as

[0121] Reward function: In s t Execute action a in state t , the state transition is s t+1 , and obtain rewards from the environment, expressed as: R = -Z(t).

[0122] S4.2. Establish a computational offloading model based on residual network and D3QN algorithm. The model flow chart is as follows: Figure 2 The calculation of the unloading model includes the following steps:

[0123] a. The state space and action space are used as the input of the model, and the input state first enters the residual network for processing;

[0124] b. The residual network extracts key features from the input state by stacking multiple convolutional layers and pooling layers;

[0125] c. After the residual network extracts features, the feature information enters the fully connected layer;

[0126] d. The fully connected layer is responsible for learning the mapping relationship from state to action, and finally outputs the Q value corresponding to each action;

[0127] e. The agent selects the optimal action based on the Q value;

[0128] f. Update the Q value through the D3QN algorithm based on the selected action and the reward obtained.

[0129] Residual network (ResNet) is a deep convolutional neural network that effectively extracts key features from the input state by stacking multiple convolutional layers and pooling layers and using nonlinear activation functions.

[0130] The fully connected layer is responsible for learning the mapping relationship from state to action, and finally outputs the Q value corresponding to each action. The calculation formula of Q value is:

[0131]

[0132] Indicates that in state s t Next, perform action a t Q value.

[0133] and A(s t ,a t ; θ, φ) are the value function and advantage function respectively, θ is the parameter of the neural network, φ are the parameters of the value function and advantage function, The advantage function is centralized in order to solve the problem of Q value uniqueness. t represents the current state of the agent, a t represents the current action of the agent, a t+1 represents the next action of the agent, and |A| represents the number of optional actions of the agent.

[0134] D3QN (Dueling Double Deep Q Network) is an edge computing offloading method that combines the Double DQN (Deep Q Network) and Dueling DQN algorithms. Double DQN and Dueling DQN algorithms are two improved algorithms for DQN. Double DQN is designed to solve the problem that DQN uses only one network to select actions and calculate target Q values, which will cause the estimated Q value to be higher than the actual value. Dueling DQN is designed to solve the problem that the Q value output by the DQN algorithm is the average of all actions, and it is impossible to distinguish the pros and cons of different actions.

[0135] The D3QN algorithm uses a dual network structure, including the current network. The action value function of the current network is expressed as and the target network, the action value function of the target network is expressed as The current network is used to learn the Q value. The target network is used to calculate the target Q value and use the gradient descent algorithm to update the current network parameters.

[0136] The target Q value calculation formula is expressed as:

[0137]

[0138] Among them, R t+1 Indicates state s t+1 The reward obtained when , γ represents the discount factor, Indicates the action corresponding to the maximum Q value in the current Q value network, θ t represents the network parameters of the current Q-value network, and θ′ represents the network parameters of the target Q-value network.

[0139] Update the current network by minimizing the loss. The loss function is the mean square error between the target Q value and the current Q value network output Q value. The calculation formula is:

[0140]

[0141] Will Simplified to Q(θ), we get the loss gradient:

[0142]

[0143] Update network parameters by gradient descent:

[0144]

[0145] Among them, ζ represents the learning rate. The model parameters are continuously updated through the gradient descent method to gradually reduce the loss function, thereby finding the strategy with the smallest loss function, that is, the optimal unloading strategy.

[0146] In the process of computing task offloading, a quasi-static system in which the mobile terminal device does not switch access points is considered, that is, the composition of user devices remains unchanged during each time when the mobile device is offloaded. In this scenario, each user has a mobile device, and each mobile device has only one task to be processed at a time.

[0147] In this embodiment, the CPU frequency allocated by the server for processing tasks does not exceed the maximum CPU frequency, that is, When the terminal device offloads the task to the server for processing, its transmission power does not exceed the maximum transmission power, that is,

[0148] The operation method of finding the strategy with the minimum loss function in step S4 comprises the following steps:

[0149] Initialize the current Q network parameters θ, initialize the target network Q′ parameters θ′, and assign the Q network parameters to the Q′ network, θ→θ′, initialize the learning rate ζ, the discount factor γ,

[0150] Input: state space S, action space A,

[0151] Output: Task offloading decision.

[0152] Start the loop:

[0153] Step 1. Observe the current state s t , perform action a t ;

[0154] Step 2. According to different uninstall strategies Calculate the reward value R;

[0155] Step 3. Input the state space and action space, and use the residual network to extract key features and input them into the fully connected layer;

[0156] Step 4. The fully connected layer outputs the Q value corresponding to each action;

[0157] Step 5. Calculate the target Q value:

[0158] Step 6. Calculate the loss function:

[0159] Step 7. Update network parameters by gradient descent:

[0160] Step 8. Select the unloading strategy with the smallest loss function;

[0161] End the loop,

[0162] Step 9. The algorithm ends.

[0163] Components not described in detail herein are prior art.

[0164] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An edge computing offloading strategy based on deep reinforcement learning, characterized by: The following steps are involved: S1. Establish a computing architecture consisting of several terminal devices, several edge servers, and several cloud servers. The set of user terminal devices is represented by N = {N1, N2, N3...N n } indicates that N i represents the terminal device of the i-th user, and the edge server is represented by M = {M1, M2, M3...M m } indicates that M j Denotes the jth edge server, and the cloud server is represented by C = {C1, C2, C3...C c } means that C k represents the kth cloud server. The terminal device connects to the base station through a wireless network and offloads its tasks to the edge server. The edge server and the cloud server are connected through a wireless network; S2. Modeling of cloud-edge resource scheduling system. In each time slot, the task vector generated by all terminal devices is U = {U1, U2, U3...U n }, each terminal device N i The computational task to be performed is modeled as a triple U i =(D i ,C i ,S i ), i∈[1,n], Where D i Indicates the task input data size, C i It indicates the task computation amount of the user terminal device, that is, the number of CPU cycles required to calculate the task, and the number of basic CPU operations required to complete the task. i Indicates the task priority. The task priority is calculated according to the remaining time of the task, and urgent tasks are given priority for calculation; S3, calculate the system cost of the task, the unloading decision of the terminal device task in time slot t is Indicates that the user offloads the task to the edge server. Indicates that the user offloads the task to the cloud server. This means that the task is processed locally on the terminal device, and the latency and energy consumption of the task on the terminal device, edge server, and cloud server are calculated respectively. The system cost is calculated by the weighted sum of latency and energy consumption. The calculation formula is: Z(t)=α·T(t)+β·E(t), (α,β≥0)^(α+β＝1) Among them, α and β represent hyperparameters set according to the offloading task types with different requirements, T(t) represents the total processing delay of the task, E(t) represents the total energy consumption of processing the task, and the minimum Z(t) offloading strategy is taken; S4, computing task offloading method, S4.

1. Convert the task computation offloading problem into a Markov decision process and define the Markov decision process (MDP) as: MDP = (S, A, P, R, γ) Among them, S is the state space; A is the action space; P is the state transfer function, R is the reward function; γ is the discount factor; S4.

2. Establish a computational offloading model based on the residual network and D3QN algorithm, and continuously update the model parameters through the gradient descent method to gradually reduce the loss function, thereby finding a strategy with the minimum loss function.

2. According to claim 1, an edge computing offloading strategy based on deep reinforcement learning is characterized in that: When calculating the system cost of the task in step S3, each user terminal N i The transmission rate is R, and the calculation formula is: Where pi represents the transmission power between the user terminal and the server, which cannot exceed the maximum transmission power p max , W represents the channel bandwidth, lb is the logarithm with base 2, N0 represents the channel noise power, the interference between users is negligible, h i It represents the channel gain of the uplink between the user terminal and the server, and is calculated as follows: Where l represents the link loss constant, d0 represents the distance parameter, and d i Indicates the link transmission distance, δ represents the link loss index; when When the computing task is executed on the local terminal device, the computing delay of the task is The calculation formula is: Among them, C i Indicates the task computation amount of the user terminal, that is, the number of CPU cycles required to compute the task. Indicates the CPU frequency of the user terminal; The energy consumption generated by executing the task on the local terminal device is: Where κ is the effective capacitance coefficient, which depends on the device structure; when When the task is U, the computational delay includes the transmission delay of offloading the task to the edge server and the execution time of the task on the edge server. i Offload to edge server M j The transmission delay is: The user terminal transmission rate R is expressed as: Among them, p i,j represents the transmission power between the user terminal and the edge server, W i,j represents the channel bandwidth between the user terminal and the edge server, h i,j represents the channel gain of the uplink between the user terminal and the edge server; Without considering the delay and energy consumption of the edge server back to the terminal, the transmission task U i The energy consumption of transmission to the edge server is: Execution time of edge server processing tasks for: Among them, f i,j ∈[0,f max ] represents the frequency of the edge server CPU, f max is the maximum CPU frequency that the edge server can achieve. The total execution time of the task offloaded to the edge server is: when When , the computation delay of the task only includes the transmission delay of the task from the terminal device to the cloud server, and the transmission delay of the task from the edge server to the edge server is: The energy consumption of transmitting tasks to the cloud server is: Among them, p i,k represents the transmission power between the user terminal and the cloud server, W i,k represents the channel bandwidth between the user terminal and the cloud server, h i,k Indicates the channel gain of the uplink between the user terminal and the cloud server.

3. The edge computing offloading strategy based on deep reinforcement learning according to claim 2 is characterized in that: The CPU frequency allocated by the server for processing tasks does not exceed the maximum CPU frequency. When the terminal device offloads the task to the server for processing, its transmission power does not exceed the maximum transmission power.

4. The edge computing offloading strategy based on deep reinforcement learning according to claim 1 is characterized in that: The calculation unloading model in step S4.2 includes the following process: a. The state space and action space are used as the input of the model, and the input state first enters the residual network for processing; b. The residual network extracts key features from the input state by stacking multiple convolutional layers and pooling layers; c. After the residual network extracts features, the feature information enters the fully connected layer; d. The fully connected layer is responsible for learning the mapping relationship from state to action, and finally outputs the Q value corresponding to each action; e. The agent selects the optimal action based on the Q value; f. Update the Q value through the D3QN algorithm based on the selected action and the reward obtained.

Images