Mobile edge computing unloading and service caching method and device based on deep reinforcement learning

Abstract

The invention discloses a mobile edge computing unloading and service caching method and device based on deep reinforcement learning. The method comprises the following steps: constructing a mobile edge computing system model; constructing a service cache model and a decision model; constructing a cost calculation model; establishing a task unloading and service cache optimization problem according to the mobile edge computing system model, the service cache model and the decision model; and solving a task unloading and service cache optimization problem based on a deep reinforcement learning method by taking the minimum cost as an optimization objective function to obtain an unloading decision and a service cache decision. According to the method and the device, the calculation complexity in the MEC calculation unloading solving process can be reduced.

Description

Technical Field

[0001] The present application relates to the field of mobile edge computing offloading technology, and in particular to a method and device for mobile edge computing offloading and service caching based on deep reinforcement learning. Background Art

[0002] In recent years, the development and popularization of 5G networks and mobile smart devices have made it possible to apply various cutting-edge technologies in real-world scenarios, such as virtual reality, augmented reality, and autonomous driving. However, the application of these technologies requires high communication and computing requirements, and mobile devices need to cache supporting services when processing corresponding computing tasks. When these technologies are applied to safety-related fields such as autonomous driving, computing tasks need to be completed within extremely low latency to ensure user safety.

[0003] The research and application of computational offloading technology has received extensive attention. It can effectively reduce the latency and energy consumption costs of computationally intensive tasks, and promote the development and application of emerging technologies in the field of the Internet of Things. In current research, computational offloading is divided into binary offloading and partial offloading. Binary offloading means that the task cannot be split and can only be completely offloaded or completely executed locally; while partial offloading means that the task can be offloaded in any proportion, and the offloaded part and the part of the task calculated locally will be processed in parallel. For the binary offloading method, when offloading computing tasks, discrete variables can be used to represent the offloading decision. Due to the limited transmission bandwidth and computing resources of edge servers, computational offloading often involves the allocation of transmission and computing resources. The decisions of these problems are represented by continuous variables when modeling.

[0004] Using caching technology during the wireless data transmission stage can reduce bandwidth and computing pressure by reusing content. Cache content is divided into two categories: task cache and service cache. The former is the cache of computing content, and the latter is the cache of services required for computing. Caching strategies can be divided into two types, static cache and dynamic cache. The difference between the two is that the cache content of static cache remains unchanged for a long time, while the dynamic cache changes with time and changes in system status.

[0005] Solving the joint decision-making problem of computation offloading and service caching is challenging. In current MEC-based mobile edge computing, there is a problem that the edge server is overloaded, resulting in computational latency and energy consumption that are unbearable, leading to task failure. In order to reduce the burden on edge servers, many optimization methods have been proposed to solve problems such as MEC computation offloading. However, existing solutions for solving MEC computation offloading have the problem of high computational complexity. Summary of the invention

[0006] The embodiments of the present application provide a method and device for mobile edge computing offloading and service caching based on deep reinforcement learning, which can reduce the computational complexity of solving the MEC computing offloading process.

[0007] The embodiment of the present application also provides a mobile edge computing offloading and service caching device based on deep reinforcement learning, including:

[0008] Build a mobile edge computing system model;

[0009] Build service cache model and decision model;

[0010] Build costing models;

[0011] Establishing task offloading and service cache optimization problems according to the mobile edge computing system model, the service cache model and the decision model;

[0012] Taking the minimum cost as the optimization objective function, the task offloading and service caching optimization problems are solved based on the deep reinforcement learning method to obtain the offloading decision and service caching decision.

[0013] Furthermore, in the above-mentioned mobile edge computing offloading and service caching method based on deep reinforcement learning, the mobile edge computing system model includes a plurality of nodes, each node is deployed with an edge server, and each edge server has a plurality of mobile devices within its service range;

[0014] The task of the mobile device is represented as in, Indicates the amount of task data, Indicates the number of CPU cycles required to process each byte. Indicates the final completion deadline. Indicates the required service cache.

[0015] Furthermore, in the above-mentioned mobile edge computing offloading and service caching method based on deep reinforcement learning, it is necessary to calculate in the decision model: computing offloading decision variables, service caching decision variables and computing energy consumption decision variables;

[0016] The computational offloading decision variables are expressed as in, Indicates the proportion of tasks calculated locally. Indicates the proportion of tasks offloaded to each edge server;

[0017] The service cache decision variable is expressed as in, Push variables to the service cache when When , it means that service s will be pushed to mobile device i, otherwise it will not be pushed. When service s is pushed, if there is no cached service s in the current time slot, Used to update the local cache of the mobile device, where: Indicates that the current mobile device has cached services and needs to clear the cached services required by the currently requested task. Indicates that the service of the current request task has been cached and the service continues to be retained; Indicates the service required for caching the current request task;

[0018] The computational energy consumption decision variable is expressed as in, To represent the computing resource allocation of user i in time slot t.

[0019] Furthermore, in the above-mentioned mobile edge computing offloading and service caching method based on deep reinforcement learning, in the cost calculation model, the cost of the local computing process also needs to be calculated;

[0020] In the local calculation, the calculation delay is expressed as:

[0021]

[0022] in, Indicates the proportion of tasks calculated locally. and They represent the amount of task data and the CPU cycles required to process each byte of data. To calculate the resource allocation ratio, Indicates the maximum computing rate of the mobile device;

[0023] The energy consumption in the process of calculating the delay is:

[0024]

[0025] Where κ represents the energy efficiency coefficient.

[0026] Furthermore, in the above-mentioned mobile edge computing offloading and service caching method based on deep reinforcement learning, in the cost calculation model, the cost of the offloading calculation process also needs to be calculated;

[0027] In the offloading calculation, the upload delay is expressed as:

[0028]

[0029] The energy consumption during the upload delay process is:

[0030]

[0031] in, is the uplink transmission rate between mobile device i and the corresponding edge server m, p i Indicates the transmission power of the mobile device;

[0032] The waiting time before task calculation is expressed as:

[0033]

[0034] Among them, l m (i, t) represents the task queue of task i arriving at edge server m in time slot t, q∈l k (i, t) is the qth task, and the waiting time is the time it takes to calculate the completion of all tasks in the task queue when it arrives;

[0035] The computation delay of the partial task assigned by mobile device i to the corresponding edge server m is:

[0036]

[0037] in, represents the proportion of tasks assigned to the corresponding edge server m by mobile device i;

[0038] The total delay calculated by the edge server is expressed as:

[0039] The transmission delay from the edge server corresponding to the mobile device to the assisting edge server k is expressed as:

[0040]

[0041] The time to wait before the assisting edge server calculates:

[0042]

[0043] Among them, l k (i, t) represents the task queue of task i arriving at edge server k in time slot t, q∈l k (i, t) is the qth task;

[0044] The latency required for the assisted edge server task is:

[0045]

[0046] The total delay of assisting in the calculation of some tasks is expressed as:

[0047] The total delay of processing a task is expressed as the maximum value of the local computing delay, the corresponding edge server computing delay, and the assisting edge server delay:

[0048]

[0049] in, Indicates the computational latency in local computing.

[0050] Furthermore, the above-mentioned mobile edge computing offloading and service caching method based on deep reinforcement learning, wherein the task offloading and service caching optimization problem is established according to the mobile edge computing system model, the service caching model and the decision model, including: constructing a cost minimization optimization objective function according to the task processing delay and the energy consumption of the mobile device under different task processing modes; constructing optimization constraints, and establishing the task offloading and service caching optimization problem according to the constraints of jointly optimizing the offloading variables and the caching variables to minimize the variables.

[0051] Furthermore, in the above-mentioned mobile edge computing offloading and service caching method based on deep reinforcement learning, the task offloading and service caching optimization problem is expressed as:

[0052] P1:

[0053]

[0054] Problem P1 represents the optimization of the computation offloading decision variable X(t), the service cache decision variable Y(t) and the computation resource allocation decision variable Z(t) in each time slot to obtain the lowest total system cost.

[0055] Furthermore, in the above-mentioned mobile edge computing offloading and service caching method based on deep reinforcement learning, the task offloading and service caching optimization problem is described as a Markov decision process, and the actions of the Markov decision process represent the computing offloading, service caching and computing resource allocation decisions of the intelligent agent, and the goal of the intelligent agent is set to maximize the strategy of the output action to obtain the maximum reward, and the reward is set to the negative value of the optimization objective function.

[0056] Furthermore, the above-mentioned mobile edge computing offloading and service caching method based on deep reinforcement learning, wherein the method further comprises: outputting the action of the Markov decision process through the actor network of the central controller, and training the actor network based on the ISAC algorithm;

[0057] The training process includes:

[0058] Collecting task information generated by all mobile devices within the service range, service cache status and wireless communication status of the mobile devices, and saving a number of decision information to form status information;

[0059] Inputting the state information into the actor network, converting the state information into actions through the actor network and outputting the actions;

[0060] Put the training samples into the experience replay pool until the number of experiences in the experience replay pool reaches the maximum number; wherein the training samples are {s, a, r, s′}, s, a, r, s′ represent the state, action, reward and state information of the narrow moment respectively;

[0061] Samples are extracted from the experience pool as training samples, the above operations are repeated, the loss function is calculated, and the network parameters of the actor network are updated based on the loss function.

[0062] The embodiment of the present application also provides a mobile edge computing offloading and service caching device based on deep reinforcement learning, including:

[0063] The first building module is used to build a mobile edge computing system model;

[0064] The second building module is used to build a service cache model and a decision model;

[0065] The third building block is used to build a cost calculation model;

[0066] The decision module is used to establish the task offloading and service cache optimization problem according to the mobile edge computing system model, the service cache model and the decision model, and solve the task offloading and service cache optimization problem based on the deep reinforcement learning method with the minimum cost as the optimization objective function to obtain the offloading decision and the service cache decision.

[0067] The present application provides a method and device for mobile edge computing offloading and service caching based on deep reinforcement learning. The present application transforms the task offloading and service caching optimization problem into a Markov decision process (MDP), and uses deep reinforcement learning (DRL) technology to explore and optimize the most effective strategy. DRL solves this challenge well. It can output computing offloading and service caching strategies through DNN (deep neural network) by taking highly variable communication states as state inputs, and reduces computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The technical solution and other beneficial effects of the present application will be made apparent by describing in detail the specific implementation methods of the present application in conjunction with the accompanying drawings.

[0069] Figure 1A flowchart of a mobile edge computing offloading and service caching method based on deep reinforcement learning provided in an embodiment of the present application.

[0070] Figure 2 Schematic diagram of the application scenario provided for this application.

[0071] Figure 3 A schematic diagram of the structure of the actor network provided for this application.

[0072] Figure 4 A schematic diagram of the training process of the actor module and the critic module provided in the embodiment of the present application.

[0073] Figure 5 A structural diagram of a mobile edge computing offloading and service caching device based on deep reinforcement learning provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0075] In current mobile edge computing based on MEC, there is a problem that the edge server is overloaded, resulting in computing delay and energy consumption that are difficult to bear, thus causing task failure. In order to reduce the burden on edge servers, many optimization methods have been proposed to solve problems such as MEC computing offloading. Existing solutions have the following problems: (1) The computational complexity is too high; (2) When the problem conditions change and need to be iterated again, the delay is too large to be applied in the actual MEC environment; (3) The general reinforcement learning algorithm has a large action space, resulting in high training costs.

[0076] To solve the above problems, the embodiments of the present application provide a method and device for mobile edge computing offloading and service caching based on deep reinforcement learning. The embodiments of the present application provide a mobile edge computing offloading and service caching device based on deep reinforcement learning, which can be integrated in an electronic device, which can be a terminal, a server and other devices, wherein the terminal can include a tablet computer, a laptop computer, a personal computer (PC, Personal Computer), a micro processing box, or other devices.

[0077] See also Figure 1 , Figure 1A flowchart of a mobile edge computing offloading and service caching method based on deep reinforcement learning provided in an embodiment of the present application, which is applied to an electronic device, and the mobile edge computing offloading and service caching method based on deep reinforcement learning includes the following steps:

[0078] S1, build a mobile edge computing system model.

[0079] First, we need to build an application scenario for mobile edge computing offloading and service caching methods based on deep reinforcement learning. Figure 2 The application scenario diagram provided for this application is as follows: Figure 2 As shown, the present application considers a scenario with multiple edge servers and multiple mobile devices. The system running time is divided into multiple fixed-length time slots t∈{1,…,T}. The edge server has a large cache capacity and caches the service types required for all tasks, while the mobile device can only cache limited services due to the limitation of cache capacity. If a task that does not have a corresponding service cached needs to be calculated, the task cannot be processed locally in the current time slot, and the required cache service needs to be downloaded from the edge server. Compared with mobile devices, edge servers have a stronger ability to process tasks. Users can offload tasks to edge servers and use the powerful computing power of edge servers to reduce latency. Each edge server can communicate and collaborate with each other to jointly process a task to balance computing resources. Each mobile device has a corresponding edge server, and the tasks generated by the device can be processed by first transmitting them to the corresponding edge server, and then transmitting them to other edge servers by the edge server.

[0080] At the beginning of each time slot, a mobile device will generate a task. Since the tasks in the same time slot correspond to mobile devices one by one, we use the set i = {1,…,I} to represent both tasks and mobile devices. Each task is represented by a four-tuple They represent the task data size (in bit), the CPU cycles required to process each bit of data (in cycle / bit), the final completion deadline (in s), and the required service cache. The service cache set is s∈{1,…,S}, and the required data size of the service cache is u s At the beginning of each time slot, the controller will confirm the cache status of mobile device i. We use To indicate whether mobile device i has cached service s in time slot t. The edge server set is represented as m = {1, ..., M}, and the computing power of the edge server is represented as

[0081] The mobile edge computing system model includes multiple nodes, each of which is deployed with an edge server. Each edge server has multiple mobile devices within its service range. The tasks of the mobile devices are represented as in, Indicates the amount of task data, Indicates the number of CPU cycles required to process each byte. Indicates the final completion deadline. Indicates the required service cache.

[0082] S2, build service cache model and decision model.

[0083] The service cache model is represented by s∈{1,…,S}, and the amount of data required for the service cache is u s At the beginning of each time slot, the controller will confirm the cache status of mobile device i. We use To indicate whether mobile device i has cached service s in time slot t.

[0084] In one embodiment, the decision model needs to calculate: computing offloading decision variables, service cache decision variables, and computing energy consumption decision variables.

[0085] The calculation of unloading decision variables is expressed as in, Indicates the proportion of tasks calculated locally. Indicates the proportion of tasks offloaded to each edge server. In order to ensure that each task can be processed completely, it is necessary to meet

[0086] The service cache decision variable is represented as in, Push variables to the service cache when When , it means that service s will be pushed to mobile device i, otherwise it will not be pushed. When service s is pushed, if there is no cached service s in the current time slot, And the cache update does not choose to cache the service, that is, Or 0, Set to 1 to cache the service when updating. Used to update the local cache of the mobile device, where: Indicates that the current mobile device has cached services and needs to clear the cached services required by the currently requested task. Indicates that the service of the current request task has been cached and the service continues to be retained; Indicates the updated status of the service required by the cached current request task If the mobile device does not have the services required for the cache task, it cannot process the task locally. If the mobile device does not have the required service cached, it will be set

[0087] Indicates that the service cache pushes variables. The time service s will push the message to the mobile device i at the end of the time slot, otherwise it will not push the message. It is used to indicate the service cache update of mobile device i, corresponding to the clearing, retention and caching of service s. The update method of the service cache status is Cache update variables The value needs to satisfy the following constraints

[0088]

[0089] The left side of the inequality restricts that only services that are cached can be cleared, while the right side of the inequality indicates that tasks can only be cached if the service is not cached and is pushed. In this application, if the mobile device does not have the service required by the cached task, it cannot process the task locally. Therefore, when If the mobile device does not have the required service cached, it will be set

[0090] The energy consumption decision variables are expressed as in, denoted by the computational resource allocation of user i in time slot t, which aims to minimize the system utility while preventing the mobile device from violating the energy consumption constraint.

[0091] S3, build a cost calculation model.

[0092] First, we need to build a mobile edge communication scenario:

[0093] Mobile devices communicate with edge servers through wireless links. In order to reduce mutual interference between devices, orthogonal frequency division multiple access (OFDMA) technology is used for uplink communication between mobile devices and servers. According to Shannon's theorem, the uplink transmission rate between mobile device i and its corresponding edge server m can be defined as

[0094]

[0095] in and represents the communication bandwidth and channel gain between mobile device i and its corresponding edge server m, p i represents the transmission power of the mobile device, σ 2 is the channel noise. When the user assigns a task to another edge server, it is first uploaded to the corresponding edge server and then transmitted to the target edge server. To simplify the model, it is assumed that the transmission speed between edge servers is

[0096] In the cost calculation model, the cost of the local computing process and the cost of the offloaded computing process need to be calculated.

[0097] 1) Local computing

[0098] The latency required for tasks assigned to local computing on mobile devices is expressed as The energy consumption of the task in the local computing process is expressed as

[0099] The latency required for tasks assigned to local computing on mobile devices is expressed as

[0100]

[0101] in, Indicates the proportion of tasks calculated locally. and They represent the amount of task data and the CPU cycles required to process each bit of data. The ratio of computing resources allocated determines the actual computing rate of the mobile device. represents the maximum computing rate of the mobile device. The energy consumption of this process is expressed as:

[0102]

[0103] Among them, κ represents the energy efficiency coefficient, which is set to 10 -27 .

[0104] 2) Offloading calculations

[0105] No matter which edge server a mobile device is offloaded to, it is necessary to first transmit part of the offloaded computation to the corresponding edge server. The transmission delay to the edge server corresponding to the device can be expressed as:

[0106]

[0107] The energy consumption of this process can be expressed as:

[0108]

[0109] Among them, p i represents the transmission power of the mobile device. Due to the limited battery capacity of the mobile device, the energy consumption of each time slot cannot exceed the threshold E i , expressed as a constraint

[0110] Considering that the edge server will receive multiple task processing requests in a time slot, the tasks are set to be processed in the order of arrival. There may be a waiting time before the task is calculated. The waiting time is expressed as:

[0111]

[0112] Among them, l m (i, t) represents the task queue of task i arriving at edge server m in time slot t, q∈l k (i, t) is the qth task, and the waiting time is the time it takes to calculate the completion of all tasks in the task queue when it arrives.

[0113] It represents the proportion of tasks assigned to the corresponding edge server m by the mobile device i. The third delay of this part of tasks is:

[0114]

[0115] Since the amount of calculation result data is small and in order to simplify the model, this method does not consider the delay caused by the calculation result return. In summary, the processing of the part of the task assigned to the corresponding edge server m by the mobile device i can be divided into three parts: upload delay Calculate the waiting time before this part of the task Computational latency required for some tasks Therefore, the total delay calculated at the corresponding edge server is expressed as:

[0116] The edge server other than the server corresponding to the mobile device is called the assisting server and is represented by k∈M\{m}. The transmission delay from the edge server corresponding to the mobile device to the assisting edge server k is expressed as:

[0117]

[0118] The time to wait before the assisting edge server calculates:

[0119]

[0120] Among them, l k (i, t) represents the task queue of task i arriving at edge server k in time slot t, q∈l k (i,t) is the qth task.

[0121] The latency required for the assisted edge server task is:

[0122]

[0123] In summary, the total delay of assisting in computing some tasks is expressed as: The total delay of processing a task is expressed as the maximum value of the local computing delay, the corresponding edge server computing delay, and the assisting edge server delay:

[0124]

[0125] S4, establishes the task offloading and service cache optimization problem based on the mobile edge computing system model, service cache model and decision model.

[0126] The task offloading and service cache optimization problem is transformed into a Markov decision process and solved using deep reinforcement learning methods, which includes the following sub-steps:

[0127] S41, transforms the computation offloading decision and service caching decision in the mobile edge computing scenario into an optimization problem of minimizing the joint optimization of offloading variables and caching variables to minimize the total cost. The cost of a task is defined as Where H(t) represents the cache hit rate, and α is a parameter that balances the difference between latency and cache hit rate. The total system cost is the sum of the costs of all tasks in the long term.

[0128] Problem P1 is expressed as follows:

[0129] P1:

[0130]

[0131] Problem P1 represents the optimization of the computation offloading decision variable X(t), the service cache decision variable Y(t) and the computation resource allocation decision variable Z(t) in each time slot to obtain the lowest total system cost. Constraints (a), (b) and (c) specify the values ​​of the optimization variables. Constraint (d) ensures that each task can be fully processed and will not be repeatedly calculated. Constraint (e) indicates that the cache capacity of all services in the mobile device cannot exceed the cache capacity. Constraint (f) indicates that the energy consumption of the mobile device in each time slot cannot exceed the threshold. Constraint (g) indicates that the task needs to be completed before the latest deadline.

[0132] S42, transform the optimization problem into a Markov decision process;

[0133] The computational offloading process of MEC can be regarded as an MDP (Markov Decision Process). The intelligent agent can interact through an unknown environment to learn the optimal action. The MDP is expressed as {S, A, P, R}. Next, the state space, action space and reward are defined.

[0134] State space S

[0135] The agent will output the corresponding action according to the observed state. The state of the MDP is represented as

[0136] S(t)={Stask (t),S com (t),S ca (t),S pre (t)}

[0137] in Contains the task information of all mobile devices in time slot t, including the task data volume, the CPU cycles required to process each bit of data, the final completion deadline, and the required service cache; is the communication status, including the channel gain between each mobile device and the corresponding edge server; Indicates the cache status, which is used to indicate the cache status of different services on mobile devices; S pre (t) = {a(t-1),…,a(tL)} contains the actions of the agent in the previous L time slots, and the agent can learn from past action decisions.

[0138] Action Space A

[0139] We use actions to represent the agent's computation offloading, service caching, and computational resource allocation decisions. The action space is represented as

[0140] a(t)={X(t),Y(t),Z(t)}

[0141] in, represents the uninstall decision, Represents the service caching decision, Represents a computational resource allocation decision.

[0142] Reward R

[0143] In MDP, the goal of the agent is to learn the strategy of outputting actions to maximize the maximum reward, and the application needs to minimize the optimization goal. Therefore, without violating the cache capacity and energy consumption constraints, the reward is set to the negative value of the optimization objective function. When the constraints are violated, a penalty is given. Let the agent try to avoid outputting actions that violate constraints during the learning process. The reward is expressed as

[0144]

[0145] S5, taking the minimum cost as the optimization objective function, solves the task offloading and service caching optimization problems based on the deep reinforcement learning method, and obtains the offloading decision and service caching decision.

[0146] The actions of the Markov decision process are output through the actor network of the central controller, and the actor network is trained based on the ISAC algorithm.

[0147] In one embodiment, the training process includes:

[0148] S51, collecting task information generated by all mobile devices within the service range, service cache status and wireless communication status of the mobile devices, and saving a number of decision information to form status information.

[0149] The central controller located at the edge server collects the task information, service cache status and wireless communication status of all mobile devices within the service range, and combines the decision information in the previous time periods stored in the database to form the state S(t) at the time t = {S task (t),S com (t),S ca (t),S pre (t)}.

[0150] S52, input the state information into the actor network, convert the state information into action a(t)={X(t), Y(t), Z(t)} through the actor network and output it.

[0151] Preferably, the present invention adopts an actor-critic framework for network parameter optimization training and decision output, wherein the actor network module is responsible for outputting an action a(t) according to an input state S(t). Figure 3 The structural diagram of the actor network provided for this application is as follows: Figure 3 As shown in Figure 1, the actor network consists of a multi-layer neural network, including an input layer, a hidden layer, and an output layer.

[0152] S53, putting the training samples into the experience replay pool until the number of experiences in the experience replay pool reaches the maximum number; wherein the training samples are {s, a, r, s′}, s, a, r, s′ respectively represent the state, action, reward and state information of the narrow moment.

[0153] S54, extract samples from the experience pool as training samples, repeat the above operations, calculate the loss function, and update the network parameters of the actor network based on the loss function.

[0154] Specifically, the DRL method - Improved Soft Policy Gradient (ISAC) algorithm can be used to train and update the network parameters of the actor network. The specific steps are as follows:

[0155] Traditional DRL algorithms are prone to fall into local optimality when solving problems in high-dimensional action spaces. The Soft Actor-Critic (SAC) algorithm introduces an entropy regularization term in the objective function. Entropy reflects the randomness of the strategy output actions, improves the algorithm's exploration ability, and reduces the risk of the strategy falling into local optimality.

[0156] The entropy term introduced is expressed as follows, where π(at |s t ) means in state s t Next output action a t Strategy:

[0157]

[0158] For the SAC algorithm, the goal is to find a strategy π that maximizes the objective function of reward plus entropy. The optimal strategy is expressed as follows:

[0159]

[0160] Where γ is the discount factor, μ is the coefficient that weighs the reward and entropy value, and a larger μ indicates a stronger exploration ability. The enhancement of the algorithm's exploration ability will sacrifice the utilization ability. In order to balance exploration and utilization, inspired by the simulated annealing algorithm, this application will dynamically adjust the entropy coefficient μ during the training process.

[0161]

[0162] Where μ0 is the value of the entropy coefficient at the beginning of training, K max The maximum number of training rounds, k is the current training round.

[0163] In DRL, the Q value is used to measure the long-term reward of the state action under a given strategy, while in SAC the soft Q value is used to measure the sum of the long-term reward and entropy, expressed as:

[0164]

[0165] In SAC, the soft V value is defined to evaluate the state s under this strategy. t , expressed as:

[0166]

[0167] In order to maximize the soft policy function, this application proposes ISAC (Improved Sotf Actor-Critic). The ISAC algorithm mainly includes the actor module and the critic module. The role of the actor module is to fit the strategy. By observing the current state, it outputs the corresponding action, including task offloading decision, cache decision and computing resource allocation decision; the critic module outputs and updates the Q value and V value according to the environmental state and the action output by the actor module to evaluate the action, so that the actor module can output better actions and fit better strategies. Figure 4 The schematic diagram of the actor module and critic module training process provided in the embodiment of the present application can be found in Figure 4 ,The training process and principles of the two modules are introduced in detail below.

[0168] The critic module includes four DNNs (deep neural networks), a V network and a targetV network, using V ω (s), In other words, ω and ω′ represent the parameters of the two networks. In addition, there are two Q networks. Indicates that, where θ j represents the parameters of the jth Q network. When updating the V network, the smaller value of the output of the two Q networks will be taken, which can alleviate the overestimation of the value function. In order to improve the utilization efficiency of training samples and break the time order relationship between training samples, an experience replay pool is introduced. It can store newly generated experiences, each of which is a tuple that can be used for training, including state, action, reward and the state at the next moment, represented as {s, a, r, s′}. Each time training, a mini-batch containing B samples is randomly selected as a training sample.

[0169] For the Q network, the training method is given below. In order to avoid bootstrapping and alleviate overestimation, a separate target soft V function is added to calculate the target soft Q, which is used as the relative true value estimate of the state s.

[0170]

[0171] In practice, action a is used to calculate the predicted value estimate of state s, and the MSE between the output soft Q value and the target soft Q value is minimized to train The loss function is:

[0172]

[0173] The gradient calculation method is expressed as:

[0174]

[0175] Update the parameters of the soft Q network by minimizing the objective function through the gradient descent method

[0176]

[0177] Among them, α is the learning rate.

[0178] For Network V ω , we train it by minimizing the mean square error (MSE). When using soft Q to estimate soft V, we take the minimum value of the two Q networks to alleviate the overestimation of the value function. The loss function is expressed as:

[0179]

[0180] The gradient of the loss function is calculated as:

[0181]

[0182] Use the gradient descent method to minimize the loss function to update the V network parameters ω

[0183]

[0184] After the V network parameters are updated, the target V network will update its own parameters through soft update, where β is the soft update coefficient

[0185] ω′=βω+(1-β)ω′

[0186] This application introduces a multi-branch structure in the actor module, assigning an independent DNN to each user to fit the user's strategy, outputting the user's decision action based on the input state, and then combining the actions of all users to form the overall action. In addition, in order to learn from previous experience, past action information is added to the input state, and LSTM is used to process this part of the input data. The DNN structure of each user is as follows: Figure 3 , the overall structure of the actor module is as follows Figure 4 .

[0187] Each mobile device uses an independent DNN network for decision making. In order to stabilize the training process and increase the exploration capability of the training process, a random Gaussian noise δ is added to each DNN. i , represents the network parameters corresponding to mobile device i, and the output action is expressed as

[0188]

[0189] By maximizing the strategy The expected soft Q value under is used to optimize the actor network parameters, so the loss function of the Actor network is defined as:

[0190]

[0191] The strategy gradient is calculated as:

[0192]

[0193] Minimize the loss function by gradient descent Update actor network parameters

[0194]

[0195] Furthermore, the trained actor network model is used for policy output. At the same time, the generated new experience will still be stored in the experience replay pool.

[0196] Preferably, after the model is put into use for decision making, the experience replay pool will continue to store the experience generated by the current decision process, and the experience replay pool will delete the old training samples while storing new training samples. The network parameters will still be periodically updated according to the training.

[0197] The trained neural network can quickly infer the action of the actor network as a decision through the input state, and does not need to be retrained when the input changes. This feature can reduce the delay required for decision-making, improve the hit rate of the service cache, and reduce the action space of each agent in the actor module by introducing a multi-branch structure in the actor module. This application transforms the task offloading and service cache optimization problems into a Markov decision process (MDP), and uses deep reinforcement learning (DRL) technology to explore and optimize the most effective strategy. DRL solves this challenge well. It can output computing offloading and service caching strategies through DNN by using highly changing communication states as state input. After training, when the state changes, the unloading and caching strategies under the current conditions can be solved without retraining the DNN. At the same time, this application improves the soft actor-critic (SAC) reinforcement learning algorithm, integrates the long short-term neural network (LSTM) to learn past experience, and improves training efficiency. This application can train and optimize the model through DRL technology within an acceptable training cost for the task information, communication status, cache status and previous decision-making experience that need to be processed in the MEC environment, thereby providing an offloading and service caching solution for the batch of tasks and obtaining the lowest computing cost.

[0198] According to the method described in the above embodiments, this embodiment will be further described from the perspective of a mobile edge computing offloading and service caching device based on deep reinforcement learning. The mobile edge computing offloading and service caching device based on deep reinforcement learning can be implemented as an independent entity or integrated in an electronic device, which can be a terminal, server and other devices. The terminal can include a tablet computer, a laptop computer, a personal computer (PC), a micro processing box, or other devices.

[0199] See also Figure 5 , Figure 5The present invention specifically describes a mobile edge computing offloading and service caching device based on deep reinforcement learning provided in an embodiment of the present application, which is applied to an electronic device. The mobile edge computing offloading and service caching device based on deep reinforcement learning may include:

[0200] The first building module is used to build a mobile edge computing system model;

[0201] The second building module is used to build a service cache model and a decision model;

[0202] The third building block is used to build a cost calculation model;

[0203] The decision module is used to establish the task offloading and service cache optimization problem according to the mobile edge computing system model, the service cache model and the decision model, and solve the task offloading and service cache optimization problem based on the deep reinforcement learning method with the minimum cost as the optimization objective function to obtain the offloading decision and the service cache decision.

[0204] In another embodiment, the mobile edge computing offloading and service caching device based on deep reinforcement learning may also include:

[0205] The data collection and transmission department collects information needed for decision-making and network training, and ensures data interaction between edge servers, mobile devices, central controllers, and databases;

[0206] Channel model building department, builds the channel model between edge mobile devices and roadside infrastructure:

[0207] Preferably, the mobile device communicates with the edge server via a wireless link. In order to reduce mutual interference between devices, orthogonal frequency division multiple access (OFDMA) technology is used for uplink communication between the mobile device and the server. The uplink transmission rate between the mobile device i and its corresponding edge server m can be defined by Shannon's theorem as:

[0208]

[0209] in, and represents the communication bandwidth and channel gain between mobile device i and its corresponding edge server m, p i represents the transmission power of the mobile device, σ 2 is the channel noise. When the user assigns a task to another edge server, it is first uploaded to the corresponding edge server and then transmitted to the target edge server. To simplify the model, it is assumed that the transmission speed between edge servers is

[0210] The decision-making department combines the local states of all mobile devices and edge servers to obtain task offloading, service decisions, and computing rate allocation decisions through actor network reasoning;

[0211] The environment interaction part interacts with the environment according to the decision actions output by the actor network, calculates the queue time of other tasks to be offloaded to the edge and the delay required for the edge server to process each task, updates the local state, and returns the reward value of the action;

[0212] Network training department, calculates loss function and updates network parameters;

[0213] The control unit is connected to the task data acquisition unit, the channel model building unit, the channel model building unit, the decision unit, the environment interaction unit and the network training unit to control their operations.

[0214] Preferably, the mobile edge task offloading and service caching device based on deep reinforcement learning provided in the present application may also include: an input display unit, which is communicatively connected to the control unit, allowing the user to input operation instructions, and displaying the input, output and intermediate processing data of the corresponding unit in the form of text, tables, graphics, static or dynamic models according to the operation instructions.

[0215] Preferably, in the mobile edge task offloading and service caching device based on deep reinforcement learning provided by the present invention, the frequency of interaction perception in the environment interaction part is 1ms.

[0216] During specific implementation, the above modules and / or units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above modules and / or units can refer to the previous method embodiments. The specific beneficial effects that can be achieved can also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0217] The above is a detailed introduction to a mobile edge computing offloading and service caching method and device based on deep reinforcement learning provided in an embodiment of the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A mobile edge computing offloading and service caching method based on deep reinforcement learning, characterized in that: The method comprises: Build a mobile edge computing system model; Build service cache model and decision model; Build costing models; Establishing task offloading and service cache optimization problems according to the mobile edge computing system model, the service cache model and the decision model; Taking the minimum cost as the optimization objective function, the task offloading and service caching optimization problems are solved based on the deep reinforcement learning method to obtain the offloading decision and service caching decision.

2. The method for mobile edge computing offloading and service caching based on deep reinforcement learning according to claim 1 is characterized in that: The mobile edge computing system model includes multiple nodes, each node is deployed with an edge server, and each edge server has multiple mobile devices within its service range; The task of the mobile device is represented as in, Indicates the amount of task data, Indicates the number of CPU cycles required to process each byte. Indicates the final completion deadline. Indicates the required service cache.

3. The method for mobile edge computing offloading and service caching based on deep reinforcement learning according to claim 1 is characterized in that: In the decision model, it is necessary to calculate: computing offloading decision variables, service cache decision variables and computing energy consumption decision variables; The computational offloading decision variables are expressed as in, Indicates the proportion of tasks calculated locally. Indicates the proportion of tasks offloaded to each edge server; The service cache decision variable is expressed as in, Push variables to the service cache when When , it means that service s will be pushed to mobile device i, otherwise it will not be pushed. When service s is pushed, if there is no cached service s in the current time slot, Used to update the local cache of the mobile device, where: Indicates that the current mobile device has cached services and needs to clear the cached services required by the currently requested task. Indicates that the service of the current request task has been cached and the service continues to be retained; Indicates the service required for caching the current request task; The computational energy consumption decision variable is expressed as in, To represent the computing resource allocation of user i in time slot t.

4. The method for mobile edge computing offloading and service caching based on deep reinforcement learning according to claim 3 is characterized in that: In the cost calculation model, the cost of the local computing process also needs to be calculated; In the local calculation, the calculation delay is expressed as: in, Indicates the proportion of tasks calculated locally. and They represent the amount of task data and the CPU cycles required to process each byte of data. To calculate the resource allocation ratio, Indicates the maximum computing rate of the mobile device; The energy consumption in the process of calculating the delay is: Where κ represents the energy efficiency coefficient.

5. The method for mobile edge computing offloading and service caching based on deep reinforcement learning according to claim 4 is characterized in that: In the cost calculation model, the cost of the offloading calculation process also needs to be calculated; In the offloading calculation, the upload delay is expressed as: The energy consumption during the upload delay process is: in, is the uplink transmission rate between mobile device i and the corresponding edge server m, p i Indicates the transmission power of the mobile device; The waiting time before task calculation is expressed as: Among them, l m (i, t) represents the task queue of task i arriving at edge server m in time slot t, q∈l k (i, t) is the qth task, and the waiting time is the time it takes to calculate the completion of all tasks in the task queue when it arrives; The computation delay of the partial task assigned by mobile device i to the corresponding edge server m is: in, represents the proportion of tasks assigned to the corresponding edge server m by mobile device i; The total delay calculated by the edge server is expressed as: The transmission delay from the edge server corresponding to the mobile device to the assisting edge server k is expressed as: The time to wait before the assisting edge server calculates: Among them, l k (i, t) represents the task queue of task i arriving at edge server k in time slot t, q∈l k (i, t) is the qth task; The latency required for the assisted edge server task is: The total delay of assisting in the calculation of some tasks is expressed as: The total delay of processing a task is expressed as the maximum value of the local computing delay, the corresponding edge server computing delay, and the assisting edge server delay: in, Indicates the computational latency in local computing.

6. The method for mobile edge computing offloading and service caching based on deep reinforcement learning according to claim 1, characterized in that: The task offloading and service cache optimization problem is established according to the mobile edge computing system model, the service cache model and the decision model, including: constructing a cost minimization optimization objective function according to the task processing delay and the energy consumption of the mobile device under different task processing modes; constructing optimization constraints, and establishing the task offloading and service cache optimization problem according to the constraints of minimizing the joint optimization offloading variables and the cache variables.

7. The method for mobile edge computing offloading and service caching based on deep reinforcement learning according to claim 6 is characterized in that: The task offloading and service cache optimization problem is expressed as: Problem P1 represents the optimization of the computation offloading decision variable X(t), the service cache decision variable Y(t) and the computation resource allocation decision variable Z(t) in each time slot to obtain the lowest total system cost.

8. The method for mobile edge computing offloading and service caching based on deep reinforcement learning according to claim 1, characterized in that: The task offloading and service cache optimization problem is described as a Markov decision process, and the actions of the Markov decision process represent the agent's computational offloading, service cache and computing resource allocation decisions. The agent's goal is set to maximize the strategy of the output action to obtain the maximum reward, and the reward is set to the negative value of the optimization objective function.

9. The method for mobile edge computing offloading and service caching based on deep reinforcement learning according to claim 8, characterized in that: The method further includes: outputting the actions of the Markov decision process through an actor network of a central controller, and training the actor network based on an ISAC algorithm; The training process includes: Collecting task information generated by all mobile devices within the service range, service cache status and wireless communication status of the mobile devices, and saving a number of decision information to form status information; Inputting the state information into the actor network, converting the state information into actions through the actor network and outputting the actions; The training samples are put into the experience replay pool until the number of experiences in the experience replay pool reaches the maximum number; wherein the training samples are {s, a, r, s ′ },s,a,r,s ′ Respectively represent the state, action, reward and state information of the narrow moment; Samples are extracted from the experience pool as training samples, the above operations are repeated, the loss function is calculated, and the network parameters of the actor network are updated based on the loss function.

10. A mobile edge computing offloading and service caching device based on deep reinforcement learning, characterized in that: include: The first building module is used to build a mobile edge computing system model; The second building module is used to build a service cache model and a decision model; The third building block is used to build a cost calculation model; The decision module is used to establish the task offloading and service cache optimization problem according to the mobile edge computing system model, the service cache model and the decision model, and solve the task offloading and service cache optimization problem based on the deep reinforcement learning method with the minimum cost as the optimization objective function to obtain the offloading decision and the service cache decision.

Images