A multi-user dependent task offloading method based on edge computing and deep reinforcement learning

Abstract

The present invention discloses a multi-user dependent task offloading method based on edge computing and deep reinforcement learning, the steps include: 1) establishing a multi-user mobile edge computing system; 2) obtaining edge computing network information, including task information generated by cloud servers, edge servers, and wireless mobile devices; 3) for the nth application of the wireless mobile device, establishing a directed acyclic graph to represent the dependency relationship between I dependent subtasks in the application; 4) according to the directed acyclic graph, establishing a dependent task offloading delay and energy consumption model; 5) according to the dependent task offloading delay and energy consumption model, establishing a task offloading decision model, and solving the task offloading decision model using a Markov decision method to obtain a task offloading decision. The present invention greatly reduces the communication delay and energy cost of the wireless mobile device, meets user needs, and saves link costs.

Description

Technical Field

[0001] The present invention relates to the field of edge computing and deep reinforcement learning, and specifically to a multi-user dependent task offloading method based on edge computing and deep reinforcement learning. Background Art

[0002] The rapid development of communication technology and the explosive growth in the number of mobile devices have accelerated the emergence of innovative applications such as augmented reality, face recognition, smart cars and smart homes. These complex applications usually require intensive computing resources and low response latency, which will put tremendous pressure on the core network. At the same time, since these mobile devices are usually limited by computing, storage and energy, it is difficult to run these applications on mobile devices. Such applications are generally carried out in the cloud, but since the cloud is far away from the mobile devices, running these applications in the cloud will result in high transmission latency.

[0003] Mobile edge computing is one of the emerging technologies in recent years that can effectively overcome this dilemma. Specifically, as a key technology of mobile edge computing, task offloading aims to offload computationally intensive tasks from mobile devices to edge servers deployed at the edge of the network. By jointly considering the nature of the tasks and the resource utilization of the edge servers, task offloading can significantly reduce the system latency and energy consumption of mobile devices and improve the quality of service to users. Therefore, task offloading plays a vital role in meeting the requirements of these computationally latency-sensitive tasks.

[0004] The currently commonly used optimization method is to offload the computing tasks of mobile terminals to edge servers, but this optimization method has some difficult-to-solve problems: First, when a large number of mobile devices are included in the mobile edge computing system, the edge server will be overloaded, which will significantly increase the processing time of computing tasks and reduce the user experience quality. Second, this method does not fully consider the collaborative capabilities of mobile terminals, edge servers, and cloud servers. It only combines a single edge server and multiple mobile terminals and ignores the service capabilities of the cloud server, resulting in low efficiency of the entire mobile edge computing system. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-user dependent task offloading method based on edge computing and deep reinforcement learning, comprising the following steps:

[0006] 1) Establish a multi-user mobile edge computing system, including a cloud server, B edge base stations, and N wireless mobile devices; each base station has an edge server;

[0007] 2) Obtain edge computing network information, including task information generated by cloud servers, edge servers, and wireless mobile devices;

[0008] 3) For the nth application of the wireless mobile device, a directed acyclic graph is established to represent the dependency relationship between I dependent subtasks in the application;

[0009] 4) Based on the directed acyclic graph, a model of latency and energy consumption of dependent task offloading is established;

[0010] 5) According to the dependent task offloading delay and energy consumption model, a task offloading decision model is established, and the task offloading decision model is solved using the Markov decision method to obtain the task offloading decision.

[0011] Further, the directed acyclic graph has an entry node and an exit node; wherein the entry node and the exit node represent the start task and the end task of the application respectively;

[0012] All tasks in the directed acyclic graph are arranged in descending order according to the rank value;

[0013] Among them, task q in The rank value rank(q in ) is as follows:

[0014]

[0015] In the formula, FT in Indicates unloading and processing task q in The waiting time; (I+1)n represents the set of exit tasks. rank(q jn ) is task q jn rank value of .

[0016] Furthermore, the steps of establishing the latency and energy consumption model of dependent task offloading include:

[0017] 4.1) According to the directed acyclic graph, the tasks A set of direct predecessors of

[0018] 4.2) Computational tasks The maximum completion time is:

[0019]

[0020] In the formula, when processing tasks locally hour, Ready time For the task The maximum completion time; The completion time of the task executed locally, on an edge server, or on a cloud server;

[0021] 4.3) Task The local execution time is expressed as The local average computing power is expressed as

[0022] Computational tasks Local execution time and completion time Right now:

[0023]

[0024]

[0025] In the formula, The CPU cycles required for the task;

[0026] 4.4) Compute tasks that are performed locally Energy consumption Right now;

[0027]

[0028] In the formula, κ is the energy consumption coefficient;

[0029] 4.5) The ready time of tasks established on the edge server Right now:

[0030]

[0031] In the formula, They represent the transmission time from the local to the edge server and from the edge server b to the edge server b′ respectively;

[0032] 4.6) Calculate the transmission time from edge server b to cloud server is the data size; is the transmission rate from edge server b to cloud server;

[0033] Computational tasks Execution time on cloud server f c The computing power of the cloud server;

[0034] Computational tasks Ready time on cloud servers Right now:

[0035]

[0036] In the formula, is the transmission time from the edge server to the cloud server;

[0037] 4.7) Calculate the weighted sum of the task offloading delay and energy of the nth application

[0038] in, is the total energy consumption of the mobile device; local energy consumption Energy consumption of upload tasks For unit energy consumption.

[0039] Furthermore, the objective function of the task offloading decision model is: the weighted sum of the total delay and total energy of task offloading is minimized.

[0040] Furthermore, the steps of solving the task offloading decision model using the Markov decision method include:

[0041] 5.1) Establish the action space A and state space S during task offloading;

[0042] 5.2) Based on the objective function of the task offloading decision model, define the system reward R for each training t ;

[0043] 5.3) Based on system reward R t , determine the task offloading decision.

[0044] Further, the state space S is as follows:

[0045] S={s|s=(G_n({I_{n}, θ_{n}), α_{1:i})}; (8)

[0046] Where G_n({I_{n}, θ_{n}) is the task embedding, including the index of the current task and the index of the immediate predecessor and successor tasks; α_{1:i} represents the task embedding from arrive uninstall decision.

[0047] In the action space A, the action A at time slot t t As shown below:

[0048] A t =0,1,2,…,|{B}+1|; (9)

[0049] In the formula, A t =0 means the task is executed locally, A t ={1, 2, ..., B} indicates that the task is offloaded to the edge server, A t =B+1 means that the task is offloaded to the cloud server.

[0050] Further, the system rewards R t As shown below:

[0051]

[0052] In the formula, ξ n The weighted sum of the total latency and total energy of the task offloading.

[0053] Furthermore, based on the system reward R t ,The steps to determine the task offloading decision include:

[0054] 5.3.1) Create an expression for action a, namely:

[0055] a={argmax}{Q(s,a′;w)}; (11)

[0056] In the formula, w is the parameter matrix; s is the state; Q is the action value Q * ;

[0057] 5.3.2) Perform action a in state s according to strategy π to obtain action value Q(s, a);

[0058] 5.3.3) Change action a, repeat step 5.3.2), obtain multiple action values ​​Q(s, a), and take the maximum action value as the optimal action value Q * (s, a), that is:

[0059] Q*(s,a)={max}[R t |s t =s,a t =a,π]; (12)

[0060] 5.3.4) For the optimal action value Q * (s, a) is optimized to obtain:

[0061] Q *(s，a) ={E}′ s [R+γ{a′}{max}Q(s′, a′)|s, a] (13)

[0062] In the formula, Q *(s，a) is the optimal action value after optimization; γ is the discount factor; s′ is the state; a′ is the action;

[0063] 5.3.5) Take the optimized best action value as the next action value Q {t+1} (s, a), that is:

[0064] Q {t+1} (s, a) = {E}′ s [R+γ{a′}{max}Q t (s′, a′)|s, a]; (14)

[0065] Where R is the reward and E is the weighted sum of delay and energy.

[0066] Furthermore, the mean square error F between the action value Q(s′, a′; w) and the optimal action value Q*(s', a') is as follows:

[0067] F=r+γ{a′}{max}Q(s′,a′;w); (15)

[0068] Where r is the reward value generated by the action.

[0069] Furthermore, in the process of determining the task offloading decision, the loss function used when iteratively updating the parameter w matrix is ​​as follows:

[0070] Loss(w)={E} {s，a，r} [{E} s [F|s,a]-Q(s′,a′;w - )] (16)

[0071] In the formula, w - is the parameter matrix of the previous iteration.

[0072] The technical effect of the present invention is unquestionable. The present invention comprehensively considers the cloud side, edge server side and mobile device side, and can make the optimal task offloading decision according to the time and energy required for task offloading.

[0073] The present invention designs a universal edge-to-edge cloud network architecture and proposes a dependency-aware task offloading method based on deep reinforcement learning to jointly offload tasks with dependencies in the considered network.

[0074] The present invention models the dependencies between tasks through a directed acyclic graph and formulates the problem as minimizing the average time-energy weighted sum of all users. In order to solve this NP-hard problem, the present invention creatively uses deep neural networks to extract system features and proposes a framework based on deep Q-network to determine the task offloading strategy.

[0075] The reinforcement learning framework proposed in the present invention comprehensively considers both energy and delay factors in solving the dependency task scheduling problem, realizes the energy-delay weighted optimization, greatly reduces the communication delay and energy cost of wireless mobile devices, meets user needs, and saves link costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 It is the system model diagram;

[0077] Figure 2 It is the task model diagram;

[0078] Figure 3This is a diagram of the reinforcement learning architecture;

[0079] Figure 4 Flowchart of the algorithm for offloading dependent tasks. DETAILED DESCRIPTION

[0080] The present invention is further described below in conjunction with the embodiments, but it should not be understood that the above subject matter of the present invention is limited to the following embodiments. Without departing from the above technical ideas of the present invention, various substitutions and changes are made according to the common technical knowledge and customary means in the art, which should all be included in the protection scope of the present invention.

[0081] Embodiment 1:

[0082] See also Figures 1 to 4 , a multi-user dependent task offloading method based on edge computing and deep reinforcement learning, comprising the following steps:

[0083] 1) Establish a multi-user mobile edge computing system, including a cloud server, B edge base stations, and N wireless mobile devices; each base station has an edge server;

[0084] 2) Obtain edge computing network information, including task information generated by cloud servers, edge servers, and wireless mobile devices;

[0085] 3) For the nth application of the wireless mobile device, a directed acyclic graph is established to represent the dependency relationship between 1 dependent subtasks in the application;

[0086] 4) Based on the directed acyclic graph, a model of latency and energy consumption of dependent task offloading is established;

[0087] 5) According to the dependent task offloading delay and energy consumption model, a task offloading decision model is established, and the task offloading decision model is solved using the Markov decision method to obtain the task offloading decision.

[0088] The directed acyclic graph has an entry node and an exit node; wherein the entry node and the exit node represent the start task and the end task of the application respectively;

[0089] All tasks in the directed acyclic graph are arranged in descending order according to the rank value;

[0090] Among them, task q in The rank value rank(q in ) is as follows:

[0091]

[0092] In the formula, FT in Indicates unloading and processing task q inThe waiting time; (I+1)n represents the set of exit tasks. rank(q jn ) is task q jn rank value of .

[0093] The steps to establish the latency and energy consumption model of dependent task offloading include:

[0094] 4.1) According to the directed acyclic graph, the tasks A set of direct predecessors of

[0095] 4.2) Computational tasks The maximum completion time is:

[0096]

[0097] In the formula, when processing tasks locally hour, Ready time For the task The maximum completion time; The completion time of the task executed locally, on an edge server, or on a cloud server;

[0098] 4.3) Task The local execution time is expressed as The local average computing power is expressed as

[0099] Computational tasks Local execution time and completion time Right now:

[0100]

[0101]

[0102] In the formula, The CPU cycles required for the task;

[0103] 4.4) Compute tasks that are performed locally Energy consumption Right now;

[0104]

[0105] In the formula, κ is the energy consumption coefficient;

[0106] 4.5) The ready time of tasks established on the edge server Right now:

[0107]

[0108] In the formula, They represent the transmission time from the local to the edge server and from the edge server b to the edge server b′ respectively;

[0109] 4.6) Calculate the transmission time from edge server b to cloud server is the data size; is the transmission rate from edge server b to cloud server;

[0110] Computational tasks Execution time on cloud server f c The computing power of the cloud server;

[0111] Computational tasks Ready time on cloud servers Right now:

[0112]

[0113] In the formula, is the transmission time from the edge server to the cloud server;

[0114] 4.7) Calculate the weighted sum of the task offloading delay and energy of the nth application

[0115] in, is the total energy consumption of the mobile device; local energy consumption Energy consumption of upload tasks For unit energy consumption.

[0116] The objective function of the task offloading decision model is to minimize the weighted sum of the total delay and total energy of task offloading.

[0117] The steps of solving the task offloading decision model using the Markov decision method include:

[0118] 5.1) Establish the action space A and state space S during task offloading;

[0119] 5.2) Based on the objective function of the task offloading decision model, define the system reward R for each training t ;

[0120] 5.3) Based on system reward R t , determine the task offloading decision.

[0121] The state space S is as follows:

[0122] S={s|s=(G_n({I_{n}, θ_{n}), α_{1:i})}; (8)

[0123] Where G_n({I_{n}, θ_{n}) is the task embedding, including the index of the current task and the index of the immediate predecessor and successor tasks; α_{1:i} represents the task embedding from arrive uninstall decision.

[0124] In the action space A, the action A at time slot t t As shown below:

[0125] A t =0,1,2,…,|{B}+1|; (9)

[0126] In the formula, A t =0 means the task is executed locally, A t ={1, 2, ..., B} indicates that the task is offloaded to the edge server, A t =B+1 means that the task is offloaded to the cloud server.

[0127] System Reward R t As shown below:

[0128]

[0129] In the formula, ξ n The weighted sum of the total latency and total energy of the task offloading.

[0130] Based on system reward R t ,The steps to determine the task offloading decision include:

[0131] 5.3.1) Create an expression for action a, namely:

[0132] a={argmax}{Q(s,a′;w)}; (11)

[0133] In the formula, w is the parameter matrix; s is the state; Q is the action value Q * ;

[0134] 5.3.2) Perform action a in state s according to strategy π to obtain action value Q(s, a);

[0135] 5.3.3) Change action a, repeat step 5.3.2), obtain multiple action values ​​Q(s, a), and take the maximum action value as the optimal action value Q * (s, a), that is:

[0136] Q*(s,a)={max}[R t |s t=s,a t =a,π]; (12)

[0137] 5.3.4) For the optimal action value Q * (s, a) is optimized to obtain:

[0138] Q *(s，a) ={E}′ s [R+γ{a′}{max}Q(s′, a′)|s, a] (13)

[0139] In the formula, Q *(s，a) is the optimal action value after optimization; γ is the discount factor; s′ is the state; a′ is the action;

[0140] 5.3.5) Take the optimized best action value as the next action value Q {t+1} (s, a), that is:

[0141] Q {t+1} (s, a) = {E}′ s [R+γ{a′}{max}Q t (s′, a′)|s, a]; (14)

[0142] Where R is the reward and E is the weighted sum of delay and energy.

[0143] The mean square error F between the action value Q(s′, a′; w) and the optimal action value Q*(s', a') is as follows:

[0144] F=r+γ{a′}{max}Q(s′,a′;w); (15)

[0145] Where r is the reward value generated by the action.

[0146] In the process of determining the task offloading decision, the loss function used when iteratively updating the parameter w matrix is ​​as follows:

[0147] Loss(w)={E} {s，a，r} [{E}_s[F|s,a]-Q(s′,a′;w - )] (16)

[0148] In the formula, w - is the parameter matrix of the previous iteration.

[0149] Embodiment 2:

[0150] A multi-user dependent task offloading method based on edge computing and deep reinforcement learning includes the following steps:

[0151] 1) Establish a multi-user mobile edge computing system, which includes a cloud server, B edge base stations and n wireless mobile devices, where each base station has an edge server and each wireless mobile device runs an application n containing I dependent subtasks.

[0152] 2) Obtain edge computing network information, including the task information generated by the cloud server, edge server and user mobile device. The CPU cycles and input data size required for the task are expressed as and

[0153] 3) For application n, the present invention describes the dependency of its tasks through a directed acyclic graph with an entry node and an exit node, where the entry node and the exit node represent the start task and the end task of the application respectively. The present invention uses θn to represent the dependency of all tasks of n.

[0154] 4) The task offloading strategy is represented as α∈{0, 1, ..., B, B+1}, where 0 means that the task is executed locally, 1 means that it is offloaded to the corresponding edge server, and B+1 means that it is offloaded to the cloud server. Note that since the download transmission speed is very high, the present invention considers that the download transmission time can be ignored. In addition, the present invention defines the ready time of a task as the time when all its direct predecessor tasks complete processing, and the completion time is defined as the time when the task completes processing. Based on this, the preparation time and completion time of executing a task locally, on an edge server, or on a cloud server can be represented as RT, respectively. l in , RT e in , RT c in and FT l in , FT e in , FT c in

[0155] 5) According to the task offloading principles of mobile terminals, edge servers and cloud ends, and according to the initialization values ​​of their respective parameters, the time and energy expectation models of task offloading execution on mobile terminals, edge servers and cloud ends are constructed, and further based on the constructed models, the total delay and energy expectation model required for the application n on the mobile device to complete the series of subtasks is calculated.

[0156] 5.1) The present invention will task A set of direct predecessors of When processed locally hour, The ready time, i.e. Should be The maximum completion time. It can also be processed locally, offloaded to edge servers, or offloaded to cloud servers, so the task The maximum completion time of should be the sum of its maximum completion time and local execution time, which can be modeled as

[0157]

[0158] 5.2) In addition, the present invention will task The local execution time is expressed as MD n The average computing power is expressed as Its upper limit is therefore, Can be Computing, Tasks The completion time can be Computation. Tasks executed locally The energy consumption can be obtained by the following formula:

[0159]

[0160] 5.3) The task completion preparation time on the edge server is modeled as

[0161]

[0162] 5.4) The present invention expresses the transmission rate from edge server b to cloud server as The transmission time from b to the cloud service can be calculated as The present invention represents the cloud computing capability as f c , then the task The execution time can be calculated as Task The ready time in the cloud can be expressed as:

[0163]

[0164] 5.5) For each application n of the wireless mobile device, the invention defines the weighted sum of delay and energy as

[0165] Where E is the total energy consumption of each mobile device, which can be obtained by

[0166] where e in represents the energy consumption of local or upload tasks, which can be given by the following formula

[0167]

[0168]

[0169] Define the optimal decision problem for offloading dependent tasks on mobile devices, and use the weighted sum of minimizing the total delay and total energy of task offloading as the final optimization goal. According to the expected time and energy model required for task offloading obtained in step 5, transform the optimal decision problem into an objective function and its constraints.

[0170] The task offloading decision modeling is transformed into a Markov decision process, the action space A and the state space S in the task offloading process are defined, and the reward R for each training is defined according to the previously obtained objective function. The task offloading decision is effectively determined through a deep reinforcement learning algorithm.

[0171] 7.1) The latency and energy consumption of the system may be affected by many factors, such as the task profile (i.e., data size, required CPU cycles), the topology of the directed acyclic graph, the headers that are suppressed due to being too long, the wireless transmission rate, and the state of the mobile edge computing system. The state of the mobile edge computing system depends on the previous task offloading decisions. Therefore, the present invention defines the state space as a set of directed acyclic graph information and partial offloading plans

[0172] S={s|s=(G_n({I_{n}, θ_{n}), α_{1:i})},

[0173] where Gn(In,θn) consists of a set of task embeddings, α1:i represents the embedding from task arrive In addition, in order to convert the directed acyclic graph into a task embedding sequence, the present invention sorts the tasks of the directed acyclic graph in descending order according to the rank value, which can be defined as

[0174]

[0175] Where FTin represents the waiting time for unloading and processing task qin, and (I+1)n represents the set of exited tasks. Each task embedding consists of three vectors, which embed the index of the current task and the index of the immediate predecessor and successor tasks respectively.

[0176] 7.2) In each time slot t, the present invention assumes that only one task can be decided. Action A t It can be expressed as

[0177] A t =0,1,2,…,|{B}+1|,

[0178] Among them A t =0 means the task is executed locally, A t ={1, 2, ..., B} indicates that the task is offloaded to the edge server, A t=B+1 means that the task is offloaded to the cloud server.

[0179] 7.3) At each time slot t, a task needs to be determined, and the system is based on the current state S t Select the most appropriate uninstall actionA t After taking the action, the state of the mobile edge computing system changes from S t Change to S t+1 In order to minimize the average CFT of all users, the present invention defines the reward at time step t as the negative increment of the weighted sum of the CETs of all users in the mobile edge computing system. Then the system reward can be calculated as

[0180]

[0181] In addition, the present invention adopts a reward shaping method to select actions that satisfy the constraints during training. That is, the present invention constructs a reward value function that encourages the agent to satisfy the constraints by setting penalties, thereby reducing the reward when the constraints are not satisfied.

[0182] A deep reinforcement learning-based framework is proposed to model the dependency task offloading problem as a Markov decision process in mobile edge computing networks.

[0183] 8.1) The reinforcement learning agent adopts a balanced approach including exploitation and exploration to learn and interact with the mobile edge computing network. The exploitation approach is to exploit relevant experience with Q-values ​​obtained through greedy learning, which can be described as

[0184] a={argmax}{Q(s,a′;w)}

[0185] where w is the parameter matrix. In addition, the exploration agent can take random actions to obtain information about the mobile edge computing environment.

[0186] 8.2) In addition, the agent's goal is to find the best action value Q by * (s, a) and find the best decision π * , maximizing the long-term cumulative reward. The agent can obtain the action value Q(s, a) by executing action a in state s according to strategy π. Then, the present invention can obtain the optimal action value Q by selecting the maximum value of Q(s, a) among all possible values. * (s, a)

[0187] Q*(s,a)={max}[R t |s t =s,a t =a,π]

[0188] The optimal policy π* satisfies the Bellman equation and can optimize the action value Q*(s, a) as follows:

[0189] Q *(s，a) ={E}′ s [R+γ{a′}{max}Q(s′, a′)|s, a] (13)

[0190] The best action value step is to take action a from all possible actions for the next step of Q(s, a) that maximizes R+Q(s′, a′), and repeat this action until the best strategy is generated. In each round of iteration, by updating the Bellman equation, the Q value continues to increase, Qt→Q* as t→∞, as

[0191] Q {t+1} (s, a) = {E}′ s [R+γ{a′}{max}Q t (s′, a′)|s, a]

[0192] 8.3) In the DQN model proposed in the present invention, in order to prevent fluctuations and errors during training, the DQN model introduces a target network with fixed parameters w-. The target network parameters can remain unchanged in some cases. Another neural network called the main network with parameters w can continuously learn from the experience buffer. Every k steps, the main network updates its parameters to the target network to minimize the loss function. The current action value with Q(s′, a′; w) is the same as the optimal Q * The mean square error between (s', a') can be replaced by the fixed term F:

[0193] F=r+γ{a′}{max}Q(s′,a′;w)

[0194] where w - Updated in the previous iteration. The loss function can be given by

[0195] Loss(w)={E} {s，a，r} [{E}_s[F|s,a]-Q(s′,a′;w - )]

[0196] Embodiment 3:

[0197] A multi-user dependent task offloading method based on edge computing and deep reinforcement learning includes the following steps:

[0198] 1) Establishing the multi-user mobile edge computing system, including a cloud server, B edge base stations and N wireless mobile devices; each base station has an edge server.

[0199] 2) Obtain edge computing network information, including task information generated by cloud servers, edge servers, and wireless mobile devices;

[0200] 3) For the nth application of the wireless mobile device, a directed acyclic graph is established to represent the dependency relationship between I dependent subtasks in the application;

[0201] 4) Based on the directed acyclic graph, a model of latency and energy consumption of dependent task offloading is established;

[0202] 5) According to the dependent task offloading delay and energy consumption model, a task offloading decision model is established, and the task offloading decision model is solved using the Markov decision method to obtain the task offloading decision.

[0203] Embodiment 4:

[0204] A multi-user dependent task offloading method based on edge computing and deep reinforcement learning, the main content of which is shown in Example 3, wherein the directed acyclic graph has an entry node and an exit node; wherein the entry node and the exit node represent the start task and the end task of the application respectively;

[0205] All tasks in the directed acyclic graph are arranged in descending order according to the rank value;

[0206] Among them, task q in The rank value rank(q in ) is as follows:

[0207]

[0208] In the formula, FT in Indicates unloading and processing task q in The waiting time; (I+1)n represents the set of exit tasks. rank(q jn ) is task q jn rank value of .

[0209] Embodiment 5:

[0210] A multi-user dependent task offloading method based on edge computing and deep reinforcement learning, the main content of which is shown in Example 3, wherein the step of establishing a dependent task offloading delay and energy consumption model includes:

[0211] 1) According to the directed acyclic graph, the tasks A set of direct predecessors of

[0212] 2) Computational tasks The maximum completion time is:

[0213]

[0214] In the formula, when processing tasks locally hour, Ready time For the task The maximum completion time; The completion time of the task executed locally, on an edge server, or on a cloud server;

[0215] 3) Task The local execution time is expressed as The local average computing power is expressed as

[0216] Computational tasks Local execution time and completion time Right now:

[0217]

[0218]

[0219] In the formula, The CPU cycles required for the task;

[0220] 4) Compute tasks that are performed locally Energy consumption Right now;

[0221]

[0222] In the formula, κ is the energy consumption coefficient;

[0223] 5) The ready time of tasks established on the edge server Right now:

[0224]

[0225] In the formula, They represent the transmission time from the local to the edge server and from the edge server b to the edge server b′ respectively;

[0226] 6) Calculate the transmission time from edge server b to cloud server is the data size; is the transmission rate from edge server b to cloud server;

[0227] Computational tasks Execution time on cloud server f c The computing power of the cloud server;

[0228] Computational tasks Ready time on cloud servers Right now:

[0229]

[0230] In the formula, is the transmission time from the edge server to the cloud server;

[0231] 7) Calculate the weighted sum of the task offloading delay and energy of the nth application

[0232] in, is the total energy consumption of the mobile device; local energy consumption Energy consumption of upload tasks For unit energy consumption.

[0233] Embodiment 6:

[0234] A multi-user dependent task offloading method based on edge computing and deep reinforcement learning, the main content of which is shown in Example 3, wherein the objective function of the task offloading decision model is: the weighted sum of the total delay and total energy of task offloading is minimized.

[0235] Embodiment 7:

[0236] A multi-user dependent task offloading method based on edge computing and deep reinforcement learning, the main content of which is shown in Example 3, wherein the step of solving the task offloading decision model using the Markov decision method includes:

[0237] 1) Establish the action space A and state space S in the task offloading process;

[0238] 2) According to the objective function of the task offloading decision model, define the system reward R for each training t ;

[0239] 3) Based on system reward R t , determine the task offloading decision.

[0240] Embodiment 8:

[0241] A multi-user dependent task offloading method based on edge computing and deep reinforcement learning, the main content of which is shown in Example 3, wherein the state space S is as follows:

[0242] S={s|s=(G_n({I_{n}, θ_{n}), α_{1:i})}; (8)

[0243] Where G_n({I_{n}, θ_{n}) is the task embedding, including the index of the current task and the index of the immediate predecessor and successor tasks; α_{1:i} represents the task embedding from arrive uninstall decision.

[0244] In the action space A, the action A at time slot t t As shown below:

[0245] A t =0,1,2,…,|{B}+1|; (9)

[0246] In the formula, A t =0 means the task is executed locally, A t ={1, 2, ..., B} indicates that the task is offloaded to the edge server, A t =B+1 means that the task is offloaded to the cloud server.

[0247] Embodiment 9:

[0248] A multi-user dependent task offloading method based on edge computing and deep reinforcement learning, the main content of which is shown in Example 3, wherein the system rewards R t As shown below:

[0249]

[0250] In the formula, ξ n The weighted sum of the total latency and total energy of the task offloading.

[0251] Embodiment 10:

[0252] A multi-user dependent task offloading method based on edge computing and deep reinforcement learning, the main content of which is shown in Example 3, wherein based on the system reward R t ,The steps to determine the task offloading decision include:

[0253] 1) Create an expression for action a, namely:

[0254] a={argmax}{Q(s,a′;w)}; (11)

[0255] Where w is the parameter matrix; s is the state; Q is the action value; a′ is the action;

[0256] 2) Execute action a in state s according to strategy π to obtain action value Q(s, a);

[0257] 3) Change action a and repeat step 2) to obtain multiple action values ​​Q(s, a), and take the maximum action value as the optimal action value Q * (s, a), that is:

[0258] Q*(s,a)={max}[R t |s t =s,a t =a,π]; (12)

[0259] 4) For the optimal action value Q * (s, a) is optimized to obtain:

[0260] Q *(s，a) ={E}′ s [R+γ{a′}{max}Q(s′, a′)|s, a] (13)

[0261] In the formula, Q *(s，a) is the optimal action value after optimization; γ is the discount factor; s′ is the state; a′ is the action;

[0262] 5) Take the optimized best action value as the next action value Q {t+1} (s, a), that is:

[0263] Q {t+1} (s, a) = {E}′ s [R+γ{a′}{max}Q t (s′, a′)|s, a]; (14)

[0264] Where R is the reward and E is the weighted sum of delay and energy.

[0265] Embodiment 11:

[0266] A multi-user dependent task offloading method based on edge computing and deep reinforcement learning, the main content of which is shown in Example 3, wherein the mean square error F between the action value Q(s′, a′; w) and the optimal action value Q*(s′, a′) is as follows:

[0267] F=r+γ{a′}{max}Q(s′,a′;w); (15)

[0268] Where r is the reward value generated by the action.

[0269] Embodiment 12:

[0270] A multi-user dependent task offloading method based on edge computing and deep reinforcement learning, the main content of which is shown in Example 3, wherein in the process of determining the task offloading decision, when iteratively updating the parameter w matrix, the loss function used is as follows:

[0271] Loss(w)={E} {s，a，r} [{E} s [F|s,a]-Q(s′,a′;w -)] (16)

[0272] In the formula, w - is the parameter matrix of the previous iteration.

Claims

1. A multi-user dependent task offloading method based on edge computing and deep reinforcement learning, characterized in that: The following steps are involved: 1) Establish a multi-user mobile edge computing system, including a cloud server, B edge base stations, and N wireless mobile devices; each base station has an edge server; 2) Obtain edge computing network information, including task information generated by cloud servers, edge servers, and wireless mobile devices; 3) For the nth application of the wireless mobile device, a directed acyclic graph is established to represent the dependency relationship between I dependent subtasks in the application; 4) Based on the directed acyclic graph, a model of latency and energy consumption of dependent task offloading is established; 5) According to the dependent task offloading delay and energy consumption model, a task offloading decision model is established, and the task offloading decision model is solved using the Markov decision method to obtain the task offloading decision; The steps to establish the latency and energy consumption model of dependent task offloading include: 4.1) According to the directed acyclic graph, the tasks The set of immediate predecessors of 4.2) Computational tasks The maximum completion time is: In the formula, when processing tasks locally hour, Ready time For the task The maximum completion time; The completion time of the task executed locally, on an edge server, or on a cloud server; 4.3) Task The local execution time is expressed as The local average computing power is expressed as Computational tasks Local execution time and completion time Right now: In the formula, The CPU cycles required for the task; 4.4) Compute tasks that are performed locally Energy consumption Right now; In the formula, κ is the energy consumption coefficient; 4.5) The ready time of tasks established on the edge server Right now: In the formula, They represent the transmission time from the local to the edge server and from the edge server b to the edge server b′ respectively; 4.6) Calculate the transmission time from edge server b to cloud server is the data size; is the transmission rate from edge server b to cloud server; Computational tasks Execution time on cloud server f c The computing power of the cloud server; Computational tasks Ready time on cloud servers Right now: In the formula, is the transmission time from the edge server to the cloud server; 4.7) Calculate the weighted sum of the task offloading delay and energy of the nth application in, is the total energy consumption of the mobile device; local energy consumption Energy consumption of upload tasks is the unit energy consumption; The steps of solving the task offloading decision model using the Markov decision method include: 5.1) Establish the action space A and state space S during task offloading; 5.2) Based on the objective function of the task offloading decision model, define the system reward R for each training t ; 5.3) Based on system reward R t , determine the task offloading decision; The state space S is as follows: S={s|s=(G_n({_{n},θ_{n}),α-{1:i})}; (7) Where G_n({I_{n},θ_{n}) is the task embedding, including the index of the current task and the index of the immediate predecessor and successor tasks; α_{1:i} represents the task embedding from task arrive Uninstallation decision; In the action space A, the action A at time slot t t As shown below: A t =0,1,2,…,|{B}+1|;(8) In the formula, A t =0 means the task is executed locally, A t ={1, 2, ..., B} indicates that the task is offloaded to the edge server, A t =B+1 means that this task is offloaded to the cloud server; System Reward R t As shown below: In the formula, ξ n The weighted sum of the total latency and total energy of the task offloading; Based on system reward R t ,The steps to determine the task offloading decision include: 5.3.1) Create an expression for action a, namely: a={argmax}{Q(s,a′;w)}; (10) Where w is the parameter matrix; s is the state; Q is the action value; a′ is the action; 5.3.2) Perform action a in state s according to strategy π to obtain action value Q(s, a); 5.3.3) Change action a, repeat step 5.3.2), obtain multiple action values ​​Q(s, a), and take the maximum action value as the optimal action value Q*(s, a), that is: Q*(s, a)={max}[R t |s t =s, a t =a,π]; (11) 5.3.4) Optimize the best action value Q*(s, a) and get: Q *(s，a) ={E}′ s [R+γ{a′}{max}Q(s′,a′)|s,a] (12) In the formula, Q *(s，a) is the optimal action value after optimization; γ is the discount factor; s′ is the state; a′ is the action; 5.3.5) Take the optimized best action value as the next action value Q {t+1} (s, a), that is: Q {t+1} (s,a)={E}′ s [R+γ{a′}{max}Q t (s′,a′)|s,a]; (13) Where R is the reward; E is the weighted sum of delay and energy.

2. According to claim 1, a multi-user dependent task offloading method based on edge computing and deep reinforcement learning is characterized in that: The directed acyclic graph has an entry node and an exit node; wherein the entry node and the exit node represent the start task and the end task of the application respectively; All tasks in the directed acyclic graph are arranged in descending order according to the rank value; Among them, task q in The rank value rank(q in ) is as follows: In the formula, FT in Indicates unloading and processing task q in The waiting time; (I+1)n represents the set of exit tasks; rank(q jn ) is task q jn rank value of .

3. According to claim 1, a multi-user dependent task offloading method based on edge computing and deep reinforcement learning is characterized in that: The objective function of the task offloading decision model is to minimize the weighted sum of the total delay and total energy of task offloading.

4. According to claim 1, a multi-user dependent task offloading method based on edge computing and deep reinforcement learning is characterized in that: The mean square error F between the action value Q(s′, a′; w) and the optimal action value Q*(s', a') is as follows: F=r+γ{a′}{max}Q(s′,a′;w); (15) Where r is the reward value generated by the action.

5. According to claim 1, a multi-user dependent task offloading method based on edge computing and deep reinforcement learning is characterized in that: In the process of determining the task offloading decision, the loss function used when iteratively updating the parameter w matrix is ​​as follows: LOss(w)={E} {s，a，r} [{E} s [F|s,a]-Q(s′,a′;w - )] (16) In the formula, w - is the parameter matrix of the previous iteration.

Images