A workflow task offloading method based on Markov decision process

Abstract

The present invention discloses a workflow task offloading method based on a Markov decision process, comprising a preprocessing stage: monitoring and recording resources of the entire cloud-edge-end system, defining an adjacency matrix table of a server set and a network topology; a solution stage: for each workflow to be offloaded, constructing a direct predecessor table, an offloaded task set, a to-be-offloaded task set, etc., building a Markov decision process model based on the workflow, and selecting the best decision in the current state according to a value function; an update stage: updating various system parameters according to the state updates generated by different reasons. The present invention uses a Markov decision process to solve the workflow offloading problem in a heterogeneous cloud-edge-end scenario, optimizes the fastest task completion time, and has broad application value and usage prospects in the field of mobile Internet of Things.

Description

Technical Field

[0001] The present invention relates to a workflow task offloading method based on a Markov decision process, and belongs to the technical fields of cloud computing application, scheduling optimization, and task offloading. Background Art

[0002] In recent years, the widespread use of mobile devices is driving the development of delay-sensitive and resource-intensive applications, such as face recognition and data stream processing. Currently, these applications are executed on mobile devices or cloud platforms. On the one hand, mobile devices have relatively small computing resources. On the other hand, running resource-intensive applications in cloud computing centers usually requires transmitting large amounts of data between mobile devices and remote servers, resulting in unpredictable communication delays [1]. To address the above issues, it is considered to offload tasks to the edge of the network [2] to make up for the shortcomings of mobile devices and remote clouds, thereby enhancing computing power while shortening transmission delays.

[0003] The rapid development of mobile devices has led to a rapid increase in the complexity of applications. Many applications are not independent tasks, but are composed of several interdependent subtasks (workflows). For example, a face recognition application can be divided into the following interdependent tasks: target acquisition, face detection, preprocessing, feature extraction, and classification [3]. In this example, the fifth subtask, classification, must be completed after the fourth step, feature extraction. This is a possible dependency between tasks. Obviously, a subtask that depends on other subtasks cannot be offloaded before its predecessor subtask is completed.

[0004] Many researchers have proposed edge offloading methods for different scenarios, but these methods all have certain limitations. Liu et al. [4] minimized the application completion time as the optimization goal, considering the dependencies between subtasks, but did not consider the resource constraints of the edge server. In fact, edge servers are resource-sensitive and heterogeneous, which needs to be considered when making offloading decisions. Yang et al. [5] used Markov decision making to consider the heterogeneity of network edge resources, but their tasks were independent. In fact, applications are becoming more and more complex, with more and more tasks with dependencies. In addition, the computing power of edge computing networks is limited. Both of the above characteristics should be considered.

[0005] The Markov decision process is a method in reinforcement learning that describes an observable system. The observable state content in the system determines the features required for decision-making, and the Markov decision process determines the next state based on these observed features. The offloading method proposed in this paper is based on the Markov decision process. Each state can be defined as the Cartesian product of task and server resource characteristics. Each state transition is determined by the edge server where the current state resides and its available resources.

[0006] [1]M.-H.Chen,B.Liang,and M.Dong,“Joint offloading and resourceallocation for computation and communication in mobile cloud with computingaccess point,”in Proc.IEEE Conf.Comput.Commun.,2017,pp.1-9.

[0007] [2]X.Wang,Y.Han,C.Wang,Q.Zhao,X.Chen,and M.Chen,“In-edge AI:Intelligentizing mobile edge computing,caching and communication by federatedlearning,”IEEE Netw.,vol.33,no.5,pp.156-165,Sep. / Oct.2019,doi:10.1109 / MNET.2019.1800286.

[0008] [3]G.Zhao,H.Xu,Y.Zhao,C.Qiao and L.Huang,″Offloading Tasks WithDependency and Service Caching in Mobile Edge Computing,”in IEEE Transactionson Parallel and Distributed Systems,vol.32,no.11,pp.2777-2792,1 Nov.2021,doi:10.1109 / TPDS.2021.3076687.

[0009] [4]L.Liu,H.Tan,S.H.-C.Jiang,Z.Han,X.-Y.Li,and H.Huang,“Dependent taskplacement and scheduling with function configuration in edge computing,”inProc.ACM Int.Symp.Qual.Serv.,2019,pp.1-10.

[0010] [5] G.Yang, L.Hou, X.He, D.He, S.Chan and M.Guizani, "Offloading TimeOptimization via Markov Decision Process in Mobile-Edge Computing," in IEEEInternet of Things Journal, vol.8, no.4, pp.2483-2493, 15Feb.15, 2021, doi: 10.1109 / JIOT.2020.3033285. Summary of the Invention

[0011] The present invention considers offloading workflow tasks in resource-constrained heterogeneous edge networks, with the goal of obtaining a set of offloading strategies with minimum completion time. The rate λ of workflows arriving at each edge server follows a Poisson distribution. l ={a l , G l}, a l Represents workflow WF l The arrival time, G l Represents workflow WF l The topological structure of G l It is further modeled as a DAG graph, namely G l =(T l , E l ), where T l ={task1, task2,...,task m} represents the task in the workflow, i.e., the vertex of the DAG graph, m is the number of tasks, E l ={e i，j |task i , task j ∈T l} represents the dependency relationship between tasks, that is, the edge of the DAG graph, such as e i，j Indicates that the task i Point to task task j The arrow represents the task i The task cannot be executed until it is completed. j Each task i ={inst1, inst2,..., inst n} is represented as a set of function instances, representing the set of function instances that need to be executed for the task, where each instance x =(type x , cpu x, mem x ,exec x ) is represented as a four-tuple, type x Indicates the type of the function, cpu x Indicates the computing resources occupied by an instance of the function, mem x Indicates the storage resources occupied by an instance of the function, exec x Indicates the execution time of an instance of this function on the server.

[0012] Purpose of the invention: To overcome the shortcomings of the existing technology, this invention provides an edge offloading method that minimizes execution time by using a Markov decision process for workflows with partial order relationships and considering resource-limited and heterogeneous cloud-edge environments. Objective function:

[0013] Technical solution: To achieve the above purpose, the technical solution adopted by the present invention is:

[0014] A workflow task offloading method based on Markov decision process includes the following stages:

[0015] A. Preprocessing phase: Record the computing resources, storage resources, and function images of each edge server to construct a server set; build an adjacency matrix table based on the network topology of the edge servers, and the elements in the table represent the distance between edge servers.

[0016] B. Solution Phase: For each workflow to be offloaded, we first construct a table of the workflow's immediate predecessors, the set of offloaded tasks, the set of tasks to be offloaded, and the candidate set based on the workflow's topology. We then perform Markov decision process modeling on the workflow, defining the state space, action set, and reward function. Finally, we evaluate the optimal decision in the current state based on the value function to form a decision set.

[0017] C. Update phase: Based on the status updates caused by different reasons, the parameters of the task set to be uninstalled, the task set that has been uninstalled, the candidate set, the server set, etc. are updated respectively.

[0018] The specific steps of the pre-processing stage are as follows:

[0019] A1. Record the computing resources, storage resources, and function images of each edge server, and define the server set M = ES, CS, where the edge server set ES = {es i , cpu i , mem i ,Inst i}, where es i Indicates that the edge server is numbered i, cpu iIndicates the number of remaining CPU cores of edge server i, mem i Indicates the remaining memory of edge server i, Inst i represents the function image stored in edge server i, 0≤i≤N, N is the total number of edge servers; the cloud server set CS={cs,∞,∞,Inst all}, Inst all Indicates that the cloud server has all types of function images.

[0020] A2. Based on the network topology, construct an adjacency matrix adj[N+1][N+1], which represents the distance between servers. The first N rows and N columns represent the N edge servers, and the N+1 row and N+1 column represent the cloud server. The distances from the cloud server to all edge servers are very large.

[0021] The solution phase includes the following steps:

[0022] B1. Construct the direct predecessor table of the workflow Pretable = {task i , pre i}, pre i Indicates task i The direct predecessor set, To directly point to the task of taski;

[0023] B2. Construct the workflow's offloaded task set Comp and the to-be-offloaded task set Wait;

[0024] B3. Construct the candidate set Cand, which represents the set of tasks to which the next state can be transferred. At the initial moment, there is only one task in the candidate set, which is the first node in the topological sorting of the workflow tasks. This node has no direct predecessor.

[0025] B4. Define the state space S = {Tasks × M}, where × represents the Cartesian product, Tasks represents the task set, i.e., all task nodes in the workflow; and M represents the server set. For example, s i =task i ,{es i ,cpu i ,mem i ,Inst i} indicates that the current state is task task i With computing resources cpu i , storage resource mem i And function mirror Inst i Serveres i Run on;

[0026] B5. Define action set A = {a1, a2, ..., a n ,...,a N}∩a c , which indicates the action taken to transfer from the current state to the next state, i.e., deciding which server to select to offload the task. m =1 means select the mth server for uninstallation, a m =0 means not to select the mth server for uninstallation; a c Indicates whether to offload to the cloud server. c =1 means offloading to the cloud server, a c =0 means not offloading to the cloud server;

[0027] B6. Define the reward function R(s,a), which represents the reward value that can be obtained in a certain state. R(s,a)=-(T trans +T exec ), where T trans Indicates the transmission time, e.g. Indicates task i From the current server i Offload to server The transmission time, T exec Indicates execution time, for example Indicates task i On the server The execution time on . Among them D i Indicates task i The amount of data, adj i,j Indicates task i Server es i and The distance between them, B represents the task i Server es i and The bandwidth between. By task i The execution time of all function instances that need to be executed is accumulated.

[0028] B7. Define the expected reward to select the optimal strategy based on it to minimize the unloading time. Define the action value function It represents the expected return of following the strategy π when the current state is s. represents the expected return when the selected strategy is π and the current state is s. represents the expected function for calculating the expected reward when the strategy is π and the current state is s; R t (s,a) indicates the execution of task tThe immediate reward of the offloading decision, γ is the discount factor, γ∈(0,1]. Specifically, the goal is to minimize the offloading time by choosing the optimal policy in δ, where δ is a set of π. The maximum expected reward Thus, the optimal strategy At this point, select the optimal strategy π under the current state * ;

[0029] B8. State transfer. This includes the following steps:

[0030] B8.1 Select a task from the candidate set Cand i As the task to be offloaded in the current state, the priority is to select the task whose required function image is the same as the function image owned by the server in the previous state. If there is no such task, a random one is selected from the candidate set;

[0031] B8.2 Traverse the edge server function image library, (1) for those who do not have task i Edge server ES of the required function image without Each edge server in (2) For those who have tasks i Edge server ES of the required function image hold Each edge server in Calculate selected server The expected rewards of the uninstalled strategies, from which a strategy with the largest expected reward π is selected * (s);

[0032] B8.3 According to strategy π * (s), transfer to the next state.

[0033] The update phase includes the following steps:

[0034] System status updates occur in the following two situations: (1) a task node arrives at a server and is about to occupy the server resources and start execution; (2) a task is completed on a server and is about to release the server resources.

[0035] C1. If the status update is caused by the start of task execution, the system will update parameters such as the set of tasks to be offloaded Wait, the set of offloaded tasks Comp, the candidate set Cand, and the server set M;

[0036] C2. If the status update is caused by the completion of the task, the system will update the parameter server set M.

[0037] C3. Go to step B8.

[0038] Beneficial effects: Compared with the prior art, the workflow task offloading method based on Markov decision process provided by the present invention has the following beneficial effects:

[0039] (1) This paper considers workflow tasks with partial ordering, aiming to obtain an offloading strategy with the minimum completion time. The execution order of workflows with dependencies is uncertain during offloading. The proposed method can select the execution order of subtasks with the minimum completion time.

[0040] (2) Edge servers have limited resources, and the resource distribution of each edge server is different and changes dynamically. The Markov decision process can quickly select the best decision in the current state based on the current resource situation. Compared with traditional heuristic rules, this method has stronger perception and adaptability to resource changes.

[0041] (3) The Markov decision process combined with reinforcement learning has learning characteristics, which makes the unloading strategy become more and more optimal as the number of iterations increases. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of workflow topology and network topology in the example of the present invention.

[0043] Figure 2 It is a schematic diagram of the scenario and basic principle of the method of the present invention.

[0044] Figure 3 Schematic diagram of the state transition of the Markov decision process of the method of the present invention.

[0045] Figure 4 It is a flowchart of a specific implementation algorithm of the method of the present invention. DETAILED DESCRIPTION

[0046] The present invention is further illustrated below with reference to the accompanying drawings and specific implementation examples. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, modifications of various equivalent forms of the present invention made by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0047] Example:

[0048] like Figure 2 The figure shows a workflow task offloading method based on the Markov decision process. It comprehensively considers constraints such as the heterogeneous and limited server resources, the dynamic arrival of workflows, and the minimization of completion time. In the process of making offloading decisions, the Markov decision process is combined with the reinforcement learning method to dynamically analyze the impact of task scheduling sequence and system resources, and minimize the completion time of each workflow.

[0049] Assume that at this time the server es1 arrives at Figure 1 The workflow shown on the left, A and B respectively represent the function instance types required for each task. The resource situation in the system is as follows Figure 1 As shown on the right, it includes the following stages:

[0050] A. Preprocessing phase: Record the computing resources, storage resources, and function images of each edge server to construct a server set; build an adjacency matrix table based on the network topology of the edge servers, and the elements in the table represent the distance between edge servers.

[0051] B. Solution phase: For each workflow to be offloaded, we first construct a table of the workflow's immediate predecessors, the set of offloaded tasks, the set of tasks to be offloaded, and the candidate set. We then perform Markov decision process modeling on the workflow, defining the state space, action set, reward function, and expected reward. Finally, we select the optimal strategy in the current state based on the reward and add it to the decision set.

[0052] C. Update Phase: System status updates occur in the following two situations: (1) a task node arrives at a server and is about to occupy the server's resources and begin execution; (2) a task is completed on a server and is about to release the server's resources. After the system status update occurs, parameters such as the set of tasks to be offloaded, the set of offloaded tasks, the candidate set, and the server resource set are updated.

[0053] The specific steps of the pre-processing stage are as follows:

[0054] A1. Record the computing resources, storage resources, and function images of each edge server and define the server set M = ES, CS, where the edge server set ES = {{es1, 6, 3, A}, {es2, 3, 2, B}, {es3, 2, 1, A}}. In this embodiment, there is no cloud server, so the cloud server set Therefore, M = {{es1, 6, 3, A}, {es2, 3, 2, B}, {es3, 2, 1, A}}.

[0055] A2. Construct an adjacency matrix table adj[3][3]=[[0, 1, 1.2], [1, 0, 0.2], [1.2, 0.2, 0]] based on the network topology, where adj[i, j] represents the distance between server i and server j.

[0056] The specific steps of the solution phase are as follows:

[0057] B1. According to the workflow topology G = (V, E), where V = {{1, A}, {2, B}, {3, A}, {4, B}, {5, A}, {6, B}}, E = {(1, 2), (1, 3), (2, 4), (2, 5), (3, 5), (4, 6), (5, 6)}, construct the workflow's direct predecessor table Pretable = {task i , pre i}, pre i Indicates task i The direct predecessor set, To point directly to the task i Task, Pretable = {1: {}; 2: {1}; 3: {1}; 4: {2}; 5: {2, 3}; 6: {4, 5}}

[0058] B2. Construct the workflow's offloaded task set Comp = {}, and the to-be-offloaded task set Wait = {1,2,3,4,5,6};

[0059] B3. Construct the candidate set Cand, which represents the set of tasks that can transition to the next state. At the initial moment, there is only one task in the candidate set, which is the first node in the topological sorting of the workflow tasks. This node has no direct predecessor, that is, Cand = {1};

[0060] B4. Define the state space s i =task i ,{es i ,cpu i ,mem i ,Inst i} indicates that the current state is task task i With computing resources cpu i , storage resource mem i And function mirror Inst i Serveres i Run on;

[0061] B5. Define the action set A = {a1, a2, a3}, which represents the action taken to transfer from the current state to the next state, that is, to decide which server to select for offloading tasks. m =1 means select the mth server for uninstallation, a m =0 means not to select the mth server for uninstallation; a c Indicates whether to offload to the cloud server. c =1 means offloading to the cloud server, a c=0 means not to offload to the cloud server. In the current embodiment, there is no cloud server, so the action set does not include a c ;

[0062] B6. Define the reward function R(s,a), which represents the reward value that can be obtained in a certain state. R(s,a)=-(T trans +T exec ), where T trans Indicates the transmission time, e.g. Indicates task i From the current server i Offload to server The transmission time, T exec Indicates execution time, for example Indicates task i On the server The execution time on . Among them D i Indicates task i The amount of data, adj i,j Indicates task i Server es i and The distance between them, B represents the task i Server es i and The bandwidth between. By task i The execution time of all function instances that need to be executed is accumulated.

[0063] B7. Define the expected reward to select the optimal strategy based on it to minimize the unloading time. Define the action value function It represents the expected return of following the strategy π when the current state is s. represents the expected return when the selected strategy is π and the current state is s. represents the expected function for calculating the expected reward when the strategy is π and the current state is s; R t (s,a) indicates the execution of task t The immediate reward for the offloading decision, γ is the discount factor, in this example, γ = 0.5. Specifically, the goal is to minimize the offloading time by choosing the best policy in δ, where δ is a set of π. Maximum expected reward Thus, the optimal strategy

[0064] B8. State transfer, specifically including the following steps:

[0065] B8.1 Select a task from the candidate set Candi =1 is the task to be offloaded in the current state. The task whose function image required by this task is the same as the function image owned by the server in the previous state is preferred. If there is no such task, a random one is selected from the candidate set.

[0066] B8.2 Traverse the edge server function image library, (1) for those who do not have task i Edge server ES of the required function image without Each edge server in (2) For those who have tasks i Edge server ES of the required function image hold Each edge server in Calculate selected server The expected rewards of the uninstalled strategies, from which a strategy with the largest expected reward π is selected * (s). In this embodiment, since the function type A required by task=1 happens to be the function image owned by server es1, the uninstallation strategy es1 is selected, that is, a1=1, and it is directly uninstalled on es1. * (s) = a1 = 1;

[0067] B8.3 According to strategy π * (s), transfer to the next state s′.

[0068] The update phase includes the following steps:

[0069] System status updates occur in the following two situations: (1) a task node arrives at a server and is about to occupy the server resources and start execution; (2) a task is completed on a server and is about to release the server resources.

[0070] C1. If the state update is caused by the start of task execution, the system will update parameters such as the pending task set Wait, the offloaded task set Comp, the candidate set Cand, and the server set M. The current state is caused by the offloading of task task = 1 to server es1. At this time, the state is S = {1, {es1, 4, 2, A}}. The candidate set Cand is updated to {2, 3}, the pending task set Wait = {2, 3, 4, 5, 6}, the offloaded task set Comp = {1}, and the server M = {{es1, 4, 2, A}, {es2, 3, 2, B}, {es3, 2, 1, A}}.

[0071] C2. If the status update is caused by the completion of the task, the system will update the parameter server set M.

[0072] C3. Go to step B8: Since the current state is on server es1, and the function type required by task = 3 in the candidate set is owned by es1, task = 3 is selected from the candidate set as the node to be uninstalled. Evaluate the value of each behavior in the action set A and select the optimal strategy, i.e., π * (s′) = a1 = 1. This process of updating state parameters, selecting nodes to offload, evaluating the value of each strategy, and selecting the optimal strategy repeats repeatedly until the set of candidate offloads is empty, and the algorithm ends. The final scheduling result is: Res = {(1, es1), (3, es1), (2, es2), (4, es2), (5, es3), (6, es2)}, and the final completion time is: 2.8 seconds.

[0073] It should be noted that the above embodiments are not intended to limit the scope of protection of the present invention, and equivalent changes or substitutions made on the basis of the above technical solutions fall within the scope of protection of the claims of the present invention.

Claims

1. A workflow task offloading method based on Markov decision process, characterized by: The following stages are included: A. Preprocessing phase: Record the computing resources, storage resources, and function images of each edge server and construct a server set; Construct an adjacency matrix table based on the network topology of the edge servers, where the elements in the table represent the distances between the edge servers; B. Solution Phase: For each workflow to be offloaded, we first construct a table of the workflow's immediate predecessors, the set of offloaded tasks, the set of tasks to be offloaded, and the candidate set. We then perform Markov decision process modeling on the workflow, defining the state space, action set, reward function, and expected reward. Finally, we select the optimal strategy in the current state based on the reward and add it to the decision set. C. Update phase: System status updates occur in the following two situations: (1) a task node arrives at a server and is about to occupy the server resources and start execution; (2) a task is completed on a server and is about to release the server resources. After the system generates a status update, the parameters of the task set to be uninstalled, the task set that has been uninstalled, the candidate set, and the server resource set are updated respectively; The solution phase includes the following steps: B1. Construct the direct predecessor table of the workflow Pretable = {task i ,pre i }, pre i Indicates task i The direct predecessor set, To point directly to the task i mission; B2. Construct the workflow's offloaded task set Comp and the to-be-offloaded task set Wait; B3. Construct the candidate set Cand, which represents the task set for the next state transition. B4. Define the state space S = {Tasks × M}, where × represents the Cartesian product, Tasks represents the task set, i.e., all task nodes in the workflow; and M represents the server set. B5. Define action set A = {a1, a2, ..., a n ,...,a N }∩a c , represents the action taken to transfer from the current state to the next state, a m =1 means select the mth server for uninstallation, a m =0 means not to select the mth server for uninstallation; a c Indicates whether to offload to the cloud server. c =1 means offloading to the cloud server, a c =0 means not offloading to the cloud server; B6. Define the reward function R(s,a), which represents the reward value that can be obtained in a certain state. R(s,a)=-(T trans +T exec ), where T trans represents the transmission time, T exec Indicates the execution time, Among them D i Indicates task i The amount of data, adj i,j Indicates task i Server es i and the distance between them; By task i The execution time of all function instances that need to be executed is accumulated; B7. Define the expected reward to select the optimal strategy based on it to minimize the unloading time; B8. State transfer, specifically including the following steps: B8.1 Select a task from the candidate set Cand i As the task to be offloaded in the current state, the priority is to select the task whose required function image is the same as the function image owned by the server in the previous state. If there is no such task, a random one is selected from the candidate set; B8.2 Traverse the edge server function image library, (1) for those who do not have task i Edge server ES of the required function image without Each edge server in (2) For those who have tasks i Edge server ES of the required function image hold Each edge server in Calculate selected server The expected rewards of the uninstalled strategies, from which a strategy with the largest expected reward π is selected * (s); B8.3 According to strategy π * (s), transfer to the next state.

2. The workflow task offloading method based on Markov decision process according to claim 1, characterized in that: A. Preprocessing stage: The specific steps are as follows: A1. Record the computing resources, storage resources, and function image libraries of each edge server and define a server set. Each element of the set includes the server ID, the number of remaining CPU cores on the server, and the remaining memory on the server. A2. Construct an adjacency matrix table based on the network topology to represent the distance between servers.

3. The workflow task offloading method based on Markov decision process according to claim 2 is characterized in that: C. Update phase: includes the following steps: C1. If the status update is caused by the start of task execution, the system will update the parameters of the task set to be offloaded, the offloaded task set, the candidate set, and the server set; C2. If the status update is caused by the completion of the task, the system will update the server set parameters; C3. After the parameters are updated, go to step B8.

4. The workflow task offloading method based on Markov decision process according to claim 3 is characterized in that: The topological structure of all workflows is the same, but the function types and number of calls required by the subtasks in each workflow are different each time. The cloud-edge server resources are heterogeneous and limited, while the cloud computing and storage resources are unlimited, and the computing speed is much faster than that of the edge server.

Images