A reinforcement learning and heuristic driven method for scheduling edge computing dependent tasks

Abstract

The present invention relates to the field of edge computing and task scheduling, and specifically discloses a reinforcement learning and heuristic driven edge computing dependent task scheduling method, comprising the following steps: first merging the tasks of multiple users, and adding a start node for all tasks without predecessor tasks; converting the edge computing and dependent task scheduling problem into a two-layer optimization problem: in the upper layer optimization, using an artificial intelligence algorithm based on reinforcement learning to modify the dependency relationship of the tasks, that is, adding new edges, thereby adding new constraints; in the lower layer optimization, using a traditional heuristic algorithm to offload and solve the tasks, and obtain the scheduling order and scheduling location of the tasks; iteratively optimizing the upper and lower layers to minimize the objective function. The present invention adopts the above-mentioned reinforcement learning and heuristic driven edge computing dependent task scheduling method, and achieves high-quality user experience by dynamically increasing the dependency relationship between tasks, and can be generalized to large-scale scenarios.

Description

Technical Field

[0001] The present invention relates to the field of edge computing and task scheduling technology, and in particular to a reinforcement learning and heuristic-driven edge computing dependent task scheduling method. Background Art

[0002] With the development of the Internet of Everything (IoE), cloud computing is increasingly struggling to meet the requirements for latency, energy consumption, application performance, and reliability. Edge computing (EC) has been proposed as a powerful complement to cloud computing. Unlike cloud computing, which theoretically has unlimited resources, EC must improve user quality of experience (QoE) within resource constraints. Therefore, resource scheduling has been a hot topic in EC research. In recent years, task offloading, as one of the most important resource scheduling techniques in EC, has been widely studied. By determining whether computationally intensive user applications (referred to as tasks) should be offloaded to edge servers (ESs), this technique can significantly reduce latency and energy consumption on user equipment (UEs), thereby improving QoE. However, selecting the appropriate ES to offload tasks to is often challenging due to numerous other considerations in reality, such as payment costs and application deadlines.

[0003] In the field of edge computing, the core goal of task offloading algorithms is to optimize the user experience by rationally allocating computing resources between the UE and the ES, deciding whether to offload tasks to the ES or schedule them locally or on the ES. In the field of edge computing, offloading tasks to edge servers is equivalent to scheduling them to them, so this paper does not distinguish between offloading and scheduling tasks.

[0004] In the field of edge computing, technologies for handling dependency tasks are mainly divided into two categories:

[0005] 1. Traditional heuristic algorithms, which handle dependent task offloading through rule-driven or local optimization strategies, can quickly find solutions but sacrifice understanding quality and have limited performance in dynamic environments, multi-objective optimization, and complex constraints.

[0006] 2. Artificial intelligence algorithms, among which deep reinforcement learning (DRL)-based dependency task offloading algorithms have been widely studied in recent years. Traditional DRL methods address dependency task offloading through Markov decision process (MDP) modeling, flattened state representation, and discrete action spaces. However, in large-scale task scenarios, exploration efficiency is low, resulting in sparse rewards. Novel methods combining graph neural networks, meta-learning, and hierarchical strategies are gradually addressing these limitations and becoming a leading direction in edge computing task scheduling.

[0007] Considering the many other factors that need to be considered in reality and the shortcomings of the above two technologies, there is an urgent need for an edge computing system-dependent task scheduling method to solve the above problems. Summary of the Invention

[0008] The purpose of this invention is to provide a reinforcement learning and heuristic-driven edge computing dependent task scheduling method, which solves the edge computing and dependent task scheduling problems, and adopts a two-layer optimization strategy to solve the reward sparsity problem of reinforcement learning in this scenario and the low solution quality problem of the heuristic algorithm.

[0009] To achieve the above objectives, the present invention provides a reinforcement learning and heuristic-driven edge computing dependent task scheduling method, comprising the following steps:

[0010] S1. Merge the tasks of multiple users and add a start node for all tasks that do not have a predecessor task;

[0011] S2. Convert edge computing and dependent task scheduling problems into a two-level optimization problem:

[0012] In the upper-level optimization, an artificial intelligence algorithm based on reinforcement learning is used to modify the dependencies of tasks, that is, to add new edges and thus add new constraints;

[0013] In the lower-level optimization, traditional heuristic algorithms are used to solve the task offloading and obtain the task scheduling order and scheduling location;

[0014] S3, iteratively optimize the upper and lower layers to minimize the objective function.

[0015] Preferably, in S1, it is assumed that each terminal device generates an application consisting of a series of dependent tasks, and there are dependencies between tasks. The tasks of multiple users are first merged, and a start node is added to all tasks without predecessor tasks. The node has no data and type.

[0016] Preferably, in S2, the upper layer optimization process is as follows:

[0017] Sa1. Construct Markov chain MDP:

[0018] Formulate the problem as a Markov chain MDP of reinforcement learning, where the state is the current directed acyclic graph DAG G k , the action is defined as the direction to G k Add an edge and get a new DAG G k+1 ;

[0019] The added edges enforce additional constraints and reduce the decision space, where x k Indicates the execution order and location of the k-th step scheduling, in G k+1The traditional heuristic algorithm is used to calculate x k+1 ;

[0020] According to x k Calculate the objective function value f (x k |G k ), define the reward as the previous objective function value f (x k |G k ) minus the new objective function value f (x k+1 |G k+1 ):

[0021] Reward = f (x k | G k )-f (x k+1 |G k+1 );

[0022] For state transition, a maximum number of actions K is set as the end condition for each iteration to prevent the algorithm from terminating;

[0023] Sa2, status code:

[0024] A graph convolutional network (GCN) is used to encode the state of the DAG representation to extract data features. Considering the DAG structure, two GCNs are used. The first GCN processes the original DAG, and the second GCN processes the DAG with all edges flipped.

[0025] Connect two GCN modules to obtain the node feature n, and use the attention pooling layer to extract the overall feature g of the graph:

[0026] n=[GCN1(G k )||GCN2(reverse(G k ))];

[0027] g=Att(n);

[0028] Among them, reverse(G k ) represents G k For a DAG with all edges flipped, is the splicing function;

[0029] Sa3. Use a proximal policy optimization algorithm consisting of a policy network ActorNet and a value network Critic Net. The policy network ActorNet is responsible for generating the probability distribution of actions, and the value network Critic Net is responsible for evaluating the long-term value of the current state.

[0030] Preferably, in the dependent task offloading problem, the policy network Actor Net in Sa3 generates the corresponding selection probability for each edge. In order to reduce the action space, the edge selection operation action is divided into two steps: selecting the starting node and selecting the destination node. The selection of the starting node and the selection of the destination node are predicted by two independent 3-layer residual network ResNet modules respectively. Among them, the input of the second ResNet also contains the feature vector of the selected starting node:

[0031] P(a1) = softmax(ResNet1([n|| g]));

[0032] P(a2| a1) =softmax(ResNet2([n || n[a1] || g]));

[0033] Among them, the subscript n[a1] represents the node feature of node a1.

[0034] Preferably, during the edge selection process, the legality of adding edges is checked. The legality determination conditions are as follows:

[0035] In a directed acyclic graph (DAG), check whether the current DAG already contains the edge to be added. If it does, the action of adding the edge is illegal.

[0036] Determine whether there is already a path between the starting node and the destination node of the edge to be added. If there is a path, the action of adding the edge is illegal;

[0037] Use topological sorting or depth-first search (DFS) to check whether adding edges will form a loop. If a loop is formed, the action of adding edges is illegal.

[0038] By adding an action mask, the probability of illegal actions is set to zero or negative infinity. The action mask ensures that the agent only selects legal actions by shielding illegal actions.

[0039] Preferably, in Sa3, the value network CriticNet predicts the expected cumulative return starting from the current state and following the current strategy;

[0040] Perform maxpool processing on all node features, concatenate the pooled features with the graph features from the state encoder, and perform value prediction by another ResNet, namely:

[0041] V(G k ) = ResNet3([maxpool(n) ||g]);

[0042] Get the predicted result of the long-term value of the current state.

[0043] Preferably, in S2, the lower-level optimization uses a traditional heuristic algorithm based on a greedy algorithm to solve the task offloading problem. The specific process is as follows:

[0044] Sb1. Put all users' tasks without predecessor nodes into the ready queue readylist and sort them in non-descending order according to the application deadline to obtain the tasks and their scheduling order for the first round of scheduling;

[0045] Sb2. Schedule the tasks in the queue in order to the location where the weighted sum of the three target values ​​of latency, energy consumption and cost is minimized. After execution, remove the task from the queue and mark it as completed.

[0046] Sb3, check whether the successor task of the scheduled task meets the conditions, that is, whether all direct predecessor tasks of the successor task have been completed. If so, put the successor task into the ready queue;

[0047] Sb4. When all tasks in the ready queue are completed, the newly added tasks are sorted in non-descending order of their application deadlines, and a new round of task scheduling begins. When all tasks are completed, the final solution is obtained.

[0048] Therefore, the present invention adopts the above-mentioned reinforcement learning and heuristic-driven edge computing dependent task scheduling method, and the beneficial effects are as follows:

[0049] (1) The present invention effectively solves the problem of dependent task scheduling in edge computing. Through a two-layer optimization architecture, the upper layer uses reinforcement learning to modify task dependencies, and the lower layer uses a heuristic algorithm to solve task offloading, overcoming the limitations of low solution quality of traditional heuristic algorithms and sparse reinforcement learning rewards.

[0050] (2) The present invention dynamically increases task dependencies, reduces the decision space, provides better constraints for the underlying heuristic algorithm, improves the efficiency and accuracy of task scheduling, achieves comprehensive optimization of multiple objectives such as latency, energy consumption, and cost, and significantly improves the quality of user experience.

[0051] (3) The parameters trained in a small-scale environment can be applied to large-scale scenarios to cope with complex and changing tasks and resource conditions, reduce the algorithm's dependence on large-scale data training, and improve the algorithm's practicality and adaptability.

[0052] (4) The present invention sets strict legal judgment conditions in the edge selection operation of the reinforcement learning strategy network to avoid invalid and erroneous operations, ensure the reasonable structure of the directed acyclic graph, and ensure the correctness of the task scheduling logic and orderly execution.

[0053] (5) The present invention adopts a proximal policy optimization algorithm, combines the policy network with the value network, and accurately guides the policy gradient direction by calculating the advantage function, thereby accelerating the algorithm convergence speed, improving learning efficiency, and reducing training time and resource consumption.

[0054] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Schematic diagram of a Markov chain for reinforcement learning in an embodiment of a method for scheduling edge computing-dependent tasks driven by reinforcement learning and heuristics of the present invention;

[0056] Figure 2 This is a flowchart of an embodiment of a reinforcement learning and heuristic driven edge computing dependent task scheduling method for solving the edge computing system dependent task problem of the present invention;

[0057] Figure 3 This is a simple applicable scenario of an embodiment of the edge computing dependent task scheduling method driven by reinforcement learning and heuristics of the present invention;

[0058] Figure 4 This is an example of a component task graph of an application generated by a user device in an embodiment of a reinforcement learning and heuristic-driven edge computing dependent task scheduling method of the present invention;

[0059] Figure 5 This is a merging operation of multi-user dependent tasks in an embodiment of the edge computing dependent task scheduling method driven by reinforcement learning and heuristics of the present invention. DETAILED DESCRIPTION

[0060] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0061] Unless otherwise defined, technical or scientific terms used in the present invention shall have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

[0062] This paper proposes a method for scheduling edge computing system-dependent tasks based on reinforcement learning and heuristics, which adopts a two-layer optimization strategy. As shown in the figure, a method for scheduling edge computing-dependent tasks based on reinforcement learning and heuristics, which adopts a two-layer optimization strategy, includes the following steps:

[0063] S1, such as Figure 3 The following is a simple application scenario of the present invention. The service type of the edge server is limited and can only process tasks of the corresponding type, i.e. Figure 3The colors in the table represent tasks of type "red". For example, a red circle represents a task of type "red", and a red square represents a service that can execute tasks of type "red" stored in the edge server.

[0064] Assume that each terminal device (i.e., user) generates an application consisting of a series of dependent tasks, such as Figure 4 In the directed acyclic graph shown, there is a dependency relationship between tasks. There is an edge from task A4 to task A7. This means that task A7 can only be executed after A4 is completed. In addition, task A7 is the direct successor of A4, and A4 is the direct predecessor of A7.

[0065] According to the above description, before solving the problem, the tasks of multiple users are merged first, such as Figure 5 As shown, a start node is added to all tasks that do not have a predecessor task. The node has no data and type.

[0066] S2. Convert edge computing and dependent task scheduling problems into a two-level optimization problem:

[0067] In the upper-level optimization, an artificial intelligence algorithm based on reinforcement learning is used to modify the dependencies of tasks, that is, to add new edges, thereby adding new constraints to further improve the performance of the heuristic algorithm.

[0068] In the lower-level optimization, traditional heuristic algorithms are used to solve the task offloading, obtain the scheduling order and scheduling location of the tasks, and iteratively optimize the upper and lower layers to achieve high-quality user experience, that is, minimize the objective function, such as Figure 2 Shown is the general framework of the present invention.

[0069] 1) The upper layer optimization process is as follows:

[0070] Sa1. Construct Markov chain MDP:

[0071] Formulate the problem as a Markov chain MDP of reinforcement learning, such as Figure 1 As shown, the state is the current directed acyclic graph DAGG k , the action is defined as the direction to G k Add an edge to get a new DAG G k+1 The added edges enforce additional constraints and reduce the decision space to improve the performance of the heuristic; where x k Indicates the execution order and location of the k-th step scheduling, in G k+1 The traditional heuristic algorithm is used to calculate x k+1 .

[0072] According to x k The objective function value f (x k |Gk ), such as user QoE, cost, application hit rate, etc. Therefore, the present invention defines the reward as the previous objective function value f (x k |G k ) minus the new objective function value f (x k+1 | G k +1 ):

[0073] Reward = f (x k | G k )-f (x k+1 |G k+1 );

[0074] For state transitions, an end condition needs to be set to prevent the algorithm from terminating. The reinforcement learning in this invention sets a maximum number of actions K as the end condition for each iteration to avoid excessive number of actions, thereby solving the problem of sparse rewards.

[0075] Sa2, status code:

[0076] After establishing the Markov chain, we need to consider the specific implementation in reinforcement learning, such as the state encoder. Its core function is to transform the original high-dimensional, complex, or unstructured environment state information into a low-dimensional, structured, and easy-to-process representation.

[0077] Due to the complexity of the DAG structure, this paper uses a graph convolutional network (GCN) to encode the state represented by the DAG to better extract data features. Considering the DAG structure, two GCNs are used. The first GCN processes the original DAG, and the second GCN processes the DAG with all edges flipped. The two GCN modules are then connected to obtain the node feature n, and the attention pooling layer is used to extract the overall feature g of the graph:

[0078] n=[GCN1(G k )||GCN2(reverse(G k ))];

[0079] g=Att(n);

[0080] Among them, reverse(G k ) represents G k For a DAG with all edges flipped, is the splicing function.

[0081] Sa3. The reinforcement learning algorithm of the present invention uses the proximal policy optimization algorithm PPO composed of the policy network ActorNet and the value network Critic Net. Therefore, the implementation of the policy network (ActorNet) and the value network (Critic Net) also needs to be specifically considered.

[0082] Specifically, the policy network ActorNet is responsible for generating the probability distribution of actions, that is, outputting the selection probability of each action based on the current environment state (such as the task dependency graph and edge server status). In the dependent task offloading problem, the policy network ActorNet generates corresponding selection probabilities for each edge. To reduce the action space, the edge selection operation is divided into two steps: selecting the starting node and selecting the destination node. The starting node selection and the destination node selection are predicted by two independent three-layer ResNet modules, where the second ResNet input also includes the feature vector of the selected starting node:

[0083] P(a1) = softmax(ResNet1([n|| g]));

[0084] P(a2| a1) =softmax(ResNet2([n || n[a1] || g]));

[0085] Among them, the subscript n[a1] represents the node feature of node a1.

[0086] During edge selection, it is necessary to check whether the action of adding edges is legal. The legality judgment conditions are as follows:

[0087] 1. Existing edges: In a directed acyclic graph (DAG), check whether the current DAG already contains the edge to be added. If it does, the action of adding the edge is illegal.

[0088] 2. Meaningless edges: Determine whether there is already a path between the starting node and the destination node of the edge to be added. If there is a path, the action of adding the edge is illegal.

[0089] 3. Loop formation: Use topological sorting or depth-first search (DFS) to detect whether a loop will be formed after adding edges. If a loop is formed, the action of adding edges is illegal.

[0090] For actions that don't meet the above three conditions, an action mask is added. The mask is applied to the action probability distribution output by the policy network, setting the probability of illegal actions to zero or negative infinity. The action mask ensures that the agent only selects legal actions by blocking illegal actions.

[0091] The value network Critic Net is responsible for evaluating the long-term value of the current state, that is, predicting the expected cumulative return starting from the current state and following the current strategy. By performing maxpool processing on all node features, the pooled features are spliced ​​with the graph features from the state encoder, and finally another ResNet performs value prediction processing, namely:

[0092] V(G k ) = ResNet3([maxpool(n) || g]);

[0093] Get the predicted result of the long-term value of the current state.

[0094] In this scenario, the present invention predicts the future comprehensive rewards, such as latency, energy consumption, cost, and application hit rate, based on the current DAG structure and edge server service types. Furthermore, through collaboration with ActorNet, the advantage function is calculated to guide the direction of the policy gradient.

[0095] 2) In the lower-level optimization, a traditional heuristic algorithm based on a greedy algorithm is used to solve the task offloading problem. The specific process is as follows:

[0096] Sb1. Put all users' tasks without predecessor nodes into the ready queue readylist, and sort them in non-descending order according to the application deadline to obtain the tasks and their scheduling order for the first round of scheduling.

[0097] Sb2. Schedule the tasks in the queue in sequence to the location where the weighted sum of the three target values ​​of latency, energy consumption and cost is minimized (local execution or edge server execution). After execution, remove the task from the queue and mark the task as completed.

[0098] Sb3. Check whether the successor task of the scheduled task meets the conditions, that is, whether all direct predecessor tasks of the successor task have been completed. If so, put the successor task into the ready queue.

[0099] Sb4. When all tasks in the ready queue are completed, the newly added tasks are sorted in non-descending order of their application deadlines, and a new round of task scheduling begins. When all tasks are completed, the final solution is obtained.

[0100] S3. Iteratively optimize the upper and lower layers to achieve a high-quality user experience, that is, minimize the objective function.

[0101] Therefore, the present invention adopts the above-mentioned reinforcement learning and heuristic-driven edge computing dependent task scheduling method to solve the edge computing and dependent task scheduling problems; adopts a two-layer optimization strategy to solve the reward sparsity problem of reinforcement learning and the low solution quality problem of the heuristic algorithm; and through the parameters obtained by training in a small-scale environment, it can be applied to large-scale problems.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A reinforcement learning and heuristic driven edge computing dependent task scheduling method, characterized in that: The following steps are involved: S1. Merge the tasks of multiple users and add a start node for all tasks that do not have a predecessor task; S2. Convert edge computing and dependent task scheduling problems into a two-level optimization problem: In the upper-level optimization, an artificial intelligence algorithm based on reinforcement learning is used to modify the dependencies of tasks, that is, to add new edges and thus add new constraints; In the lower-level optimization, traditional heuristic algorithms are used to solve the task offloading and obtain the task scheduling order and scheduling location; S3, iteratively optimize the upper and lower layers to minimize the objective function; In S2, the upper layer optimization process is as follows: Sa1. Construct Markov chain MDP: Formulate the problem as a Markov chain MDP of reinforcement learning, where the state is the current directed acyclic graph DAG G k , the action is defined as the direction to G k Add an edge and get a new DAG G k+1 ; The added edges enforce additional constraints and reduce the decision space, where x k Indicates the execution order and location of the k-th step scheduling, in G k+1 The traditional heuristic algorithm is used to calculate x k+1 ; According to x k Calculate the objective function value f (x k | G k ), define the reward as the previous objective function value f (x k | G k ) minus the new objective function value f (x k+1 | G k+1 ): Reward = f (x k | G k )−f (x k+1 | G k+1 ); For state transition, a maximum number of actions K is set as the end condition for each iteration to prevent the algorithm from terminating; Sa2, status code: A graph convolutional network (GCN) is used to encode the state of the DAG representation to extract data features. Considering the DAG structure, two GCNs are used. The first GCN processes the original DAG, and the second GCN processes the DAG with all edges flipped. Connect two GCN modules to obtain the node feature n, and use the attention pooling layer to extract the overall feature g of the graph: n=[GCN1(G k )||GCN2(reverse(G k ))]; g=Att(n); Among them, reverse(G k ) represents G k For a DAG with all edges flipped, is the splicing function; Sa3, using a proximal policy optimization algorithm consisting of a policy network Actor Net and a value network Critic Net. The policy network Actor Net is responsible for generating the probability distribution of actions, and the value network Critic Net is responsible for evaluating the long-term value of the current state. In S2, the lower-level optimization uses a traditional heuristic algorithm based on a greedy algorithm to solve the task offloading problem. The specific process is as follows: Sb1. Put all users' tasks without predecessor nodes into the ready queue (ready list) and sort them in non-descending order according to the application deadline to obtain the tasks for the first round of scheduling and their scheduling order. Sb2. Schedule the tasks in the queue in order to the location where the weighted sum of the three target values ​​of latency, energy consumption and cost is minimized. After execution, remove the task from the queue and mark it as completed. Sb3, check whether the successor task of the scheduled task meets the conditions, that is, whether all direct predecessor tasks of the successor task have been completed. If so, put the successor task into the ready queue; Sb4. When all tasks in the ready queue are completed, the newly added tasks are sorted in non-descending order of their application deadlines, and a new round of task scheduling begins. When all tasks are completed, the final solution is obtained.

2. The method for scheduling edge computing-dependent tasks based on reinforcement learning and heuristic driving according to claim 1, characterized in that: In S1, it is assumed that each terminal device generates an application consisting of a series of dependent tasks. There are dependencies between tasks. First, the tasks of multiple users are merged, and a start node is added to all tasks without predecessor tasks. This node has no data and type.

3. The method for scheduling edge computing-dependent tasks based on reinforcement learning and heuristic driving according to claim 1, characterized in that: In the dependency task offloading problem, the policy network Actor Net in Sa3 generates a corresponding selection probability for each edge. To reduce the action space, the edge selection operation action is divided into two steps: selecting the starting node and selecting the destination node. The selection of the starting node and the selection of the destination node are predicted by two independent three-layer residual network ResNet modules, respectively. The input of the second ResNet also contains the feature vector of the selected starting node: P(a1) = softmax(ResNet1([n || g])); P(a2 | a1) = softmax(ResNet2([n || n[a1] || g])); Among them, the subscript n[a1] represents the node feature of node a1.

4. The method for scheduling edge computing-dependent tasks based on reinforcement learning and heuristic driving according to claim 3, characterized in that: During edge selection, check whether the action of adding edges is legal. The legality judgment conditions are as follows: In a directed acyclic graph (DAG), check whether the current DAG already contains the edge to be added. If it does, the action of adding the edge is illegal. Determine whether there is already a path between the starting node and the destination node of the edge to be added. If there is a path, the action of adding the edge is illegal; Use topological sorting or depth-first search (DFS) to check whether adding edges will form a loop. If a loop is formed, the action of adding edges is illegal. By adding an action mask, the probability of illegal actions is set to zero or negative infinity. The action mask ensures that the agent only selects legal actions by shielding illegal actions.

5. The method for scheduling edge computing-dependent tasks based on reinforcement learning and heuristic driving according to claim 1, characterized in that: In Sa3, the value network Critic Net predicts the expected cumulative return starting from the current state and following the current strategy; Perform maxpool processing on all node features, concatenate the pooled features with the graph features from the state encoder, and perform value prediction by another ResNet, namely: V(G k ) = ResNet3([maxpool(n) || g]); Get the predicted result of the long-term value of the current state.

Images