A heterogeneous dynamic scheduling method based on reinforcement learning

Abstract

The application discloses a heterogeneous dynamic scheduling strategy based on reinforcement learning, comprising the following steps: firstly, modeling the scheduling problem as a Markov decision process, defining state, action, transition function and reward function, using a graph convolution network to extract task graph structure features and constructing state representation; outputting action probability distribution through a policy network, combining advantage function and entropy regularization for policy optimization, and simultaneously using a value network to estimate state value and training through minimizing Bellman error; in system interaction, the agent selects task scheduling action according to the state and continuously optimizes the policy, and finally realizes the goal of minimizing the task completion time. The application can make allocation and scheduling decisions according to the system state in runtime, and significantly improves the scheduling efficiency and adaptability.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a heterogeneous dynamic scheduling method based on reinforcement learning. Background Technology

[0002] Task scheduling is a typical combinatorial optimization problem, characterized by NP-hardness, especially in dynamic environments where its complexity increases further. For example, in practical applications, some task graph structures (DAGs) may gradually unfold with the addition of new resources or computing nodes, making the scheduling process highly uncertain and requiring real-time processing. This type of problem is widely present in complex computing environments such as parallel computing, cloud computing, and edge computing.

[0003] Existing scheduling algorithms mainly include static heuristic methods (such as HEFT) and some rule-based adaptive methods. However, these algorithms are usually limited to specific task graph structures or problem sizes, and are difficult to effectively cope with uncertainties such as task duration, communication latency, and performance differences of heterogeneous platforms, lacking a general strategy with broad adaptability.

[0004] As the scale and complexity of computing systems continue to increase, traditional scheduling methods face problems such as weak adaptability and poor scalability. In recent years, reinforcement learning, as an intelligent method with self-learning and policy optimization capabilities, has gradually shown its application potential in the field of dynamic scheduling. Although existing research has shown that reinforcement learning has advantages in handling small-scale or static task graphs, its policy generalization ability and training stability still need further improvement when facing dynamic, varied, and heterogeneous scheduling environments. Therefore, there is an urgent need for an intelligent scheduling method that integrates graph structure modeling and reinforcement learning mechanisms, can adapt to dynamic task changes, and improve scheduling efficiency. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide an intelligent scheduling method that integrates graph structure modeling and reinforcement learning mechanisms, which can adapt to dynamic changes in tasks and improve scheduling efficiency.

[0006] The technical solution of this invention is as follows: A heterogeneous dynamic scheduling method based on reinforcement learning includes the following steps: Step 1, Problem Definition and Modeling: The scheduling problem is formalized as task scheduling given a directed acyclic graph (DAG) and heterogeneous computing units. With the goal of minimizing task completion time, the dynamic scheduling problem is modeled as a Markov decision process, defining state S, action A, transition function P, and reward function R. The state S contains information about the ready task and its descendants within a certain depth, as well as the state of computing resources. The task information is processed by a graph convolutional network (GCN). Each task is represented by the original features, and the state is enriched by stacking GCN layers to represent the DAG. Action A is defined as: when computing resources are available, select a task from the ready tasks to run on that resource, or select to keep it idle; The transfer function P is automatically transferred from one state to another by the computer system, and reinforcement learning does not involve explicit modeling. The reward function R is defined as follows: before the DAG scheduling is completed, the immediate reward for each step is 0; after the scheduling is completed, the reward is the normalized difference between the completion time of the RL algorithm and the completion time of the HEFT algorithm. Step 2: Select the A2C algorithm and train the RL agent, including the following steps: Step 2.1, Network Structure Design: The A2C algorithm includes two neural networks: Policy Network: Input state s, output action probability distribution , represents the probability of choosing action a in state s, where θ is the parameter of the policy network. By optimizing θ, the policy network can adjust the action probability distribution to maximize the cumulative reward. Value network: Input state s, output state value , representing the expected cumulative return of the state, where, The parameters of the value network are optimized. Value networks can estimate state value more accurately; Step 2.2, Neural Network Optimization, including: Policy network optimization: The objective function combines the policy gradient, advantage function, and entropy regularization, and is expressed as follows: , , , , in, Let be the objective function of the policy network, representing the expected return of policy π under parameter θ. This function is used to optimize the policy network, aiming to maximize π through gradient ascent. This improves the strategy; E[·] is the expectation operator, which means taking the expectation of the distribution of state s and action a; The advantage function measures the merits of choosing action a in state s compared to the average action. Action-state value represents the immediate reward obtained after taking action a in state s. Add the future value after discounts; State value represents the expected cumulative reward of state s, output by the value network. γ is the discount factor used to balance immediate and future returns, representing the immediate return obtained at time step t. For the next state The state value is given by s, where s is the current state and a is the current action. Policy entropy measures the uncertainty of a policy in state s. The higher the entropy, the more random the policy; the lower the entropy, the more deterministic the policy. Let β be the action probability distribution, and let β be the entropy regularization hyperparameter, which controls the influence of the entropy term on the optimization and prevents the policy from converging to a local optimum too early. Value network optimization: The value network is optimized by minimizing the Bellman error, expressed as: , in, The loss function of the value network is used to optimize the network parameters. The goal is to minimize This allows the value network to more accurately estimate state values; E[·] is the expectation operator. γ is the discount factor for the immediate return obtained at time step t; Step 2.3, the training process, includes: The agent interacts with the environment to generate a series of state-action-reward sequences. ; After collecting data over multiple steps, θ and φ are updated using stochastic gradient descent. ; Repeat the interaction and optimization until the strategy converges; Step 3: Build the RL proxy architecture, including the following steps: Step 3.1, Data Input: Send the current DAG task graph and resource status to the agent; Step 3.2, DAG Information Processing: Extract DAG features using stacked graph convolutional layers, expressed as: , in, Let be the feature vector of node v at layer l+1, representing the updated features of node v after processing by the GCN at layer l+1, used to capture information about the node and its neighbors. is the weight matrix of the l-th layer, used to perform linear transformation on the aggregated features; Let u be the feature vector of node u in the l-th layer. Let v be the set of neighboring nodes; Each layer of GCN aggregates node and neighbor information to generate richer feature representations; The final GCN layer outputs the global representation of the DAG. ; Step 3.3, Resource State Embedding, including: Embed computing resource states as vectors ; Combined with the global representation of DAG sum vector To obtain the complete state representation ; Step 3.4, Calculate the action probability: Input h into the fully connected layer of the policy network, the expression is: , in, The weights of the fully connected layer, For bias; Step 3.5, Action Sampling and Execution: From If an action is sampled, a ready task is selected and assigned to the current computing resources for execution; if the sampling result is 0 actions, no task scheduling is performed. Step 3.6, Status Update and Loop: Update the task status and resource utilization, determine whether there are ready tasks, and repeat the scheduling until all tasks are completed.

[0007] Furthermore, in step 1, a Graph Convolutional Network (GCN) is used to process task information. Each task is represented by original features, and the state is enriched by stacking GCN layers to enrich the representation of the DAG. , in, The node feature matrix of the (l+1)th layer. Let l be the node feature matrix of the l-th layer. For the adjacency matrix with added self-loops, Let A be the original adjacency matrix and I be the identity matrix. for The corresponding degree matrix, is the learnable weight matrix of the l-th layer, used to map the input features to the feature space of the next layer.

[0008] Furthermore, the expression for the reward function R in step 1 is: , in, Let be the completion time of the RL algorithm. Let be the completion time of the HEFT algorithm, when < If R > 0, it means that the RL algorithm is superior to the HEFT algorithm.

[0009] The beneficial effects of this invention are: 1. Improve scheduling efficiency and optimize task completion time: This invention introduces the A2C algorithm from reinforcement learning, which enables the agent to adaptively select the optimal scheduling action under different states, effectively reducing the overall completion time of task scheduling. It outperforms traditional heuristic algorithms such as HEFT in most task graphs. 2. Adaptable to dynamically changing scheduling environments: This invention models the scheduling problem as a Markov decision process, combining dynamic graph structure input and resource status, which can dynamically respond to the gradual unfolding of the task graph structure and the real-time changes in resource status, thus possessing strong environmental adaptability. 3. Supports heterogeneous resource scheduling and multi-task parallelism: By combining resource state embedding with policy networks, this invention enables intelligent agents to reasonably distinguish the characteristics of different computing resources (such as CPU and GPU), realize the collaborative scheduling of heterogeneous resources, and improve the overall resource utilization of the system. 4. The reinforcement learning strategy has self-learning and generalization capabilities: The RL agent proposed in this invention gradually optimizes the scheduling strategy during multi-round interactive training, does not rely on fixed rules, and can show strong generalization capabilities when facing different task sizes, different DAG structures and unknown task graphs. 5. Introducing Graph Convolutional Networks to Enhance Task Graph Structure Understanding: This invention utilizes GCN to perform multi-layer embedding learning on DAG task graphs, capturing the topological dependencies between nodes, resulting in richer state representations, more reasonable policy decisions, and significantly improved scheduling decision accuracy. 6. Co-optimization of policy and value networks to improve stability and convergence speed: This invention adopts a dual-network structure of the A2C algorithm, introduces a value benchmark in policy optimization to improve the stability of gradient estimation; and avoids the policy from getting trapped in local optima too early through entropy regularization, thereby improving the overall training efficiency and the diversity of scheduling policies. 7. High portability and applicability to various scheduling scenarios: The method of this invention has no task graph-specific dependencies and policy learning capabilities, and is applicable to various heterogeneous scheduling environments such as cloud computing, edge computing, and high-performance parallel systems. It has good engineering practicality and deployability. Attached Figure Description

[0010] Figure 1 This is a basic flowchart of a heterogeneous dynamic scheduling method based on reinforcement learning according to the present invention.

[0011] Figure 2 This is a schematic diagram illustrating the principle of a heterogeneous dynamic scheduling method based on reinforcement learning according to the present invention. Detailed Implementation

[0012] like Figure 1-2 As shown, a heterogeneous dynamic scheduling method based on reinforcement learning includes the following steps: Step 1, Problem Definition and Modeling: The scheduling problem is formalized as task scheduling given a directed acyclic graph (DAG) and heterogeneous computing units. With the goal of minimizing task completion time, the dynamic scheduling problem is modeled as a Markov decision process, defining state S, action A, transition function P, and reward function R. The state S contains information about the ready task and its descendants within a certain depth, as well as the state of computing resources. The task information is processed by a graph convolutional network (GCN). Each task is represented by the original features, and the state is enriched by stacking GCN layers to represent the DAG. Action A is defined as: when computing resources are available, select a task from the ready tasks to run on that resource, or select to keep it idle; The transfer function P is automatically transferred from one state to another by the computer system, and reinforcement learning does not involve explicit modeling. The reward function R is defined as follows: before the DAG scheduling is completed, the immediate reward for each step is 0; after the scheduling is completed, the reward is the normalized difference between the completion time of the RL algorithm and the completion time of the HEFT algorithm. Step 2: Select the A2C algorithm and train the RL agent, including the following steps: Step 2.1, Network Structure Design: The A2C algorithm includes two neural networks: Policy Network: Input state s, output action probability distribution , represents the probability of choosing action a in state s, where θ is the parameter of the policy network. By optimizing θ, the policy network can adjust the action probability distribution to maximize the cumulative reward. Value network: Input state s, output state value , representing the expected cumulative return of the state, where, The parameters of the value network are optimized. Value networks can estimate state value more accurately; Step 2.2, Neural Network Optimization, including: Policy network optimization: The objective function combines the policy gradient, advantage function, and entropy regularization, and is expressed as follows: , , , , in, Let be the objective function of the policy network, representing the expected return of policy π under parameter θ. This function is used to optimize the policy network, aiming to maximize π through gradient ascent. This improves the strategy; E[·] is the expectation operator, which means taking the expectation of the distribution of state s and action a; The advantage function measures the merits of choosing action a in state s compared to the average action. Action-state value represents the immediate reward obtained after taking action a in state s. Add the future value after discounts; State value represents the expected cumulative reward of state s, output by the value network. γ is the discount factor used to balance immediate and future returns, representing the immediate return obtained at time step t. For the next state The state value is given by s, where s is the current state and a is the current action. Policy entropy measures the uncertainty of a policy in state s. The higher the entropy, the more random the policy; the lower the entropy, the more deterministic the policy. Let β be the action probability distribution, and let β be the entropy regularization hyperparameter, which controls the influence of the entropy term on the optimization and prevents the policy from converging to a local optimum too early. Value network optimization: The value network is optimized by minimizing the Bellman error, expressed as: , in, The loss function of the value network is used to optimize the network parameters. The goal is to minimize This allows the value network to more accurately estimate state values; E[·] is the expectation operator. γ is the discount factor for the immediate return obtained at time step t; Step 2.3, the training process, includes: The agent interacts with the environment to generate a series of state-action-reward sequences. ; After collecting data over multiple steps, θ and φ are updated using stochastic gradient descent. ; Repeat the interaction and optimization until the strategy converges; Step 3: Build the RL proxy architecture, including the following steps: Step 3.1, Data Input: Send the current DAG task graph and resource status to the agent; Step 3.2, DAG Information Processing: Extract DAG features using stacked graph convolutional layers, expressed as: , in, Let be the feature vector of node v at layer l+1, representing the updated features of node v after processing by the GCN at layer l+1, used to capture information about the node and its neighbors. is the weight matrix of the l-th layer, used to perform linear transformation on the aggregated features; Let u be the feature vector of node u in the l-th layer. Let v be the set of neighboring nodes; Each layer of GCN aggregates node and neighbor information to generate richer feature representations; The final GCN layer outputs the global representation of the DAG. ; Step 3.3, Resource State Embedding, including: Embed computing resource states as vectors ; Combined with the global representation of DAG sum vector To obtain the complete state representation ; Step 3.4, Calculate the action probability: Input h into the fully connected layer of the policy network, the expression is: , in, The weights of the fully connected layer, For bias; Step 3.5, Action Sampling and Execution: From If an action is sampled, a ready task is selected and assigned to the current computing resources for execution; if the sampling result is 0 actions, no task scheduling is performed. Step 3.6, Status Update and Loop: Update the task status and resource utilization, determine whether there are ready tasks, and repeat the scheduling until all tasks are completed.

[0013] Furthermore, in step 1, a Graph Convolutional Network (GCN) is used to process task information. Each task is represented by original features, and the state is enriched by stacking GCN layers to enrich the representation of the DAG. , in, The node feature matrix of the (l+1)th layer. Let l be the node feature matrix of the l-th layer. For the adjacency matrix with added self-loops, Let A be the original adjacency matrix and I be the identity matrix. for The corresponding degree matrix, is the learnable weight matrix of the l-th layer, used to map the input features to the feature space of the next layer.

[0014] Furthermore, the expression for the reward function R in step 1 is: , in, Let be the completion time of the RL algorithm. Let be the completion time of the HEFT algorithm, when < If R > 0, it means that the RL algorithm is superior to the HEFT algorithm.

[0015] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and modifications can be made to the present invention without departing from the spirit and scope thereof, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A heterogeneous dynamic scheduling method based on reinforcement learning, characterized in that, Includes the following steps: Step 1, Problem Definition and Modeling: The scheduling problem is formalized as task scheduling given a directed acyclic graph (DAG) and heterogeneous computing units. With the goal of minimizing task completion time, the dynamic scheduling problem is modeled as a Markov decision process, defining state S, action A, transition function P, and reward function R. The state S contains information about the ready task and its descendants within a certain depth, as well as the state of computing resources. The task information is processed by a graph convolutional network (GCN). Each task is represented by the original features, and the state is enriched by stacking GCN layers to represent the DAG. Action A is defined as: when computing resources are available, select a task from the ready tasks to run on that resource, or select to keep it idle; The transfer function P is automatically transferred from one state to another by the computer system, and reinforcement learning does not involve explicit modeling. The reward function R is defined as follows: before the DAG scheduling is completed, the immediate reward for each step is 0; after the scheduling is completed, the reward is the normalized difference between the completion time of the RL algorithm and the completion time of the HEFT algorithm. Step 2: Select the A2C algorithm and train the RL agent, including the following steps: Step 2.1, Network Structure Design: The A2C algorithm includes two neural networks: Policy Network: Input state s, output action probability distribution , represents the probability of choosing action a in state s, where θ is the parameter of the policy network. By optimizing θ, the policy network can adjust the action probability distribution to maximize the cumulative reward. Value network: Input state s, output state value , representing the expected cumulative return of the state, where, The parameters of the value network are optimized. Value networks can estimate state value more accurately; Step 2.2, Neural Network Optimization, including: Policy network optimization: The objective function combines the policy gradient, advantage function, and entropy regularization, and is expressed as follows: ， ， ， ， in, Let be the objective function of the policy network, representing the expected return of policy π under parameter θ. This function is used to optimize the policy network, aiming to maximize π through gradient ascent. This improves the strategy; E[·] is the expectation operator, which means taking the expectation of the distribution of state s and action a; The advantage function measures the merits of choosing action a in state s compared to the average action. Action-state value represents the immediate reward obtained after taking action a in state s. Add the future value after discounts; State value represents the expected cumulative reward of state s, output by the value network. γ is the discount factor used to balance immediate and future returns, representing the immediate return obtained at time step t. For the next state The state value is given by s, where s is the current state and a is the current action. Policy entropy measures the uncertainty of a policy in state s. The higher the entropy, the more random the policy; the lower the entropy, the more deterministic the policy. Let β be the action probability distribution, and let β be the entropy regularization hyperparameter, which controls the influence of the entropy term on the optimization and prevents the policy from converging to a local optimum too early. Value network optimization: The value network is optimized by minimizing the Bellman error, expressed as: ， in, The loss function of the value network is used to optimize the network parameters. The goal is to minimize This allows the value network to more accurately estimate state values; E[·] is the expectation operator. γ is the discount factor for the immediate return obtained at time step t; Step 2.3, the training process, includes: The agent interacts with the environment to generate a series of state-action-reward sequences. ; After collecting data over multiple steps, θ and φ are updated using stochastic gradient descent. ; Repeat the interaction and optimization until the strategy converges; Step 3: Build the RL proxy architecture, including the following steps: Step 3.1, Data Input: Send the current DAG task graph and resource status to the agent; Step 3.2, DAG Information Processing: Extract DAG features using stacked graph convolutional layers, expressed as: ， in, Let be the feature vector of node v at layer l+1, representing the updated features of node v after processing by the GCN at layer l+1, used to capture information about the node and its neighbors. is the weight matrix of the l-th layer, used to perform linear transformation on the aggregated features; Let u be the feature vector of node u in the l-th layer. Let v be the set of neighboring nodes; Each layer of GCN aggregates node and neighbor information to generate richer feature representations; The final GCN layer outputs the global representation of the DAG. ; Step 3.3, Resource State Embedding, including: Embed computing resource states as vectors ; Combined with the global representation of DAG sum vector To obtain the complete state representation ; Step 3.4, Calculate the action probability: Input h into the fully connected layer of the policy network, the expression is: ， in, The weights of the fully connected layer, For bias; Step 3.5, Action Sampling and Execution: From If an action is sampled, a ready task is selected and assigned to the current computing resources for execution; if the sampling result is 0 actions, no task scheduling is performed. Step 3.6, Status Update and Loop: Update the task status and resource utilization, determine whether there are ready tasks, and repeat the scheduling until all tasks are completed.

2. The heterogeneous dynamic scheduling method based on reinforcement learning according to claim 1, characterized in that, In step 1, a Graph Convolutional Network (GCN) is used to process task information. Each task is represented by original features, and the state is enriched by stacking GCN layers to enrich the expression of the DAG representation. ， in, The feature matrix of the nodes in the (l+1)th layer. Let l be the node feature matrix of the l-th layer. For the adjacency matrix with added self-loops, Let A be the original adjacency matrix and I be the identity matrix. for The corresponding degree matrix, is the learnable weight matrix of the l-th layer, used to map the input features to the feature space of the next layer.

3. The heterogeneous dynamic scheduling method based on reinforcement learning according to claim 1, characterized in that, The expression for the reward function R in step 1 is: , in, Let be the completion time of the RL algorithm. Let be the completion time of the HEFT algorithm, when < If R > 0, it means that the RL algorithm is superior to the HEFT algorithm.

Images