Near-end strategy optimization method oriented to heterogeneous supercomputing cluster job flow dynamic scheduling

Abstract

The invention discloses a near-end strategy optimization method for heterogeneous supercomputing cluster job flow dynamic scheduling, which comprises the following steps of: abstracting a directed graph job flow scheduling problem in a heterogeneous supercomputing cluster, determining a scheduling target and an evaluation index, and constructing a reinforcement learning framework; and adopting a near-end strategy to optimize and realize heterogeneous supercomputing cluster job flow dynamic scheduling. According to the method, a hierarchical scheduling method is provided, processing of a scheduler on state information is reduced through a sparse scheduling decision, efficient task scheduling is achieved, task dependency relationship characteristics are extracted through a GCN and serve as a part of the state information, and communication overhead between tasks is reduced by introducing resource inheritance and an intra-task parallel mechanism; and the optimal scheduling strategy is trained and learned by using the improved PPO method, and the trained scheduling strategy can make an optimal scheduling action decision according to the dynamic change task and the state information of the computing resources.

Description

Technical Field

[0001] The present invention belongs to the technical field of heterogeneous supercomputing cluster resource scheduling, and in particular relates to a proximal strategy optimization method for dynamic scheduling of heterogeneous supercomputing cluster job flows. Background Art

[0002] With the rapid development of supercomputers, the scale of simulation experiments in the field of engineering research is also expanding. However, simulation computing tasks often cause idle waiting of computing resources and low utilization efficiency due to the complex data dependencies between tasks. At the same time, modern supercomputers have evolved into multi-resource heterogeneous computing architectures that can meet the needs of computing tasks with different characteristics. However, the introduction of heterogeneous computing resources has also significantly increased the complexity of mapping between tasks and resources, which poses a higher challenge to the design of scheduling algorithms. According to the supercomputing cluster utilization results published by TOP500 in 2024, among the top 500 supercomputing clusters in the world, more than 95% of the cluster resource utilization is less than 90%, and as many as 40% of the cluster resource utilization is only about 50%. This also fully illustrates the shortcomings of the current scheduling algorithm and the urgent need for more efficient resource utilization solutions.

[0003] Complex scientific applications are often defined by scientists as scientific workflows through workflow management systems (WMS), which consist of a series of tasks and their dependencies. Under the premise of ensuring task dependencies, the reasonable allocation of computing resources to specific tasks can effectively shorten the completion time of the entire workflow. At the same time, dynamically allocating and recycling computing resources according to the execution stage of the workflow and the current parallelism can avoid the ineffective occupation of resources by tasks, thereby improving resource utilization. However, most of the existing research focuses on the scheduling problem of a single job on a heterogeneous computing cluster. Usually, fixed resources are allocated to the job before execution until the job is completed. When the resources are idle, they are not recycled and reallocated to other jobs. The importance of dynamic scheduling is ignored, resulting in a large number of computing resources being idle and low computing resource utilization, which in turn causes a long response time for other jobs in the system.

[0004] Scheduling workflows on heterogeneous computing clusters is an NP-complete problem. A large number of studies have proposed corresponding solutions based on heuristic algorithms and optimization algorithms. For example, first-come, first-served (FCFS) and fair scheduling algorithms are common heuristic methods. They are easy to implement and have low computational complexity, so they are suitable for simple scheduling scenarios. However, these methods can only achieve relatively basic scheduling goals. In contrast, scheduling methods based on optimization algorithms have been widely used in scheduling problems because they can achieve more advanced scheduling goals, but their computational complexity increases exponentially with the increase in the size of the problem. In addition, both heuristic and optimization algorithms have limitations that are difficult to overcome, namely, these static scheduling algorithms require the status of all jobs and resources to be known before execution, and cannot cope with dynamic changes in environmental status. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes a proximal strategy optimization method for dynamic scheduling of heterogeneous supercomputing cluster job flows, aiming to solve the problems of long workflow job completion time and low resource utilization existing in the heterogeneous supercomputing cluster scheduling algorithm.

[0006] The technical solution adopted by the present invention is: a proximal strategy optimization method for dynamic scheduling of heterogeneous supercomputing cluster job flows, and the specific steps are as follows:

[0007] S1. Abstract of directed graph job flow scheduling problem in heterogeneous supercomputing clusters;

[0008] S2. Based on step S1, determine the scheduling objectives and evaluation indicators;

[0009] S3. Based on the scheduling problem of step S1 and the scheduling goal of step S2, a reinforcement learning framework is constructed;

[0010] S4. Based on the reinforcement learning framework constructed in step S3, the proximal strategy is used to optimize and realize the dynamic scheduling of heterogeneous supercomputing cluster job flows.

[0011] Furthermore, the step S1 is specifically as follows:

[0012] In a heterogeneous supercomputing cluster environment, jobs submitted by users arrive randomly and continuously in the form of a stream, waiting for the scheduler to allocate computing resources. i (V,E) represents the i-th job arriving at the platform workflow.

[0013] in, Represents the set of task nodes for job i; represents the set of task dependencies within job i, and the fine-grained parallelism criteria within the task, task node Represents an asynchronous subtask st vThe asynchronous subtasks in the same task use the same operation to perform different data calculations, and the same workload wl is used to describe each subtask; d j,k Indicates the task Only in missions The execution can only start after completion. The set of all predecessor tasks of task v is denoted as ζ(v), and the set of all successor tasks is denoted as ξ(v).

[0014] Heterogeneous supercomputing cluster computing platform P = {p1, p2, ..., p |P|} consists of |P| types of heterogeneous processors, each type of processor has a different number of computing cores CC, and the computing rate of each CC is cs. Each CC is the basic unit of scheduling, that is, the basic unit for executing subtasks.

[0015] Where |P| represents the number of processor types of platform P.

[0016] Furthermore, the step S2 is specifically as follows:

[0017] The workflow submitted by users continuously arrives at the heterogeneous supercomputing cluster computing platform in a Poisson process. The scheduling goal is to determine the start time of each task, assign CCs to the tasks, and optimize the response time, span time and resource utilization of the job while satisfying resource and dependency constraints.

[0018] Among them, the CC assigned to the task includes: type and quantity; the indicators for evaluating the performance of the scheduling algorithm are as follows:

[0019] 1) Average job response time 2) Average job completion time 3) Resource Utilization

[0020] Where N represents the number of jobs, represents the waiting time of the i-th job, Respectively represent the operation G i The moment it arrives at the system, obtains resources and starts execution. Indicates the span time of the job, Indicates job G i The time when the execution ends. et represents the effective working time of the resource, and tt represents the total time.

[0021] Furthermore, the step S3 is specifically as follows:

[0022] S31, status information representation;

[0023] The status information includes dynamically changing information about tasks to be scheduled and available resources. Interdependent tasks to be scheduled change dynamically with the scheduling process, so a graph convolutional neural network (GCN) is used to process such dynamically scaling graph data. Job nodes and global nodes are added to the design to represent the information of each job and the summary information of all jobs respectively, and the existing one-way information aggregation GCN is improved to a two-way information aggregation BiGCN.

[0024] S311, task node embedding;

[0025] The original feature x of the task node v v As input, BiGCN is used to output the embedding features of task nodes.

[0026] Upward embedding features of task node v Aggregate the information of all child nodes, the expression is as follows:

[0027]

[0028] Among them, g(·), h(·), and f(·) represent three nonlinear transfer functions σ(WX), W represents the learnable parameter matrix, and σ represents the activation function LeakyReLU.

[0029] Let X represent the original features of the job, which is composed of the original features of each task node, including: (i) the number of CCs that can be applied, (ii) whether the available CCs belong to this job, (iii) the type of occupied CCs, (iv) the number of occupied CCs, (v) the number of remaining subtasks, and (vi) the workload of each subtask.

[0030] Downward embedding features of task node v Aggregate the information of all parent nodes, the expression is as follows:

[0031]

[0032] Here, g′(·) and h′(·) represent functions with the same form as g(·) and h(·) but different parameters.

[0033] The leaf node task has no successor task, and its upward embedding vector is obtained by its original features through two layers of nonlinear transformation f(·) and g(·), that is, the low-dimensional space features of the leaf node task are mapped to the high-dimensional space, and the information is aggregated to the parent node during the upward embedding process as the upward embedding feature of the parent node, and so on, to obtain the upward embedding features of all task nodes. Similarly, the downward embedding process is to aggregate the features of the parent node to the child node, and finally obtain the downward embedding features of all task nodes.

[0034] S312, directed graph job embedding;

[0035] Add a virtual node to each directed acyclic graph DAG and embed the features of the task node And the original feature x v Aggregate to virtual node y i , represents the embedding feature of the entire job, and its expression is as follows:

[0036]

[0037] S313, global job embedding;

[0038] Similar to directed graph jobs, add a virtual global node z i , which represents the aggregation of the embedded feature information of all jobs as the embedded feature of the global job in the system. Its expression is as follows:

[0039]

[0040] S32, design scheduling action;

[0041] In the design, a two-stage scheduling action is adopted. The first stage selects a task from the schedulable tasks, and the second stage selects a CC type and its quantity from the available computing resources. The scheduling action can be triggered when any of the following situations occur: (i) a new job arrives at the system; (ii) a task execution ends.

[0042] S321, task selection;

[0043] 1) At a certain time t, the scheduling action selects a task from n schedulable tasks, and the policy network calculates a scalar value for all schedulable tasks v by embedding features

[0044] Here, p(·) represents a shallow neural network.

[0045] 2) Using the softmax(·) function, according to the scalar value p v The probability of each task being selected is calculated as follows:

[0046]

[0047] Among them, A t Represents the set of schedulable tasks at time t.

[0048] 3) According to the probability distribution calculated by formula (5), the scheduling strategy determines the tasks to be scheduled.

[0049] S322. Computing resource selection

[0050] 1) First select the type of CC;

[0051] Similar to the task selection stage, the neural network q(y i ,z,CC abil ) Calculate the score q for each CC type type .

[0052] Among them, CC abil Indicates the computing power of different types of resources, q type Indicates the priority of computing resources of type type that are suitable for the current job environment.

[0053] Use the softmax(·) function to convert q type Converted into probability distribution, the scheduling strategy determines the appropriate CC type.

[0054] 2) Allocate the appropriate amount from the available amount of CC;

[0055] The scheduling strategy determines the appropriate CC type for each possible number of CCs in type. The q value is calculated and converted into a probability distribution using the softmax(·) function. The scheduling strategy determines the number of CCs to be allocated, num.

[0056] The q value represents the number of allocatable CCs of type type, num represents the number of actually allocated CCs, and num≤q.

[0057] S323, scheduling action coding;

[0058] Encode the scheduling action into a four-tuple (job idx ,task idx ,type,num).

[0059] Among them, job idx 、task idx They represent the job index number and task index number of the task selected in the first stage respectively.

[0060] S33, reward function design;

[0061] The optimization target indicators of the scheduling strategy include: AJRT and AJCT, then the reward function design expression is as follows:

[0062] r k =-(t k -t k-1 )(μM k +νN k ) (6)

[0063] Among them, t k represents the time when the kth scheduling action is generated, M k 、N kRespectively represent the time interval [t k-1 ,t k ), μ and ν represent the hyperparameters of relative weights, and the agent receives a reward value r after making the kth action. k .

[0064] The training and learning process is to schedule the policy π θ Cumulative rewards for the generated tracks Solving the maximization problem is equivalent to minimizing the number of all jobs and the number of unresponsive jobs in the scheduling system; according to Little's law, the learning process for the scheduling goal is equivalent to reducing the average job completion time and response time.

[0065] Among them, k, τ, and θ represent the time step, scheduling trajectory, and strategy parameters, respectively.

[0066] Furthermore, the step S4 is specifically as follows:

[0067] S41, Proximal Strategy Optimization PPO Advantage Function Improvement;

[0068] set up It represents the advantage function calculated by the generalized advantage estimation GAE algorithm, which is used to evaluate the relative quality of the current action in the policy update and provide the direction of the policy update. Its expression is as follows:

[0069]

[0070] Among them, r t represents the reward obtained by the t-th action, T represents the total number of scheduled actions, γ represents the discount factor, which controls the weight of future returns; λ represents the smoothing coefficient, which helps balance the deviation and variance when updating the strategy; V(s t ) represents the state value function, which represents the state s t expected return.

[0071] Then the PPO advantage function improvement expression is set as follows:

[0072]

[0073] Among them, e and β represent the natural base and hyperparameter respectively, and d t Indicates the time interval between continuous actions.

[0074] S42, based on step S41, realizing dynamic scheduling of heterogeneous supercomputing cluster job flows;

[0075] The scheduling system simulates the hardware composition structure of the supercomputing center and builds a simulation computing platform with multiple heterogeneous computing cores in a single computing node. It also generates multiple DAG jobs with different structures and scales, and uses the Poisson arrival process to simulate the job arrival time for training PPOM-DSJF-HSC.

[0076] First, the Actor and Critic neural networks are randomly initialized. The Actor receives state information input and outputs scheduling actions, while the Critic outputs state values ​​for the state information. The agent interacts with the environment to generate scheduling trajectories, and then updates the network parameters through trajectory data. In each scheduling round, the reward value r is calculated according to formula (6), and the environment state is changed to obtain the next state information s′. The trajectory generated by this round of scheduling is stored in τ until all tasks are scheduled.

[0077] Then, a resource inheritance mechanism is proposed, which is only used between subtasks within a task and between successive tasks within the same job, that is, when a subtask is executed and the occupied computing core is released, the subtasks to be scheduled on the same task node can directly inherit and use the released computing core; when a task node is completed, its released computing core is directly inherited and used by its successor task node. And the resource scheduling between different jobs is decided by the scheduler based on the current state.

[0078] At the same time, the randomly arriving DAG job flows are scheduled at three levels:

[0079] 1) The lowest level is to move CCs between subtasks within the same task node. The trigger condition is that a subtask is completed, but the task node to which it belongs still has subtasks to be executed. The scheduler directly moves the CC to other parallel subtasks without the need to obtain actions through state processing.

[0080] 2) The middle layer is to move CCs between different task nodes in the same job, which includes two cases.

[0081] Case 1: When a task is completed and its successor task node is unlocked, the scheduler moves the CC it occupies to the successor task node without the need for state processing to obtain an action;

[0082] Case 2: When a subtask in a task is completed and there are no remaining unexecuted subtasks, the scheduler allocates its CC to other task nodes that need CC in this job based on the current status information.

[0083] 3) The top layer is to move CCs between different jobs. When a new job arrives or a job is completed, the scheduler gives a scheduling action based on the current status information and assigns the available CCs to specific tasks.

[0084] Beneficial effects of the present invention: The method of the present invention first abstracts the directed graph job flow scheduling problem in a heterogeneous supercomputing cluster, then determines the scheduling objectives and evaluation indicators, and constructs a reinforcement learning framework, and uses a proximal strategy to optimize and realize the dynamic scheduling of heterogeneous supercomputing cluster job flows. The method of the present invention proposes a hierarchical scheduling method, which reduces the processing of state information by the scheduler with sparse scheduling decisions, realizes efficient task scheduling, uses GCN to extract task dependency features, and uses it as part of the state information. By introducing resource inheritance and intra-task parallel mechanisms, the communication overhead between tasks is reduced, the utilization rate of computing resources is improved, and the optimal scheduling strategy is trained and learned using an improved PPO method. The trained scheduling strategy can make the best scheduling action decision based on dynamically changing tasks and computing resource status information. The method of the present invention meets the multi-level parallelism between task nodes within a job and between subtasks within a task node, reduces the response time of jobs by sharing global computing resources between jobs, and provides a more efficient solution for the scheduling of computing resources in heterogeneous supercomputing centers, which can provide resource support for different computing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 The present invention is a flowchart of a proximal strategy optimization method for dynamic scheduling of heterogeneous supercomputing cluster job flows.

[0086] Figure 2 Schematic diagram of the task feature embedding process in an embodiment of the present invention.

[0087] Figure 3 Schematic diagram of the scheduling trajectory generation process in an embodiment of the present invention.

[0088] Figure 4 This is a flow chart of the scheduling algorithm for interactive learning between the agent and the environment in an embodiment of the present invention.

[0089] Figure 5 Schematic diagram of a hierarchical scheduling model in an embodiment of the present invention.

[0090] Figure 6 This is a flow chart of the algorithm for processing subtask completion events in an embodiment of the present invention.

[0091] Figure 7 This is a flow chart of the Actor and Critic neural network model training algorithm in an embodiment of the present invention.

[0092] Figure 8 Schematic diagram of the corresponding relationship between the calculation platform load and the average job arrival interval time in an embodiment of the present invention.

[0093] Fig. 9 This is a schematic diagram of an environment configuration with better model training results in an embodiment of the present invention.

[0094] Fig.10 Schematic diagram of algorithm performance comparison under different workloads in an embodiment of the present invention.

[0095] Fig.11 Schematic diagram of algorithm performance comparison under the same environment in embodiments of the present invention.

[0096] Fig.12 Schematic diagram showing the comparison of performance indicators of PPO before and after improvement in an embodiment of the present invention.

[0097] Fig.13 Schematic diagram of the comparison of the performance of training models in environments with workloads that differ by 10 times according to an embodiment of the present invention. DETAILED DESCRIPTION

[0098] The method of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0099] like Figure 1 As shown in the figure, a flow chart of a proximal policy optimization method for dynamic scheduling of job flows in heterogeneous supercomputer clusters (PPOM-DSJF-HSC) of the present invention is shown, and the specific steps are as follows:

[0100] S1. Abstract of directed graph job flow scheduling problem in heterogeneous supercomputing clusters;

[0101] S2. Based on step S1, determine the scheduling objectives and evaluation indicators;

[0102] S3. Based on the scheduling problem of step S1 and the scheduling goal of step S2, a reinforcement learning framework is constructed;

[0103] S4. Based on the reinforcement learning framework constructed in step S3, the proximal strategy is used to optimize and realize the dynamic scheduling of heterogeneous supercomputing cluster job flows.

[0104] In this embodiment, the step S1 is specifically as follows:

[0105] In a heterogeneous supercomputing cluster environment, jobs submitted by users arrive randomly and continuously in the form of streams (scientific workflows), waiting for the scheduler to allocate computing resources. i (V,E) represents the i-th job arriving at the platform workflow.

[0106] in, Represents the set of task nodes for job i; represents the set of task dependencies within job i, and the fine-grained parallelism criteria within the task, task node Represents an asynchronous subtask st v The asynchronous subtasks in the same task use the same operation to perform different data calculations, and the same workload wl is used to describe each subtask; d j,k Indicates the task Only in missions The execution can only start after completion. The set of all predecessor tasks of task v is denoted as ζ(v), and the set of all successor tasks is denoted as ξ(v).

[0107] Heterogeneous supercomputing cluster computing platform P = {p1, p2, ..., p |P| It consists of |P| types of heterogeneous processors, each type of processor has a different number of computing cores (CC), and the computing rate of each CC is cs. Each CC is the basic unit of scheduling, that is, the basic unit for executing subtasks.

[0108] Where |P| represents the number of processor types of platform P.

[0109] In this embodiment, step S2 is specifically as follows:

[0110] The workflow submitted by users continuously arrives at the heterogeneous supercomputing cluster computing platform in a Poisson process. The scheduling goal is to determine the start time of each task, assign CCs to the tasks, and optimize the response time, span time and resource utilization of the job while satisfying resource and dependency constraints.

[0111] Among them, the CC assigned to the task includes: type and quantity; the indicators for evaluating the performance of the scheduling algorithm are as follows:

[0112] 1) Average job response time 2) Average job completion time 3) Resource Utilization

[0113] Where N represents the number of jobs, represents the waiting time of the i-th job, Respectively represent the operation G i The moment it arrives at the system, obtains resources and starts execution. Indicates the span time of the job, Indicates job G i The time when the execution ends. et represents the effective working time of the resource, and tt represents the total time.

[0114] In this embodiment, step S3 is specifically as follows:

[0115] S31, status information representation;

[0116] The status information includes dynamically changing information about tasks to be scheduled and available resources. Interdependent tasks to be scheduled change dynamically with the scheduling process, and the graph convolutional neural network GCN is used to process such dynamically scaling graph data. Job nodes and global nodes are added to the design to represent the information of each job and the summary information of all jobs respectively. The existing one-way information aggregation GCN is improved to the two-way information aggregation BiGCN to further enhance the task node information representation capability.

[0117] S311, such as Figure 2 As shown in (a), the task nodes are embedded;

[0118] The original feature x of the task node v v As input, BiGCN is used to output the embedding features of task nodes.

[0119] Upward embedding features of task node v Aggregate the information of all child nodes, the expression is as follows:

[0120]

[0121] Among them, g(·), h(·), and f(·) represent three nonlinear transfer functions σ(WX), W represents the learnable parameter matrix, and σ represents the activation function LeakyReLU. Let X represent the original features of the job, which is composed of the original features of each task node, including: (i) the number of CCs that can be applied, (ii) whether the available CCs belong to this job, (iii) the type of occupied CCs, (iv) the number of occupied CCs, (v) the number of remaining subtasks, and (vi) the workload of each subtask.

[0122] Downward embedding features of task node v Aggregate the information of all parent nodes, the expression is as follows:

[0123]

[0124] Here, g′(·) and h′(·) represent functions with the same form as g(·) and h(·) but different parameters.

[0125] The leaf node task has no successor task, and its upward embedding vector is obtained by its original features through two layers of nonlinear transformation f(·) and g(·), that is, the low-dimensional space features of the leaf node task are mapped to the high-dimensional space, and the information is aggregated to the parent node during the upward embedding process as the upward embedding feature of the parent node, and so on, to obtain the upward embedding features of all task nodes. Similarly, the downward embedding process is to aggregate the features of the parent node to the child node, and finally obtain the downward embedding features of all task nodes.

[0126] S312, such as Figure 2 (b) shows the directed graph job embedding;

[0127] Add a virtual node to each directed acyclic graph (DAG) and embed the task node’s features And the original feature x v Aggregate to virtual node y i , represents the embedding feature of the entire job, and its expression is as follows:

[0128]

[0129] S313, such as Figure 2 (b) shows,global job embedding;

[0130] Similar to directed graph jobs, add a virtual global node z i , which represents the aggregation of the embedded feature information of all jobs as the embedded feature of the global job in the system. Its expression is as follows:

[0131]

[0132] S32, design scheduling action;

[0133] In the design, a two-stage scheduling action is adopted to balance the decision complexity and action trajectory length. The first stage selects a task from the schedulable tasks, and the second stage selects a CC type and its number from the available computing resources. The scheduling action can be triggered when any of the following situations occur: (i) a new job arrives at the system, or (ii) a task execution ends.

[0134] S321, task selection;

[0135] 1) At a certain time t, the scheduling action selects a task from n schedulable tasks, and the policy network calculates a scalar value for all schedulable tasks v by embedding features

[0136] Here, p(·) represents a shallow neural network.

[0137] 2) Using the softmax(·) function, according to the scalar value p v The probability of each task being selected is calculated as follows:

[0138]

[0139] Among them, A t Represents the set of schedulable tasks at time t.

[0140] 3) According to the probability distribution calculated by formula (5), the scheduling strategy determines the tasks to be scheduled.

[0141] S322, computing resource selection;

[0142] 1) First select the type of CC;

[0143] Similar to the task selection stage, the neural network q(y i ,z,CC abil ) Calculate the score q for each CC type type .

[0144] Among them, CC abil Indicates the computing power of different types of resources, q type Indicates the priority of computing resources of type type that are suitable for the current job environment.

[0145] Use the softmax(·) function to convert q type Converted into probability distribution, the scheduling strategy determines the appropriate CC type.

[0146] 2) Allocate the appropriate amount from the available amount of CC;

[0147] The scheduling strategy determines the appropriate CC type for each possible number of CCs in type. The q value is calculated and converted into a probability distribution using the softmax(·) function. The scheduling strategy determines the number of CCs to be allocated, num.

[0148] The q value represents the number of allocatable CCs of type type, num represents the number of actually allocated CCs, and num≤q.

[0149] S323, scheduling action coding;

[0150] Encode the scheduling action into a four-tuple (job idx ,task idx ,type,num).

[0151] Among them, job idx 、task idx They represent the job index number and task index number of the task selected in the first stage respectively.

[0152] S33, reward function design;

[0153] The optimization target indicators of the scheduling strategy include: AJRT and AJCT, then the reward function design expression is as follows:

[0154] r k =-(t k -tk-1 )(μM k +νN k ) (6)

[0155] Among them, t k represents the time when the kth scheduling action is generated, M k 、N k Respectively represent the time interval [t k-1 ,t k ), μ and ν represent the hyperparameters of relative weights, and the agent receives a reward value r after making the kth action. k .

[0156] The training and learning process is to schedule the policy π θ The cumulative reward obtained by the generated trajectory Solving the maximization problem is equivalent to minimizing the number of all jobs and the number of unresponsive jobs in the scheduling system; according to Little's law, the learning process for the scheduling goal is equivalent to reducing the average job completion time and response time.

[0157] Among them, k, τ, and θ represent the time step, scheduling trajectory, and strategy parameters, respectively.

[0158] In this embodiment, step S4 is specifically as follows:

[0159] S41, Proximal Policy Optimization (PPO) advantage function improvement;

[0160] PPO is a policy gradient-based reinforcement learning method, the core of which is to decide how to adjust policy parameters to increase future rewards. It represents the advantage function calculated by the generalized advantage estimation (GAE) algorithm, which is used to evaluate the relative goodness of the current action in the policy update and provide the direction of the policy update. Its expression is as follows:

[0161]

[0162] Among them, r t represents the reward obtained by the t-th action, T represents the total number of scheduled actions, γ represents the discount factor, which controls the weight of future returns; λ represents the smoothing coefficient, which helps balance the deviation and variance when updating the strategy; V(s t ) represents the state value function, which represents the state s t expected return.

[0163] Combined with the background of scheduling problems, jobs arrive at the system in the form of a stream according to the Poisson process, the size of the task set has a large variance, and the arrival time of the job is random. The scheduling action occurs when a new job arrives at the system or when a task is executed. The reward value obtained is also related to the interval time of the scheduling action. During scheduling, larger tasks in some jobs will occupy computing resources for a long time, which will lead to an increase in the scheduling interval and a larger absolute value of the reward. In fact, such a reward value cannot fully represent the quality of this round of scheduling actions, but will greatly affect the advantage value. In addition, fixed γ and λ are difficult to avoid the impact of the variance of the task size on the advantage value. Combining the above analysis, the improved expression of the PPO advantage function is as follows:

[0164]

[0165] Among them, e and β represent the natural base and hyperparameter respectively, and d t Indicates the time interval between continuous actions.

[0166] After the PPO advantage function is improved, when d t The bigger the The smaller it is, the stronger the discount on the reward value obtained from large-scale tasks is, which weakens its impact on the advantage value and makes the direction and magnitude of the strategy update more accurate.

[0167] S42, based on step S41, realizing dynamic scheduling of heterogeneous supercomputing cluster job flows;

[0168] The scheduling system simulates the hardware composition structure of the supercomputing center and builds a simulation computing platform with multiple heterogeneous computing cores in a single computing node. It also generates multiple DAG jobs with different structures and scales, and uses the Poisson arrival process to simulate the job arrival time for training PPOM-DSJF-HSC.

[0169] like Figure 3 As shown in the figure, the scheduling trajectory generation process describes the design of the two main modules, Actor and Critic, and the generation process of τ. First, the two neural networks, Actor and Critic, are randomly initialized. Actor receives state information input and outputs scheduling actions, while Critic outputs state value for state information. The agent interacts with the environment to generate scheduling trajectories, and then updates the network parameters through trajectory data. Figure 4 The generation process of scheduling trajectory τ is described for interactive learning scheduling algorithm between the agent and the environment. The boxes in steps 3, 4, 5, 6, and 8 represent neural networks, which constitute the Actor. In each scheduling round, the reward value r is calculated according to formula (6), and the state of the environment is changed to obtain the next state information s′. The trajectory generated by this round of scheduling is stored in τ until all tasks are scheduled.

[0170] In order to simplify or reduce the latency overhead of data communication between tasks (without considering the impact of communication bandwidth between processors), this embodiment proposes a resource inheritance mechanism that is only used between subtasks within a task and between successive tasks within the same job, that is: when a subtask is executed and the occupied computing core is released, the subtasks to be scheduled on the same task node can directly inherit and use the released computing core; when a task node is completed, its released computing core is directly inherited and used by its successor task node. And the resource scheduling between different jobs is made by the scheduler according to the current state.

[0171] At the same time, if Figure 5 As shown in the figure, the hierarchical scheduling model schedules randomly arriving DAG job flows at three levels. The directed arrows of different colors in the figure represent the movement of CCs between tasks:

[0172] 1) If Figure 5 As shown in (a), the bottom layer is to move CC between subtasks in the same task node. The trigger condition is that a subtask is completed, but the task node to which it belongs still has subtasks to be executed (i.e. Figure 6 As shown in step 2 in ), the scheduler directly moves CC to other parallel subtasks without the need to obtain actions through state processing.

[0173] 2) If Figure 5 As shown in (b), the middle layer is to move CCs between different task nodes in the same job, which includes two cases.

[0174] Case 1: If Figure 4 As shown in step 11 of , when a task is completed, its successor task node is unlocked, such as Figure 5 As shown by the red arrow in (b), the scheduler moves the occupied CC to the subsequent task node without the need to obtain the action through state processing;

[0175] Case 2: If Figure 6 As shown in step 17 of , when a task has subtasks completed and there are no remaining unexecuted subtasks, Figure 5 As shown by the green arrow in (b), the scheduler allocates its CC to other task nodes that need CC in this job based on the current status information.

[0176] 3) If Figure 5 As shown in (c), the top layer is to move CC between different jobs, such as Figure 4 As shown in steps 2 to 10 in , when a new job arrives or a job is completed, Figure 5 As shown by the orange arrow in (c), the scheduler gives scheduling actions based on the current state information and assigns the available CCs to specific tasks.

[0177] Figure 7Describe the Actor and Critic update process. The training data is divided into multiple batches, and in each batch, the loss functions of the Actor and Critic are calculated alternately and the model parameters are updated.

[0178] This embodiment is further verified by simulation experiments, as follows:

[0179] According to the scientific computing workflow in real scenarios, 1,000 DAG jobs of different structures and sizes are randomly generated to train the model of the present invention. The job sequence is modeled as a Poisson process with an average arrival interval of t minutes. The simulation computing platform simulates the heterogeneous multi-core computing nodes in real applications and consists of two heterogeneous computing cores. Different types of computing cores have different computing rates cs.

[0180] Set the estimated workload of each subtask to be known, measure it with the standard execution time st, and calculate the actual execution time based on the computing core type obtained The generated jobs are evenly distributed from the simplest 5 task nodes to the complex 85 task nodes. The number of subtasks within a task node is normally distributed from 1 to 200, and the ST of the subtasks is normally distributed from 2 minutes to 15 minutes.

[0181] In order to maintain the resource competition relationship between jobs and verify the effectiveness of the scheduling strategy, this embodiment configures a heterogeneous computing node with 10 high-speed computing cores and 10 low-speed computing cores as a simulation computing platform. The corresponding relationship between the computing platform load and the average job arrival interval in the experiment is as follows: Figure 8 This embodiment found through multiple groups of experiments that Fig. 9 The environment configuration shown has better model training results.

[0182] Then compare the performance indicators, such as Fig.10 As shown, Fig.10 (a) Comparison of the AJCT indicators of PPOM-DSJF-HSC and other methods under different workloads shows that PPOM-DSJF-HSC is significantly better than other methods, especially when the workload is larger. When the workload is 100%, PPOM-DSJF-HSC has a 31% performance improvement over decima-rev.

[0183] Fig.10(b) Comparing the AJRT index, it can be seen that FCFS is significantly higher than other methods. This is because this strategy allocates almost all idle resources to a new job when it arrives, which causes the subsequent jobs to wait for a long time. PPOM-DSJF-HSC adds a penalty for job response time during the training process, and AJRT is reduced by 49% compared to decima-rev when the workload is 100%. fair performs best in this index test because the fair scheduling strategy evenly distributes computing resources to each job, sacrificing the job completion time index to ensure the fast response of the job.

[0184] like Fig.11 As shown, this embodiment compares the computing core utilization of several strategies in an environment with a workload of 60%. Fig.11 (a) is the fair algorithm, Fig.11 (b) is the FCFS first-come, first-served algorithm, Fig.11 (c) is decima-rev: Decima correction algorithm, Fig.11 (d) is the method of the present invention. As can be seen from the figure, the RU of fair and FCFS are relatively low, and the load of each computing core of fair is obviously unbalanced, while the load of the computing core of decima-rev is the most balanced.

[0185] The load of high-speed computing cores (numbered less than or equal to 10) obtained by PPOM-DSJF-HSC is higher than that of low-speed computing cores (numbered greater than 10), and is more balanced. This is because PPOM-DSJF-HSC takes the completion time of the job as the goal and learns the strategy of prioritizing the scheduling of high-speed computing cores during the training process. Although PPOM-DSJF-HSC seems to be slightly worse than decima-rev in terms of RU indicators, it is worth noting that in an environment with a workload of 60%, PPOM-DSJF-HSC achieves 20% and 53% performance improvements over decima-rev's AJCT and AJRT indicators, respectively. This also shows that although decima-rev has achieved a high and balanced resource utilization, it has not scheduled resources to the most critical tasks.

[0186] like Fig.12 As shown in the figure, it reflects the stability of the strategy learning process and describes the role of the improvement measures of PPO in the model training process. It can be seen that the convergence rate of the improved model is significantly improved, it can converge to a better solution, and the oscillation amplitude of the training curve is also significantly reduced.

[0187] like Fig.13As shown in the figure, the scalability of the model is reflected. The experimental results show that the models trained in workload environments with a difference of 10 times have basically the same performance in AJCT and AJRT indicators. This shows that PPOM-DSJF-HSC has strong scalability.

[0188] In the method of the present invention, GCN is used to extract task dependency features and use them as part of the state information; resource inheritance and intra-task parallelism mechanisms are introduced to reduce inter-task communication overhead and improve computing resource utilization. At the same time, the improved PPO method is used to train and learn the optimal scheduling strategy. The trained scheduling strategy can make the best scheduling action decision based on dynamically changing tasks and computing resource status information. The above experimental results show that the method of the present invention improves the job completion time and response performance indicators by more than 50% compared with the existing scheduling algorithms.

[0189] In summary, there are many studies on the problem of job flow scheduling in heterogeneous environments, but most of the studies do not consider the resource inheritance between successive tasks and the fine-grained parallelism within the tasks. The hierarchical scheduling method proposed by the method of the present invention reduces the processing of state information by the scheduler with sparser scheduling decisions, thereby achieving efficient task scheduling. Through the above experimental verification, the method of the present invention can achieve significantly better performance than other methods in terms of the average completion time, average response time, utilization rate of computing resources and load balancing of the job, and has strong training stability and scalability. The method of the present invention satisfies the multi-level parallelism between task nodes within the job and between subtasks within the task node, reduces the response time of the job by sharing global computing resources between jobs, and provides a more efficient solution for the computing resource scheduling of heterogeneous supercomputing centers, which can provide resource support for different computing tasks.

[0190] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A proximal strategy optimization method for dynamic scheduling of heterogeneous supercomputing cluster job flows, the specific steps are as follows: S1. Abstract of directed graph job flow scheduling problem in heterogeneous supercomputing clusters; S2. Based on step S1, determine the scheduling objectives and evaluation indicators; S3. Based on the scheduling problem of step S1 and the scheduling goal of step S2, a reinforcement learning framework is constructed; S4. Based on the reinforcement learning framework constructed in step S3, the proximal strategy is used to optimize and realize the dynamic scheduling of heterogeneous supercomputing cluster job flows.

2. According to claim 1, a proximal strategy optimization method for dynamic scheduling of heterogeneous supercomputing cluster job flows is characterized in that: The step S1 is specifically as follows: In a heterogeneous supercomputing cluster environment, jobs submitted by users arrive randomly and continuously in the form of a stream, waiting for the scheduler to allocate computing resources. i (V,E) represents the i-th job arriving at the platform workflow; in, Represents the set of task nodes for job i; represents the set of task dependencies within job i, and the fine-grained parallelism criteria within the task, task node Represents an asynchronous subtask st v The asynchronous subtasks in the same task use the same operation to perform different data calculations, and the same workload wl is used to describe each subtask; d j,k Indicates the task Only in missions Execution can only start after completion; the set of all predecessor tasks of task v is recorded as ζ(v), and the set of all successor tasks is recorded as ξ(v); Heterogeneous supercomputing cluster computing platform P = {p1, p2, ..., p P It consists of |P| types of heterogeneous processors, each type of processor has a different number of computing cores CC, and the computing rate of each CC is cs; each CC is the basic unit of scheduling, that is, the basic unit for executing subtasks; Where |P| represents the number of processor types of platform P.

3. The method for proximal strategy optimization for dynamic scheduling of heterogeneous supercomputing cluster job flows according to claim 1, characterized in that: The step S2 is specifically as follows: The workflow submitted by the user continuously arrives at the heterogeneous supercomputing cluster computing platform in a Poisson process. The scheduling goal is to determine the start time of each task, assign CCs to the tasks, and optimize the response time, span time, and resource utilization of the job under the premise of satisfying resource and dependency constraints; Among them, the CC assigned to the task includes: type and quantity; the indicators for evaluating the performance of the scheduling algorithm are as follows: 1) Average job response time 2) Average job completion time 3) Resource Utilization Where N represents the number of jobs, represents the waiting time of the i-th job, Respectively represent the operation G i The moment it arrives at the system, obtains resources and starts execution; Indicates the span time of the job, Indicates job G i The moment when the execution ends; et represents the effective working time of the resource, and tt represents the total time.

4. The method for proximal strategy optimization for dynamic scheduling of heterogeneous supercomputing cluster job flows according to claim 1, characterized in that: The step S3 is specifically as follows: S31, status information representation; The state information includes dynamically changing information about tasks to be scheduled and available resources. Interdependent tasks to be scheduled change dynamically with the scheduling process, so a graph convolutional neural network (GCN) is used to process such dynamically scalable graph data. Job nodes and global nodes are added to the design to represent each job information and the summary information of all jobs respectively, and the existing one-way information aggregation GCN is improved to a two-way information aggregation BiGCN. S311, task node embedding; The original feature x of the task node v v As input, BiGCN is used to output the embedding features of the task nodes; Upward embedding features of task node v Aggregate the information of all child nodes, the expression is as follows: Among them, g(·), h(·), and f(·) represent three nonlinear transfer functions σ(WX), W represents the learnable parameter matrix, and σ represents the activation function LeakyReLU; Let X represent the original features of the job, which is composed of the original features of each task node, including: (i) the number of CCs that can be applied, (ii) whether the available CCs belong to this job, (iii) the type of occupied CCs, (iv) the number of occupied CCs, (v) the number of remaining subtasks, and (vi) the workload of each subtask; Downward embedding features of task node v Aggregate the information of all parent nodes, the expression is as follows: Here, g′(·) and h′(·) represent functions with the same form as g(·) and h(·) but different parameters, respectively; The leaf node task has no successor task, and its upward embedding vector is obtained by its original features through two layers of nonlinear transformations f(·) and g(·), that is, the low-dimensional space features of the leaf node task are mapped to the high-dimensional space, and the information is aggregated to the parent node during the upward embedding process as the upward embedding feature of the parent node. Similarly, the upward embedding features of all task nodes are obtained; similarly, the downward embedding process is to aggregate the features of the parent node to the child node, and finally obtain the downward embedding features of all task nodes; S312, directed graph job embedding; Add a virtual node to each directed acyclic graph DAG and embed the features of the task node And the original feature x v Aggregate to virtual node y i , represents the embedding feature of the entire job, and its expression is as follows: S313, global job embedding; Similar to directed graph jobs, add a virtual global node z i , which represents the aggregation of the embedded feature information of all jobs as the embedded feature of the global job in the system. Its expression is as follows: S32, design scheduling action; In the design, a two-stage scheduling action is adopted. The first stage selects a task from the schedulable tasks, and the second stage selects a CC type and its quantity from the available computing resources. The scheduling action can be triggered when any of the following situations occurs: (i) a new job arrives at the system; (ii) a task execution ends; S321, task selection; 1) At a certain time t, the scheduling action selects a task from n schedulable tasks, and the policy network calculates a scalar value for all schedulable tasks v by embedding features Where p(·) represents a shallow neural network; 2) Using the softmax(·) function, according to the scalar value p v The probability of each task being selected is calculated as follows: Among them, A t represents the set of schedulable tasks at time t; 3) According to the probability distribution calculated by formula (5), the scheduling strategy determines the tasks to be scheduled; S322. Computing resource selection 1) First select the type of CC; Similar to the task selection stage, the neural network q(y i ,z,CC abil ) Calculate the score q for each CC type type ; Among them, CC abil Indicates the computing power of different types of resources, q type Indicates the priority of type computing resources that are suitable for the current job environment; Use the softmax(·) function to convert q type Converted into probability distribution, the scheduling strategy determines the appropriate CC type; 2) Allocate the appropriate amount from the available amount of CC; Determine the appropriate CC type for the scheduling policy type Calculate the q value for each possible CC number in, use the softmax(·) function to convert the q value into a probability distribution, and the scheduling strategy determines the number of CCs to be allocated num; The q value indicates the number of allocatable CCs of type type, num indicates the number of actually allocated CCs, and num≤q; S323, scheduling action coding; Encode the scheduling action into a four-tuple (job idx ,task idx ,type,num); Among them, job idx 、task idx They represent the job index number and task index number of the task selected in the first stage respectively; S33, reward function design; The optimization target indicators of the scheduling strategy include: AJRT and AJCT, then the reward function design expression is as follows: r k =-(t k -t k-1 )(μM k +νN k ) (6) Among them, t k represents the time when the kth scheduling action is generated, M k 、N k Respectively represent the time interval [t k-1 ,t k ), μ and ν represent the hyperparameters of relative weights, and the agent receives a reward value r after making the kth action. k ; The training and learning process is to schedule the policy π θ Cumulative rewards for the generated tracks Solving the maximization problem is equivalent to minimizing the number of all jobs and the number of unresponsive jobs in the scheduling system. According to Little's law, the learning process of the scheduling goal is equivalent to reducing the average job completion time and response time. Among them, k, τ, and θ represent the time step, scheduling trajectory, and strategy parameters, respectively.

5. The method for proximal strategy optimization for dynamic scheduling of heterogeneous supercomputing cluster job flows according to claim 1, characterized in that: The step S4 is specifically as follows: S41, Proximal Strategy Optimization PPO Advantage Function Improvement; set up It represents the advantage function calculated by the generalized advantage estimation GAE algorithm, which is used to evaluate the relative quality of the current action in the policy update and provide the direction of the policy update. Its expression is as follows: Among them, r t represents the reward obtained by the t-th action, T represents the total number of scheduled actions, γ represents the discount factor, which controls the weight of future returns; λ represents the smoothing coefficient, which helps balance the deviation and variance when the strategy is updated; V(s t ) represents the state value function, which represents the state s t expected return; Then the PPO advantage function improvement expression is set as follows: Among them, e and β represent the natural base and hyperparameter respectively, and d t Indicates the time interval between consecutive actions; S42, based on step S41, realizing dynamic scheduling of heterogeneous supercomputing cluster job flows; The scheduling system simulates the hardware structure of the supercomputing center, builds a simulation computing platform with multiple heterogeneous computing cores in a single computing node, and generates multiple DAG jobs with different structures and scales. It uses the Poisson arrival process to simulate the job arrival time for training PPOM-DSJF-HSC. First, the Actor and Critic neural networks are randomly initialized. The Actor receives state information input and outputs scheduling actions, while the Critic outputs state values ​​for the state information. The agent interacts with the environment to generate scheduling trajectories, and then updates the network parameters through trajectory data. In each scheduling round, the reward value r is calculated according to formula (6), and the environment state is changed to obtain the next state information s′. The trajectory generated by this round of scheduling is stored in τ until all tasks are scheduled. Then, a resource inheritance mechanism is proposed, which is only used between subtasks within a task and between successive tasks within the same job, namely: when a subtask is executed and the occupied computing core is released, the subtasks to be scheduled on the same task node can directly inherit and use the released computing core; when a task node is completed, its released computing core is directly inherited and used by its successor task node; and the resource scheduling between different jobs is decided by the scheduler based on the current status; At the same time, the randomly arriving DAG job flows are scheduled at three levels: 1) The lowest level is to move CCs between subtasks in the same task node. The trigger condition is that a subtask is completed, but the task node to which it belongs still has subtasks to be executed. The scheduler directly moves the CC to other parallel subtasks without the need to obtain actions through state processing; 2) The middle layer is to move CCs between different task nodes in the same job. This layer includes two cases; Case 1: When a task is completed and its successor task node is unlocked, the scheduler moves the CC it occupies to the successor task node without the need for state processing to obtain an action; Case 2: When a subtask in a task is completed and there are no remaining unexecuted subtasks, the scheduler allocates its CC to other task nodes that need CC in this job based on the current status information; 3) The top layer is to move CCs between different jobs. When a new job arrives or a job is completed, the scheduler gives a scheduling action based on the current status information and assigns the available CCs to specific tasks.

Images