Historical task data driven deep learning task scheduling simulation method and system

Abstract

The application discloses a historical task data driven deep learning task scheduling simulation method and system, and the method comprises the following steps: a control module modifies a cluster resource management table according to newly added or deleted task execution modules, and updates deep learning task scheduling historical data; a simulation scheduling module generates a to-be-scheduled task set according to the deep learning task scheduling historical data, generates a newly added scheduling task set of the current round through a task admission strategy, and adds the newly added scheduling task set to a current round scheduling task set; a task scheduling strategy is used for sorting the current round scheduling task set, so that a cluster scheduling task set is obtained; a task distribution strategy is used for generating a task sequence that can be run and a task set that needs to be suspended according to the cluster scheduling task set, and a cluster running information table and a task running state information table are updated; and a task state database is refreshed according to records of the updated cluster running information table and the task running state information table.

Description

Technical Field

[0001] The present invention belongs to the field of distributed deep learning task scheduling algorithm design, and specifically relates to a deep learning task scheduling simulation method and system driven by historical task data. Background Art

[0002] Scheduling frameworks like Apache Mesos and Hadoop YARN handle workloads consisting of several short-running tasks or long-running big data jobs. These tasks or services run at high priority and are therefore typically not preempted. Deep learning training tasks typically run for long periods of time, with computations repeated across numerous iterations. Therefore, unlike big data scheduling algorithms, deep learning task scheduling algorithms must frequently preempt running jobs to better utilize cluster computing resources.

[0003] At the same time, deep learning task schedulers typically require access to application-level metrics such as function loss, gradient values, and throughput to support specific properties of deep learning tasks, such as completion time fairness or gradient-based elasticity, which are not easily implemented in existing scheduling frameworks. Therefore, although previous deep learning task scheduling algorithms have been implemented as plugins on Kubernetes or YARN, these systems typically require the design of additional deep learning task functionality to support iteration-level preemption or application-level metric collection, resulting in low task scheduling efficiency. Summary of the Invention

[0004] In response to the above-mentioned deficiencies in the prior art, the historical task data-driven deep learning task scheduling simulation method and system provided by the present invention solve the problems of low task scheduling efficiency, complex configuration of the scheduling algorithm development and verification environment, and high implementation cost of the existing deep learning task scheduling algorithms.

[0005] In order to achieve the above-mentioned purpose of the invention, the technical solution adopted by the present invention is: a deep learning task scheduling simulation method driven by historical task data, comprising the following steps:

[0006] S1. Modify the cluster resource management table based on the newly added or deleted task execution modules through the control module, and update the deep learning task scheduling historical data;

[0007] S2. Generate a set of tasks to be scheduled based on the deep learning task scheduling historical data through the simulation scheduling module, generate a set of newly schedulable tasks in this round through the task admission strategy, and add it to the set of tasks to be scheduled in this round;

[0008] S3. Sort the set of tasks to be scheduled in this round by the task scheduling strategy to obtain the cluster scheduling task set;

[0009] S4. Generate a runnable task sequence and a task set that needs to be suspended based on the cluster scheduling task set through the task distribution strategy;

[0010] S5. Call the task execution module according to the executable task sequence and the task set that needs to be suspended, and update the cluster operation information table and the task operation status information table;

[0011] S6. Refresh the task status database according to the updated records in the cluster operation information table and the task operation status information table.

[0012] Furthermore: in said S1, the method for modifying the cluster resource management table is specifically as follows:

[0013] Add the IP address, process ID, and number of accelerator cards of the newly added task execution module to the cluster resource management table. Delete the records of deleted or faulty task execution modules from the cluster resource management table.

[0014] Furthermore, in S2, the task admission strategy method is specifically as follows:

[0015] According to the generated set of tasks to be scheduled, the tasks in the set of tasks to be scheduled are obtained in a first-in-first-out order. The set of newly schedulable tasks in this round is generated under the constraints. The specific expression of the constraints is:

[0016]

[0017] Where, f(N jk ) is the N newly available scheduling tasks in this round jk The number of accelerator cards that need to be occupied, n is the number of newly scheduled tasks that can be added in this round, f(R jk ) is the running task R jk The number of accelerator cards that need to be occupied, m is the number of tasks to be scheduled, TotalAccNum is the number of accelerator cards available in the entire cluster, LoadThreshold is the maximum load factor of the cluster, LoadThreshold∈(1.0,2.0].

[0018] Furthermore, in S3, the task scheduling strategy is specifically as follows:

[0019] Calculate the first and second indicators of the scheduling tasks, sort the set of tasks to be scheduled in this round according to the values ​​of the first and second indicators, and use the sorted set of scheduling tasks as the cluster scheduling task set;

[0020] Among them, the first indicator jpr(S ji ) Specifically for task S ji The running priority of

[0021] The expression of the second indicator is specifically:

[0022] jei(S ji )*jit(S ji )*jad(S ji )

[0023] In the formula, jei(S ji ) is task S ji The number of executed iterations, jit(S ji ) is task S ji The single-round iteration running time, jad(S ji ) is task S ji The number of accelerator cards required.

[0024] Further: in said S4, the method of task distribution strategy includes compact task distribution strategy and decentralized task distribution strategy;

[0025] The compact task allocation strategy specifically allocates accelerator card computing resources to the scheduled tasks to be run according to the natural order of the idle accelerator card numbers;

[0026] The decentralized task allocation strategy is to allocate accelerator card computing resources to the scheduled tasks to be run according to the load balancing principle.

[0027] Further: in said S5, the cluster operation information table includes the execution node identifier, IP address, number of accelerator cards and accelerator card information;

[0028] Accelerator card information includes accelerator card ID, local accelerator card ID, execution node ID, memory capacity, usage, and running task list;

[0029] The task running status information table includes the timestamp of this round of scheduling, the list of tasks to be run, the statistics of running task time, the list of completed tasks and the list of suspended tasks.

[0030] A deep learning task scheduling simulation system driven by historical task data, including:

[0031] The simulation scheduling module is used to generate tasks to be scheduled based on the historical data of deep learning task scheduling, trigger task admission strategy, task scheduling strategy and task distribution strategy, and obtain cluster operation information and task operation status information;

[0032] The control module is used to manage cluster resource information, including CPU, memory, disk, accelerator card, and network. It is also used to manage the running status information of scheduled tasks, including start, stop, preemption, and waiting status. It is also used to distribute the scheduled tasks to the responsible task execution module.

[0033] The task execution module is used to trigger the running of tasks and provide management of the accelerator card usage status. It is also used to record the task ID, duration, accelerator card utilization and memory consumption of the current accelerator card task.

[0034] The task status database is used to store status information of suspended or started tasks when the system is running.

[0035] The beneficial effects of the present invention are:

[0036] (1) The present invention provides a deep learning task scheduling simulation method and system driven by historical task data. The present invention uses historical task scheduling data to drive the operation of the entire system, uses task admission strategy, task scheduling strategy and task distribution strategy to manage scheduling tasks, uses a task status database to record the data generated by the new simulation scheduling process, and constructs a complete and universal deep learning task scheduling method, so that the research on deep learning task scheduling methods no longer needs to occupy a large amount of computing power cluster resources, thereby improving the efficiency of verification and comparison of existing scheduling algorithms.

[0037] (2) Based on a large amount of deep learning task scheduling historical data, the present invention drives the simulation operation of the deep learning task scheduling algorithm, and provides a low-cost, high-efficiency task scheduling algorithm verification and comparison system, which helps to create new deep learning task scheduling algorithms and improve the development and testing efficiency of existing scheduling algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flowchart of the historical task data-driven deep learning task scheduling simulation method of the present invention.

[0039] Figure 2 This is a structural diagram of the historical task data-driven deep learning task scheduling simulation system of the present invention. DETAILED DESCRIPTION

[0040] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0041] like Figure 1 As shown, in one embodiment of the present invention, a deep learning task scheduling simulation method driven by historical task data includes the following steps:

[0042] S1. Modify the cluster resource management table based on the newly added or deleted task execution modules through the control module, and update the deep learning task scheduling historical data;

[0043] S2. Generate a set of tasks to be scheduled based on the deep learning task scheduling historical data through the simulation scheduling module, generate a set of newly schedulable tasks in this round through the task admission strategy, and add it to the set of tasks to be scheduled in this round;

[0044] S3. Sort the set of tasks to be scheduled in this round by the task scheduling strategy to obtain the cluster scheduling task set;

[0045] S4. Generate a runnable task sequence and a task set that needs to be suspended based on the cluster scheduling task set through the task distribution strategy;

[0046] S5. Call the task execution module according to the executable task sequence and the task set that needs to be suspended, and update the cluster operation information table and the task operation status information table;

[0047] S6. Refresh the task status database according to the updated records in the cluster operation information table and the task operation status information table.

[0048] In S1, the method for modifying the cluster resource management table is specifically as follows:

[0049] Add the IP address, process ID, and number of accelerator cards of the newly added task execution module to the cluster resource management table. Delete the records of deleted or faulty task execution modules from the cluster resource management table.

[0050] In S2, the task admission strategy method is specifically as follows:

[0051] According to the generated set of tasks to be scheduled, the tasks in the set of tasks to be scheduled are obtained in a first-in-first-out order. The set of newly schedulable tasks in this round is generated under the constraints. The specific expression of the constraints is:

[0052]

[0053] Where, f(N jk ) is the N newly available scheduling tasks in this round jk The number of accelerator cards that need to be occupied, n is the number of newly scheduled tasks that can be added in this round, f(R jk ) is the running task R jk The number of accelerator cards that need to be occupied, m is the number of tasks to be scheduled, TotalAccNum is the number of accelerator cards available in the entire cluster, LoadThreshold is the maximum load factor of the cluster, LoadThreshold∈(1.0,2.0].

[0054] In this embodiment, the generated set of tasks to be scheduled is defined as {S j1 ,S j2 ...S jm}, the set of all running tasks is defined as {R j1 ,R j2 ...R jn}, the task admission strategy obtains tasks from the set of tasks to be scheduled in the first-in-first-out order, and generates a set of newly schedulable tasks in this round, which is defined as {N j1 ,N j2 ...N jn}.

[0055] In S3, the task scheduling strategy is specifically as follows:

[0056] Calculate the first and second indicators of the scheduling tasks, sort the set of tasks to be scheduled in this round according to the values ​​of the first and second indicators, and use the sorted set of scheduling tasks as the cluster scheduling task set;

[0057] Among them, the first indicator jpr(S ji ) Specifically for task S ji The running priority of

[0058] The expression of the second indicator is specifically:

[0059] jei(S ji )*jit(S ji )*jad(S ji )

[0060] In the formula, jei(S ji ) is task S ji The number of executed iterations, jit(S ji ) is task S ji The single-round iteration running time, jad(S ji ) is task S ji The number of accelerator cards required.

[0061] In this embodiment, all tasks to be scheduled in this round are defined as the set of tasks to be scheduled in this round {S j1 ,S j2 ...S jn}, sort the tasks in the task set to be scheduled in this round as the actual order of cluster scheduling tasks, and get the cluster scheduling task set, which is defined as {S jl ,S jn ...S jm}.

[0062] In said S4, the task distribution strategy method includes a compact task allocation strategy and a decentralized task allocation strategy;

[0063] The compact task allocation strategy specifically allocates accelerator card computing resources to the scheduled tasks to be run according to the natural order of the idle accelerator card numbers;

[0064] The decentralized task allocation strategy is to allocate accelerator card computing resources to the scheduled tasks to be run according to the load balancing principle.

[0065] In this embodiment, for all scheduled task sequences that are not in the running state, they are defined as {J1, J2...J n}, all task execution modules including idle accelerator cards are defined as {Executor0, Executor1...Executor m}, the accelerator card set of a single task execution module is defined as {Accelerator0, Accelerator1...Accelerator7}, and these tasks are assigned to the accelerator cards of the task execution module according to the actual task allocation strategy in the simulated scheduling algorithm. The task distribution strategy adopted by the present invention includes a compact task allocation strategy and a dispersed task allocation strategy. The compact task allocation strategy is to allocate accelerator card computing resources to the tasks to be run according to the natural order of the idle accelerator card numbers. The dispersed task allocation strategy is to allocate accelerator card computing resources to the tasks to be run according to the load balancing principle. Since some task resources may not be allocated to accelerator card resources, after the allocation strategy is completed, an executable task sequence {Active j1 ,Active j2 ...Active jn} and the set of tasks that need to be suspended {Suspend j1 ,Suspend j2 ...Suspend jn}.

[0066] In S5, the cluster operation information table includes the execution node identifier, IP address, number of accelerator cards, and accelerator card information;

[0067] Accelerator card information includes accelerator card ID, local accelerator card ID, execution node ID, memory capacity, usage, and running task list;

[0068] The task running status information table includes the timestamp of this round of scheduling, the list of tasks to be run, the statistics of running task time, the list of completed tasks and the list of suspended tasks.

[0069] In this embodiment, the control module will call the task execution module's task running submodule to obtain the executable task sequence {Active j1 ,Active j2...Active jn} and the set of tasks that need to be suspended {Suspend j1 ,Suspend j2 ...Suspend jn}, call the task execution module to update the cluster operation information table and task operation status information table. The information that needs to be updated is shown in the following table.

[0070] Table 1 Cluster operation information table

[0071] Node Id Execution node identifier IP Address IP address Accelerator Num Number of accelerator cards Accelerator Infos Accelerator card information

[0072] Table 2 Accelerator card information table

[0073] Accelerator Id Accelerator card identification Local Accelerator Id Local accelerator card identifier Node Id Execution node identifier Memory Memory capacity In Use Is it in use? Job Ids List of running tasks

[0074] Table 3 Task running status information table

[0075] Schedule Timestamp Timestamp of this round of scheduling Active Job Ids List of tasks to be run Job Runtime Stats Running task time statistics Finished Job Ids Completed Task List Suspend Job Ids Suspended Tasks List

[0076] like Figure 2 As shown in Figure 1, the deep learning task scheduling simulation system driven by historical task data includes:

[0077] The simulation scheduling module is used to generate tasks to be scheduled based on the historical data of deep learning task scheduling, trigger task admission strategy, task scheduling strategy and task distribution strategy, and obtain cluster operation information and task operation status information;

[0078] The control module is used to manage cluster resource information, including CPU, memory, disk, accelerator card, and network. It is also used to manage the running status information of scheduled tasks, including start, stop, preemption, and waiting status. It is also used to distribute the scheduled tasks to the responsible task execution module.

[0079] The task execution module is used to trigger the running of tasks and provide management of the accelerator card usage status. It is also used to record the task ID, duration, accelerator card utilization and memory consumption of the current accelerator card task.

[0080] The task status database is used to store status information of suspended or started tasks when the system is running.

[0081] In this embodiment, the deep learning task scheduling history data includes the task history, the CPU, memory and accelerator card usage of the cluster, wherein the task history includes recording the task ID, the task scheduling time, start time, end time, the accelerator card used and the user name to which the task belongs.

[0082] The beneficial effects of the present invention are: the present invention provides a deep learning task scheduling simulation method and system driven by historical task data. The present invention uses historical task scheduling data to drive the operation of the entire system, uses task access strategy, task scheduling strategy and task distribution strategy to manage scheduling tasks, and uses the task status database to record the data generated by the new simulation scheduling process, constructing a complete and universal deep learning task scheduling method, so that the research on deep learning task scheduling methods no longer needs to occupy a large amount of computing power cluster resources, thereby improving the efficiency of verification and comparison of existing scheduling algorithms.

[0083] Based on a large amount of deep learning task scheduling historical data, this invention drives the simulation operation of deep learning task scheduling algorithms, providing a low-cost, high-efficiency task scheduling algorithm verification and comparison system, which helps to create new deep learning task scheduling algorithms and improve the development and testing efficiency of existing scheduling algorithms.

[0084] In the description of the present invention, it should be understood that the terms "center", "thickness", "upper", "lower", "horizontal", "top", "bottom", "inner", "outer", "radial", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only and cannot be understood as indicating or implying the relative importance or the number of technical features implicitly specified. Therefore, the features defined by "first", "second", and "third" may explicitly or implicitly include one or more of such features.

Claims

1. A deep learning task scheduling simulation method driven by historical task data, characterized by: The following steps are involved: S1. Modify the cluster resource management table based on the newly added or deleted task execution modules through the control module, and update the deep learning task scheduling historical data; S2. Generate a set of tasks to be scheduled based on the deep learning task scheduling historical data through the simulation scheduling module, generate a set of newly schedulable tasks in this round through the task admission strategy, and add it to the set of tasks to be scheduled in this round; In S2, the task admission strategy method is specifically as follows: According to the generated set of tasks to be scheduled, the tasks in the set of tasks to be scheduled are obtained in a first-in-first-out order. The set of newly schedulable tasks in this round is generated under the constraints. The specific expression of the constraints is: Where, f (N jk ) is the N newly available scheduling tasks in this round jk The number of accelerator cards required, n is the number of newly scheduled tasks that can be added in this round, For the running task R jk The number of accelerator cards required, m is the number of tasks to be scheduled, TotalAccNum is the number of accelerator cards available in the entire cluster, LoadThreshold is the maximum load factor of the cluster, ; S3. Sort the set of tasks to be scheduled in this round by using the task scheduling strategy to obtain a cluster scheduling task set. The specific method of the task scheduling strategy is: Calculate the first and second indicators of the scheduling tasks, sort the set of tasks to be scheduled in this round according to the values ​​of the first and second indicators, and use the sorted set of scheduling tasks as the cluster scheduling task set; Among them, the first indicator Specifically, task S ji The running priority of The expression of the second indicator is specifically: In the formula, jei(S ji ) is task S ji The number of executed iterations, jit(S ji ) is task S ji The single-round iteration running time, jad(S ji ) is task S ji The number of accelerator cards required; S4. Generate a runnable task sequence and a task set that needs to be suspended based on the cluster scheduling task set through the task distribution strategy; S5. Call the task execution module according to the executable task sequence and the task set that needs to be suspended, and update the cluster operation information table and the task operation status information table; S6. Refresh the task status database according to the updated records in the cluster operation information table and the task operation status information table.

2. The historical task data-driven deep learning task scheduling simulation method according to claim 1 is characterized in that: In S1, the method for modifying the cluster resource management table is specifically as follows: Add the IP address, process ID, and number of accelerator cards of the newly added task execution module to the cluster resource management table. Delete the records of deleted or faulty task execution modules from the cluster resource management table.

3. The historical task data-driven deep learning task scheduling simulation method according to claim 1 is characterized in that: In said S4, the task distribution strategy method includes a compact task allocation strategy and a decentralized task allocation strategy; The compact task allocation strategy specifically allocates accelerator card computing resources to the scheduled tasks to be run according to the natural order of the idle accelerator card numbers; The decentralized task allocation strategy is to allocate accelerator card computing resources to the scheduled tasks to be run according to the load balancing principle.

4. The historical task data-driven deep learning task scheduling simulation method according to claim 1 is characterized in that: In S5, the cluster operation information table includes the execution node identifier, IP address, number of accelerator cards, and accelerator card information; Accelerator card information includes accelerator card ID, local accelerator card ID, execution node ID, memory capacity, usage, and running task list; The task running status information table includes the timestamp of this round of scheduling, the list of tasks to be run, the statistics of running task time, the list of completed tasks and the list of suspended tasks.

5. A historical task data-driven deep learning task scheduling simulation system, applied to the historical task data-driven deep learning task scheduling simulation method according to any one of claims 1 to 4, characterized in that: The system includes: The simulation scheduling module is used to generate tasks to be scheduled based on the historical data of deep learning task scheduling, trigger task admission strategy, task scheduling strategy and task distribution strategy, and obtain cluster operation information and task operation status information; The control module is used to manage cluster resource information, including CPU, memory, disk, accelerator card, and network. It is also used to manage the running status information of scheduled tasks, including start, stop, preemption, and waiting status. It is also used to distribute the scheduled tasks to the responsible task execution module. The task execution module is used to trigger the running of tasks and provide management of the accelerator card usage status. It is also used to record the task ID, duration, accelerator card utilization and memory consumption of the current accelerator card task. The task status database is used to store status information of suspended or started tasks when the system is running.

Images