A Hybrid Load Priority Distributed Scheduling Method Based on Global Time Wall

Abstract

This invention discloses a hybrid load priority distributed scheduling method based on a global time wall. It achieves hybrid load priority distributed scheduling by setting short and long job time walls to record the time slices required for all job executions; setting short and long job queues to record all computational tasks; the job task manager receiving and parsing tasks; and the task scheduler prioritizing short job scheduling and long job scheduling strategies. This invention subdivides hybrid load scenarios and designs priority scheduling strategies for each, minimizing short job timeouts and blocking / waiting of low-priority long jobs.

Description

Technical Field

[0001] This invention belongs to the field of parallel computing, specifically relating to a hybrid load priority distributed scheduling method based on a global time wall. Background Technology

[0002] In the field of modern computing services, there is an increasing need to consider task scheduling requirements under mixed load conditions, namely the mixed scheduling of short and long tasks, in order to improve the overall utilization of computing cluster resources. Task scheduling is the process of mapping task allocation to computing resources. Traditional centralized scheduling schemes are mainly designed for task allocation in small clusters. When the number of tasks increases, centralized task scheduling has obvious bottlenecks, while distributed scheduling extends this process to the level of thousands of nodes in large-scale distributed systems.

[0003] Currently, most distributed schedulers rarely consider the scheduling of computing resources under mixed task load conditions. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and to represent the distributed scheduling of tasks for hybrid loads in a heterogeneous environment by using a time-dimensional constraint description method, which improves the quality of computing services, task execution efficiency and overall system performance.

[0005] To achieve the above-mentioned objectives, this invention provides a hybrid load priority distributed scheduling method based on a global time wall, comprising the following steps: (1) setting up a short job task time wall and a long job task time wall to record the time slices required for the execution of all job tasks; (2) setting up a short job task queue and a long job task queue to record all computation tasks; (3) the job task manager receives and parses the tasks, adds the new tasks to the corresponding task queues, and adds the time slices required for task execution to the corresponding time wall; (4) the task scheduler prioritizes the short job task priority scheduling strategy, selects computation tasks from the short job queues for execution, and deletes the corresponding time slices from the time wall after execution. When the time slices in the short job time wall are cleared, the job scheduler obtains the sum of the remaining service time slices in the long job task time wall. When this value meets the preset conditions, the task scheduler selects a task from the long job task queue for execution. (5) When the long job task is scheduled for execution, when the sum of the remaining service time slices of the tasks in the long job task time wall is less than the system preset lower limit, the task scheduler selects a task from the long job task queue for execution. When the sum of the remaining service time slices of the tasks in the long job task time wall is greater than the system preset upper limit, the task scheduler waits until the sum of the remaining service time slices of the tasks in the long job task time wall is less than the system preset lower limit, and then selects a task from the job queue for execution again.

[0006] Furthermore, the time wall refers to an ordered queue of task time slices; the task time slice refers to an overestimated timeout required for task execution, which should be significantly greater than the actual execution time required by the task.

[0007] Further, the short job priority scheduling strategy in step (4) includes the following steps: (41) When the short job task queue is not empty, the task is ranked according to the remaining service time of the task time slice in the short job task time wall, and the task with the smallest remaining service time score is selected, that is, the task with the shortest remaining service time; (42) The task with the shortest remaining service time is sent to the resource scheduler, the resource scheduler matches the task with the optimal executor that meets the constraints and returns to the task manager; (43) When the task manager sends the task to the executor for execution, the short job task time wall intermittent polling waiting time Ts is calculated and updated according to the estimated execution timeout of the current task; (44) The task manager sends the task to the executor for execution, and the time slice of the current task in the time wall is cleared after the task is executed.

[0008] Furthermore, the calculation process of the short job task time wall intermittent polling waiting time Ts using the PI control algorithm is as follows:

[0009] Ts = Sp e(t) + Si∫e(t)dt; where Sp and Si are parameter values, and e(t) is the estimated execution time required for the short job task; then, the short job task time wall is polled intermittently based on the waiting time Ts to select the task scheduler.

[0010] Further, the long job scheduling step in step (5) includes the following steps: (51) When the long job scheduling is executed, calculate the sum of all remaining service time slice values ​​Sn in the short job time wall. If Sn is less than or equal to 0, it means that there are no short job tasks to be executed, and a long job task is selected for scheduling and execution; otherwise, wait for the short job task to be executed; (52) When Sn is less than or equal to 0, that is, when the short job time slices in the job task time wall are cleared, calculate the remaining service time slice value Sm of the long job task within the time slice starting range of the long job time wall according to the system preset; (53) Compare Sm with the system preset value. When Sm is less than or equal to the lower limit of the system preset value, select the long job task to be executed; when Sm is greater than the upper limit of the system preset value, wait for it to be executed. After the long job task is completed, another long job task is selected for dequeue execution; (54) When Sm is less than or equal to the lower limit of the system preset value, the task time slice in the long job task time wall is scored and ranked according to the remaining service time of the task time slice, and the task with the smallest remaining service time score is selected, that is, the task with the shortest remaining service time; (55) The task with the shortest remaining service time is selected and sent to the resource scheduler. The resource scheduler matches the task with the executor that meets the constraints and is optimal and returns to the task manager; (56) When the task manager sends the task to the executor for execution, the short job task time wall intermittent polling waiting time Tl is calculated based on the estimated execution timeout of the current task and updated; (57) The task manager sends the task to the executor for execution. After the task is executed, the current task time slice in the long job task time wall is cleared.

[0011] Further, the calculation process of Sn is as follows: Sn = sum(zrange(e(t),n,m)-kT / 1000); where e(t) represents the estimated execution time required for the short job task, T represents the current system timestamp, n and m represent the coordinate values ​​of the starting range of the short job task time wall queue, and k represents the number of time slices in the starting range between n and m; here n = 0, m = -1, and the value of k is equal to the total number of time slices in the short job task time wall. The sum of the remaining service time values ​​of all short job tasks in the short job task time wall is calculated by the formula.

[0012] Furthermore, when Sn is less than or equal to 0, it means that the time slices of short tasks in the short task time wall have been cleared, and long tasks are selected for scheduling and execution.

[0013] Furthermore, the calculation process of Sm is as follows: Sm = sum(zrange(e(t),n,m)-kT / 1000); where e(t) represents the estimated execution time required for long jobs, T represents the current system timestamp, n and m represent the coordinates of the starting range of the long job time wall queue, and k represents the number of long job time slices in the starting range between n and m. The sum of the remaining service time slices of long jobs in the preset starting range of the system, Sm, is obtained by calculating the formula.

[0014] Furthermore, when Sm is greater than the upper limit of the system preset value, polling and waiting is performed; when Sm is less than the lower limit of the system preset value, a task is selected from the long job task queue for scheduling and execution.

[0015] Furthermore, the calculation process of Tl using the PI control algorithm is as follows: Tl = Lp e(t) + Li∫e(t)dt; where Lp and Li are parameter values, and e(t) is the estimated execution time required for the short task; then according to T l Use polling to select task scheduling from the time wall of long-running tasks.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0017] 1. The proposed scheduling strategy under mixed load can combine resource monitoring and feedback mechanisms to ensure the final completion of tasks in the scheduling process of computing tasks in complex environments, thereby improving resource utilization and meeting the optimal system throughput performance index.

[0018] 2. This scheduling algorithm allocates tasks to the corresponding executor resources, solving the performance bottleneck of cluster computing resources and improving the quality of computing services.

[0019] 3. The mixed load situation is subdivided and priority scheduling strategies are designed for each, which minimizes the timeout of short jobs and the blocking and waiting of low-priority long jobs. Attached Figure Description

[0020] Figure 1 This is an architecture diagram / flowchart of one embodiment of the present invention.

[0021] Figure 2 This is a flowchart of a short job priority scheduling strategy in one embodiment of the present invention.

[0022] Figure 3 This is a flowchart of a long-job task priority scheduling strategy in one embodiment of the present invention. Detailed Implementation

[0023] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0024] like Figures 1-3 As shown, an embodiment of the hybrid load priority distributed scheduling system based on a global time wall of the present invention includes the following steps: (1) setting up a short job task time wall and a long job task time wall to record the time slices required for the execution of all job tasks; (2) setting up a short job task queue and a long job task queue to record all computation tasks; (3) the job task manager receives and parses the tasks, adds the new tasks to the corresponding task queues, and adds the time slices required for the execution of the tasks to the corresponding time wall; (4) the task scheduler prioritizes the short job task priority scheduling strategy, selects computation tasks from the short job queues for execution, and deletes the corresponding time slices from the time wall after execution. When the time slices in the short job time wall are cleared, the job scheduler obtains the sum of the remaining service time slices in the long job task time wall. When this value meets the preset conditions, the task scheduler selects a task from the long job task queue for execution. (5) When the long job task is scheduled for execution, when the sum of the remaining service time slices of the tasks in the long job task time wall is less than the system preset lower limit, the task scheduler selects a task from the long job task queue for execution. When the sum of the remaining service time slices of the tasks in the long job task time wall is greater than the system preset upper limit, the task scheduler waits until the sum of the remaining service time slices of the tasks in the long job task time wall is less than the system preset lower limit, and then selects a task from the job queue for execution again.

[0025] In one embodiment, the time wall refers to an ordered queue of task time slices; the task time slice refers to an overestimated timeout required for task execution, which should be significantly greater than the actual execution time required for the task.

[0026] In one embodiment, the short job priority scheduling strategy in step (4) includes the following steps: (41) When the short job task queue is not empty, the task is ranked according to the remaining service time of the task time slice in the short job task time wall, and the task with the smallest remaining service time score is selected, that is, the task with the shortest remaining service time; (42) The task with the shortest remaining service time is sent to the resource scheduler, the resource scheduler matches the task with the optimal executor that meets the constraints and returns to the task manager; (43) When the task manager sends the task to the executor for execution, the short job task time wall intermittent polling waiting time Ts is calculated based on the estimated execution timeout of the current task and updated; (44) The task manager sends the task to the executor for execution, and the time slice of the current task in the time wall is cleared after the task is executed.

[0027] In one embodiment, the calculation process of the short job task time wall intermittent polling waiting time Ts using the PI control algorithm is as follows:

[0028] Ts = Sp e(t) + Si∫e(t)dt; where Sp and Si are parameter values, and e(t) is the estimated execution time required for the short job task; then, the short job task time wall is polled intermittently based on the waiting time Ts to select the task scheduler.

[0029] In one embodiment, the long job scheduling step in step (5) includes the following steps: (51) When the long job scheduling is executed, calculate the sum of all remaining service time slice values ​​Sn in the short job time wall. If Sn is less than or equal to 0, it means that there are no short job tasks to be executed, and a long job task is selected for scheduling and execution; otherwise, wait for the short job task to be executed; (52) When Sn is less than or equal to 0, that is, when the short job time slices in the job task time wall are cleared, calculate the remaining service time slice value Sm of the long job task within the time slice starting range of the long job time wall according to the system preset starting range of the time slice; (53) Compare Sm with the system preset value. When Sm is less than or equal to the lower limit of the system preset value, select the long job task for execution; when Sm is greater than the upper limit of the system preset value, wait. (54) When Sm is less than or equal to the lower limit of the system preset value, the task time slice in the long task time wall is scored and ranked according to the remaining service time of the task time slice. The task with the smallest remaining service time score is selected, that is, the task with the shortest remaining service time. (55) The task with the shortest remaining service time is sent to the resource scheduler. The resource scheduler matches the task with the executor that meets the constraints and is optimal and returns to the task manager. (56) When the task manager sends the task to the executor for execution, the short task time wall intermittent polling waiting time Tl is calculated based on the estimated execution timeout of the current task and updated. (57) The task manager sends the task to the executor for execution. After the task is executed, the current task time slice in the long task time wall is cleared.

[0030] In one embodiment, the calculation process of Sn is as follows: Sn = sum(zrange(e(t),n,m)-kT / 1000); where e(t) represents the estimated execution time required for the short job task, T represents the current system timestamp, n and m represent the coordinates of the starting range of the short job task time wall queue, and k represents the number of time slices in the starting range between n and m; here n = 0, m = -1, and the value of k is equal to the total number of time slices in the short job task time wall. The sum of the remaining service time values ​​of all short job tasks in the short job task time wall is calculated by the formula.

[0031] In one embodiment, when Sn is less than or equal to 0, it means that the time slices of short jobs in the short job time wall have been cleared, and long jobs are selected for scheduling and execution.

[0032] In one embodiment, the calculation process of Sm is as follows: Sm = sum(zrange(e(t),n,m)-kT / 1000); where e(t) represents the estimated execution time required for a long job, T represents the current system timestamp, n and m represent the coordinates of the starting range of the long job timewall queue, and k represents the number of long job time slices in the starting range between n and m. The sum of the remaining service time slices of long jobs in the system's preset starting range is calculated using the formula.

[0033] In one embodiment, when Sm is greater than the upper limit of the system preset value, polling and waiting is performed; when Sm is less than the lower limit of the system preset value, a task is selected from the long job task queue for scheduling and execution.

[0034] In one embodiment, the calculation process of Tl using the PI control algorithm is as follows: Tl = Lp e(t) + Li∫e(t)dt; where Lp and Li are parameter values, and e(t) is the estimated execution time required for the short job task; then according to T l Use polling to select task scheduling from the time wall of long-running tasks.

[0035] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A hybrid load priority distributed scheduling method based on a global time wall, characterized in that, Includes the following steps: (1) Set up short task timewalls and long task timewalls to record the time slices required for the execution of all tasks; (2) Set up short job task queues and long job task queues to record all computation tasks; (3) The job task manager receives and parses the task, adds the new task to the corresponding task queue, and adds the time slice required for task execution to the corresponding time wall; (4) The task scheduler prioritizes the short job task priority scheduling strategy, selects computing tasks from the short job queue for execution, and deletes the corresponding time slices from the time wall after execution. When the time slices in the short job time wall are cleared, the job scheduler obtains the sum of the remaining service time slice values ​​of the long job task time wall. When this value meets the preset conditions, it selects tasks from the long job task queue for execution. (5) When scheduling and executing long-running tasks, if the sum of the remaining service time slices of tasks within the long-running task time frame is less than the system's preset lower limit, the task scheduler selects a task from the long-running task queue for execution; if the sum of the remaining service time slices of tasks within the long-running task time frame is greater than the system's preset upper limit, the task scheduler waits until the sum of the remaining service time slices of tasks within the long-running task time frame is less than the system's preset lower limit, at which point it reselects a task from the task queue for execution. The long-running task scheduling steps include the following steps: (51) When scheduling and executing a long job task, calculate the sum of all remaining service time slices Sn in the short job task time wall. If Sn is less than or equal to 0, it means that there are no short job tasks to be executed, and a long job task is selected for scheduling and execution; otherwise, wait for the short job task to be executed. (52) When Sn is less than or equal to 0, that is, when the short task time slice in the task time wall is cleared, calculate the remaining service time slice value and Sm of the long task within the preset starting range of the time slice in the long task time wall. (53) Compare Sm with the system preset value. When Sm is less than or equal to the lower limit of the system preset value, select the long job task to execute. When Sm is greater than the upper limit of the system preset value, wait for other long job tasks to be completed before selecting the long job task to be dequeued for execution. (54) When Sm is less than or equal to the lower limit of the system preset value, the task is ranked and scored according to the remaining service time of the task time slice in the long task time wall, and the task with the smallest remaining service time score is selected, that is, the task with the shortest remaining service time. (55) Select the task with the shortest remaining service time and send it to the resource scheduler. The resource scheduler will match the task with the optimal executor that meets the constraints and return to the task manager. (56) When the Task Manager sends a task to the executor for execution, it calculates the short job task timewall intermittent polling waiting time Tl based on the estimated execution timeout of the current task and updates it; (57) The Task Manager sends the task to the executor for execution. After the task is completed, the current task time slice in the long job time wall is cleared. The calculation process for Sn is as follows: Sn=sum(zrange(e(t),n,m)-kT / 1000); Where e(t) represents the estimated execution time required for a short job task, T represents the current system timestamp, n and m represent the coordinates of the starting range of the short job task timewall queue, and k represents the number of time slices in the starting range between n and m; here n = 0, m = -1, and the value of k is equal to the total number of time slices within the short job task timewall. The sum of the remaining service time values ​​Sn of all short jobs within the short job task timewall is calculated by the formula.

2. The hybrid load priority distributed scheduling method based on a global time wall according to claim 1, characterized in that, The time wall refers to an ordered queue of task time slices; the task time slice refers to an overestimated timeout required for task execution, which should be significantly greater than the actual execution time required for the task.

3. The hybrid load priority distributed scheduling method based on a global time wall according to claim 1 or 2, characterized in that, The shortest job priority scheduling strategy described in step (4) includes the following steps: (41) When the short job task queue is not empty, the task is ranked and scored according to the remaining service time of the task time slice in the short job task time wall, and the task with the smallest remaining service time score is selected, that is, the task with the shortest remaining service time. (42) Select the task with the shortest remaining service time and send it to the resource scheduler. The resource scheduler will match the task with the optimal executor that meets the constraints and return to the task manager. (43) When the task manager sends a task to the executor for execution, it calculates and updates the short job task time wall intermittent polling waiting time Ts based on the estimated execution timeout of the current task. (44) The Task Manager sends a task to the executor for execution. After the task is completed, the time slice of the current task in the time wall is cleared.

4. The hybrid load priority distributed scheduling method based on a global time wall according to claim 3, characterized in that, The calculation process of the intermittent polling waiting time Ts for the short job task time wall using the PI control algorithm is as follows: Ts=Sp e(t)+Si∫e(t)dt; Where Sp and Si are parameter values, and e(t) is the estimated execution time of the short job task; Then, based on the intermittent polling waiting time Ts of the short task time wall, the task scheduling is selected by polling the short task time wall.

5. The hybrid load priority distributed scheduling method based on a global time wall according to claim 1, characterized in that, When Sn is less than or equal to 0, it means that the time slices of short jobs in the short job time wall have been cleared, and long jobs are selected for scheduling and execution.

6. The hybrid load priority distributed scheduling method based on a global time wall according to claim 1, characterized in that, The calculation process for Sm is as follows: Sm=sum(zrange(e(t),n,m)-kT / 1000); Where e(t) represents the estimated execution time required for a long job, T represents the current system timestamp, n and m represent the coordinates of the starting range of the long job timewall queue, and k represents the number of long job time slices in the starting range between n and m. The total remaining service time slices Sm of long jobs in the system's preset starting range are calculated using the formula.

7. The hybrid load priority distributed scheduling method based on a global time wall according to claim 6, characterized in that, When Sm is greater than the upper limit of the system preset value, poll and wait; when Sm is less than the lower limit of the system preset value, select a task from the long job task queue for scheduling and execution.

8. The hybrid load priority distributed scheduling method based on a global time wall according to claim 1, characterized in that, The calculation process of Tl using the PI control algorithm is as follows: Tl=Lp e(t)+Li∫e(t)dt; Where Lp and Li are parameter values, and e(t) is the estimated execution time of the short job task; Then according to T l Use polling to select task scheduling from the long-running task timeline.

Images