Tail latency optimization job scheduling method and system based on heterogeneous GPU cluster

Abstract

The present invention discloses a tail delay optimization job scheduling method and system based on a heterogeneous GPU cluster, which relates to the field of artificial intelligence technology, including: a node manager obtains an initial scheduling sequence generated by each scheduler, and the initial scheduling sequence adopts a short job priority strategy to sort tasks to obtain a sequence; for a target scheduler with a heavy tail load, the corresponding initial scheduling sequence is arranged in ascending order to obtain an ordered sequence, and a heavy tail load task set is constructed based on the ordered sequence; all possible job arrangements are generated according to the heavy tail load task set, with the goal of minimizing the job completion time of the scheduler with the largest job completion time, to obtain an optimal job scheduling sequence; jobs are allocated to each scheduler according to the optimal job scheduling sequence for distributed job scheduling; the job scheduling method dynamically selects the optimal scheduling strategy under mixed workloads, ensures strict tail delay service level targets, and improves system resource utilization and versatility.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for tail delay optimization job scheduling based on a heterogeneous GPU cluster. Background Art

[0002] With the rapid development of cloud computing and big data technologies, data centers have become the core infrastructure supporting high-concurrency, real-time applications such as web search, e-commerce, and social networking. These applications place stringent quality of service (QoS) requirements on microsecond tail latency, such as meeting a 99.9% upper bound on service level objectives (SLOs). However, data center resource scheduling faces multiple technical bottlenecks:

[0003] 1. Dynamic imbalance between resource supply and demand: The burstiness and dynamic nature of user requests lead to significant fluctuations in resource demand, resulting in alternating periods of short-term resource overload and idleness.

[0004] 2. Tail effect of distributed architecture: Modern applications are based on a modular microservices architecture. End-to-end requests must be processed collaboratively across multiple distributed servers. The overall response time is limited by the tail latency of the worst-performing node, creating a "barrel effect."

[0005] 3. Challenges of mixed heterogeneous workloads: Light-tailed tasks (such as real-time inference) and heavy-tailed tasks (such as distributed training) coexist in real scenarios, and a single scheduling strategy is difficult to adapt to the mixed workload characteristics.

[0006] Current mainstream scheduling algorithms are based on classical queuing theory, such as Shortest Job First (SJF), First-Come-First-Served (FCFS), and Processor Sharing (PS). Previous studies have shown that SJF minimizes average job completion time (JCT) in light-tail workload scenarios, while PS achieves fairness in heavy-tail workloads through a round-robin mechanism. However, mixed workloads are common in real-world production environments, and a single strategy cannot balance the low latency requirements of light-tail tasks with the resource isolation requirements of heavy-tail tasks, resulting in tail latencies (such as P99 latency) frequently exceeding SLO thresholds. Furthermore, while traditional optimization methods (such as single-queue priority scheduling or preemptive scheduling) can partially improve tail latencies, they sacrifice up to 30% of system throughput. This throughput loss increases exponentially with finer task granularity (such as containerized microservices).

[0007] In addition, the dynamic nature of data center resources further exacerbates scheduling complexity:

[0008] 1. Insufficient resource elasticity: Medium-sized data centers are limited by their physical resource scale. Burst requests can easily lead to cluster-level overload. Prediction models based on historical data (such as ARIMA and LSTM) cannot capture non-periodic demand fluctuations (such as holiday traffic peaks and black swan events), rendering resource reservation strategies ineffective.

[0009] 2. Local load hotspots amplify tail fluctuations: Network partitions, firmware compatibility errors, or uneven task distribution can trigger high-load windows for local resources (such as GPU memory contention and NVLink bandwidth saturation), increasing tail latency fluctuations by 2-5 times.

[0010] 3. The trade-off between service quality and throughput: Existing systems typically employ service degradation strategies (such as limiting resource quotas for low-priority tasks and discarding timed-out requests) to control tail latency. However, this directly leads to a degraded user experience and revenue loss (for example, in e-commerce scenarios, a 1% increase in request latency can lead to a 5% loss of orders).

[0011] Therefore, the existing technology lacks a scheduling mechanism that can dynamically adapt to the characteristics of mixed workloads, take into account both tail latency and throughput optimization, and has the ability to dynamically adjust resources in real time. Summary of the Invention

[0012] Based on the technical problems existing in the background technology, the present invention proposes a tail latency optimization job scheduling method and system based on a heterogeneous GPU cluster, which dynamically selects the optimal scheduling strategy under mixed workloads, thereby improving system resource utilization and versatility while ensuring strict tail latency service level targets.

[0013] The present invention proposes a method for tail latency optimization job scheduling based on a heterogeneous GPU cluster, comprising:

[0014] The node manager obtains the initial scheduling sequence generated by each scheduler, wherein the initial scheduling sequence is a sequence obtained by sorting tasks using the shortest job priority strategy;

[0015] For the target scheduler with heavy tail load, the corresponding initial scheduling sequence is sorted in ascending order to obtain an ordered sequence, and the heavy tail load task set is constructed based on it;

[0016] Generate all possible job arrangements based on the heavy tail load task set, with the goal of minimizing the job completion time of the scheduler with the largest job completion time, and obtain the optimal job scheduling sequence;

[0017] According to the optimal job scheduling sequence, jobs are assigned to various schedulers for distributed job scheduling.

[0018] Furthermore, the heavy tail load task set The construction formula is as follows:

[0019] ;

[0020] in, For the homework, is the target scheduler, For the best The completion time of the job, To get an ordered sequence by arranging all the jobs to be scheduled on the target scheduler in ascending order of completion time, is the total number of jobs to be scheduled on the target scheduler, is the weight parameter, is the upper limit of the number of short jobs filtered out, Indicates a round-down operation. Indicates that the first character is cut off from the ordered sequence The tasks corresponding to the completion time.

[0021] Furthermore, the process of generating the optimal job scheduling sequence is as follows:

[0022] Generate all possible job permutations using a set of heavy tail-load tasks ;

[0023] For each arrangement , get the maximum job completion time based on the job completion time of each scheduler;

[0024] Find the order that minimizes the job completion time of the scheduler that maximizes the job completion time, assign the order as the optimal job scheduling sequence to each scheduler, and return the scheduling results to each scheduler.

[0025] Furthermore, the optimal job scheduling sequence The generation formula is as follows:

[0026] ;

[0027] in, is the job completion time of the scheduler with the largest job completion time, is the sorting method corresponding to the optimal job scheduling sequence, To get the maximum value of the job completion time of each scheduler, is the total number of schedulers in the heterogeneous GPU cluster, The index of the scheduler.

[0028] Furthermore, each computing node deploys an independent node manager, which is responsible for monitoring the local resource status and performing initial task scheduling;

[0029] Multiple schedulers form a scheduler cluster. The scheduler cluster adopts a decentralized design. Each scheduler runs independently and manages the job sequence based on a preset short job priority strategy.

[0030] Tail latency optimization job scheduling system based on heterogeneous GPU clusters, including scheduler, node manager and heterogeneous GPU clusters;

[0031] Multiple schedulers form a distributed scheduler cluster. Each scheduler runs independently and sorts tasks based on the preset short-job priority strategy to obtain the initial scheduling sequence.

[0032] The node manager obtains the initial scheduling sequences generated by each scheduler. For the target scheduler with heavy tail load, the corresponding initial scheduling sequences are sorted in ascending order to obtain an ordered sequence, which is used to construct a heavy tail load task set. Based on the heavy tail load task set, all possible job arrangements are generated, with the goal of minimizing the completion time of the job of the scheduler with the longest job completion time, to obtain the optimal job scheduling sequence.

[0033] According to the optimal job scheduling sequence, jobs are assigned to various schedulers for distributed job scheduling.

[0034] Furthermore, the heavy tail load task set The construction formula is as follows:

[0035] ;

[0036] in, For the homework, is the target scheduler, For the best The completion time of the job, To get an ordered sequence by arranging all the jobs to be scheduled on the target scheduler in ascending order of completion time, is the total number of jobs to be scheduled on the target scheduler, is the weight parameter, is the upper limit of the number of short jobs filtered out, Indicates that the first character is cut off from the ordered sequence The tasks corresponding to the completion time.

[0037] Furthermore, the process of the node manager generating the optimal job scheduling sequence is as follows:

[0038] Generate all possible job permutations using a set of heavy tail-load tasks ;

[0039] For each arrangement , get the maximum job completion time based on the job completion time of each scheduler;

[0040] Find the order that minimizes the job completion time of the scheduler that maximizes the job completion time, assign the order as the optimal job scheduling sequence to each scheduler, and return the scheduling results to each scheduler.

[0041] Furthermore, the optimal job scheduling sequence The generation formula is as follows:

[0042] ;

[0043] in, is the job completion time of the scheduler with the largest job completion time, is the sorting method corresponding to the optimal job scheduling sequence, To get the maximum value of the job completion time of each scheduler, is the total number of schedulers in the heterogeneous GPU cluster, The index of the scheduler.

[0044] Each computing node deploys an independent node manager, which is responsible for monitoring the local resource status and performing initial task scheduling.

[0045] The advantages of the tail latency optimization job scheduling method and system based on heterogeneous GPU clusters provided by the present invention are: a scheduling mechanism that can dynamically adapt to the characteristics of mixed workloads, take into account both tail latency and throughput optimization, and has the ability to dynamically adjust resources in real time; for the problem of local scheduler overload caused by heavy tail loads, the node manager monitors the queue status of each scheduler in real time and implements load rebalancing through dynamic task migration across schedulers. While ensuring local scheduling autonomy, the tail latency bottleneck caused by heavy tail loads is effectively eliminated through lightweight global intervention, ultimately achieving cluster-level load balancing; by introducing a two-level scheduling mechanism: (b1) local autonomous scheduling and global load rebalancing (step 2); (b2) the system only implements cross-node migration (steps 3 and 4) of low-resource-consuming tasks (i.e., small tasks) in the affected scheduler when a JCT anomaly caused by heavy tail loads is detected. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of the process of the present invention;

[0047] Figure 2 Schematic diagram of the tail latency optimization job scheduling solution. (a) is a diagram of a centralized node manager, and (b) is a diagram of a distributed scheduler cluster.

[0048] Figure 3 Schematic diagrams of three types of job scheduling optimization: a) is the traditional short-job-based optimization subgraph, b) is the traditional deadline optimization subgraph, and c) is the tail delay optimization subgraph of this embodiment;

[0049] Figure 4 A scheduling diagram for scheduling a sequence for a job. DETAILED DESCRIPTION

[0050] The technical solutions of the present invention are described in detail below through specific embodiments. Numerous specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0051] like Figures 1 to 4 As shown, the tail delay optimization job scheduling method based on heterogeneous GPU cluster proposed in the present invention includes the following steps:

[0052] Step 1: The node manager obtains the initial scheduling sequence generated by each scheduler, wherein the initial scheduling sequence is a sequence obtained by sorting tasks using the shortest job priority strategy;

[0053] Step 2: For the target scheduler with heavy tail load, sort the corresponding initial scheduling sequence in ascending order to obtain an ordered sequence, and construct the heavy tail load task set based on it;

[0054] Step 3: Generate all possible job arrangements based on the heavy tail load task set, with the goal of minimizing the job completion time of the scheduler with the longest job completion time, and obtain the optimal job scheduling sequence;

[0055] Step 4: Allocate the jobs to each scheduler according to the optimal job scheduling sequence for distributed job scheduling.

[0056] This embodiment optimizes both job completion time and job deadline simultaneously and dynamically selects the optimal scheduling strategy under mixed workloads, aiming to reduce the tail job completion time and average job completion time in a high-utilization data center by reducing the variance of the total waiting time between job tasks.

[0057] The purpose of the tail delay task local optimization strategy designed in this embodiment is to (1) obtain the initial job scheduling sequence of the short job priority strategy that has been proven to be the best under light tail workload. The tail-delayed jobs in the node manager are centrally managed to obtain the optimal job scheduling sequence after the tail-delayed task scheduling is optimized. , For the The assignment in A sequence of schedules on available schedulers, , , and are the total number of jobs and available schedulers, respectively. (2) They are then allocated to various schedulers for distributed job scheduling to optimize the job completion time (JCT) while maintaining the job deadline requirements.

[0058] In one embodiment, step one is specifically:

[0059] The tail delay task scheduling of this embodiment is composed of multi-level collaborative components, including a distributed scheduler, a node manager, and a heterogeneous GPU cluster. Each computing node deploys an independent node manager, which is responsible for coordinating local resources and performing task scheduling. When a job is submitted to the system, the main controller assigns it to a specific scheduler based on the job type and resource requirements. Each scheduler uses the Shortest Job First (SJF) strategy to schedule tasks locally, and obtains the initial job scheduling sequence in the scheduler. ,like Figure 4 As shown in (a), this method maximizes the scheduling efficiency in the light tail load scenario.

[0060] In one embodiment, the node manager further processes the distributed scheduling result (i.e., the initial job scheduling sequence) ) to perform global optimization, that is, step 2 is specifically as follows:

[0061] To address the problem of local scheduler overload caused by heavy tail load (manifested as a significant increase in job completion time or job deadline timeout), the node manager monitors the queue status of each scheduler in real time and identifies small tasks that occupy less resources. , and achieve load rebalancing through dynamic task migration across schedulers, specifically:

[0062] Assume existence Jobs (such as deep learning jobs) constitute a collection , in heterogeneous GPU clusters Available scheduler resources constitute a set , For the homework, For the Available schedulers. For the initial schedule sequence generated in the scheduler , for target schedulers with heavy tail loads , to filter out the heavy tail load task set .

[0063] It should be noted that in this embodiment, the scheduler is the GPU. Scheduler is a function-based name, and GPU is a physical device-based name. Therefore, GPU resources in a heterogeneous GPU cluster refer to scheduler resources.

[0064] Among them, the target scheduler is set Existence The set of jobs to be scheduled is , For the The completion time of the jobs to be scheduled is sorted in ascending order by completion time to obtain an ordered sequence ,in , 、 and They are ranked Location, Location, The time the job was completed for the location.

[0065] Define a set of heavy tail load tasks for:

[0066] ;(1)

[0067] in: Indicates the target scheduler On the homework, For homework The completion time, To get an ordered sequence by arranging all the jobs to be scheduled on the target scheduler in ascending order of completion time, is the total number of jobs to be scheduled on the target scheduler, is the weight parameter, and its default value is set to 50%, that is, , indicating the shortest screening completion time Proportional jobs, which are small tasks that take up less resources, is the upper limit of the number of short jobs filtered out, Indicates that the first character is cut off from the ordered sequence The tasks corresponding to the completion time, Indicates a round-down operation to ensure that the number of selected jobs is an integer.

[0068] To address local scheduler overloads caused by heavy tail loads, the node manager monitors the status of each scheduler's queues in real time and rebalances the load through dynamic task migration across schedulers. While ensuring local scheduling autonomy, lightweight global intervention effectively eliminates tail latency bottlenecks caused by heavy tail loads, ultimately achieving cluster-wide load balancing.

[0069] In one embodiment, the total job completion time of the same batch of job tasks depends on the scheduler with the longest job completion time. Therefore, the optimization goal is to minimize the job completion time of the scheduler with the longest job completion time. That is, step three is specifically as follows:

[0070] (a1) The set of heavy-tail load tasks obtained using formula (1) Generate all possible job permutations ;

[0071] (a2) For each permutation , will Scheduler The required completion time of the jobs in the scheduler is added together to get The completion time of the job , find the maximum job completion time: ;

[0072] (a3) Find the scheduler that maximizes the job completion time and minimizes the job completion time. ,in The sorting method corresponding to the optimal job scheduling sequence;

[0073] (a4) Sort the The optimal job scheduling sequence is assigned to each scheduler, and the scheduling results are returned to each scheduler.

[0074] Based on (a1) to (a4), the job scheduling sequence obtained by the tail delay task optimization scheduling algorithm is shown in formula (2):

[0075] ;

[0076] in, is the job completion time of the scheduler with the largest job completion time, To get the maximum value of the job completion time of each scheduler, is the total number of schedulers in the heterogeneous GPU cluster, The index of the scheduler.

[0077] It should be noted that when calculating the optimal job scheduling sequence, the tasks will be re-planned to schedulers that are not task-intensive. Therefore, after obtaining the optimal job scheduling sequence, the jobs can be directly assigned to various schedulers for distributed job scheduling. The assigned schedulers are idle or task-less schedulers.

[0078] This embodiment focuses on tail-latency tasks generated during distributed scheduling—small tasks that accumulate in local schedulers due to heavy-tail workloads. This optimization objective is achieved by introducing a two-level scheduling mechanism: (b1) local autonomous scheduling and global load rebalancing (step 2); and (b2) the system only migrates low-resource-intensive tasks (small tasks) across nodes in the affected schedulers when a JCT anomaly caused by heavy-tail workloads is detected (steps 3 and 4).

[0079] This embodiment uses a decentralized scheduling method to coordinate the work of multiple schedulers, resulting in a shorter overall completion time, but the task volume remains unchanged, thereby reducing the waiting time of tail tasks and improving the overall efficiency of the job. In addition, this embodiment will evaluate the effectiveness of queue reordering technology (step 2) in optimizing the completion time of tail tasks. This method can ensure that even small tasks can obtain timely resource allocation and reduce waiting time. At the same time, the research will explore how to combine the characteristics of jobs and tasks to optimize scheduling decisions to achieve more accurate and efficient task scheduling. The research will also consider the scalability and adaptability of scheduling strategies to ensure efficient scheduling under different workloads and resource configurations. This is particularly important for processing large-scale, dynamically changing jobs (such as deep learning jobs).

[0080] As an embodiment;

[0081] Table 1 shows an example of a heterogeneous GPU cluster consisting of 4 jobs and 2 different types of GPUs. The execution time and deadline of the jobs (job1, job2, job3, job4) on the scheduler (i.e., GPUs: 1080Ti, V100);

[0082] Table 1

[0083]

[0084] Assume that the deadline of the job is already specified when the job arrives (as shown in Table 1). Figure 3 The vertical dotted lines in a), b), and c) are the deadlines for each assignment.

[0085] like Figure 3 Figure a) shows the scheduling result under the short job priority scheduling algorithm. The short job priority scheduling algorithm is a strategy that prioritizes jobs with short processing time for scheduling to reduce overall waiting time. Because the scheduling strategy of the short job priority scheduling algorithm is only related to the parameter job completion time and completely ignores the job deadline, job1 still times out even though it has notified the scheduler of the deadline. Therefore, from Figure a), it can be seen that the job completion time of the short job priority algorithm may exceed the job timeout.

[0086] Figure 3 Figure b) shows the scheduling result under the deadline priority scheduling algorithm. The deadline priority scheduling algorithm is a method that sorts jobs according to their deadlines and prioritizes jobs with earlier deadlines to ensure task timeliness. It can be seen that since only a single parameter, deadline priority, is considered instead of JCT as the optimization goal, the job completion time of the deadline priority scheduling algorithm is 15, which is 1.5 times that of the tail delay optimization scheduling algorithm proposed in this application. In actual production, this causes energy consumption (locally) or operating cost (on the cloud) problems. From Figure b), it can be seen that the heavy tail load in the deadline priority algorithm leads to low job completion time efficiency.

[0087] Figure 3 c) is the local optimization of tail-delay tasks designed in this embodiment. Since heavy-tail-load tasks are the most important cause of deadline timeouts, this embodiment has two optimization goals: deadline and job completion time. If it is detected that a timeout will occur according to the scheduler's initial scheduling sequence, it will be determined as a heavy-tail task and the scheduling policy will be updated in the node manager, significantly improving the optimization effect.

[0088] As another embodiment;

[0089] The scheduling strategy of this embodiment is based on a multi-level collaborative architecture, combining local autonomous scheduling with a global dynamic load balancing mechanism to optimize the tail latency of jobs (such as deep learning jobs). The specific implementation is as follows:

[0090] (c1) System architecture and component deployment;

[0091] The system consists of a distributed scheduler cluster, node managers, and heterogeneous computing resources (such as GPU clusters). Each compute node deploys an independent node manager, responsible for monitoring local resource status and performing initial task scheduling. The scheduler cluster adopts a decentralized design, with each scheduler instance running independently and managing the job queue based on a short-job-first policy. The master controller assigns newly submitted jobs to specific schedulers based on job type and resource requirements, using a short-job-first policy to ensure initial load distribution.

[0092] (c2) local autonomous scheduling strategy;

[0093] At the local scheduling layer, each scheduler orchestrates jobs using a short-job priority strategy. For light-tail workloads (such as real-time inference), tasks with shorter execution times are prioritized to minimize the average job completion time (JCT). The scheduler maintains local node resource status information (such as GPU memory and bandwidth utilization) in real time and dynamically updates the job sequence. When assigning tasks, the scheduler selects the optimal node for deployment based on real-time resource availability and estimated wait times.

[0094] (c3) Global load balancing mechanism;

[0095] The node manager achieves global load optimization through cross-scheduler collaborative communication. The specific process is as follows:

[0096] Status monitoring: The node manager continuously collects queue status, resource utilization, and task execution progress of each scheduler to identify local resource overloads caused by heavy tail loads (such as significantly increased job completion time or queue backlogs).

[0097] Tail task identification: For target schedulers with heavy tail loads, small tasks with low resource usage (such as short inference requests) are screened and marked as tail latency optimization targets.

[0098] Dynamic task migration: Migrates marked tasks to scheduler nodes with idle resources. A lightweight data synchronization mechanism ensures complete migration of task context. The migration process adheres to the principle of minimal disruption to avoid impacting ongoing heavy-tail tasks (such as distributed training jobs). The minimal disruption principle ensures that ongoing jobs are not affected during the migration process. For example, if the node manager identifies job 2 as requiring scheduling but it is already running, it will not be scheduled and will wait for completion to minimize disruption.

[0099] (c4) Integration with existing scheduling frameworks;

[0100] To achieve compatibility with mainstream container orchestration systems, this embodiment expands the existing scheduling framework:

[0101] Multi-scheduler instantiation: Deploy multiple scheduler instances in a distributed scheduler cluster, with each instance independently managing a subset of node resources. Newly submitted tasks are bound to a specific scheduler based on pre-defined rules (such as job type or resource tags).

[0102] Dynamic updates of resource wait times: The scheduler maintains a global resource view and records the estimated wait time of each node in real time. When a task is completed or migrated, the resource status cache of all schedulers is updated through a collaborative communication protocol to ensure the timeliness of scheduling decisions.

[0103] Elastic Queue Management: Node-local queues use a hybrid priority and timestamp sorting strategy (first prioritizing shortest jobs (priority), then identifying heavy tail loads (jobs with deadlines exceeding), and finally using the node manager for scheduling, ensuring job deadlines (DDLs) based on wait times and completion times). When resources are released, the longest-waiting tasks are prioritized, and queue lengths are dynamically adjusted to prevent resource overload.

[0104] Through the above implementation, the system ensures low latency for light-tail tasks while effectively alleviating the tail latency bottleneck caused by heavy-tail tasks, achieving a dual improvement in cluster-level resource utilization and service quality.

[0105] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A tail latency optimization job scheduling method based on heterogeneous GPU clusters, characterized by: include: The node manager obtains the initial scheduling sequence generated by each scheduler, wherein the initial scheduling sequence is a sequence obtained by sorting tasks using the shortest job priority strategy; For the target scheduler with heavy tail load, the corresponding initial scheduling sequence is sorted in ascending order of job completion time to obtain an ordered sequence, and the heavy tail load task set is constructed based on it; Generate all possible job permutations using a set of heavy tail-load tasks ; For each permutation , based on the job completion time of each scheduler, the maximum job completion time is obtained; find the order corresponding to the minimum job completion time of the scheduler that makes the job completion time the largest, and use this order as the optimal job scheduling sequence; Allocate jobs to various schedulers for distributed job scheduling according to the optimal job scheduling sequence; The heavy tail load task set The construction formula is as follows: ； in, For the homework, is the target scheduler, For the best The completion time of the job, To get an ordered sequence by arranging all the jobs to be scheduled on the target scheduler in ascending order of completion time, is the total number of jobs to be scheduled on the target scheduler, is the weight parameter, is the upper limit of the number of short jobs filtered out, Indicates a round-down operation. Indicates that the first character is cut off from the ordered sequence The tasks corresponding to the completion time.

2. The method for tail delay optimization job scheduling based on heterogeneous GPU cluster according to claim 1, characterized in that: Optimal job scheduling sequence The generation formula is as follows: ； in, is the job completion time of the scheduler with the largest job completion time, is the sorting method corresponding to the optimal job scheduling sequence, To get the maximum value of the job completion time of each scheduler, is the total number of schedulers in the heterogeneous GPU cluster, The index of the scheduler.

3. The method for tail delay optimization job scheduling based on heterogeneous GPU cluster according to claim 1, characterized in that: Each computing node deploys an independent node manager, which is responsible for monitoring local resource status and performing initial task scheduling; Multiple schedulers form a distributed scheduler cluster. The distributed scheduler cluster adopts a decentralized design. Each scheduler runs independently and manages the job sequence based on a preset short job priority strategy.

4. Tail latency optimization job scheduling system based on heterogeneous GPU cluster, characterized by: Including scheduler, node manager and heterogeneous GPU cluster; Multiple schedulers form a distributed scheduler cluster. Each scheduler runs independently and sorts tasks based on the preset short-job priority strategy to obtain the initial scheduling sequence. The node manager obtains the initial scheduling sequence generated by each scheduler. For the target scheduler with heavy tail load, the corresponding initial scheduling sequence is sorted in ascending order of job completion time to obtain an ordered sequence, and the heavy tail load task set is constructed based on it; the heavy tail load task set is used to generate all possible job arrangements. ; For each permutation , based on the job completion time of each scheduler, the maximum job completion time is obtained; find the order corresponding to the minimum job completion time of the scheduler that makes the job completion time the largest, and use this order as the optimal job scheduling sequence; Allocate jobs to various schedulers for distributed job scheduling according to the optimal job scheduling sequence; The heavy tail load task set The construction formula is as follows: ； in, For the homework, is the target scheduler, For the best The completion time of the job, To get an ordered sequence by arranging all the jobs to be scheduled on the target scheduler in ascending order of completion time, is the total number of jobs to be scheduled on the target scheduler, is the weight parameter, is the upper limit of the number of short jobs filtered out, Indicates that the first character is cut off from the ordered sequence The tasks corresponding to the completion time.

5. The tail delay optimization job scheduling system based on heterogeneous GPU cluster according to claim 4 is characterized in that: Optimal job scheduling sequence The generation formula is as follows: ； in, is the job completion time of the scheduler with the largest job completion time, is the sorting method corresponding to the optimal job scheduling sequence, To get the maximum value of the job completion time of each scheduler, is the total number of schedulers in the heterogeneous GPU cluster, The index of the scheduler.

6. The tail delay optimization job scheduling system based on heterogeneous GPU cluster according to claim 4 is characterized in that: Each computing node deploys an independent node manager, which is responsible for monitoring the local resource status and performing initial task scheduling.

Images