Job scheduling system of distributed training cluster

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a distributed training cluster job scheduling system which comprises a queue management service, a job queue, a job deployment management service, a job analysis service, a job termination management service and a fault-tolerant service. The number of the job queues is multiple. According to the method, a plurality of queues are managed for a large-scale distributed AI training scene, the training jobs are analyzed, corresponding processing is adaptively executed, the training jobs are reasonably scheduled, it is ensured that the training jobs have high availability and reliability, and the training efficiency and the training effect can also be ensured when a large number of training jobs exist.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a job scheduling system of a distributed training cluster. BACKGROUND

[0002] For modern solutions in the autonomous driving industry, an AI (artificial intelligence) job scheduler is critical as it ensures efficient, reliable, and cost-effective execution of highly complex and resource-intensive LLM (large language model) based training jobs using TB (terabyte) level training data on GPU (graphics processing unit) cluster assets across different cloud computing regions and data centers, which requires efficient utilization of resources when executing training jobs.

[0003] In a large-scale distributed artificial intelligence training job scenario, multiple engineers submit required training jobs, and each engineer also submits multiple training jobs per day, such as hundreds of training jobs per day, and each training job involves TB level data. How to reasonably schedule these training jobs to ensure stable operation of each training job is a problem that needs to be solved. SUMMARY

[0004] Therefore, the present application provides a job scheduling system of a distributed training cluster to realize scheduling and stable operation of distributed training jobs.

[0005] In a first aspect, the present application provides a job scheduling system of a distributed training cluster, which comprises a queue management service, a job queue, a job deployment management service, a job analysis service, a job termination management service, and a fault-tolerant service; the number of the job queues is multiple;

[0006] The queue management service is configured to obtain a training job, select a target job queue corresponding to the obtained training job from the multiple job queues, and push the obtained training job to the target job queue;

[0007] The target job queue is configured to insert the training job pushed by the queue management service into a corresponding position of the target job queue, and push the training job in the target job queue to the job deployment management service;

[0008] The job deployment management service is configured to allocate required resources for each training job pushed by the target job queue;

[0009] The job analysis service is configured to analyze each training job according to job running data of the training job, send a first training job requiring termination among the training jobs to the job termination management service, and push a second training job having a fault or failure among the training jobs to the fault tolerance service.

[0010] The job termination management service is configured to terminate the first training job and release resources allocated to the first training job.

[0011] The fault tolerance service is configured to perform fault tolerance processing on the second training job to execute the second training job again in a case where the second training job allows fault tolerance.

[0012] In some optional embodiments, the process in which the queue management service selects a target job queue corresponding to an acquired training job from a plurality of job queues includes:

[0013] For an acquired third training job, determining candidate job queues among the plurality of job queues that match training resource requirements of the third training job;

[0014] Calculating a queue score of each of the candidate job queues, and taking a candidate job queue corresponding to a maximum queue score as a target job queue corresponding to the third training job.

[0015] In some optional embodiments, the process of calculating the queue score of each of the candidate job queues includes:

[0016] Determining a usage rate and a fault rate of the candidate job queue;

[0017] Determining a relative position between the candidate job queue and a target storage node configured to store training data required by the third training job;

[0018] Determining the queue score of the candidate job queue according to the usage rate, the fault rate, and the relative position of the candidate job queue.

[0019] In some optional embodiments, the process in which the target job queue inserts a training job pushed by the queue management service into a corresponding position of the target job queue includes:

[0020] For a fourth training job pushed by the queue management service, calculating a job score of the fourth training job;

[0021] Inserting the fourth training job into a corresponding position of the target job queue according to the job score of the fourth training job; and each training job in the target job queue is sorted according to the size of the job score.

[0022] In some optional embodiments, the calculating the job score of the fourth training job comprises:

[0023] determining a job priority and a model size corresponding to the fourth training job;

[0024] determining a retry number of the fourth training job according to historical training data of the fourth training job;

[0025] determining the job score of the fourth training job according to the job priority, the model size and the retry number of the fourth training job.

[0026] In some optional embodiments, the process of allocating required resources for each training job pushed by the target job queue by the job deployment management service comprises:

[0027] for a fifth training job pushed by the target job queue, judging whether available resources corresponding to the target job queue exceed required resources of the fifth training job;

[0028] in a case where the available resources corresponding to the target job queue exceed the required resources of the fifth training job, allocating required resources for the fifth training job pushed by the target job queue.

[0029] In some optional embodiments, the allocating required resources for the fifth training job pushed by the target job queue in a case where the available resources corresponding to the target job queue exceed the required resources of the fifth training job comprises:

[0030] in a case where the remaining resources corresponding to the target job queue exceed the required resources of the fifth training job, allocating required resources for the fifth training job pushed by the target job queue according to the remaining resources;

[0031] in a case where the remaining resources corresponding to the target job queue do not exceed the required resources of the fifth training job, and the sum of the remaining resources and preemptible resources corresponding to the target job queue exceeds the required resources of the fifth training job, releasing part or all of the preemptible resources, and allocating required resources for the fifth training job pushed by the target job queue according to the remaining resources and the released preemptible resources;

[0032] wherein the remaining resources and the preemptible resources both belong to the available resources.

[0033] In some optional embodiments, the job deployment management service is further configured to:

[0034] in a case where available resources corresponding to the target job queue do not exceed resources required by the fifth training job, determining whether preheating is required;

[0035] in a case where preheating is required, preheating the fifth training job and resubmitting the fifth training job to the queue management service;

[0036] in a case where preheating is not required, resubmitting the fifth training job to the queue management service.

[0037] In some optional embodiments, the job analysis service is further configured to:

[0038] select, as the first training job, a training job that has completed training among the training jobs;

[0039] select, as the first training job, a training job that has a training duration exceeding a preset duration and / or a training quality indicator converging among the training jobs.

[0040] In some optional embodiments, the fault-tolerant service is configured to perform fault-tolerant processing on the second training job, including:

[0041] in a case where the second training job has a hardware error and / or a software error, performing corresponding hardware fault tolerance and / or software fault tolerance on the second training job;

[0042] restarting a latest checkpoint for saving a training state of the second training job, or resubmitting the second training job to the queue management service; the latest checkpoint is a last checkpoint before the error of the second training job.

[0043] In some optional embodiments, the job scheduling system further includes a prediction service;

[0044] the job analysis service is further configured to send a sixth training job to be predicted to the prediction service;

[0045] the prediction service is configured to predict a running state of the sixth training job; the running state includes at least one of resource usage, failure prediction, and hardware performance.

[0046] In some optional embodiments, the running state includes resource usage; and the job deployment management service is configured to allocate required resources for each training job pushed by the target job queue, including:

[0047] determining required reserved resources for each training job according to resource usage of each training job predicted by the prediction service;

[0048] determining whether a difference between available resources corresponding to the target job queue and the reserved resources exceeds resources required by the training job;

[0049] if the difference exceeds the resources required by the training job, allocating the resources required by the training job pushed by the target job queue.

[0050] The present application is based on the job scheduling system of the distributed training cluster. For a large number of submitted training jobs, the queue management service is used to allocate the training jobs to corresponding job queues, to realize the association between the training jobs and the job queues, and each job queue pushes the training jobs to the job deployment management service in sequence, so as to complete the resource allocation and deployment. For the running training jobs, the job analysis service is used to perform real-time analysis based on job running data, and then the job termination management service is used to terminate the training jobs in time, to release the resources, and the fault tolerance service is used to retry the faulty training jobs, to perform scheduling as required, to ensure that each training job has high availability and reliability, and to ensure the training efficiency and training effect when there are a large number of training jobs. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the specific embodiments or related art, the drawings needed to be used in the specific embodiments or related art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0052] Figure 1 is a schematic diagram of a job scheduling system of a distributed training cluster according to an embodiment of the present application;

[0053] Figure 2 is another schematic diagram of a job scheduling system of a distributed training cluster according to an embodiment of the present application;

[0054] Figure 3 is still another schematic diagram of a job scheduling system of a distributed training cluster according to an embodiment of the present application.

[0055] Legend of reference signs:

[0056] 10, queue management service; 20, target job queue; 30, job deployment management service; 40, job analysis service; 50, job termination management service; 60, fault tolerance service; 70, prediction service. DETAILED DESCRIPTION

[0057] To make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0058] In the description of the present application, it should be noted that the terms "first", "second", "third" are only for descriptive purposes and cannot be understood or implied as indicating or implying relative importance. In addition, the technical features involved in the different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0059] Efficient scheduling of training jobs in large-scale distributed training clusters is crucial for efficient resource utilization and ensuring training reliability.

[0060] Specifically, training LLM (Large Language Model) on TB-level data requires a large amount of computing resources, including GPU clusters with thousands of high-end GPU nodes, large fast storage, and high-speed networks, etc. Without an efficient scheduler, resources may be underutilized or overused, leading to bottlenecks and capacity waste.

[0061] Large-scale datasets often require intensive preprocessing, which must be synchronized with training work to avoid delays; TB-level data transfer of clusters or storage systems can be a major bottleneck. Through the scheduler, data and computation can be strategically co-located to minimize these costs.

[0062] In the autonomous driving industry, timely model updates are crucial for early and ongoing success, for example, faster job completion helps shorten the time to market for new vehicle models. An effective scheduler prioritizes related jobs to meet service level agreements.

[0063] Training LLMs consumes a large amount of energy. By optimizing resource utilization, the scheduler helps achieve sustainability and reduce energy consumption, significantly reducing costs. Enterprises and research institutions often run multiple LLM training or fine-tuning jobs simultaneously, which requires balancing workloads across clusters without affecting efficiency.

[0064] Training jobs on TB-level data can take days or weeks, which requires ensuring that the system remains operational even when infrastructure fails (e.g., GPU, node, rack, cluster, or even data center fails), allowing critical training jobs to be completed first.

[0065] Researchers often need to train multiple models or perform hyperparameter tuning. By allocating resources through a scheduler, the speed of experiments can be accelerated; and, in addition to basic model training, fine-tuning for specific tasks can be performed to maximize hardware utilization, leaving more hardware resources for other tasks.

[0066] Companies or research laboratories with efficient training processes can iterate models faster, thus innovating and deploying advanced artificial intelligence applications faster. In scenarios that require real-time model updates (e.g., daily incremental training, partial model retraining), a scheduler can ensure that models are retrained and deployed without delay.

[0067] Some related scheduling systems, such as Kubernetes, can automatically deploy, scale, and manage containerized applications, providing job scheduling capabilities such as job running concurrency and job replacement, and providing security such as multi-tenancy and container-level isolation. However, for large-scale distributed training scenarios, these scheduling systems have very limited scheduling capabilities for training jobs, making it difficult to adapt to complex scheduling problems, such as difficulty in scheduling large-scale AI job workloads and difficulty in achieving job fault tolerance.

[0068] The embodiments of the present application provide a job scheduling system for a distributed training cluster, which manages multiple queues for large-scale distributed AI training scenarios, analyzes training jobs, adaptively performs corresponding processing, schedules each training job, ensures high availability and reliability of each training job, and guarantees training efficiency and training effect when there are a large number of training jobs.

[0069] In the present embodiment, a job scheduling system for a distributed training cluster is provided for scheduling training jobs, Figure 1 The flowchart of the job scheduling system according to the embodiments of the present application is shown in Figure 1 The job scheduling system includes a queue management service 10, job queues, a job deployment management service 30, a job analysis service 40, a job termination management service 50, and a fault tolerance service 60; wherein the number of job queues is multiple.

[0070] The job queue (Job Queue) is used to provide queue services for multiple training jobs, the queue management service 10 is a service for managing each training job, the job deployment management service 30 is a service for managing the deployment of each training job, the job analysis service 40 is a service for analyzing the running of each training job, the job termination management service 50 is a service for terminating training jobs, and the fault tolerance service 60 is a service for fault tolerance processing of training jobs. Each of the above services can be deployed in a corresponding server or server cluster, which is not limited in the present embodiment.

[0071] The queue management service 10 is configured to: obtain a training job, select a target job queue 20 corresponding to the obtained training job from a plurality of job queues, and push the obtained training job to the target job queue 20.

[0072] The target job queue 20 is configured to: insert the training job pushed by the queue management service 10 into a corresponding position of the target job queue 20, and push the training job in the target job queue 20 to the job deployment management service 30. It can be understood that the target job queue 20 is one of the plurality of job queues, and similar processing is performed for other job queues, for example, each job queue is configured to: insert the training job pushed by the queue management service 10 into a corresponding position of the job queue, and push the training job in the job queue to the job deployment management service in sequence. For simplicity of description, the target job queue 20 is mainly taken as an example for description in this embodiment.

[0073] The job deployment management service 30 is configured to: allocate required resources for each training job pushed by the target job queue 20.

[0074] The job analysis service 40 is configured to: analyze the corresponding training job according to the job running data of each training job; send a first training job in which training is required to terminate to the job termination management service 50, and push a second training job that fails or fails to the fault tolerance service 60.

[0075] The job termination management service 50 is configured to: terminate the first training job, and release the resources allocated to the first training job.

[0076] The fault tolerance service 60 is configured to: perform fault tolerance processing on the second training job in the case that the second training job allows fault tolerance, so as to execute the second training job again.

[0077] In this embodiment, the queue management service 10 can obtain the training jobs submitted by each party, and one of the training jobs A is taken as an example for description for convenience. For example, each user (such as an AI engineer) can submit a training job that needs to be trained, and these training jobs are pushed to the queue management service 10; for example, any one of the training jobs can be taken as the training job A. For the training job A, the queue management service 10 selects a job queue most suitable for the training job A from a plurality of job queues, takes the job queue as the target job queue 20, and pushes the training job A to the target job queue 20, so as to be able to schedule the training job A based on the target job queue 20.

[0078] Each job queue is used to record the pushed training jobs on demand. Continuing to take training job A as an example, for the target job queue 20, after the target job queue 20 obtains the training job A pushed by the queue management service 10, the position of the training job A in the queue can be determined, and then the training job A is inserted into the corresponding position of the target job queue 20. Moreover, each training job in the target job queue 20 is arranged in sequence, and each training job in the target job queue 20 is pushed to the job deployment management service 30 in sequence.

[0079] For example, each training job can be inserted into the queue in sequence according to the order in which the training job is obtained; and each training job in the target job queue 20 is arranged in sequence, and each time the first training job is pushed to the job deployment management service 30, after pushing one training job, the second training job in the remaining training jobs becomes the first training job and is then pushed to the job deployment management service 30, and so on; when the training job A is in the first position, the training job A is pushed to the job deployment management service 30.

[0080] For example, when the target job queue 20 receives the training job A, if there is no other training job in the target job queue 20 at this time, for example, the training job A is the first training job received, or the previously pushed training job has been executed, the training job A can be arranged in the first position, i.e., the first position, so as to be directly pushed subsequently. If the target job queue 20 receives the training job A while there is another training job in the target job queue 20, the training job A needs to be inserted into the appropriate position in sequence and pushed in sequence.

[0081] After the job deployment management service 30 receives the training jobs pushed by each job queue, the job deployment management service 30 allocates resources required for executing each training job to each training job; taking the training job A pushed by the target job queue 20 as an example, after the job deployment management service 30 receives the training job A, the job deployment management service 30 allocates resources required by the training job A to the training job A according to the actual situation of the training job A, and the resources may include, for example, various hardware resources required by the AI training job, including CPU, GPU, storage, cache, network, etc.

[0082] The job analysis service 40 can obtain job running data of each training job, and then analyze the corresponding training job according to the job running data of each training job. After the job deployment management service 30 allocates resources to the training job, the training job can be activated and executed; the job running data obtained by the job analysis service 40 is related data of the activated training job, in other words, for a certain training job, only after the resources are allocated and the training job is activated, the job running data of the training job is obtained, and the training job can also be called an activated training job, which is a running training job.

[0083] The job running data is key data representing a training job running condition, which can include logs, indicators, tracking, and training progress, etc. Through analysis of the job running data, the performance of the job scheduling program can be enhanced, and the reliability and user experience can be improved.

[0084] If there is a training job that needs to be terminated in the activated training jobs, for example, a training job that has completed training, such training job is referred to as a first training job, and is sent to the job termination management service 50. Similarly, if there is a training job that has failed or failed in the activated training jobs, such training job is referred to as a second training job, and is sent to the fault tolerance service 60.

[0085] After the job termination management service 50 obtains the first training job pushed by the job analysis service 40, the first training job can be terminated, and the resources allocated to the first training job are released, so that the resources are allocated to other training jobs in the future.

[0086] Similarly, after the fault tolerance service 60 obtains the second training job pushed by the job analysis service 40, if the second training job allows fault tolerance, the second training job is fault-tolerant to execute the second training job again. The fault tolerance service 60 can directly fault-tolerate the second training job pushed by the job analysis service 40, that is, the second training job can be fault-tolerant by default; or the fault tolerance service 60 can further determine whether the second training job can be fault-tolerant, if it can be fault-tolerant, the fault-tolerant processing is performed; if it cannot be fault-tolerant, a corresponding message is returned to the job analysis service 40, or the job termination management service 50 is directly notified to terminate the second training job. Through fault-tolerant processing, the high availability, reliability, and fault recovery capability of the training task can be ensured.

[0087] The job scheduling system of the distributed training cluster provided in the embodiment is used for a large number of submitted training jobs, which are allocated to corresponding job queues based on the queue management service 10, the association between the training jobs and the job queues is realized, each job queue pushes the training job to the job deployment management service 30 in sequence, so that the resource allocation and deployment are completed; for the running training job, the job analysis service 40 performs real-time analysis based on the job running data, and then uses the job termination management service 50 to terminate the training job in time, releases the resources, and uses the fault tolerance service 60 to retry the error training job, so that each training job has high availability and reliability, and the training efficiency and training effect can be ensured when there are a large number of training jobs.

[0088] In the embodiment, another job scheduling system of a distributed training cluster is provided, which is used for scheduling training jobs, Figure 2is a flowchart of a job scheduling system according to an embodiment of the present application, as shown in Figure 2 The job scheduling system comprises a queue management service 10, a plurality of job queues, a job deployment management service 30, a job analysis service 40, a job termination management service 50, a fault-tolerant service 60 and a prediction service 70; wherein the number of job queues is plural.

[0089] The queue management service 10 is mainly used for selecting a suitable job queue for each training job, and details can be referred to the related description of the embodiments shown in Figure 1

[0090] In some optional embodiments, the process of selecting a target job queue 20 corresponding to the acquired training job from the plurality of job queues by the queue management service 10 specifically comprises the following steps A1 to A2.

[0091] Step A1: For the acquired third training job, determine the candidate job queues in the plurality of job queues that match the training resource requirements of the third training job.

[0092] Step A2: Calculate the queue scores of each candidate job queue, and take the candidate job queue corresponding to the maximum queue score as the target job queue 20 corresponding to the third training job.

[0093] Specifically, when each training job is executed, certain training resources are required, which may include, for example, model resources supporting the training job, distributed training framework resources, training pipeline resources, data resources, etc. As described above, a plurality of job queues are provided in advance, and each training job can be marked with supported training resources. The metadata of the training job can record the required training resources, based on which the queue management service 10 can determine which job queues can meet the training resource requirements of the training job, and these job queues are referred to as candidate job queues.

[0094] The distributed training cluster comprises a plurality of data centers, and each data center is deployed with a plurality of GPU nodes. For each job queue, a group of GPU nodes is bound. For example, one or more job queues can be allocated to each team, and in general, each job queue is responsible for a team; according to the quota of the team in the corresponding data center, the job queue can be bound to a group of GPU nodes in a data center. Since the number of job queues is large, if there are permission restrictions on the users allowed to use some job queues, for example, only training jobs submitted by members of the team itself can be pushed to the corresponding job queue, based on the identity information of the owner of the training job, a part of the job queues that are not suitable can also be excluded, and finally the matching candidate job queues are selected. In general, the number of candidate job queues is plural.​

[0095] For the convenience of description, the third training job obtained is taken as an example. It can be understood that any training job obtained by the queue management service 10 can be used as the third training job, or in other words, the third training job is the training job obtained by the queue management service 10. For each candidate job queue, the queue score of the candidate job queue is calculated, and the queue score is used to indicate the degree of matching between the candidate job queue and the third training job; wherein, for a certain candidate job queue, its queue score can be determined according to its actual status, or it can be a score related to the third training job, which is specifically based on the calculation method of the queue score. The higher the queue score of the candidate job queue, the more the candidate job queue matches the third training job, so the candidate job queue corresponding to the maximum queue score is used as the job queue corresponding to the third training job, that is, the target job queue 20.

[0096] In this embodiment, the target job queue is selected from the candidate job queues to ensure that the distributed training framework of the target job queue has strict parallelism constraints (e.g., data parallelism, tensor parallelism, and / or pipeline parallelism). AI training jobs typically involve fine-tuning after basic model training. By allocating priority resources to each task in the job pipeline under the selected framework, the optimal resource queue is selected to achieve mixed workloads.

[0097] Optionally, the above step A2 of “calculating the queue score of each candidate job queue” may include steps A21 to A23.

[0098] Step A21 : Determine the utilization rate and failure rate of the candidate job queue.

[0099] Step A22: Determine the relative position between the candidate job queue and the target storage node; the target storage node is used to store the training data required for the third training job.

[0100] Step A23 : determining the queue score of the candidate job queue according to the utilization rate, failure rate, and relative position of the candidate job queue.

[0101] Specifically, for a candidate job queue, based on the actual usage of the candidate job queue, the usage rate and failure rate of the candidate job queue can be determined. Figure 3 As shown, the job scheduling system also provides a job statistics service, which can count the usage of each job queue, including which resources the job queue uses, as well as failure conditions, etc. Based on the statistical information of the job statistics service, the queue management service 10 can determine the usage rate and failure rate of each job queue (including candidate job queues).

[0102] And, a certain training data is needed to perform the third training job to complete the job of model training. The training data is stored in a certain storage node, i.e., a target storage node; and the candidate job queue generally corresponds to a group of GPU nodes, so the relative position between them can be determined. The closer the relative position, the faster the candidate job queue can obtain the training data, and the more suitable it is as the target job queue.

[0103] After determining the usage rate, failure rate and relative position of the candidate job queue, the queue score of the candidate job queue can be calculated. For example, the larger the usage rate, the smaller the failure rate, and the closer the relative position, the higher the queue score of the corresponding candidate job queue.

[0104] Among them, the usage rate, failure rate and relative position can have different weights. Generally, the usage rate has a higher weight, i.e., the target job queue is selected according to the usage rate first.

[0105] For the third training job, it is first assigned to the candidate job queue with a high usage rate, and if it fails, it will try to use the candidate job queue with a low usage rate, which can ensure that the job queue with a large amount of unused resources can be provided to future large AI training jobs. The candidate job queue with a high failure rate will be set to a lower priority to be assigned to other unimportant training jobs. In addition, the candidate job queue corresponding to the same data center as the training data required by the training job has a higher priority to quickly obtain data, preheat, preprocess and / or process, etc.

[0106] In this embodiment, the target job queue 20 is mainly used for sorting and managing each training job in the queue. For details, please refer to the related description of the embodiment shown in Figure 1 The related description of the embodiment shown in

[0107] Among them, the appropriate job queue is allocated to the corresponding team. The job queue should have the same configuration in hardware and software to avoid unpredictable compatibility problems; the job queue can have scalability to adapt to changes in future team quotas. The job queue should be able to flexibly release its resources to other job queues in the same data center that have more resource needs.

[0108] A team can have multiple job queues in different data centers according to its total quota and the quota of each data center to achieve fault isolation and high availability. And, the time-sharing quota of the job queue with other job queues owned by other teams should be approved by the relevant team and OPS (operation and maintenance system).

[0109] In some optional embodiments, the process of inserting the training job pushed by the queue management service 10 into the corresponding position of the target job queue 20 by the target job queue 20 includes the following steps B1 to B2.

[0110] Step B1, for the fourth training job pushed by the queue management service 10, calculating the job score of the fourth training job.

[0111] Step B2, inserting the fourth training job into the corresponding position of the target job queue 20 according to the job score of the fourth training job; each training job in the target job queue 20 is sorted according to the size of the job score.

[0112] For the convenience of description, the fourth training job pushed by the queue management service 10 is taken as an example for illustration, it can be understood that any training job pushed by the queue management service 10 can be taken as the fourth training job, or in other words, the fourth training job is the training job pushed by the queue management service 10. Specifically, after the job queue receives each training job, sorting is realized based on the job score of each training job. Taking the target job queue 20 as an example, after the target job queue 20 receives the fourth training job, the job score of the fourth training job is calculated, which is used to represent the priority of execution. The larger the job score is, the higher the priority is, and the fourth training job should be executed preferentially, that is, the position of the fourth training job in the target job queue 20 is earlier, and the fourth training job can be pushed to the job deployment management service 30 earlier.

[0113] After the job score of the fourth training job is determined, the job score of the fourth training job can be compared with the job scores of other training jobs already in the target job queue 20, and according to the size relationship between the job scores of each training job, the fourth training job can be inserted into the corresponding position of the target job queue 20, so that each training job in the target job queue 20 is sorted according to the size of the job score.

[0114] For example, each training job in the target job queue 20 is sorted in the order of the job score from large to small. After the job score S0 of the fourth training job is determined, it can be compared with the job score S1 of the first training job in the target job queue 20. If the job score S0 > the job score S1, the fourth training job is set in the first position. If the job score S0 < the job score S1, the comparison with the job score S2 of the first training job in the target job queue 20 is continued, and the same is true for the subsequent training jobs, until the training job with a job score less than the job score S0 of the fourth training job is found, and the fourth training job is inserted. It can be understood that if the job score S0 of the fourth training job is the smallest, the fourth training job is inserted into the tail of the target job queue 20.

[0115] Optionally, the step B1 of calculating the job score of the fourth training job can specifically include steps B11 to B13.

[0116] In step B11, the job priority and the model size corresponding to the fourth training job are determined.

[0117] In step B12, the number of retries of the fourth training job is determined according to the historical training data of the fourth training job.

[0118] In step B13, the job score of the fourth training job is determined according to the job priority, the model size and the number of retries of the fourth training job.

[0119] Specifically, for the fourth training job, the job priority and the model size corresponding to the fourth training job can be determined. The job priority represents the importance of the fourth training job. For example, the job priority of a training job is divided into three levels by default: low priority for experiments, normal priority for model training early work, and high priority for model release deadline or other urgent situations, and can be extended to more priority levels as needed.

[0120] The model size is the size of the model corresponding to the training job. Generally, the job of a larger model has a higher job score than the job of a smaller model with the same job priority, to ensure that the large model is trained first.

[0121] In addition, based on the training data before the fourth training job, i.e. the historical training data, the number of retries of the fourth training job, i.e. the number of times the fourth training job is re-executed, can be determined. For example, as shown in Figure 3 The job statistics service can determine the training data of each training job and provide it to the corresponding job queue, so that the job queue can determine the number of retries of each training job.

[0122] After determining the job priority, the model size and the number of retries of the fourth training job, the job score of the fourth training job can be calculated. For example, the higher the task priority, the larger the model size, and the fewer the number of retries, the higher the job score of the fourth training job.

[0123] The job priority can have a high or even absolute weight, i.e. regardless of the model size and the number of retries, the training job with high job priority is executed first. For multiple training jobs with the same job priority, the model size and the number of retries are sorted to ensure that the training job with high job priority is executed first to meet business needs.

[0124] For training jobs from different teams with different priorities, fairness and priority are balanced by considering the job priority of the training job, retry penalty, and model size to ensure fairness while meeting the training task deadline. Resources are selected for a given training job by calculating a queue score and a job score in real time to determine the most suitable job queue and insertion position and achieve quota sharing between teams.

[0125] The job deployment management service 30 is mainly used to allocate corresponding resources for each training job, complete job deployment, and perform corresponding management. For details, please refer to the related description of the embodiments shown in Figure 1

[0126] In some optional embodiments, the process of the job deployment management service 30 allocating required resources for each training job pushed by the target job queue 20 can include steps C1 to C2.

[0127] Step C1, for the fifth training job pushed by the target job queue 20, it is determined whether the available resources corresponding to the target job queue 20 exceed the resources required by the fifth training job.

[0128] Step C2, in the case that the available resources corresponding to the target job queue 20 exceed the resources required by the fifth training job, the fifth training job pushed by the target job queue 20 is allocated with the required resources.

[0129] Specifically, after the job deployment management service 30 obtains the training jobs pushed by each job queue, it first determines whether the resources can be allocated to the training jobs, that is, whether the job queue has enough resources to allocate to a given training job.

[0130] For the fifth training job pushed by the target job queue 20, the resources required by the fifth training job can be determined, and based on the current available resources corresponding to the target job queue 20, the size relationship between the two resources can be compared to determine whether the available resources corresponding to the target job queue 20 exceed the resources required by the fifth training job. The resources may, for example, include GPU, CPU, memory, storage network, etc. For example, the memory can be the main memory for loading model data, weights, etc.; the storage can be used as the storage space for training data, model weights, checkpoints, etc.; the network specifically includes the bandwidth for transmitting data between GPU, node, data center, and cloud storage, etc.

[0131] ​If the available resources corresponding to the target job queue 20 exceed the resources required by the fifth training job, it indicates that the target job queue 20 currently has sufficient resources allocated to the fifth training job, and thus the required resources can be allocated to the fifth training job at this time, that is, the resources corresponding to the required resources of the fifth training job in the available resources corresponding to the target job queue 20 are allocated to the fifth training job for use when executing the fifth training job.

[0132] The job deployment management service 30 can limit the quota of resources required by each training job and allocate based on actual conditions to ensure efficient and fair use of resources, especially in large-scale training environments shared among teams and team members.

[0133] Optionally, the above step C2 of allocating the required resources to the fifth training job pushed by the target job queue 20 when the available resources corresponding to the target job queue 20 exceed the resources required by the fifth training job includes steps C21 to C22.

[0134] Step C21, when the remaining resources corresponding to the target job queue 20 exceed the resources required by the fifth training job, allocating the required resources to the fifth training job pushed by the target job queue 20 according to the remaining resources.

[0135] Step C22, when the remaining resources corresponding to the target job queue 20 do not exceed the resources required by the fifth training job, and the sum of the remaining resources and the preemptible resources corresponding to the target job queue 20 exceeds the resources required by the fifth training job, releasing part or all of the preemptible resources, and allocating the required resources to the fifth training job pushed by the target job queue 20 according to the remaining resources and the released preemptible resources.

[0136] The remaining resources and the preemptible resources both belong to the available resources.

[0137] Specifically, as described above, each job queue is allocated a certain amount of resources, and when executing the corresponding training job, it will occupy a certain amount of resources, and the remaining unused resources are the remaining resources of the job queue. In addition, in order to ensure that training jobs with high priority can be executed first, resources that can be preempted by other training jobs, i.e., preemptible resources, are pre-allocated.

[0138] For example, the preemptible resources can be part of the resources allocated to other training jobs with lower priority than the fifth training job. For example, other training jobs with lower priority than the fifth training job are allocated 8GB of GPU resources, and if the required base resources (minimum resources) are 6GB, the remaining 2GB of GPU resources can be used as preemptible resources for the fifth training job.

[0139] If the remaining resources corresponding to the target job queue 20 exceed the resources required by the fifth training job, i.e., remaining resources > resources required by the job, the resources of the fifth training job can be allocated without occupying the resources of other training jobs, and thus a part of the remaining resources is used as the resources required by the fifth training job, and the allocation is performed.

[0140] If the remaining resources corresponding to the target job queue 20 do not exceed the resources required by the fifth training job, it is further determined whether the sum of the remaining resources and the preemptible resources corresponding to the target job queue 20 exceeds the resources required by the fifth training job; if (remaining resources + preemptible resources) > resources required by the job > remaining resources, it is indicated that a part of the preemptible resources is occupied, and the resource allocation can also be completed, and thus a part or all of the preemptible resources in the target job queue 20 can be released. The amount of the released preemptible resources is determined based on the size of the resources required by the job and the remaining resources, and only needs to ensure that (remaining resources + released preemptible resources) > resources required by the job.

[0141] After the preemptible resources are released, the fifth training job pushed by the target job queue 20 can be allocated the resources required by the fifth training job based on the remaining resources and the released preemptible resources, so that the resources required by the fifth training job can be normally allocated.

[0142] The resource allocator can be configured to allocate available resources, such as CPU, GPU, memory, storage, and the like, to each training job in an effective manner.

[0143] For long-term or experimental AI jobs, the corresponding preemptible resources are relatively more. When the training tasks and emergency tasks conflict, the low-priority jobs can be preempted to process the emergency tasks, and the risk of resources can be avoided through fault prediction. The prediction function can be implemented based on the subsequent prediction service 70.

[0144] Optionally, the job deployment management service 30 is further configured to perform the following steps C3 to C5.

[0145] In a case where the available resources corresponding to the target job queue 20 do not exceed the resources required by the fifth training job, it is determined whether preheating is needed.

[0146] In a case where preheating is needed, the fifth training job is preheated, and the fifth training job is resubmitted to the queue management service 10.

[0147] In a case where preheating is not needed, the fifth training job is resubmitted to the queue management service 10.

[0148] As Figure 3As shown, when the job deployment management service 30 judges whether the available resources corresponding to the target job queue 20 exceed the resources required by the fifth training job, if not, for example, the remaining resources corresponding to the target job queue 20, and the sum of the remaining resources corresponding to the target job queue 20 and the preemptible resources, all do not exceed the resources required by the fifth training job, it is further judged whether the fifth training job needs to be warmed up, that is, a warm-up check is performed.

[0149] If the fifth training job does not need to be warmed up, the fifth training job is directly resubmitted to the queue management service 10, that is, the training job that has not obtained resources is resubmitted, and the training job is sorted again. Among them, the training job resubmitted due to insufficient resources can be set to a higher priority so that subsequent attempts to execute the training job can be made as soon as possible.

[0150] If the fifth training job needs to be warmed up, in addition to resubmitting the fifth training job to the queue management service 10, the fifth training job is also subjected to warm-up processing.

[0151] Among them, warm-up is an important step in large-scale artificial intelligence training, which can improve the training stability and performance in the initial stage of model training, for example, transferring training data to the target data center to reduce the delay of data acquisition, testing related nodes before deployment, loading training data from storage to main memory, etc.

[0152] Since many AI jobs involve TB-level or even PB-level data sets, such as a large number of high-resolution camera lenses, this makes data transmission costly and time-consuming. Through warm-up processing, data transmission can be arranged as a warm-up task before job execution, thereby asynchronously minimizing data transmission delay, effectively caching and retaining training data and checkpoints, achieving data localization, and reducing preprocessing delay.

[0153] The job analysis service 40 is mainly used for analyzing the running of each training job, so that different training jobs can be executed accordingly. For details, please refer to the related description of the embodiments shown in Figure 1 In addition, as shown, the job scheduling system further includes a prediction service 70; at this time, the job analysis service 40 is further used for: sending a sixth training job to be predicted to the prediction service 70, and the sixth training job will be described later. Figure 2 As shown, the job scheduling system further includes a prediction service 70; at this time, the job analysis service 40 is further used for: sending a sixth training job to be predicted to the prediction service 70, and the sixth training job will be described later.

[0154] In some optional embodiments, the process of determining the first training job by the job analysis service 40 can specifically include: taking a training job that has completed training in each training job as the first training job; taking a training job whose training duration exceeds a preset duration and / or whose training quality indicator converges as the first training job.

[0155] In this embodiment, during the running of each activated training job, the state of the training job is recorded in real time, such as running, failure, completion, etc. If the state of a training job is completion, the training job does not need to be executed again, and therefore can be taken as a first training job to be terminated based on the job termination management service 50, so as to avoid occupying resources.

[0156] Alternatively, if the training duration of a training job exceeds a preset duration, it indicates that the training job occupies resources for a long time, and the training job can be terminated. The preset duration of each training duration can be determined based on actual conditions. In addition, if the training quality index of a training job has converged, for example, the training curve no longer changes, even if the training continues, a higher training result cannot be obtained, and therefore the training job can be terminated to improve resource utilization. The efficient and reliable use of resources can be ensured, and the waste of computing power and proper handling of faults can be avoided.

[0157] The job termination management service 50 is mainly used for performing termination processing on each first training job that needs to be terminated. For details, please refer to the related description of the embodiment shown in Figure 1 , which will not be described here again.

[0158] The job termination management service 50 can directly terminate the first training job pushed by the job analysis service 40. Alternatively, the job termination management service 50 can further judge the first training job pushed by the job analysis service 40, and terminate it if it needs to be terminated. For example, for a first training job whose training duration exceeds a preset duration, an inquiry can be initiated to an operation and maintenance personnel to determine whether the first training job needs to be terminated.

[0159] The fault tolerance service 60 is mainly used for selecting a suitable job queue for each training job. For details, please refer to the related description of the embodiment shown in Figure 1 , which will not be described here again.

[0160] In some optional embodiments, the process of the fault tolerance service 60 performing fault tolerance processing on the second training job includes: performing corresponding hardware fault tolerance and / or software fault tolerance on the second training job in the case that the second training job has a hardware error and / or a software error; restarting the latest checkpoint for saving the training state of the second training job, or resubmitting the second training job to the queue management service 10. The latest checkpoint is the last checkpoint before the error of the second training job occurs.

[0161] In particular, fault tolerance capability in large-scale GPU clusters is critical to ensure high availability, reliability, and fault recovery capability. Training jobs can have a variety of errors, which can be hardware errors or software errors; if these errors are not related to the training job itself, fault tolerance can be performed.

[0162] Among them, the hardware-level fault tolerance can include, for example: for ECC (Error Correcting Code) memory, detect and correct memory errors in the GPU to prevent crashes. Health monitoring and predictive maintenance: use monitoring tools to predict failures and trigger preventive measures to achieve health monitoring and predictive maintenance. Use UPS and redundant cooling systems to prevent failures due to overheating or power outages.

[0163] The software-level fault tolerance can include, for example: containerization is performed, and workloads are run in containers to isolate faults and achieve fast recovery. Automatic scaling and self-repairing are implemented in containers. Use a distributed framework with built-in fault tolerance, for example, use a distributed training framework with retry functionality.

[0164] In this embodiment, the computing state of the training job is saved regularly, and if a failure occurs, the system can restart from the last checkpoint instead of starting from scratch, achieving job rollback. And redundant computing can be run across different GPUs or nodes to mitigate failures; use a distributed file system or erasure coding to store data on multiple nodes to achieve data redundancy and prevent single-point failures.

[0165] In addition, the fault tolerance service 60 can also implement job scheduling and load balancing. For example, if a GPU fails, the scheduler will redistribute tasks to available GPUs. Continuously monitor GPU health and migrate workloads when failures occur. If some GPUs fail, dynamically scale down workloads while maintaining partial functionality.

[0166] In this embodiment, the prediction service 70 in the job scheduling system is configured to: predict a running state of the sixth training job; the running state includes at least one of resource usage, fault prediction, and hardware performance.

[0167] Failure prediction in large-scale GPU clusters is critical to minimize downtime and optimize performance. The prediction service 70 can collect GPU health metrics, such as GPU temperature, power consumption, memory errors, fan speed, processing core and memory utilization, PCIe bandwidth and errors, GPU driver and errors, etc. Based on the GPU health metrics, failure prediction can be made based on thresholds or machine learning models, etc. to predict whether the corresponding training job will fail in the future. And when a failure is predicted, redundant execution or early checkpointing is implemented so that the computation will not start from scratch.

[0168] And the prediction service 70 can predict resource usage to be able to take a proactive approach. For example, when a GPU shows early signs of performance degradation, the workload can be migrated to a healthy node before a failure occurs. When the temperature is overheated or the power supply voltage fluctuates, the fan speed or cooling mechanism can be adjusted by prediction, or the GPU failure related to the power supply is predicted, etc.

[0169] In some optional embodiments, in the case where the running state includes resource usage, the process of the job deployment management service 30 allocating the required resources for each training job pushed to the target job queue 20 can include steps D1 to D3.

[0170] Step D1, according to the resource usage of each training job predicted by the prediction service 70, determine the reserved resources required by each training job.

[0171] Step D2, judge whether the difference between the available resources corresponding to the target job queue 20 and the reserved resources exceeds the resources required by the training job.

[0172] Step D3, if it exceeds, allocate the required resources for the training job pushed to the target job queue 20.

[0173] As shown in Figure 3 The resource usage of each training job predicted by the prediction service 70 can be sent to the job deployment management service 30, so that the job deployment management service 30 can determine the reserved resources required to ensure the running of each training job according to the predicted resource usage; the reserved resources are the resources required for running the training job in the future. For example, if it is determined based on the prediction service 70 that a certain GPU will probably fail in the future, the resources corresponding to the GPU belong to the reserved resources; or a certain training job needs additional resources for training in the future, the additional resources required also belong to a kind of reserved resources.

[0174] When the job deployment management service 30 allocates resources, it is determined whether the difference between the available resources corresponding to the target job queue 20 and the reserved resources exceeds the resources required by the training job. If the available resources include the above-mentioned remaining resources and preemptible resources, it can be determined first whether the difference between the remaining resources corresponding to the target job queue 20 and the reserved resources exceeds the resources required by the training job, that is, whether (remaining resources - reserved resources) > resources required by the job is established. If not, it is further determined whether the difference between the sum of the remaining resources and the preemptible resources corresponding to the target job queue 20 and the reserved resources exceeds the resources required by the training job, that is, whether (remaining resources - reserved resources + preemptible resources) > resources required by the job is established.

[0175] If the difference between the available resources (such as remaining resources and preemptible resources) corresponding to the target job queue 20 and the reserved resources exceeds the resources required by the training job, the required resources can be allocated to the training job pushed by the target job queue 20.

[0176] The job scheduling system of the distributed training cluster provided in the embodiment can determine whether the deployment can be performed according to the current resource usage of each training job and the estimated condition of the prediction service 70, can cope with the change of resources such as CPU, GPU, storage, cache, network, and dynamically adjust resource allocation according to the changing needs of the job (such as job type, data volume, training stage), and ensure that the resources can be reasonably utilized.

[0177] Although the embodiments of the present application are described in conjunction with the drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes should be covered within the protection scope of the present application.

Claims

1. A distributed training cluster job scheduling system, characterized in that: The job scheduling system includes: a queue management service (10), a job queue, a job deployment management service (30), a job analysis service (40), a job termination management service (50) and a fault tolerance service (60); the number of the job queues is multiple; The queue management service (10) is used to: obtain a training job, select a target job queue (20) corresponding to the obtained training job from a plurality of job queues, and push the obtained training job to the target job queue (20); The target job queue (20) is used to: insert the training job pushed by the queue management service (10) into the corresponding position of the target job queue (20), and push the training job in the target job queue (20) to the job deployment management service (30); The job deployment management service (30) is used to: allocate required resources for each training job pushed by the target job queue (20); The job analysis service (40) is used to: analyze the corresponding training jobs according to the job running data of each training job; send the first training job that needs to be terminated among the training jobs to the job termination management service (50); and push the second training job that is faulty or failed among the training jobs to the fault tolerance service (60); The job termination management service (50) is used to: terminate the first training job and release resources allocated to the first training job; The fault-tolerant service (60) is used to perform fault-tolerant processing on the second training job in the case where the second training job allows fault tolerance, so as to execute the second training job again.

2. The job scheduling system according to claim 1, characterized in that: The process of the queue management service (10) selecting a target job queue (20) corresponding to the acquired training job from the plurality of job queues includes: For the acquired third training job, determining a candidate job queue from the plurality of job queues that matches the training resource requirement of the third training job; The queue scores of the candidate job queues are calculated, and the candidate job queue corresponding to the maximum queue score is used as the target job queue corresponding to the third training job (20).

3. The job scheduling system according to claim 2, characterized in that: Calculating the queue score of each candidate job queue includes: determining a utilization rate and a failure rate of the candidate job queue; Determining a relative position between the candidate job queue and the target storage node; the target storage node is used to store the training data required for the third training job; The queue score of the candidate job queue is determined according to the usage rate, failure rate and relative position of the candidate job queue.

4. The job scheduling system according to claim 1, wherein: The process of the target job queue (20) inserting the training job pushed by the queue management service (10) into the corresponding position of the target job queue (20) includes: For the fourth training job pushed by the queue management service (10), calculating a job score of the fourth training job; The fourth training job is inserted into a corresponding position of the target job queue (20) according to the job score of the fourth training job; and the training jobs in the target job queue (20) are sorted according to the size of the job scores.

5. The job scheduling system according to claim 4, characterized in that: Calculating the job score of the fourth training job includes: Determining a job priority and a model scale corresponding to the fourth training job; determining a number of retries for the fourth training operation based on historical training data of the fourth training operation; Determine a job score for the fourth training job according to the job priority, model size, and number of retries of the fourth training job.

6. The job scheduling system according to claim 1, characterized in that: The process of the job deployment management service (30) allocating required resources for each training job pushed by the target job queue (20) includes: For the fifth training job pushed by the target job queue (20), determining whether the available resources corresponding to the target job queue (20) exceed the resources required by the fifth training job; In a case where available resources corresponding to the target job queue (20) exceed resources required for the fifth training job, the required resources are allocated to the fifth training job pushed by the target job queue (20).

7. The job scheduling system according to claim 6, characterized in that: When the available resources corresponding to the target job queue (20) exceed the resources required for the fifth training job, allocating the required resources for the fifth training job pushed by the target job queue (20) includes: In a case where the remaining resources corresponding to the target job queue (20) exceed the resources required for the fifth training job, allocating the required resources to the fifth training job pushed by the target job queue (20) according to the remaining resources; In the case where the remaining resources corresponding to the target job queue (20) do not exceed the resources required for the fifth training job, and the sum of the remaining resources and the preemptible resources corresponding to the target job queue (20) exceeds the resources required for the fifth training job, releasing part or all of the preemptible resources, and allocating the required resources for the fifth training job pushed by the target job queue (20) according to the remaining resources and the released preemptible resources; The remaining resources and the preemptible resources both belong to the available resources.

8. The job scheduling system according to claim 6, characterized in that: The job deployment management service (30) is also used to: If the available resources corresponding to the target job queue (20) do not exceed the resources required for the fifth training job, determining whether preheating is required; If preheating is required, preheating the fifth training job and resubmitting the fifth training job to the queue management service (10); In the case where no warm-up is required, the fifth training job is resubmitted to the queue management service (10).

9. The job scheduling system according to claim 1, wherein: The job analysis service (40) is also used to: The completed training task among the training tasks is used as the first training task; Among the training operations, the training operation whose training duration exceeds the preset duration and / or whose training quality indicator converges is used as the first training operation.

10. The job scheduling system according to claim 1, wherein: The process of the fault-tolerant service (60) performing fault-tolerant processing on the second training job includes: In the case where the second training job has a hardware error and / or a software error, performing corresponding hardware fault tolerance and / or software fault tolerance on the second training job; Restarting the latest checkpoint for saving the training state of the second training job, or resubmitting the second training job to the queue management service (10); the latest checkpoint is the last checkpoint before an error occurs in the second training job.

11. The job scheduling system according to claim 1, wherein: The industry scheduling system further includes: a forecasting service (70); The job analysis service (40) is further configured to: send the sixth training job to be predicted to the prediction service (70); The prediction service (70) is used to predict the running status of the sixth training job; the running status includes at least one of resource usage, fault prediction, and hardware performance.

12. The job scheduling system according to claim 11, characterized in that: The running status includes resource usage; and the process of the job deployment management service (30) allocating required resources to each training job pushed by the target job queue (20) includes: Determining the reserved resources required for each training job based on the resource usage of each training job predicted by the prediction service (70); Determining whether the difference between the available resources corresponding to the target job queue (20) and the reserved resources exceeds the resources required for the training job; If it exceeds, the required resources are allocated to the training job pushed by the target job queue (20).

Images