Scheduling method of artificial intelligence model training task based on container composer

Abstract

The invention relates to the technical field of artificial intelligence model training task scheduling, in particular to an artificial intelligence model training task scheduling method based on a container composer. The method comprises: receiving a job request; the container group configuration information is analyzed, and whether cluster resources meet the requirement of the minimum member number or not is checked; if yes, all tasks of the job are scheduled, and the job state is updated to be in operation; if not, further judging whether the cluster resource at least meets the minimum number of successful members; when the elastic strategy is started and the cluster resource meets the minimum number of successful members, scheduling a task corresponding to the minimum number of successful members, and setting a job state to be in partial operation; when the operation state is the partial operation state, the change condition of cluster resources is continuously monitored, and when enough available resources exist, the remaining tasks are automatically scheduled. The waiting time of operation is shortened, and the overall utilization rate of cluster resources and the flexibility of task scheduling are greatly improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence model training task scheduling technology, and in particular to a scheduling method for artificial intelligence model training tasks based on a container orchestrator. Background Technology

[0002] With the rapid development of artificial intelligence technology, the demand for training deep learning models is increasing daily. AI model training typically requires substantial computing resources, especially high-performance hardware such as GPUs. Traditional methods suffer from the following problems: In existing technologies, to ensure that all task processes in a distributed job can start simultaneously and work collaboratively, and to avoid deadlocks or performance degradation caused by insufficient resources for some tasks, the Gang Scheduling scheduling strategy is commonly used. This strategy follows the all-or-nothing principle, meaning that all computational tasks requested by a job must meet all resource requirements and be successfully scheduled before the job can actually begin execution. If any task cannot be scheduled due to insufficient resources, all tasks in the entire job will remain in a waiting state.

[0003] While this strict scheduling mechanism ensures consistency in job execution, it reveals significant limitations in real-world large-scale cluster environments, especially under conditions of high cluster load, intense resource contention, or resource fragmentation. Many jobs, particularly large training tasks requiring numerous computing units, may remain in a waiting state for extended periods because they cannot acquire all necessary resources at once. This not only increases the average job wait time and reduces the user's computational efficiency experience but also prevents the effective utilization of fragmented, non-contiguous available resources within the cluster, resulting in low overall resource utilization. Summary of the Invention

[0004] To address the above problems, this application provides a scheduling method for training AI models based on container orchestrators, the method comprising: Receive a job request, the job request containing container group configuration information, wherein the container group configuration information defines the minimum number of members and the elasticity policy, and the elasticity policy specifies the minimum number of successful members; Parse the container group configuration information and check whether the cluster resources meet the minimum member requirement; When the cluster resources meet the minimum number of members, schedule all tasks of the job and update the job status to running. When the cluster resources do not meet the minimum number of members, it is further determined whether the elasticity policy is enabled and whether the cluster resources at least meet the minimum number of successful members. When the elastic policy is enabled and the cluster resources meet the minimum number of successful members, the task corresponding to the minimum number of successful members is scheduled, and the job status is set to partially running. During the period when the job is in a partially running state, the cluster resources are continuously monitored for changes. When sufficient available resources are detected, the remaining tasks are automatically scheduled to complete the job execution.

[0005] Specifically, the container orchestrator is a Kubernetes cluster, and the scheduling method is implemented by integrating the Volcano scheduler. The container group configuration information corresponds to the definition of the container group custom resource in the Volcano scheduler. The container group custom resource defines the minimum number of members and the elasticity policy. The elasticity policy includes an enabled status field and a minimum number of successful members field, which are used to allow some tasks to start when resources are insufficient.

[0006] Specifically, the method further includes receiving a training task creation request submitted by the user through an application programming interface before receiving the job request. The training task creation request includes parameters such as region, task name, data path, code path, output path, container image, product specifications, and number of nodes. It also verifies the user's access permissions to the storage container and the validity of the product specifications, obtains the corresponding CPU, memory, and GPU resource quota information from the product configuration library, deducts inventory and creates an order record when resources are sufficient, creates a training task record in the database and sets the initial state to waiting, and finally triggers the resource scheduling process.

[0007] Specifically, when parsing the container group configuration information and checking whether the cluster resources meet the minimum number of members requirement, GPU resources are scheduled, and GPU resource requests are declared in the job configuration, enabling the GPU topology-aware scheduling strategy.

[0008] Specifically, the Kubernetes Informer listener continuously monitors changes in cluster resources. The Kubernetes Informer listener continuously listens for state change events of job resources in the cluster and triggers an event handler when the state changes. It automatically updates the state field of the training task in the database and then pushes a state change notification to the user via WebSocket or Webhook. At the same time, it records state change logs for auditing and troubleshooting.

[0009] Specifically, the method further includes automatically creating and managing persistent volume declarations and persistent volumes before scheduling tasks. After training the data storage path required by the task declaration, persistent volume declarations are automatically created, corresponding persistent volumes are created according to the persistent volume declarations, and the persistent volume declarations are mounted to the job's Pod container. After the task is completed, persistent volume declarations and persistent volumes are retained or automatically reclaimed according to a policy.

[0010] Specifically, the activation of the elastic policy is determined based on the enabled field in the container group configuration. When this field is set to true, elastic scheduling is allowed.

[0011] Specifically, the job status includes waiting, running, partially running, completed, and failed, where the partially running status indicates that some tasks of the job have been started and executed, but some tasks are still in the waiting scheduling state.

[0012] Specifically, the method also includes providing a unified RESTful interface for user interaction through an API gateway layer and interacting with a MySQL database through a data access layer.

[0013] This application has the following technical effects: By introducing an elastic scheduling mechanism, the rigid constraints of the traditional all-or-nothing scheduling strategy are effectively overcome. This allows distributed training jobs to start some tasks for computation based on a preset minimum number of successful members when cluster resources are insufficient. This significantly reduces job waiting time and makes full use of fragmented resources in the cluster. While ensuring the basic effectiveness of job execution, it greatly improves the overall utilization of cluster resources and the flexibility of task scheduling. Attached Figure Description

[0014] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily understood by reading the following detailed description with reference to the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, and the same or corresponding reference numerals denote the same or corresponding parts.

[0015] Figure 1 This is a flowchart of a scheduling method for an artificial intelligence model training task based on a container orchestrator, as described in an embodiment of this application. Detailed Implementation

[0016] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0017] With the rapid development of artificial intelligence technology, the demand for training deep learning models is increasing daily. AI model training typically requires substantial computing resources, especially high-performance hardware such as GPUs. Traditional methods suffer from the following problems: traditional training systems lack effective resource scheduling mechanisms, failing to fully utilize cluster resources and resulting in wasted idle resources. Furthermore, the system struggles to dynamically adjust resource allocation based on load.

[0018] While the Kubernetes native scheduler (kube-scheduler) provides basic container orchestration capabilities, it still requires specialized optimization and customization for complex scenarios like AI training, which demand GPU resources, data storage, and network configuration. This is because Kubernetes schedules on a per-pod basis, lacking the ability to coordinate the scheduling of batch jobs (such as distributed machine learning training and big data processing). This can lead to some Pods of a job being successfully scheduled, while others remain in a pending state due to insufficient resources, ultimately resulting in wasted resources for the scheduled Pods and the entire job failing to execute effectively. Furthermore, existing technologies typically use the Volcano scheduler to address these issues, which introduces PodGroup and Gang Scheduling plugins. Gang Scheduling follows an "All-or-Nothing" principle. It defines the minimum number of tasks a job can run through the minMember field in the PodGroup. Only when cluster resources can meet the minimum number of tasks will the scheduler schedule all tasks under that PodGroup; otherwise, all tasks will remain in a waiting state.

[0019] Therefore, this solution uses the Volcano scheduler to uniformly schedule tasks. In addition, it optimizes the system to address the Volcano scheduler's requirement that all tasks be ready at the same time, otherwise all tasks will be suspended. However, in AI training, sometimes 90% of the tasks are ready to start.

[0020] Specifically, this application proposes a scheduling method for training AI models based on container orchestrators. In one embodiment of this application, such as... Figure 1 As shown, the method includes: Receive job requests. The job requests contain PodGroup container group configuration information, in which the PodGroup container group configuration information defines the minimum number of members (minMember) and the elasticity policy (elasticPolicy). The elasticity policy (elasticPolicy) specifies the minimum number of successful members (minSuccessMember). Parse the PodGroup container group configuration information and check whether the cluster resources meet the minimum member requirement (minMember). When the cluster resources meet the minimum number of members (minMember), schedule all tasks of the job and update the job status to Running. When the cluster resources do not meet the minimum number of members (minMember), further determine whether the elastic policy (elasticPolicy) is enabled and whether the cluster resources at least meet the minimum number of successful members (minSuccessMember). When the elastic policy is enabled and the cluster resources meet the minimum number of successful members (minSuccessMember), the task corresponding to the minimum number of successful members (minSuccessMember) is scheduled, and the job status is set to PartialRunning. While the job is in a partially running state, the cluster resources are continuously monitored for changes. When sufficient available resources are detected, the remaining tasks are automatically scheduled to complete the job execution.

[0021] In this embodiment, the entire process of creating and managing the training task also includes: Task creation phase: Receive training task creation requests submitted by users via API. The request includes parameters such as region, task name, data path, code path, output path, container image, product specification SKU, and number of nodes. Verify the user's access permissions to the specified storage container; if permissions are insufficient, return an error and terminate the process. Verify the validity of the product specification SKU and obtain the corresponding CPU, memory, and GPU resource quota information from the product configuration library. If the specification is invalid, return an error and terminate the process. Check if the inventory resources are sufficient; if the inventory is insufficient, return an error and terminate the process. Deduct the corresponding SKU's inventory quantity and create an order record; Create a training task record in the database, and set its initial status to "waiting"; Trigger the resource scheduling process and call the Kubernetes service to create a Volcano Job task.

[0022] K8S job creation and scheduling phase: Retrieve user configuration parameters from training task records; Extract CPU, memory, and GPU resource requirements based on the product specification SKU; Create a YAML configuration template for a Volcano Job, and set the number of task replicas and restart strategy; Declare GPU resource requests in the Job configuration and enable GPU topology-aware scheduling policy; Configure data storage volume mapping to mount the user-specified data path, code path, and output path as PVC volumes within the container; Call the Kubernetes API to create a PVC object and bind it to the Job's Pod template; Submit the Job resource object via the Kubernetes Batch API; Bind the returned Job UID to the training task record and verify that the Job creation status is "successful".

[0023] Status monitoring and management phase: Start the monitoring service and initialize the Kubernetes Informer listener; Continuously monitor the state change events of Job resources in the Kubernetes cluster using Informer; Determine if the current Job status has changed; if not, return to step S22 to continue listening. If the state changes, the state change event handler is triggered; Update the status field of the corresponding training task in the database according to the job status (such as Pending, Running, Succeeded, Failed); Push status change notifications to users via WebSocket or Webhook; Record status change logs for auditing and troubleshooting purposes; Return to step S17 and continue monitoring.

[0024] Storage management phase: The training task declares the required data storage path; The PVC management module automatically creates a PersistentVolumeClaim (PVC) based on the path. Kubernetes dynamic providers create corresponding PersistentVolumes (PVs) based on PVCs. The PVC is attached to the Job's Pod container to enable data access; Once the task is completed, PVC / PV can be retained or automatically recycled according to the policy.

[0025] Specifically, in this embodiment, the container orchestrator is a Kubernetes cluster, and the scheduling method is implemented by integrating the Volcano scheduler. The container group configuration information corresponds to the definition of the container group custom resource in the Volcano scheduler. The container group custom resource defines the minimum number of members and the elasticity policy. The elasticity policy includes an enabled status field and a minimum number of successful members field, which are used to allow some tasks to start when resources are insufficient.

[0026] In this embodiment, the method further includes receiving a training task creation request submitted by the user through an application programming interface before receiving the job request. The training task creation request includes parameters such as region, task name, data path, code path, output path, container image, product specifications, and number of nodes. The method also verifies the user's access permissions to the storage container and the validity of the product specifications, obtains the corresponding CPU, memory, and GPU resource quota information from the product configuration library, deducts inventory and creates an order record when resources are sufficient, creates a training task record in the database and sets the initial state to waiting, and finally triggers the resource scheduling process.

[0027] In this embodiment, when parsing the container group configuration information and checking whether the cluster resources meet the minimum number of members requirement, GPU resources are scheduled, and GPU resource requests are declared in the job configuration to enable the GPU topology-aware scheduling strategy.

[0028] In this embodiment, the Kubernetes Informer listener continuously monitors changes in cluster resources. The Kubernetes Informer listener continuously listens for status change events of job resources in the cluster and triggers an event handler when the status changes. It automatically updates the status field of the training task in the database and then pushes the status change notification to the user via WebSocket or Webhook. At the same time, it records status change logs for auditing and troubleshooting.

[0029] In this embodiment, the method further includes automatically creating and managing persistent volume declarations and persistent volumes before scheduling tasks. After training the data storage path required by the task declaration, persistent volume declarations are automatically created, corresponding persistent volumes are created based on the persistent volume declarations, and persistent volume declarations are mounted to the job's Pod container. Persistent volume declarations and persistent volumes are retained or automatically reclaimed according to a policy after the task is completed.

[0030] In this embodiment, the enabling of the elastic policy is determined based on the enabled field in the container group configuration. When this field is set to true, elastic scheduling is allowed.

[0031] In this embodiment, the job status includes waiting, running, partially running, completed, and failed. The partially running status indicates that some tasks of the job have been started and executed, but some tasks are still in the waiting scheduling state.

[0032] In this embodiment, the method also includes providing a unified RESTful interface for user interaction through an API gateway layer, and interacting with a MySQL database through a data access layer.

[0033] Obviously, the embodiments described above are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0034] It should be understood that when the terms "first," "second," etc., are used in the claims, description, and drawings of this application, they are only used to distinguish different objects and not to describe a specific order. The terms "comprising" and "including" used in the description and claims of this application indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

Claims

1. A scheduling method for training tasks of artificial intelligence models based on container orchestrators, characterized in that, The method includes: Receive a job request, the job request containing container group configuration information, wherein the container group configuration information defines the minimum number of members and the elasticity policy, and the elasticity policy specifies the minimum number of successful members; Parse the container group configuration information and check whether the cluster resources meet the minimum member requirement; When the cluster resources meet the minimum number of members, schedule all tasks of the job and update the job status to running. When the cluster resources do not meet the minimum number of members, it is further determined whether the elasticity policy is enabled and whether the cluster resources at least meet the minimum number of successful members. When the elastic policy is enabled and the cluster resources meet the minimum number of successful members, the task corresponding to the minimum number of successful members is scheduled, and the job status is set to partially running. During the period when the job is in a partially running state, the cluster resources are continuously monitored for changes. When sufficient available resources are detected, the remaining tasks are automatically scheduled to complete the job execution.

2. The method according to claim 1, characterized in that, The container orchestrator is a Kubernetes cluster, and the scheduling method is implemented by integrating the Volcano scheduler. The container group configuration information corresponds to the definition of the container group custom resource in the Volcano scheduler. The container group custom resource defines the minimum number of members and the elasticity policy. The elasticity policy includes an enabled status field and a minimum number of successful members field, which are used to allow some tasks to start when resources are insufficient.

3. The method according to claim 1, characterized in that, The method further includes receiving a training task creation request submitted by the user through an application programming interface before receiving the job request. The training task creation request includes parameters such as region, task name, data path, code path, output path, container image, product specifications, and number of nodes. It also verifies the user's access permissions to the storage container and the validity of the product specifications, obtains the corresponding CPU, memory, and GPU resource quota information from the product configuration library, deducts inventory and creates an order record when resources are sufficient, creates a training task record in the database and sets the initial state to waiting, and finally triggers the resource scheduling process.

4. The method according to claim 1, characterized in that, When parsing the container group configuration information and checking whether the cluster resources meet the minimum number of members requirement, the GPU resources are scheduled, and the GPU resource request is declared in the job configuration, enabling the GPU topology-aware scheduling strategy.

5. The method according to claim 1, characterized in that, The Kubernetes Informer listener continuously monitors changes in cluster resources. It listens for status change events of job resources in the cluster and triggers an event handler when the status changes. The event handler automatically updates the status field of the training task in the database and then pushes the status change notification to the user via WebSocket or Webhook. At the same time, it records status change logs for auditing and troubleshooting.

6. The method according to claim 1, characterized in that, The method further includes automatically creating and managing persistent volume declarations and persistent volumes before scheduling tasks. After training the data storage path required by the task declaration, persistent volume declarations are automatically created, corresponding persistent volumes are created according to the persistent volume declarations, and the persistent volume declarations are mounted to the job's Pod container. After the task is completed, persistent volume declarations and persistent volumes are retained or automatically reclaimed according to a policy.

7. The method according to claim 1, characterized in that, The activation of the elastic policy is determined based on the enabled field in the container group configuration. Elastic scheduling is allowed when this field is set to true.

8. The method according to claim 1, characterized in that, The job status includes waiting, running, partially running, completed, and failed. The partially running status indicates that some tasks of the job have been started and executed, but some tasks are still waiting to be scheduled.

9. The method according to claim 1, characterized in that, The method also includes providing a unified RESTful interface for user interaction through an API gateway layer, and interacting with a MySQL database through a data access layer.

Images