Training method and device for multi-level model and medium

Abstract

The invention discloses a training method and device for a multi-level model and a medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: firstly, receiving a YAML file declaring multi-stage, resource and priority information; then analyzing and creating a Pod Group object for unified management, wherein the object defines the minimum number of copies and the priority; distributing the task group to a scheduling queue of a preset resource quota and a preemption strategy; after being sorted according to the priority, the scheduler checks whether the cluster can meet the requirement of simultaneous operation of the whole task group or not; if so, atomically scheduling all Pods; if not, a preemption process is triggered, and the low-priority Pod is expelled to release resources; finally, the system can monitor and expel the state of the Pod, and automatically rescheduling the Pod according to a pre-configured strategy, thereby ensuring the final completeness of the task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a training method for a multi-level model, a device and a medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, model training tasks are becoming increasingly complex, showing multi-stage and multi-level characteristics. A typical model training process includes multiple stages such as data preprocessing, distributed training, model fine-tuning, and compression inference. The types, scales, and priorities of computing resources required by different stages differ significantly. Currently, Kubernetes-based container orchestration systems have become the mainstream platform for deploying and scheduling training tasks. Although the native scheduler of Kubernetes can complete basic resource allocation and load balancing, it is not optimized for the characteristics of multi-stage training tasks.

[0003] In the prior art, the default scheduler of Kubernetes takes a Pod as the smallest scheduling unit, lacks native support for Gang Scheduling of multiple Pods, and causes partial blocking of distributed training tasks due to insufficient resources, resulting in idle resources and task stagnation. Although batch scheduling frameworks such as Volcano and Kueue attempt to address this deficiency by introducing PodGroup and queue mechanisms, they still do not achieve true multi-level task orchestration and dynamic priority adjustment. In particular, when dealing with priority inversion, fine-grained resource preemption, and task automatic recovery, existing solutions often rely on manual intervention or simple restart strategies, which cannot meet the efficiency and fairness requirements in high-concurrency and multi-tenant scenarios.

[0004] Therefore, how to achieve fine-grained resource scheduling and dynamic priority preemption in a multi-stage and multi-priority model training scenario has become a technical problem that needs to be solved by those skilled in the art. SUMMARY

[0005] The embodiments of the present application provide a training method for a multi-level model, a device and a medium, to solve the technical problem of how to achieve fine-grained resource scheduling and dynamic priority preemption in a multi-stage and multi-priority model training scenario.

[0006] In a first aspect, the embodiments of the present application provide a training method for a multi-level model, the method comprising: receiving a training task definition file submitted by a user; wherein the training task definition file declares a plurality of training stages, resource requirements and priority information of each training stage contained in the training task in YAML format; parsing the training task definition file to generate Pods corresponding to each training stage, and creating a PodGroup object for association management of a plurality of Pods belonging to the same training stage; wherein the PodGroup object declares a minimum number of replicas minMember field and a priority priorityClassName field; according to the queue declaration in the training task definition file, assigning the PodGroup object to a specified scheduling queue; wherein the scheduling queue is pre-configured with resource quota and preemption strategy; according to the priority value represented by the priorityClassName field of the PodGroup object, sorting all to-be-scheduled PodGroup objects using a preset scheduler; for each PodGroup object after sorting, using the scheduler to check whether there are node resources in the cluster that can simultaneously run the number of Pods required by the minMember field of the corresponding PodGroup object; if there are node resources that meet the condition, scheduling all Pods in the PodGroup object to the corresponding node and starting running; if there are no node resources that meet the condition, triggering a preemption process for high-priority PodGroup objects, and selecting Pods in one or more low-priority PodGroup objects to be evicted to release resources; monitoring the state of the evicted Pods, and automatically rescheduling the evicted Pods according to a pre-configured restart strategy.

[0007] In an embodiment of the present application, the training task definition file is parsed to generate Pods corresponding to each training stage, and a PodGroup object is created for association management of a plurality of Pods belonging to the same training stage, specifically comprising: identifying a plurality of training stages defined in the training task definition file; wherein each training stage corresponds to a subtask; generating an independent Kubernetes Job object or defining a Task entry in a VolcanoJob object for each training stage; creating a PodGroup object for each training stage, and associating all Pods corresponding to the training stage with the PodGroup object.

[0008] In an embodiment of the present application, the PodGroup object is assigned to a specified scheduling queue according to the queue declaration in the training task definition file, specifically including: reading the specified queue name in the PodGroup object; associating the PodGroup object with a LocalQueue custom resource object; and associating the LocalQueue object with a ClusterQueue custom resource object with predefined total resource amount and preemption rules.

[0009] In an embodiment of the present application, the preemption process is triggered for a high-priority PodGroup object, and a Pod in one or more low-priority PodGroup objects is selected to be evicted to release resources, specifically including: identifying a high-priority PodGroup that cannot be scheduled by the scheduler, and screening out nodes in the cluster that meet the resource requirements of the high-priority PodGroup; calculating the amount of resources to be released to meet the scheduling requirements of the high-priority PodGroup, and selecting one or more low-priority Pods occupying the required resources from the selected nodes; wherein the priority value of the selected Pods is lower than that of the high-priority PodGroup; sending an eviction instruction to the selected Pods and setting an eviction termination time for them.

[0010] In an embodiment of the present application, the method further includes: performing fine-grained preemption on GPU resources, specifically including: dividing physical GPU resources into multiple virtual GPU instances by a GPU virtualization plug-in; monitoring the / vgpu-memory and / vgpu-number resource states of the nodes by the scheduler; when it is necessary to preempt GPU resources for a high-priority PodGroup, selecting a low-priority Pod occupying the required vGPU memory or vGPU number to be evicted by the scheduler; and the high-priority Pod exclusively uses the released vGPU instance after being scheduled.

[0011] In an embodiment of the present application, the method further includes: monitoring the running state of all Pods in the current training stage; when it is monitored that all Pods in the current training stage have successfully completed running, triggering the creation and scheduling of the PodGroup object of the next training stage; the PodGroup object of the next training stage is created and added to the scheduling queue, waiting for scheduling.

[0012] In an embodiment of the present application, the Pod that is evicted is automatically rescheduled according to a preconfigured restart strategy, specifically including: detecting a Pod eviction event; querying a policies field defined in a VolcanoJob object to which the evicted Pod belongs; resetting the entire Job state to Pending according to a RestartJob strategy configured in the policies field for the PodEvicted event; resubmitting a PodGroup object corresponding to the Job to a scheduling queue for rescheduling.

[0013] In an embodiment of the present application, the method further includes: receiving a priority change instruction issued by a user or a system, and identifying a PodGroup object corresponding to a running task that needs to change the priority based on the priority change instruction; terminating all Pods associated with the PodGroup object; recreating the PodGroup object using a new priorityClassName field value; and resubmitting the newly created PodGroup object to the scheduling queue.

[0014] In a second aspect, the embodiments of the present application further provide a training device for a multi-level model, the device comprising: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any one of the training methods for a multi-level model described above.

[0015] In a third aspect, the embodiments of the present application further provide a non-volatile computer storage medium storing computer executable instructions, and the computer executable instructions are executed to implement any one of the training methods for a multi-level model described above.

[0016] The training method for a multi-level model, the device and the medium provided by the embodiments of the present application have the following beneficial effects: by introducing a multi-stage task management mechanism based on PodGroup, a priority-driven hierarchical scheduling queue, and a fine-grained resource preemption and automatic recovery strategy, the scheduling efficiency and resource utilization of the multi-level model training task in a shared computing cluster are significantly improved; the fine-grained sharing and isolation of GPU resources are achieved through GPU virtualization and vGPU resource monitoring, the preemption cost is effectively reduced, and resource fragmentation is avoided; at the same time, with the help of dynamic priority adjustment and the controller design with inter-stage state awareness, the timely scheduling of high-priority tasks and the dependency continuity of multi-stage tasks are guaranteed, and the overall execution reliability and system throughput of the training task are improved. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0018] Figure 1 A training method flow chart for a multi-level model provided by an embodiment of the application;

[0019] Figure 2 An internal structure schematic diagram of a training device for a multi-level model provided by an embodiment of the application. DETAILED DESCRIPTION

[0020] In order to make the objects, technical solutions and advantages of the application clearer, the technical solutions of the application will be described below in detail with reference to the embodiments of the application and the corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the application, but not all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the application.

[0021] The embodiments of the application provide a training method, device and medium for a multi-level model, to solve the technical problem of how to realize fine-grained resource scheduling and dynamic priority preemption in a multi-stage and multi-priority model training scenario.

[0022] The technical solutions of the embodiments of the application will be described in detail below with reference to the accompanying drawings.

[0023] Figure 1 A training method flow chart for a multi-level model provided by an embodiment of the application. As shown in Figure 1 The training method for a multi-level model provided by the embodiments of the application specifically includes the following steps:

[0024] Step 101, receiving a training task definition file submitted by a user.

[0025] In this embodiment, the training task definition file declares a plurality of training stages, resource requirements and priority information of each training stage included in the training task in YAML format.

[0026] In this embodiment, the training task definition file is a configuration file written by the user according to the specific multi-stage machine learning workflow requirements of the user. It can be understood that this file is the working input and the only basis for the entire scheduling system, and the integrity and accuracy of its content directly determines the behavior of subsequent resource scheduling and task arrangement. It should be noted that the system receives the file submitted by the user through a resident API server (such as a HTTP-based RESTful API) or a dedicated command line tool (CLI). For example, the user can use the kubectl apply-f training-job.yaml command, or programmatically call the API interface and submit the file content in the request body.

[0027] Specifically, the file is written in YAML (YAML Ain't Markup Language) format, mainly based on its clear structure, strong readability and easy to be parsed by program. In the YAML file, the user needs to explicitly declare all multiple training stages contained in a complete training task. For example, a complete model life cycle may contain multiple stages such as "data-preprocessing" (data preprocessing), "distributed-training" (distributed training), "model-evaluation" (model evaluation) and "model-export" (model export) in turn, each of which is a logically relatively independent and sequentially executable subtask.

[0028] It should be noted that for each declared training stage, the user must define its resource requirements in detail in the YAML file. This is usually achieved by declaring the resources.requests and resources.limits fields under the corresponding stage. Specifically, the resource requirements mainly include computing resources (such as CPU core number, memory size) and accelerator resources (such as GPU type, GPU number). For example, in the "distributed-training" stage, the user may request 4 NVIDIA A100 GPUs and 64 GiB of memory.

[0029] In addition, the user also needs to explicitly specify the priority information for each stage. This is achieved through the priorityClassName field, the value of which is a string pointing to a PriorityClass custom resource object created in advance in the Kubernetes cluster. It can be understood that different PriorityClasses correspond to different integer value priorities, thereby establishing quantifiable comparison standards for different importance task stages (such as high-priority online service inference tasks and low-priority offline batch training tasks) within the system. After the system receives this file, it will be stored persistently and generate the corresponding work resource object, thereby triggering the subsequent parsing and scheduling process.

[0030] Step 102, parsing the training task definition file to generate Pods corresponding to each training stage, and creating PodGroup objects to associate and manage multiple Pods belonging to the same training stage.

[0031] In this embodiment, the PodGroup object declares a minimum number of replicas minMember field and a priority priorityClassName field.

[0032] In an embodiment of the present application, the training task definition file is parsed to generate Pods corresponding to each training stage, and PodGroup objects are created to associate and manage multiple Pods belonging to the same training stage, specifically including: identifying multiple training stages defined in the training task definition file; wherein each training stage corresponds to a subtask; generating an independent Kubernetes Job object for each training stage or defining a Task entry in a VolcanoJob object; creating a PodGroup object for each training stage and associating all Pods corresponding to the training stage with the PodGroup object.

[0033] In this embodiment, this step is executed by a custom controller (CustomController) deployed in the Kubernetes cluster. It should be noted that the controller continuously monitors (Watches) whether a new training task definition file is submitted, and as soon as it detects the training task definition file created in step 101, it will immediately start the parsing process. Specifically, the controller uses a YAML parsing library (such as yaml.v2 for Go) to deserialize (Unmarshal) the file content, converting it into a structured data object in memory for subsequent processing by the program. It can be understood that the core purpose of this parsing process is to identify the task structure defined by the user and create corresponding resource objects that can be managed by the scheduler in the cluster.

[0034] Specifically, the controller first identifies and extracts all the training stages defined in the YAML file. As an example, a training task can be parsed into four stages: "preprocess", "train", "validate", and "export". For each identified training stage, the controller generates one or more corresponding Pod templates (PodTemplate) based on its declared image, command, environment variables, and most importantly, resource requirements (CPU, Memory, GPU) in the YAML. A Pod template is a blueprint in Kubernetes for defining the running specification of a Pod, which ensures that the tasks in this stage run in the container as expected by the user.

[0035] It is important to note that one important implementation of this step is to "generate a separate Kubernetes Job object or define a Task entry in a VolcanoJob object for each training stage". As an example, for a one-time stage like "train" that needs to ensure the completion of the task, the controller will create a Kubernetes Job resource for it. The Job object will reference the generated Pod template and be responsible for managing the execution of the Pod until it is successfully completed. For more complex scenarios that require support for advanced features such as Gang scheduling, the controller will choose to create a VolcanoJob resource and define a Task entry in its spec.tasks list for the stage. Similarly, it points to the generated Pod template and specifies the number of Pod instances required for the stage through the replicas field.

[0036] Further, to implement Gang scheduling (i.e., "all or nothing" scheduling semantics), the controller creates a PodGroup custom resource object for each training phase that requires multiple Pods to run in parallel. It is understood that PodGroup is a core concept in advanced schedulers such as Volcano, which treats multiple Pods as a logical whole for scheduling. When creating a PodGroup, the controller sets its minMember field to the minimum number of Pod instances required to start the phase, which is usually consistent with the replicas of Tasks in VolcanoJob or the parallelism of Jobs, thereby ensuring that the scheduler will only schedule when all Pod instance resource requirements for the phase can be met, avoiding deadlocks and resource waste caused by partial instance startup failures in distributed tasks. At the same time, the controller reads the priority information declared by the user in the YAML for the phase, and fills the corresponding priority class name into the priorityClassName field of the PodGroup, thereby converting the user-defined business priority into a scheduler-recognizable scheduling priority. Finally, the controller associates all Pods generated for the phase with the created PodGroup object through the Label or OwnerReference mechanism, completing the overall association management.

[0037] Step 103, according to the queue declaration in the training task definition file, assign the PodGroup object to the specified scheduling queue.

[0038] In this embodiment, the scheduling queue is pre-configured with resource quotas and preemption policies.

[0039] In an embodiment of the present application, according to the queue declaration in the training task definition file, the PodGroup object is assigned to the specified scheduling queue, specifically including: reading the specified queue name in the PodGroup object; associating the PodGroup object with a LocalQueue custom resource object; and associating the LocalQueue object with a ClusterQueue custom resource object that has predefined total resources and preemption rules.

[0040] In this embodiment, this step is responsible for performing by the queue manager component in the system, the core function of which is to bind the PodGroup object created in step 102 to the appropriate scheduling queue. It should be noted that the "scheduling queue" here is not a simple data structure, but a custom resource object in the Kubernetes cluster (such as Volcano's Queue or Kueue's LocalQueue), which serves as a logical boundary for resource isolation and multi-tenant management. Specifically, the controller will parse the specific annotations (Annotations) or labels (Labels) in the PodGroup object, or directly read the queue name field specified when it is created, to determine its target scheduling queue.

[0041] It can be understood that the allocation mechanism of the queue provides users with flexible resource configuration strategies. For example, users can label the task with queue: high-priority-queue through the metadata.labels field when submitting the training task definition file, or directly declare the queue it belongs to through the queueName field in the spec of PodGroup. The controller will capture this information and perform the binding operation.

[0042] Specifically, the binding process is divided into two levels: first, the controller will find the LocalQueue custom resource object corresponding to the queue name, and set the PodGroup as the owner or associate it through reference of the LocalQueue; Subsequently, the LocalQueue object itself belongs to a higher level ClusterQueue. ClusterQueue is a cluster-level resource, which is pre-configured by the system administrator, and defines the upper limit of the total amount of resources that the queue can use (i.e. resource quota), the resource allocation weight relative to other queues, and the most important preemption policies, such as whether to allow tasks in the queue to preempt tasks in other queues, or whether to allow being preempted by other queues.

[0043] It's important to note that this hierarchical queue design (PodGroup -> LocalQueue -> ClusterQueue) is key to achieving fine-grained management. For example, a ClusterQueue named "research" might be configured with a total quota of 100 GPUs and a "preemption allowed" policy. Multiple LocalQueues, such as "user-alice" and "user-bob," can be created under it, each configured with different weights to ensure fairness among users. Once the controller successfully associates a PodGroup with a LocalQueue, the subsequent scheduling behavior of that task group will be entirely governed by the queue's pre-configured policy. This means that whether a PodGroup can be scheduled depends not only on the current global resource status of the cluster, but also on whether the resource quota of its queue is sufficient, whether its weight allows it to acquire resources, and whether it can preferentially acquire resources or be forced to relinquish resources according to the preset preemption rules during resource contention.

[0044] Step 104: Sort all PodGroup objects to be scheduled using the preset scheduler based on the priority value represented by the priorityClassName field of the PodGroup object.

[0045] In this embodiment, this step is executed automatically and periodically by an extended scheduler (such as Volcano-scheduler or Kueue-scheduler) deployed in the cluster. It should be noted that the "preset scheduler" here is not the default Kubernetes scheduler, but a specialized scheduler that has been specifically enhanced to understand advanced concepts such as PodGroups and queues. Specifically, the scheduler continuously monitors the Kubernetes API Server to obtain all PodGroup objects that are in a Pending state and have been successfully bound to a queue, forming a queue to be scheduled.

[0046] Understandably, the sorting is based on the priority value represented by the `priorityClassName` field in the `PodGroup` object. Specifically, the scheduler reads this field, which is a string pointing to a predefined `PriorityClass` custom resource object defined in the cluster. Then, the scheduler queries the `value` integer field of this `PriorityClass` object, which directly represents the absolute priority of the `PodGroup`. For example, a `PriorityClass` named `high-priority` might be set to 10000, while a `PriorityClass` named `low-priority` might be set to 1000. In this way, the scheduler converts user-defined logical priority names (such as "high") into integer priorities that can be compared numerically within the scheduling system.

[0047] It's important to note that the sorting algorithm typically arranges PodGroups in descending order of priority (from highest to lowest). This means that PodGroups with higher priority values ​​(e.g., those for urgent production tasks) will be placed at the front of the scheduling queue, while PodGroups with lower priority values ​​(e.g., those for development and testing tasks) will be placed later. For example, if there are three PodGroups in the scheduling queue with priority values ​​of 10000, 5000, and 1000 respectively, the scheduler will strictly follow this order after sorting. This sorting mechanism ensures that when cluster resources are contested, higher-priority task groups will always have priority over lower-priority task groups in obtaining scheduling opportunities, thus meeting the business requirements for task urgency and importance and laying a fair foundation for subsequent scheduling decisions.

[0048] Step 105: For each sorted PodGroup object, use the scheduler to check if there are node resources in the cluster that can run the number of Pods simultaneously as required by the minMember field of the corresponding PodGroup object.

[0049] In this embodiment, this step is executed by the preset extended scheduler core algorithm module and is a critical decision-making step for each task group to be scheduled in the sorted PodGroup queue. It should be noted that this check is not a simple comparison of total resources, but a complex resource simulation allocation process based on constraint satisfaction. Specifically, the scheduler processes each PodGroup object in the sorted queue sequentially, first reading the value of its minMember field. This field explicitly specifies the minimum number of Pod instances that the task group must start simultaneously, which is directly related to Gang scheduling semantics and ensures the atomicity of distributed task execution.

[0050] Understandably, the scheduler will then iterate through all nodes in the cluster, performing a two-phase resource feasibility analysis. The first phase is filtering. The scheduler will consider the resource requirements declared by each Pod template in the PodGroup (such as CPU, memory, GPU type and quantity, and possible extended resources such as vgpu-memory), combined with the node's allocable resource capacity, the NodeSelector, and affinity / anti-affinity rules, to filter out all candidate nodes that may be able to run a single Pod. For example, a Pod requesting 4 GPUs will only be considered for scheduling on a node with at least 4 available GPUs.

[0051] More crucially, the second phase involves global resource verification based on the minMember requirement. The scheduler needs to solve a combinatorial optimization problem: can it find a set of node resource allocation schemes from the candidate node set such that exactly minMember Pods in the PodGroup can be simultaneously placed on them, and the remaining resources after allocation on each node still satisfy the running constraints of its other Pods? For example, for a PodGroup with a minMember of 4, the scheduler must find at least 4 nodes (or one large node that can accommodate 4 Pods) whose current idle resources can simultaneously satisfy the resource requests of these 4 Pods; none can be missing. If such a resource combination cannot be found, even if the total idle resources of the cluster are much greater than the needs of the PodGroup, if the resources are fragmented and cannot meet the condition of "simultaneously" starting minMember Pods, the scheduler will determine that the current cluster resources do not meet the scheduling conditions.

[0052] Step 106: If there are node resources that meet the conditions, schedule all Pods in the PodGroup object to the corresponding node and start running them.

[0053] In this embodiment, this step is a crucial operation performed by the scheduler after successfully finding a resource allocation scheme that meets the minMember requirement for the PodGroup. It's important to note that "scheduling" here does not refer to the sequential binding of individual Pods, but rather an atomic, batch operation on the entire PodGroup. Specifically, based on the optimal resource allocation scheme calculated in step 105, the scheduler calls the Kubernetes API Server interface to perform binding operations on all Pod objects within the PodGroup almost simultaneously, setting the nodeName field of each Pod to the selected target node name. This atomic operation is the final guarantee for achieving the "all or none" semantics of Gang scheduling; it ensures that all worker instances of the distributed task can be simultaneously allocated to compute nodes, thus laying the foundation for subsequent simultaneous startup.

[0054] For example, for a PodGroup with a minMember of 4 that requires distributed training, the scheduler will bind four Worker Pods to four selected GPU nodes at once. Subsequently, the Kubelet component on each target node in the cluster will detect the new Pod being bound to its node and begin the Pod startup process in parallel, including pulling the specified container image, creating the container runtime environment, mounting the required storage volumes, and finally starting the processes within the container. Understandably, this near-simultaneous startup is crucial for distributed training tasks because it avoids communication timeouts or coordination failures caused by large differences in Worker startup time. The entire process is completed collaboratively by the scheduler and the cluster infrastructure, without user intervention, thus accurately and reliably translating the task requirements declared by the user in the YAML file into the actual workload running in the cluster. This signifies that the training phase represented by the PodGroup has successfully transitioned from a "waiting for scheduling" state to an "in execution" state.

[0055] Step 107: If no node resources meet the conditions, a preemption process is triggered for the high-priority PodGroup object, and one or more Pods in the low-priority PodGroup object are selected to be evicted to release resources.

[0056] In one embodiment of this application, a preemption process is triggered for a high-priority PodGroup object, and one or more Pods in a low-priority PodGroup object are selected to be evicted to release resources. Specifically, this includes: using the scheduler to identify high-priority PodGroups that cannot be scheduled, and filtering out nodes in the cluster whose resource occupancy meets the resource requirements of the high-priority PodGroups; calculating the amount of resources to be released to meet the scheduling requirements of the high-priority PodGroups, and selecting one or more low-priority Pods occupying the required resources from the selected nodes; wherein the priority value of the selected Pods is lower than the priority value of the high-priority PodGroups; sending an eviction command to the selected Pods, and setting an eviction termination time for them.

[0057] In one embodiment of this application, the method further includes: fine-grained preemption of GPU resources, specifically including: dividing physical GPU resources into multiple virtual GPU instances through a GPU virtualization plugin; monitoring the resource status of / vgpu-memory and / vgpu-number of nodes using a scheduler; when GPU resources need to be preempted for a high-priority PodGroup, using the scheduler to select and evict low-priority Pods that occupy the required vGPU memory or vGPU number; after being scheduled, the high-priority Pod exclusively uses the released vGPU instance.

[0058] In this embodiment, this step is a crucial remedial mechanism executed by the scheduler when it cannot find sufficient resources for a high-priority PodGroup. It should be noted that the prerequisite for triggering the preemption process is that, after a detailed check in step 105, it is confirmed that no node or combination of nodes in the cluster can simultaneously meet the minMember resource requirement of the high-priority PodGroup. Specifically, at this point, the scheduler will initiate a preemption algorithm. This algorithm first precisely calculates the total amount of resources required to release to satisfy the scheduling of the high-priority PodGroup, including the specific amounts of various resources such as CPU, memory, and GPU. This is directly related to the feature in claim 4 that "the amount of resources required to release to satisfy the scheduling of the high-priority PodGroup" is calculated.

[0059] Understandably, the scheduler will then scan all nodes in the cluster to find nodes that can free up the required resources by evicting existing Pods. For example, the scheduler will prioritize nodes running Pods with priority values ​​lower than the currently scheduled PodGroup, and follow the "lowest cost" principle, that is, select a group of Pods that can just free up the required resources, and whose priority is as low as possible and whose number is as small as possible.

[0060] It's important to note that when GPU resources are involved, preemption is performed at a fine-grained level. The scheduler monitors the ` / vgpu-memory` and ` / vgpu-number` resource status of nodes through integrated GPU virtualization plugins (such as the `vGPU` plugin). This allows it to precisely calculate the amount of vGPU memory or the number of vGPU instances that need to be preempted, and then selects to evict low-priority Pods that are occupying these specific vGPU resources, rather than the entire Pod. This enables high-priority tasks to preempt only the portion of GPU resources they need without evicting entire low-priority Pods, achieving fine-grained resource management.

[0061] Finally, the scheduler sends a deletion request to the selected Pods to be evicted via the Kubernetes API and sets a graceful termination period based on the cluster configuration, allowing them to complete the necessary cleanup work before termination.

[0062] Step 108: Monitor the status of the evicted Pods and automatically reschedule the evicted Pods according to the pre-configured restart policy.

[0063] In one embodiment of this application, the automatic rescheduling of an evicted Pod according to a pre-configured restart policy specifically includes: detecting a Pod evictation event; querying the policies field defined in the VolcanoJob object to which the evicted Pod belongs; resetting the entire Job state to Pending according to the RestartJob policy configured for the PodEvicted event in the policies field; and resubmitting the PodGroup object corresponding to the Job to the scheduling queue to wait for rescheduling.

[0064] In this embodiment, this step is continuously executed by the Controller Manager in the cluster and is a key mechanism to ensure system fault tolerance and task completion. It should be noted that the monitoring process is implemented by listening for state change events of Pod objects on the Kubernetes API Server. Specifically, when the preemption process in step 107 triggers the eviction of a Pod, the state of these Pods changes to Failed, and their reason field is marked as Evicted. The controller captures these event changes in real time and identifies the evicted Pod and its associated upper-level management workload (such as a VolcanoJob or a standard Kubernetes Job). It is understood that the purpose of monitoring is not merely to record the state, but to trigger subsequent automated recovery processes.

[0065] Specifically, upon detecting an eviction event, the controller queries the restart policy defined in the parent managed workload object to which the Pod belongs. For example, for Volcano-based tasks, the controller queries the `policies` field of the `VolcanoJob` object; for standard Kubernetes Jobs, it queries their `restartPolicy` field. These pre-configured policies explicitly define the specific actions to be taken when a failure event such as PodEvicted occurs. It's worth noting that a common and critical policy is `RestartJob`, which instructs the controller to reset the entire Job (or Task) to a Pending state and reschedule it. The controller performs a series of operations based on this policy: first, it terminates all remaining Pods (if any) associated with the currently failed task to ensure a clean environment; next, it updates the state of the `VolcanoJob` or Job object, resetting it to an unstarted state; finally, the controller recreates the corresponding PodGroup object (if it has been deleted) and resubmits it to the scheduling queue.

[0066] Understandably, this mechanism prevents preempted low-priority tasks from being simply discarded. Instead, they are re-entered into the scheduling loop in a controlled manner. Once new resources become available in the cluster (e.g., high-priority tasks complete and release resources, or the cluster is expanded), these waiting tasks will be re-attempted and executed in subsequent scheduling cycles, thus ensuring task completion and effectively avoiding task starvation caused by resource contention.

[0067] In one embodiment of this application, the method further includes: monitoring the running status of all Pods in the current training phase; when it is detected that all Pods in the current training phase have successfully completed running, triggering the creation and scheduling of the PodGroup object for the next training phase; the PodGroup object for the next training phase is created and added to the scheduling queue, waiting for scheduling.

[0068] In one embodiment of this application, the method further includes: receiving a priority change instruction issued by a user or the system, and identifying the PodGroup object corresponding to the running task whose priority needs to be changed based on the priority change instruction; terminating all Pods associated with the PodGroup object; recreating the PodGroup object using the new priorityClassName field value; and resubmitting the newly created PodGroup object to the scheduling queue.

[0069] In this embodiment, the system continuously runs two independent but crucial collaborative processes through a dedicated controller. First, to manage dependencies between multi-stage tasks, the controller monitors the running status of all Pods in the current training stage. Specifically, the controller achieves this by listening for the completion events of Pods and Jobs. For example, when the controller detects that all Pods in a certain stage (such as "training") have successfully completed their runs (their status is all Completed), it automatically triggers the creation and scheduling of PodGroup objects for the next training stage (such as "model export"). This process is accomplished by updating the status of custom resources or directly calling the API to create resources for the next stage. The newly created PodGroup objects are then added to the scheduling queue, waiting for the scheduler to process them according to their priority and resource availability, thereby realizing an automated pipeline between stages.

[0070] Meanwhile, the system supports dynamically adjusting task priorities in response to external commands. It's important to note that this process is handled by a separate controller component. When the controller receives a priority change command from a user or the system (this command may be issued by updating annotations or specific CRD objects), it immediately identifies the PodGroup object corresponding to the running task whose priority needs to be changed. Subsequently, the controller executes a safe update strategy: first, it terminates all Pods associated with the PodGroup object; then, it recreates the PodGroup object with the new priorityClassName field value; and finally, it resubmits the newly created PodGroup object to the scheduling queue. Understandably, this "termination first, then reconstruction with new priority" approach adheres to Kubernetes' restriction against online updates to the priorities of running Pods while simultaneously achieving the business goal of dynamic priority adjustment, ensuring that tasks can participate in subsequent scheduling competition with the new priority.

[0071] The above are embodiments of the method proposed in this application. Based on the same inventive concept, embodiments of this application also provide a training device for multi-level models, the structure of which is as follows: Figure 2 As shown.

[0072] Figure 2 This is a schematic diagram of the internal structure of a training device for a multi-level model, provided as an embodiment of this application. Figure 2 As shown, the device includes:

[0073] At least one processor 201;

[0074] And a memory 202 that is communicatively connected to at least one processor;

[0075] The memory 202 stores instructions executable by at least one processor, which are executed by at least one processor 201 to enable at least one processor 201 to:

[0076] The system receives a training task definition file submitted by the user. This file, in YAML format, declares multiple training stages, resource requirements for each stage, and priority information. It parses the training task definition file to generate Pods corresponding to each training stage and creates PodGroup objects for managing multiple Pods belonging to the same training stage. Each PodGroup object declares a minimum replica count (minMember) and a priority (priorityClassName) field. Based on the queue declaration in the training task definition file, the system assigns the PodGroup objects to the specified scheduling queue. This scheduling queue is pre-configured with resource quotas and preemption policies. Finally, it assigns the PodGroup objects to the specified scheduling queue based on the priorityClassName of the PodGroup object. The priority value represented by the field is used to sort all PodGroup objects to be scheduled using a preset scheduler. For each sorted PodGroup object, the scheduler checks whether there are node resources in the cluster that can run the number of Pods required by the minMember field of the corresponding PodGroup object. If there are node resources that meet the conditions, all Pods in the PodGroup object are scheduled to the corresponding node and started running. If there are no node resources that meet the conditions, a preemption process is triggered for the high-priority PodGroup object, and one or more Pods in the low-priority PodGroup object are selected to be evicted to release resources. The status of the evicted Pods is monitored, and the evicted Pods are automatically rescheduled according to the pre-configured restart policy.

[0077] Some embodiments of this application provide corresponding to Figure 1A non-volatile computer storage medium stores computer-executable instructions, which are configured to: receive a training task definition file submitted by a user; wherein the training task definition file declares multiple training stages included in the training task, the resource requirements of each training stage, and priority information in YAML format; parse the training task definition file to generate Pods corresponding to each training stage, and create PodGroup objects for association management of multiple Pods belonging to the same training stage; wherein the PodGroup object declares a minimum replica count (minMember) field and a priority (priorityClassName) field; allocate the PodGroup objects to a specified scheduling queue according to the queue declaration in the training task definition file; wherein the scheduling queue is pre-configured with resource quotas and preemption policies; and allocate the PodGroup objects to a specified scheduling queue according to the queue declaration in the training task definition file. The priority value represented by the priorityClassName field of the PodGroup is used to sort all PodGroup objects to be scheduled using a preset scheduler. For each sorted PodGroup object, the scheduler checks whether there are node resources in the cluster that can run the number of Pods required by the minMember field of the corresponding PodGroup object. If there are node resources that meet the conditions, all Pods in the PodGroup object are scheduled to the corresponding node and started running. If there are no node resources that meet the conditions, a preemption process is triggered for the high-priority PodGroup object, and one or more Pods in the low-priority PodGroup object are selected to be evicted to release resources. The status of the evicted Pods is monitored, and the evicted Pods are automatically rescheduled according to the pre-configured restart policy.

[0078] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for IoT devices and media are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0079] The systems, media, and methods provided in this application are one-to-one correspondences. Therefore, the systems and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the systems and media will not be repeated here.

[0080] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0081] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0082] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0083] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0084] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0085] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0086] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0087] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0088] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A training method for a multi-level model, characterized by, The method comprises: receiving a user-submitted training task definition file; wherein the training task definition file declares multiple training stages, resource requirements of each training stage, and priority information contained in the training task in YAML format; parsing the training task definition file to generate Pods corresponding to each training stage, and creating PodGroup objects to associate and manage multiple Pods belonging to the same training stage; wherein the PodGroup object declares a minimum number of replicas minMember field and a priority priorityClassName field; According to the queue declaration in the training task definition file, the PodGroup object is assigned to the specified scheduling queue; wherein the scheduling queue is pre-configured with resource quota and preemption strategy; According to the priority value represented by the priorityClassName field of the PodGroup object, all the PodGroup objects to be scheduled are sorted by using the preset scheduler; For each PodGroup object after sorting, the scheduler is used to check whether there is node resource in the cluster that can simultaneously run the number of Pods required by the minMember field of the corresponding PodGroup object; If there is node resource that meets the condition, all Pods in the PodGroup object are scheduled to the corresponding node and started to run; If there is no node resource that meets the condition, the preemption process is triggered for the PodGroup object with high priority, and the Pods in one or more PodGroup objects with low priority are selected to be evicted to release resources; Monitor the state of the evicted Pods, and automatically reschedule the evicted Pods according to the preconfigured restart strategy. 2.The training method for a multi-level model of claim 1, wherein, Parsing the training task definition file to generate Pods corresponding to each training stage, and creating PodGroup objects to associate and manage multiple Pods belonging to the same training stage, specifically comprising: Identify multiple training stages defined in the training task definition file; wherein each training stage corresponds to a subtask; Generate an independent Kubernetes Job object or define a Task entry in a VolcanoJob object for each training stage; Create a PodGroup object for each training stage, and associate all Pods corresponding to the training stage with the PodGroup object. 3.The training method for a multi-level model of claim 1, wherein, According to the queue declaration in the training task definition file, the PodGroup object is assigned to the specified scheduling queue, specifically comprising: Read the queue name specified in the PodGroup object; Associate the PodGroup object with a LocalQueue custom resource object; Associate the LocalQueue object with a ClusterQueue custom resource object which is pre-defined with total resource amount and preemption rules. 4.The training method for a multi-level model of claim 1, wherein, trigger a preemption process for the high-priority PodGroup object, and select a pod in one or more low-priority PodGroup objects to be evicted to release resources, specifically including: identifying a high-priority PodGroup that cannot be scheduled by the scheduler, and selecting a node in the cluster that meets the resource requirements of the high-priority PodGroup; calculating the amount of resources to be released to meet the scheduling requirements of the high-priority PodGroup, and selecting one or more low-priority pods that occupy the required resources from the selected node; wherein the priority value of the selected pods is lower than the priority value of the high-priority PodGroup; sending an eviction instruction to the selected pods and setting an eviction termination time for them. 5.The training method for a multi-level model of claim 4, wherein, The method further comprises: fine-grained preemption for GPU resources, specifically including: dividing physical GPU resources into multiple virtual GPU instances through a GPU virtualization plug-in; monitoring the / vgpu-memory and / vgpu-number resource states of the node using the scheduler; when GPU resources need to be preempted for a high-priority PodGroup, using the scheduler to select low-priority pods that occupy the required vGPU memory or vGPU number to be evicted; the high-priority Pod is scheduled to exclusively use the released vGPU instance. 6.The training method for a multi-level model of claim 1, wherein, The method further comprises: monitoring the running state of all pods in the current training stage; when it is detected that all pods in the current training stage have successfully completed running, triggering the creation and scheduling of the PodGroup object of the next training stage; the PodGroup object of the next training stage is created and added to the scheduling queue, waiting for scheduling. 7.The training method for a multi-level model of claim 1, wherein, automatically rescheduling the evicted pods according to the preconfigured restart policy, specifically including: detecting a pod evicted event; querying the policies field defined in the VolcanoJob object to which the evicted pod belongs; resetting the entire Job state to Pending according to the RestartJob policy configured in the policies field for the PodEvicted event; resubmitting the PodGroup object corresponding to the Job to the scheduling queue for rescheduling. 8.The training method for a multi-level model of claim 1, wherein, The method further comprises: receiving a priority change instruction issued by a user or system, and identifying the PodGroup object corresponding to the running task that needs to change priority based on the priority change instruction; terminating all pods associated with the PodGroup object; recreating the PodGroup object using a new priorityClassName field value; resubmitting the newly created PodGroup object to the scheduling queue.

9. A training device for a multi-level model, the training device comprising: The device comprises: at least one processor; and a memory connected in communication with the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the training method for the multi-level model according to any one of claims 1-8.

10. A non-transitory computer storage medium storing computer-executable instructions, the computer-executable instructions comprising instructions for: receiving a request to access a file; determining whether the file is stored in a cache; and in response to determining that the file is stored in the cache, retrieving the file from the cache. The computer executable instructions, when executed, implement the training method for the multi-level model according to any one of claims 1-8.

Images