Resource scheduling method and device, electronic equipment, storage medium and program product

Abstract

Embodiments of the present disclosure provide a resource scheduling method and device, electronic equipment, storage medium and program product. The method comprises: determining a to-be-processed task; in response to the to-be-processed task being an online inference task and the idle resources of a federal cluster not meeting the starting condition of the online inference task, obtaining the priority of the online inference task; in response to the priority of the online inference task being higher than the priority of a target task, performing a preemption operation on the resources currently occupied by the target task; in response to the to-be-processed task being an offline task, obtaining the scheduling step length of the offline task; and performing cyclic expansion according to the scheduling step length of the offline task until the amount of resources occupied by the offline task reaches a target value. The present scheme can improve resource utilization and resource scheduling flexibility.

Description

Technical Field

[0001] This disclosure relates to the field of cluster management technology, and more specifically, to a resource scheduling method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] With the rapid growth of AI (Artificial Intelligence) services such as large model training, online inference, and offline data refresh, GPU (Graphics Processing Unit) clusters have gradually evolved from single training clusters or single inference clusters into integrated resource pools that combine training, inference, and offline data refresh.

[0003] Different types of tasks have different resource requirements. Therefore, there is an urgent need for a dynamic resource scheduling method in an integrated resource pool for hybrid deployments.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] This disclosure provides a resource scheduling method, apparatus, electronic device, storage medium, and program product to improve resource utilization and resource scheduling flexibility.

[0006] According to a first aspect of the present disclosure, a resource scheduling method is provided, applied to a federated scheduler, comprising: determining a task to be processed; in response to the task to be processed being an online inference task and the idle resources of the federated cluster not meeting the start conditions of the online inference task, obtaining the priority of the online inference task; in response to the priority of the online inference task being higher than the priority of a target task, performing a preemption operation on the resources currently occupied by the target task; in response to the task to be processed being an offline task, obtaining the scheduling step size of the offline task; and performing cyclic expansion according to the scheduling step size of the offline task until the amount of resources occupied by the offline task reaches a target value.

[0007] In some embodiments, the offline task includes an offline training task. Performing cyclic expansion based on the scheduling step size of the offline task until the resource usage of the offline task reaches a target value includes: determining the current number of replicas based on the amount of idle resources in the federated cluster and the scheduling step size of the offline training task; wherein the current number of replicas and the scheduling step size of the offline training task satisfy a multiple constraint; scheduling the resources corresponding to the current number of replicas to the offline training task; if the current number of replicas is less than the target value, in response to a resource release message, returning to the step of determining the current number of replicas based on the amount of idle resources in the federated cluster and the scheduling step size of the offline training task, until the resource usage of the offline task reaches the target value.

[0008] In some embodiments, determining the current number of replicas based on the amount of idle resources in the federated cluster and the scheduling step size of the offline training task includes: for each candidate sub-cluster of the federated cluster, determining the number of sub-replicas that satisfy the multiple constraint of the scheduling step size based on the amount of idle resources in the candidate sub-cluster; and summing the number of sub-replicas in each candidate sub-cluster to obtain the current number of replicas.

[0009] In some embodiments, scheduling the resources corresponding to the current number of replicas to the offline training task includes: scheduling the resources corresponding to the current number of replicas to the offline training task in response to the current number of replicas meeting the start conditions of the offline training task; the method further includes: enqueuing the offline training task into a backoff queue in response to the current number of replicas not meeting the start conditions of the offline training task.

[0010] In some embodiments, scheduling the resources corresponding to the current number of replicas to the offline training task includes: scheduling the resources corresponding to the number of sub-replicas in the candidate sub-clusters to the offline training task in descending order of the number of sub-replicas.

[0011] In some embodiments, the offline task includes an offline data refresh task; performing cyclic expansion according to the scheduling step size of the offline task until the amount of resources occupied by the offline task reaches the target value includes: obtaining the minimum number of replicas and the maximum number of replicas corresponding to the offline data refresh task; obtaining the amount of idle resources in the federated cluster; in response to the idle resources in the federated cluster being greater than the minimum number of replicas and less than the maximum number of replicas, determining the resources to be allocated according to at least one of the idle resources in the federated cluster, the resource pool, the GPU type, the label constraint, and the scheduling step size of the offline data refresh task; and scheduling the resources to be allocated to the offline data refresh task.

[0012] In some embodiments, the method further includes: in response to the amount of idle resources in the federated cluster being less than the minimum number of replicas, enqueuing the offline data refresh task into a backoff queue; and in response to the amount of idle resources in the federated cluster being greater than the maximum number of replicas or greater than the current remaining demand, scheduling the resources corresponding to the maximum number of replicas or the current remaining demand to the offline data refresh task.

[0013] In some embodiments, in response to the online inference task having a higher priority than the target task, performing a preemption operation on the resources currently occupied by the target task includes: obtaining the available resource quantity of each candidate sub-cluster, the available resource quantity including idle resource quantity and preemptible resource quantity; selecting a target sub-cluster from multiple candidate sub-clusters based on the available resource quantity; sending a scheduling instruction to the local scheduler of the target sub-cluster, the scheduling instruction being used to instruct the local scheduler to select a target node from multiple candidate nodes of the target sub-cluster, and performing a preemption operation on the resources occupied by the target task in the target node.

[0014] In some embodiments, selecting a target sub-cluster from multiple candidate sub-clusters based on available resources includes: determining whether the available resources of each candidate sub-cluster meet the resource requirements of the online inference task; in response to the availability of resources of multiple candidate sub-clusters meeting the resource requirements of the online inference task, determining the preemption cost of each candidate sub-cluster; and selecting the candidate sub-cluster with the lowest preemption cost as the target sub-cluster.

[0015] In some embodiments, the scheduling instruction is used to instruct the local scheduler to determine the target preempted Pod (container group) set corresponding to each candidate node, and to determine the candidate node corresponding to the target preempted Pod set as the target node; the target preempted Pod set includes: the Pod set with the fewest preempted Pods, or the Pod set with the lowest priority preempted Pods.

[0016] In some embodiments, the scheduling instruction is further used to instruct the local scheduler to record the correspondence between the target preempted Pod set and the Pods deploying online inference tasks on the target node, and to schedule the Pods deploying online inference tasks to the target node in response to the resource release event corresponding to the target preempted Pod set.

[0017] According to a second aspect of the present disclosure, a resource scheduling apparatus is provided, configured in a federated scheduler, comprising: a task determination module for determining a task to be processed; a priority determination module for obtaining the priority of an online inference task in response to the task being an online inference task and the idle resources of the federated cluster not meeting the start conditions of the online inference task; a resource preemption module for performing a preemption operation on the resources currently occupied by the target task in response to the priority of the online inference task being higher than the priority of the target task; a scheduling step size determination module for obtaining the scheduling step size of an offline task in response to the task being an offline task; and a resource scheduling module for performing cyclic expansion according to the scheduling step size of the offline task until the amount of resources occupied by the offline task reaches a target value.

[0018] According to a third aspect of the present disclosure, an electronic device is provided, characterized in that it includes: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the executable instructions to implement the resource scheduling method of any of the above.

[0019] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform any of the resource scheduling methods or any of the video quality assessment methods described above.

[0020] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that is executed by a processor using any of the above-described resource scheduling methods.

[0021] The resource scheduling method provided in this disclosure is applied to a federated scheduler. In response to the fact that the task to be processed is an online inference task and the idle resources of the federated cluster do not meet the startup conditions of the online inference task, the priority of the online inference task is obtained. In response to the fact that the priority of the online inference task is higher than the priority of the target task, a preemption operation is performed on the resources currently occupied by the target task to prevent high-priority tasks from being blocked by low-priority tasks for a long time, thereby improving the resource guarantee capability of high-priority tasks. In response to the fact that the task to be processed is an offline task, the scheduling step size of the offline task is obtained. A cyclic expansion is performed according to the scheduling step size of the offline task until the amount of resources occupied by the offline task reaches a target value, so that the GPU released by the online inference task can be absorbed by the offline task in a timely manner, avoiding GPU idleness and improving resource utilization.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0024] Figure 1 This is a flowchart illustrating a resource scheduling method according to an exemplary embodiment.

[0025] Figure 2 This is a flowchart illustrating a resource scheduling method for an online inference task according to an exemplary embodiment.

[0026] Figure 3 This is a flowchart illustrating another resource scheduling method for an online inference task according to an exemplary embodiment.

[0027] Figure 4 This is an application flowchart illustrating an exemplary resource preemption method according to an exemplary embodiment.

[0028] Figure 5 This is a flowchart illustrating a resource scheduling method for an offline training task according to an exemplary embodiment.

[0029] Figure 6 This is a flowchart illustrating another resource scheduling method for an offline training task according to an exemplary embodiment.

[0030] Figure 7This is a flowchart illustrating a resource scheduling method for an offline data refresh task according to an exemplary embodiment.

[0031] Figure 8 This is a flowchart illustrating another resource scheduling method for an offline data refresh task according to an exemplary embodiment.

[0032] Figure 9 This is a comparison chart of resource scheduling effects shown according to an exemplary embodiment.

[0033] Figure 10 This is a block diagram illustrating a resource scheduling apparatus according to an exemplary embodiment.

[0034] Figure 11 This is a schematic diagram illustrating the structure of an electronic device suitable for implementing exemplary embodiments of the present disclosure, according to an exemplary embodiment. Detailed Implementation

[0035] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0036] The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0037] The accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus omitting repeated descriptions of them. Some block diagrams shown in the drawings do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in at least one hardware module or integrated circuit, or in different network and / or processor devices and / or microcontroller devices.

[0038] The flowchart shown in the accompanying drawings is merely illustrative and does not necessarily include all content and steps, nor does it require execution in the described order. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0039] In this specification, the terms “a,” “an,” “the,” “the,” and “at least one” are used to indicate the presence of at least one element / component / etc.; the terms “comprising,” “including,” and “having” are used to indicate an open-ended inclusion and to mean that there may be other elements / components / etc. in addition to the listed elements / components / etc.; the terms “first,” “second,” and “third,” etc., are used only as markings and are not a limitation on the number of objects.

[0040] First, the terms used in this disclosure will be explained.

[0041] GPU: Accelerator card resources used for AI training, online inference, and offline data refresh tasks.

[0042] SLO (Service Level Objective): Service assurance requirements for online inference services regarding latency, availability, throughput, and other metrics.

[0043] Pod: The basic scheduling unit in Kubernetes (an open-source container orchestration system). A Pod can contain one or more containers.

[0044] Pipeline Parallelism: A common parallel method used in training large models, which divides the model training task into multiple stages (blocks), and the different stages are executed in a pipeline manner.

[0045] For integrated resource pools with hybrid deployments, online inference tasks typically have strict requirements for latency and availability, and their resource supply needs to be prioritized during peak business periods; offline training tasks and offline data refresh tasks are relatively less sensitive to latency, but have higher requirements for throughput and resource utilization.

[0046] In real-world production environments, online inference services exhibit distinct diurnal tidal characteristics. During the day, traffic increases, leading to expansion of the online inference service and the consumption of more GPU resources, thus compressing the resources available for offline tasks. At night, traffic decreases, releasing a large amount of GPU resources from the online inference service, resulting in a low-tidal resource flow back to the cluster. If the scheduling system cannot handle these tidal changes in a timely manner, two problems will arise: First, during periods of increased traffic, high-priority online tasks may be blocked by low-priority offline tasks, leading to excessively long queuing times and increased latency for online tasks. Second, during periods of decreased traffic, the GPU resources released by online inference cannot be absorbed by offline tasks in a timely manner, resulting in idle GPU resources and low overall utilization.

[0047] Federated cluster scheduling systems typically employ a two-tier scheduling structure. The federated scheduler maintains a global resource view and is responsible for allocating tasks to one or more sub-clusters; the local scheduler of each sub-cluster is responsible for scheduling specific Pods onto nodes. After a task is submitted, the upper-layer platform generates a federated task object. The federated scheduler selects a target sub-cluster based on factors such as the amount of idle resources in each sub-cluster. Then, the federated scheduler creates Work objects in the target sub-cluster, and finally, the local scheduler completes the node-level scheduling.

[0048] In the above scheme, the federated scheduler typically determines whether a task is schedulable based on currently available idle resources. When a high-priority online task arrives, if there are insufficient idle resources, the task enters a queuing or retry state. Even if a low-priority offline task is occupying reclaimable GPU resources, the federated scheduler will not actively consider these resources as reclaimable resources and use them for the feasibility assessment of high-priority tasks.

[0049] Some scheduling systems support priority and preemption, but preemption typically occurs during the single-cluster or single-pod scheduling phase, lacking a unified judgment at the federation layer regarding preemptible resources, candidate sub-clusters, and preemption costs. The federation layer cannot accurately determine whether preemption should occur, in which sub-clusters, and how much resource to preempt; the sub-cluster layer also lacks an event-driven wake-up mechanism for federated tasks, which can easily lead to problems such as tasks waiting for a long time after preemption, repeated preemption, or excessive preemption costs.

[0050] Furthermore, large model training tasks often employ pipelined parallelism. Such tasks cannot be effectively scaled up simply by adding a single GPU card; instead, resource allocation must be done in integer multiples of the pipeline stage width or batch size. For example, a training task with a stage width of 8 can effectively utilize 8, 16, or 24 GPU cards, but cannot effectively utilize 10 GPU cards. If the scheduler allocates 10 GPU cards to this training task, two of them may not be able to form a complete pipeline block, rendering them unusable for the training task—meaning they are idle.

[0051] Schedulers typically focus on whether the amount of idle resources meets the task startup conditions, rather than using batch size or pipeline stage width as scheduling semantics. Even if the training framework is aware of this constraint, the cluster scheduler still allocates resources based on the number of GPUs.

[0052] The number of GPUs allocated by the scheduler may not meet the effective granularity of pipeline parallel training, resulting in allocated but unusable GPU fragments. When the training framework handles batch size constraints, the federated scheduler has already completed resource allocation and cannot reserve unusable GPU fragments for other tasks from the perspective of the entire cluster. In large-scale federated GPU clusters, events such as node failures, online inference reclamation, and offline task completion frequently generate non-integer multiples of GPU resources. If the federated scheduler is unaware of step size constraints, it will further amplify fragmentation and redundant reservations.

[0053] To address at least one of the aforementioned technical problems, this disclosure provides a resource scheduling method applied to a federated scheduler. In response to the pending task being an online inference task, and the idle resources of the federated cluster not meeting the startup conditions for the online inference task, the method obtains the priority of the online inference task. In response to the online inference task having a higher priority than the target task, a preemption operation is performed on the resources currently occupied by the target task to prevent high-priority tasks from being blocked by low-priority tasks for extended periods, thereby improving the resource guarantee capability of high-priority tasks. In response to the pending task being an offline task, the method obtains the scheduling step size of the offline task. A cyclic expansion is performed based on the scheduling step size of the offline task until the amount of resources occupied by the offline task reaches a target value, ensuring that the GPU released by the online inference task can be absorbed by the offline task in a timely manner, preventing GPU idleness and improving resource utilization.

[0054] The steps of the resource scheduling method in the exemplary embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings and examples. The method provided in the embodiments of this disclosure can be executed by a federated scheduler located at the federation layer in a federated cluster.

[0055] Figure 1 This is a flowchart illustrating a resource scheduling method according to an exemplary embodiment. For example... Figure 1 As shown, the resource scheduling method provided in this embodiment mainly includes steps S102-S110.

[0056] In S102, the tasks to be processed are identified.

[0057] The federated cluster consists of multiple sub-clusters, each containing multiple GPU nodes. Each sub-cluster has its own local scheduler. The federated cluster uses a global scheduler (i.e., the federated scheduler) to manage the resources of each sub-cluster in a unified manner and schedule them across clusters.

[0058] Tasks pending processing refer to tasks that have been submitted to the federated cluster and are awaiting resource allocation. Depending on the task type, tasks pending processing can include, but are not limited to: online inference tasks, offline training tasks, and offline data refresh tasks.

[0059] In AI application scenarios, determining tasks to be processed includes: receiving task requests submitted by users to the federated cluster through a client, and designating the tasks corresponding to the task requests as tasks to be processed.

[0060] In one possible implementation, the task configuration information of the pending tasks submitted by the upper-layer platform to the federated cluster is obtained. For example, this can be obtained through a FedJob instance. The task configuration information is used to declare the task's runtime parameters.

[0061] Specifically, task configuration information includes, but is not limited to: task resource requirements, priority, task type, minimum number of replicas, maximum number of replicas, scheduling step size, etc. Task resource requirements can refer to the number and specifications of GPU instances required for task execution; priority characterizes the task's scheduling priority level in the federated cluster, used to determine the order of preemption or being preempted; task type indicates whether the task to be processed is an online inference task, an offline training task, or an offline data refresh task; minimum number of replicas indicates the minimum number of replicas required for task startup, i.e., the task is allowed to start when idle or allocable resources reach the minimum number of replicas; maximum number of replicas indicates the maximum elastic number of replicas for the task, i.e., the upper limit of the number of replicas that the task is allowed to expand to during operation; scheduling step size indicates the task's scheduling step size, i.e., the number of replicas added in a single expansion operation for an offline task.

[0062] In S104, in response to the fact that the task to be processed is an online inference task and the idle resources of the federated cluster do not meet the start conditions of the online inference task, the priority of the online inference task is obtained.

[0063] Online inference tasks can refer to real-time inference services for users, including but not limited to: online image recognition, online dialogue response, etc. For example, in a hybrid deployment of an integrated resource pool, online inference tasks have the highest priority.

[0064] In a federated cluster, idle resources can refer to resources that are currently idle and not running tasks. These resources include, but are not limited to, GPU cards, CPU cores, etc. In this embodiment, GPU cards are used as an example for illustration.

[0065] The idle resources of the federated cluster do not meet the start conditions for the online inference task, including: the number of idle GPUs in the federated cluster is less than the minimum number of replicas indicated in the task configuration information of the online inference task.

[0066] When an online inference task submits a task request, it declares its resource requirements in the task configuration information, for example, it requires 8 GPU cards. The federated scheduler continuously monitors the number of idle GPU cards in the federated cluster, which is always 6. If the number of idle GPU cards is less than the minimum number of replicas required by the online inference task, it is determined that the start condition for the online inference task has not been met, triggering a preemption process to acquire the priority of the online inference task.

[0067] Obtaining the priority of an online inference task includes reading its priority from the task configuration information corresponding to the online inference task.

[0068] In S106, in response to the fact that the priority of the online inference task is higher than that of the target task, a preemption operation is performed on the resources currently occupied by the target task.

[0069] A target task can refer to one or more tasks that are currently running and consuming GPU resources. In a hybrid deployment of a unified resource pool, target tasks have a lower priority than online inference tasks. The resources currently consumed by a target task can refer to the GPU resources that the target task has acquired and continues to consume during execution.

[0070] In one possible implementation, the federated scheduler determines that the priority of the online inference task is higher than that of the target task and sends a scheduling instruction to the local scheduler of the sub-cluster where the target task resides. The scheduling instruction carries the task identifier and the amount of resources required by the online inference task. In response to the scheduling instruction, the local scheduler releases the resources occupied by the target task and allocates the released resources to the online inference task.

[0071] During periods of high online inference traffic (i.e., the peak period), the idle GPU resources in the federated cluster are insufficient to meet the startup requirements of newly submitted online inference tasks. Low-priority offline tasks, which are not sensitive to latency, may be running within the federated cluster. The federated scheduler can instruct the release of GPU resources occupied by these low-priority offline tasks and reallocate the released GPU resources to online inference tasks. This resolves the issue of high-priority online inference tasks being blocked by low-priority offline tasks during the peak period, enabling high-priority online inference tasks to quickly obtain scheduling opportunities based on reclaimable resources and improving resource availability for high-priority online inference tasks during the peak period.

[0072] In S108, in response to the task to be processed being an offline task, the scheduling step size of the offline task is obtained.

[0073] Offline tasks refer to computational tasks that are insensitive to response latency and can tolerate longer waiting times and execution interruptions. In AI scenarios, offline tasks include, but are not limited to: offline training tasks (e.g., pre-training and fine-tuning of large models) and offline data refresh tasks (e.g., feature engineering, data cleaning, and index building). It should be noted that offline tasks typically have a lower priority than online inference tasks.

[0074] The scheduling step size can refer to the number of replicas added in a single scaling operation, i.e., the number of GPUs added during each round of scaling for an offline task. The scheduling step size can be declared through the batch size parameter in the task configuration information when submitting an offline task. Furthermore, batch size can include, but is not limited to: pipeline stage width, gang size, tensor parallel size, data parallelgroup size, number of GPUs per node, rack-level topology groups, or the effective parallel granularity declared by other training frameworks.

[0075] Determining the scheduling step size for an offline task may include: reading the batch size parameter from the task configuration information corresponding to the offline task and using the batch size parameter as the scheduling step size for the offline task.

[0076] For example, the scheduling step size is 1 for offline data refresh tasks; for pipeline parallel training tasks, the scheduling step size is the width of the parallel segment or the minimum effective scaling granularity required by the training framework.

[0077] In S110, a cyclic expansion is performed based on the scheduling step size of the offline tasks until the amount of resources occupied by the offline tasks reaches the target value.

[0078] The target value can refer to the maximum number of copies required for an offline task.

[0079] The federated scheduler selects a number of idle resources from the federated cluster that are multiples of the scheduling step size, according to the scheduling step size, and allocates the selected idle resources to offline tasks. It then determines whether the number of replicas currently allocated to the offline tasks has reached the maximum number of replicas. If not, the federated scheduler monitors for resource release events in the federated cluster. Upon detecting a resource release event, it returns to the above steps of scaling up according to the scheduling step size until the number of replicas allocated to the offline tasks reaches the maximum number of replicas.

[0080] Furthermore, after completing this round of expansion, the federated scheduler recalculates the difference between the number of replicas already allocated to offline tasks and the maximum number of replicas. If the difference is less than the scheduling step size, the difference is used as the number of replicas for the next round of expansion. If the difference is greater than the scheduling step size, the scheduling step size is used as the number of replicas for the next round of expansion.

[0081] When the federated scheduler determines that a task to be processed is an offline task, resource preemption is not required because offline tasks are not sensitive to response latency. A cyclic expansion strategy based on scheduling steps is adopted for offline tasks, that is, the number of replicas of the offline task is gradually increased by the scheduling step size until its maximum number of replicas is reached. By cyclically expanding offline tasks, GPUs released from online inference tasks can be quickly converted into the computing throughput of offline tasks, rather than remaining idle for a long time, thus improving the efficiency of GPU reclamation during low-peak periods.

[0082] In some embodiments, "in response to the fact that the priority of the online inference task is higher than that of the target task, performing a preemption operation on the resources currently occupied by the target task" includes the following steps S202-S206.

[0083] In S202, the available resources of each candidate sub-cluster are obtained. The available resources include idle resources and preemptible resources.

[0084] Candidate subclusters can refer to multiple subclusters in a federated cluster that meet the basic scheduling conditions of the task to be processed. For example, a candidate subcluster is a subcluster in the federated cluster whose GPU model is the same as the GPU model required by the task to be processed.

[0085] Available resources refer to the total amount of resources that a candidate sub-cluster can currently allocate to pending tasks, including idle resources and preemptible resources. For example, resources can refer to the number of GPU cards. Idle resources can refer to the number of GPU cards in a candidate sub-cluster that are completely idle, while preemptible resources can refer to the number of GPU cards in a candidate sub-cluster that are currently occupied by low-priority tasks but are allowed to be preempted by higher-priority online inference tasks.

[0086] When increased online inference traffic leads to a decrease in the number of idle GPUs within the federated cluster, high-priority online inference tasks cannot be launched using idle GPUs. In this situation, the federated scheduler calculates the idle resource amount R_idle and the resource amount R_low occupied by low-priority offline tasks and allowed to be reclaimed for each candidate sub-cluster. The sum of the idle resource amount R_idle and the preemptible resource amount R_low for each candidate sub-cluster is taken as the available resource amount for that candidate sub-cluster.

[0087] In one possible implementation, each candidate sub-cluster independently calculates the sum of its idle resource quantity R_idle and preemptible resource quantity R_low, which serves as the available resource quantity for that candidate sub-cluster. Each candidate sub-cluster then sends its calculated available resource quantity to the federated scheduler, which makes scheduling decisions based on the available resource quantity. This scheme can reduce the federated layer's reliance on low-priority Pod details.

[0088] In S204, the target sub-cluster is selected from multiple candidate sub-clusters based on the amount of available resources.

[0089] The target sub-cluster can refer to the selected sub-cluster used to deploy the tasks to be processed.

[0090] In one possible implementation, the federated scheduler uses the candidate sub-cluster corresponding to the maximum idle resource amount R_idle as the target sub-cluster, or uses the candidate sub-cluster corresponding to the maximum available resource amount as the target sub-cluster.

[0091] In some embodiments, “selecting a target sub-cluster from multiple candidate sub-clusters based on available resources” includes the following steps S302-S306.

[0092] In S302, it is determined whether the available resources of each candidate sub-cluster meet the startup conditions of the online inference task.

[0093] The federated scheduler obtains the available resources of each candidate sub-cluster and compares the available resources of each candidate sub-cluster with the minimum number of replicas for the online inference task. If the available resources of a candidate sub-cluster are greater than or equal to the minimum number of replicas, the candidate sub-cluster is determined to meet the start conditions for the online inference task. If the available resources of a candidate sub-cluster are less than the minimum number of replicas, the candidate sub-cluster is determined to not meet the start conditions for the online inference task and is excluded from the list of candidate sub-clusters.

[0094] In S304, in response to the availability of resources in multiple candidate sub-clusters satisfying the launch conditions of the online inference task, the preemption cost of each candidate sub-cluster is determined.

[0095] If the available resources of multiple candidate sub-clusters all meet the minimum number of replicas required for the online inference task, it indicates that the online inference task is feasible for preemptive scheduling. Therefore, the preemption cost of each candidate sub-cluster that meets the start conditions of the online inference task can be determined.

[0096] In one possible implementation, the preemption cost of a candidate sub-cluster is determined based on one or more of the following cost factors: the expected number of Pods to be preempted, the priority of the target task, the runtime of the target task, the business type of the target task, and the resource pool attributes to which the candidate sub-cluster belongs.

[0097] The expected number of preempted Pods is the number of Pod instances that need to be terminated or suspended in the candidate sub-cluster to meet the minimum number of replicas required for online inference tasks. The preemption cost is positively correlated with the number of preempted Pods; the fewer the expected number of preempted Pods, the lower the preemption cost.

[0098] The cost of preemption is positively correlated with the priority of the target task. The lower the priority of the target task, the lower the requirement for scheduling timeliness and the smaller the cost of preemption.

[0099] The runtime of the target task represents the cumulative running time of the target task from its start to the current moment. The shorter the runtime, the less computing power is wasted due to preemption, and the lower the cost of preemption.

[0100] The impact of preemption on business continuity varies depending on the type of business task. The federated scheduler determines the corresponding preemption impact coefficient based on the business type.

[0101] In S306, the candidate sub-cluster with the lowest preemption cost is selected as the target sub-cluster.

[0102] In one possible implementation, the federated scheduler calculates the preemption cost score of each candidate sub-cluster based on the aforementioned multiple cost factors and their respective weight coefficients, and selects the candidate sub-cluster with the lowest preemption cost score as the target sub-cluster.

[0103] In this embodiment, the reclaimable resources occupied by low-priority tasks are included in the schedulability judgment of high-priority tasks, and the sub-cluster with the lowest preemption cost is selected globally to reduce the impact on low-priority services.

[0104] In S206, a scheduling instruction is sent to the local scheduler of the target sub-cluster. The scheduling instruction is used to instruct the local scheduler to select the target node from multiple candidate nodes in the target sub-cluster and to perform a preemption operation on the resources occupied by the target task in the target node.

[0105] A local scheduler can refer to a scheduler deployed within a sub-cluster, used to receive scheduling instructions issued by the federated scheduler and perform resource allocation and task scheduling operations within the sub-cluster. Multiple local schedulers in sub-clusters communicate with the federated scheduler, forming a two-tier scheduling architecture.

[0106] A scheduling instruction can be a control instruction used to trigger the local scheduler to perform local scheduling operations. A scheduling instruction carries at least one of the following information: the task identifier of the task to be processed, resource requirements, priority information, and preemption indication. In one possible implementation, the scheduling instruction is sent to the local scheduler in the form of Placement resources and Work resources.

[0107] Candidate nodes can refer to computing devices within the target sub-cluster, with each candidate node configured with one or more GPU cards. Target nodes can refer to the computing nodes selected by the local scheduler from the candidate nodes for deploying online inference tasks.

[0108] After the federated scheduler completes the selection of the target sub-clusters at the federation level, it sends scheduling instructions to the local scheduler of the target sub-cluster, which then executes the specific resource allocation and preemption operations. This two-level collaborative scheduling mechanism can balance global decision-making with local execution accuracy.

[0109] In some embodiments, the scheduling instruction is used to instruct the local scheduler to determine the target preempted Pod (container group) set corresponding to each candidate node, and to determine the candidate node corresponding to the target preempted Pod set as the target node; the target preempted Pod set includes: the Pod set with the fewest preempted Pods, or the Pod set with the lowest priority preempted Pods.

[0110] In this embodiment, the online inference task is encapsulated in a Pod as a high-priority Pod. After receiving the scheduling instruction, the local scheduler in the target sub-cluster cluster reads the high-priority Pod from the scheduling instruction. The local scheduler first enters the normal scheduling process, which includes the pre-filtering, filtering, scoring, and binding stages.

[0111] If the normal scheduling process returns "NoFeasibleNodes", and a high-priority Pod meets the preset preemption trigger conditions, then the high-priority Pod is marked as a preemptor (PreemptCandidate) and enters the preemption processor (PreemptProcessor) to execute the preemption processing flow. The preemption trigger conditions include: the preemption feature gate is enabled, the task is not of the online strong guarantee type, and the scheduling failure is due to the sub-cluster's idle resources not meeting the startup conditions for the online inference task.

[0112] In one possible implementation, the process of the local scheduler performing preemption operations is explained.

[0113] First, conduct a pre-emptive check.

[0114] Before executing a preemption operation, the local scheduler first performs a preemption pre-check. The preemption pre-check includes, but is not limited to: determining whether a preemption operation is still necessary, determining whether the online inference task still meets the preemption triggering conditions, and determining whether the current time is within the preemption suppression window. The preemption suppression window is used to prevent the same task from frequently triggering preemption operations within a short period of time.

[0115] If any of the above conditions are not met, the preemption process will be terminated; if the preemption pre-check passes, the preemption operation will be executed.

[0116] Second, execute the simulated preemption and filtering logic on the candidate nodes.

[0117] The local scheduler simulates the removal of one or more low-priority Pods on candidate nodes and re-executes the filtering logic after removal to determine which low-priority Pods can satisfy the resource requirements of high-priority Pods after removal, so as to determine the target preempted Pod set, that is, the target preempted Pod set includes one or more Pods that can satisfy online inference tasks after being preempted.

[0118] Third, calculate the target Pod set to preempt.

[0119] The local scheduler calculates the target preemption set for each candidate node. The target preemption set includes at least one of the following: the set of Pods with the fewest preempted Pods; if the number of preempted Pods is the same, the set of Pods with lower preemption priority is selected. If the preemption priority is still the same, at least one of the following factors is further compared: resource release amount, business impact, and node health.

[0120] In some embodiments, the scheduling instruction is further used to instruct the local scheduler to record the correspondence between the target preempted Pod set and the Pods deploying online inference tasks on the target node, and to schedule the Pods deploying online inference tasks to the target node in response to the resource release event corresponding to the target preempted Pod set.

[0121] In one possible implementation, the preemption operation performed by the local scheduler also includes the following steps: Fourth, perform the expulsion operation and record the mapping relationship.

[0122] The local scheduler releases the resources occupied by the target preempted Pod set or deletes the target preempted Pod set, and records the mapping relationship between the preemptor and the preempted in the PreemptVictimTracker.

[0123] Fifth, listen for resource release events and trigger retry scheduling.

[0124] When the Informer detects that a preempted Pod has been successfully deleted or its resources have been released, the Preemption Victim Tracker finds the corresponding preemptor Pod based on the recorded mapping relationship, marks the preemptor Pod as retry-ready, and puts it back into the preemptor processor or scheduling queue so that it can quickly retry scheduling.

[0125] For example, periodic retries, fixed-interval polling, re-enqueue events in the scheduling queue, or resource level change events can be used to wake up the preemptive Pod, enabling it to quickly retry scheduling.

[0126] Sixth, clean up the mapping relationships.

[0127] After a preemptor Pod is successfully scheduled or gives up due to a timeout, the local scheduler calls the UntrackPreemptor interface to clean up the mapping between the preemptor and the preempted Pod, preventing residual state from affecting subsequent scheduling decisions.

[0128] In this embodiment, the high-priority task is woken up based on the Pod deletion event, thereby shortening the time between the release of preempted resources and the rescheduling of high-priority tasks.

[0129] One possible implementation is, such as Figure 4 As shown, the preemption operation includes the following steps: the federated scheduler executes S401-S402; the local scheduler of the target sub-cluster executes S403-S406; and the preemption processor of the target sub-cluster executes S407-S410.

[0130] In S401, Placement resources are received.

[0131] Receive the Placement resource sent down from the upper layer. This Placement resource carries the task configuration information of the task to be processed.

[0132] In S402, resource verification.

[0133] The system checks whether the target sub-cluster exists within the federated cluster. If it does, the check passes, and a scheduling instruction is sent to the local scheduler of the target sub-cluster. If it does not exist, the check fails, and the cluster enters a waiting queue.

[0134] Join the team in S403.

[0135] The local scheduler will queue online inference tasks in the sub-cluster scheduling queue.

[0136] In S404, pre-scheduling verification.

[0137] In S405, the normal scheduling process is executed.

[0138] In S406, it is determined whether the normal scheduling process was successful. If so, the scheduling process ends. If no available nodes are returned, it jumps to S407, where the preemption processor performs the preemption operation.

[0139] In S407, it is marked as a preemptive candidate cluster.

[0140] In S408, the target is selected to preempt the Pod set.

[0141] In S409, the target preempted Pod set is removed.

[0142] In S410, UntrackPreemptor is called to clear the mapping relationship.

[0143] In some embodiments, the offline task includes an offline training task, and "performing cyclic expansion according to the scheduling step size of the offline task until the amount of resources occupied by the offline task reaches the target value" includes the following steps S502-S506.

[0144] In S502, the current number of replicas is determined based on the amount of idle resources in the federated cluster and the scheduling step size of the offline training task; wherein the current number of replicas and the scheduling step size of the offline training task satisfy a multiple constraint.

[0145] The aforementioned multiple constraint refers to an integer multiple constraint, meaning that the current number of replicas is an integer multiple of the scheduling step size.

[0146] The federated scheduler obtains the number of currently idle GPUs in the federated cluster and the scheduling step size of the offline training task, which is declared by the offline training task in the task configuration information.

[0147] The federated scheduler determines the current number of replicas based on the amount of idle resources and the scheduling step size. This includes dividing the number of idle cards by the scheduling step size and rounding the quotient down to obtain the current number of replicas, which must be an integer multiple of the scheduling step size. When the offline training task is a pipeline parallel training task, the number of replicas added each time is an integer multiple of the scheduling step size, ensuring that the newly added replicas in each round can fully form one or more pipeline parallel groups, avoiding GPU resource fragmentation caused by insufficient replicas.

[0148] In some embodiments, determining the current number of replicas based on the amount of idle resources in the federated cluster and the scheduling step size of the offline training task includes: for each candidate sub-cluster of the federated cluster, determining the number of sub-replicas that satisfy the multiple constraint of the scheduling step size based on the amount of idle resources in the candidate sub-cluster; and summing the number of sub-replicas in each candidate sub-cluster to obtain the current number of replicas.

[0149] Assuming that the number of allocatable instances in the candidate sub-cluster c is Ac and the scheduling step size declared for the offline training task is B, the number of sub-replicas that can actually be used for the offline training task in the sub-cluster is as shown in formula (1).

[0150] A'c = floor(Ac / B) × B(1)

[0151] A'c represents the number of sub-replicas in candidate sub-cluster c that can be used for the offline training task, Ac represents the number of instances that can be allocated in candidate sub-cluster c, B represents the scheduling step size of the offline training task, and floor(Ac / B) represents the floor operation of the quotient of Ac / B.

[0152] The federated scheduler sums the number of sub-replicas of all candidate sub-clusters to obtain the current number of replicas, as shown in formula (2).

[0153] A'=Σ A'c(2)

[0154] Where A' represents the current number of replicas.

[0155] In S504, the resources corresponding to the current number of replicas are scheduled to the offline training task.

[0156] The federated scheduler selects an equal number of idle GPUs from the idle resources of the federated cluster, allocates the selected GPUs to the offline training task, updates the resource allocation record of the federated cluster, and updates the current number of allocated replicas of the offline training task.

[0157] In some embodiments, scheduling the resources corresponding to the current number of replicas to the offline training task includes: scheduling the resources corresponding to the number of sub-replicas in the candidate sub-clusters to the offline training task in descending order of the number of sub-replicas.

[0158] The number of replicas required for offline training tasks is distributed across multiple sub-clusters of the federated cluster. When the available replicas in a given round span multiple sub-clusters, the federated scheduler determines the number of replicas for each sub-cluster. The number of replicas in each candidate sub-cluster is used as the priority for allocation, and resource scheduling is performed sequentially in descending order of the number of replicas. This means that sub-clusters with larger numbers of replicas are prioritized for allocation to offline training tasks, ensuring that these tasks receive larger blocks of GPU resources and can begin training. Furthermore, prioritizing the allocation of larger blocks of GPU resources further reduces GPU resource fragmentation and improves resource utilization.

[0159] Optionally, after completing the resource scheduling of a candidate sub-cluster, the federated scheduler processes the resource scheduling of the next candidate sub-cluster in order of sorting, until the resources corresponding to the number of sub-replicas in all candidate sub-clusters have been scheduled or the number of allocated replicas of the offline training task has reached the maximum number of replicas.

[0160] When allocating resources across sub-clusters, large-scale resource blocks are prioritized for scheduling from candidate sub-clusters to offline training tasks, enabling offline training tasks to obtain a sufficient number of GPU resources to start training more quickly, while reducing GPU resource fragmentation.

[0161] In one optional implementation, if the number of sub-replicas of each candidate sub-cluster is the same, the federated scheduler determines the resource scheduling order of the candidate sub-clusters according to different strategies such as MostFit, BestFit, LeastAllocated, load balancing, fault domain shuffling, network topology affinity, GPU model priority, and service isolation.

[0162] In S506, if the current number of replicas is less than the target value, in response to the resource release message, the process returns to the step of determining the current number of replicas based on the amount of free resources in the federated cluster and the scheduling step size of the offline training task, until the amount of resources occupied by the offline task reaches the target value.

[0163] The federated scheduler determines whether the current number of allocated replicas for the offline training task has reached the maximum number of replicas. If the current number of allocated replicas is less than the maximum number of replicas, the federated scheduler calculates the difference between the maximum number of replicas and the current number of allocated replicas as the remaining demand and listens for resource release messages from the federated cluster.

[0164] When the federated scheduler receives the resource release message, it returns to the above steps of "determining the current number of replicas based on the amount of idle resources in the federated cluster and the scheduling step size of the offline training task", recalculates the current number of replicas, and performs the next round of resource allocation.

[0165] Through the above iterative process, the global scheduler gradually expands the number of replicas of offline training tasks from the currently allocated number to the maximum number of replicas, using the scheduling step size as the granularity.

[0166] In some embodiments, "performing cyclic expansion according to the scheduling step size of offline tasks until the amount of resources occupied by offline tasks reaches the target value" includes the following steps S602-S608.

[0167] In S602, the current number of replicas is determined based on the amount of idle resources in the federated cluster and the scheduling step size of the offline training task.

[0168] In S604, in response to the current number of replicas meeting the start conditions of the offline training task, the resources corresponding to the current number of replicas are scheduled to the offline training task.

[0169] If the current number of replicas A' is greater than or equal to the minimum number of replicas Mmin, it means that the minimum start condition for the task is met after alignment according to the batch size. The federated scheduler allocates resources from the candidate sub-clusters from the most available to the least available until the task requirements are met or the maximum number of replicas is reached.

[0170] In S606, in response to the current number of replicas not meeting the start conditions for an offline training task, the offline training task is enqueued into a backoff queue.

[0171] If the current number of replicas A' is less than the minimum number of replicas Mmin, it indicates that the alignment according to the minimum parallel granularity is insufficient to meet the minimum start condition of the task, the current round of scheduling fails and enters the backoff queue.

[0172] In S608, if the current number of replicas is less than the target value, in response to the resource release message, the process returns to the step of determining the first number of replicas based on the amount of free resources in the federated cluster and the scheduling step size of the offline training task, until the amount of resources occupied by the offline task reaches the target value.

[0173] For example, a training task declares replicas=12, minAvailable=4, and batchSize=4. If a sub-cluster currently provides 10 GPUs, a traditional scheduler might allocate all 10 GPUs, but the task can only effectively use 8, leaving the remaining 2 as unusable fragments. In this embodiment, the number of allocable replicas for the sub-cluster is rounded down to 8, and the remaining 2 GPUs are reserved for other tasks or subsequent scheduling.

[0174] In this embodiment, GPUs are allocated according to integer multiples of the batch size, ensuring that the allocation results can be actually used by the training task. This reduces GPU fragmentation in pipeline parallel training; in other words, the actual number of allocatable replicas for each sub-cluster is calculated as an integer multiple of the batch size, avoiding the occurrence of allocated but unusable GPU fragments in pipeline parallel training.

[0175] In some embodiments, offline tasks include offline data refresh tasks; "perform cyclic expansion according to the scheduling step size of offline tasks until the amount of resources occupied by offline tasks reaches the target value" includes the following steps S702-S708.

[0176] In S702, obtain the minimum and maximum number of replicas corresponding to the offline data refresh task.

[0177] In this implementation, when the task to be processed is an offline data refresh task, the minimum number of replicas Mmin and the maximum number of replicas Mmax are read from the task configuration information submitted by the upper-layer platform. The minimum number of replicas represents the minimum number of GPU cards required to start the offline data refresh task, and the maximum number of replicas represents the maximum number of GPU cards that the offline data refresh task is allowed to use.

[0178] In S704, obtain the amount of free resources in the federated cluster.

[0179] The federated scheduler sends resource status query requests to the local schedulers of each sub-cluster in the federated cluster. Each local scheduler responds to the query request and returns the number of currently idle GPU cards in its sub-cluster. The federated scheduler sums up the number of idle GPU cards returned by each sub-cluster to obtain the amount of idle resources in the federated cluster.

[0180] In one implementation, the federated scheduler directly reads the current amount of idle resources of the federated cluster from the global resource information table, which is maintained and updated by the resource status information periodically reported by each sub-cluster.

[0181] In S706, in response to the fact that the amount of idle resources in the federated cluster is greater than the minimum number of replicas and less than the maximum number of replicas, the resources to be allocated are determined based on at least one of the following: the amount of idle resources in the federated cluster, the resource pool, the GPU type, the label constraint, and the scheduling step size of the offline data refresh task.

[0182] The federated scheduler determines whether the current amount of idle resources in the federated cluster is greater than the minimum number of replicas and less than the maximum number of replicas. If so, it indicates that the amount of idle resources in the federated cluster can meet the resource requirements of some offline data refresh tasks.

[0183] In one implementation, in response to the fact that the amount of idle resources in the federated cluster is greater than the minimum number of replicas and less than the maximum number of replicas, idle GPUs in the federated cluster are filtered according to the GPU type and label constraints declared by the offline data refresh task. Idle GPUs with matching GPU type and label attributes are included in the candidate set of resources to be allocated. Alternatively, idle GPUs can be selected as resources to be allocated from the same resource pool to which the offline data refresh task belongs, in order to avoid the additional network overhead introduced by cross-resource pool resource scheduling.

[0184] In S708, resources to be allocated are scheduled to offline data refresh tasks.

[0185] The federated scheduler allocates the identified idle GPUs to the offline data refresh task, updates the resource allocation record of the federated cluster, and sends a start command to the executor corresponding to the offline data refresh task to perform the data refresh operation using the resources to be allocated.

[0186] Furthermore, after completing this resource scheduling, the federated scheduler calculates the remaining demand of the offline data refresh task. Specifically, it takes the difference between the maximum number of replicas and the number of replicas currently allocated to the offline data refresh task as the remaining demand of the offline data refresh task, and returns to execution step S704 to reacquire the idle resources of the federated cluster, and repeats the above allocation process until the number of replicas allocated to the offline data refresh task reaches the maximum number of replicas.

[0187] In some embodiments, offline tasks include offline data refresh tasks; "perform cyclic expansion according to the scheduling step size of offline tasks until the amount of resources occupied by offline tasks reaches the target value" includes the following steps S802-S808.

[0188] In S802, obtain the minimum and maximum number of replicas corresponding to the offline data refresh task.

[0189] In S804, obtain the amount of free resources in the federated cluster.

[0190] In S806, in response to the fact that the amount of idle resources in the federated cluster is greater than the minimum number of replicas and less than the maximum number of replicas, the resources to be allocated are determined based on at least one of the following: the amount of idle resources in the federated cluster, the resource pool, the GPU type, the label constraint, and the scheduling step size of the offline data refresh task.

[0191] In S808, the resources to be allocated are scheduled to the offline data refresh task.

[0192] In S810, in response to the amount of idle resources in the federated cluster being less than the minimum number of replicas, offline data refresh tasks are enqueued into a backoff queue.

[0193] The federated scheduler obtains the amount of idle resources in the federated cluster and compares it with the minimum number of replicas for the offline data refresh task. If the amount of idle resources is less than the minimum number of replicas, it means that the current idle resources of the federated cluster do not meet the minimum start conditions for the offline data refresh task. In this case, the federated scheduler adds the offline data refresh task to the backoff queue and waits for it to be retried in a subsequent scheduling cycle.

[0194] The backoff strategies mentioned above can include exponential backoff, fixed backoff, linear backoff, exponential backoff with random jitter, or dynamic adjustment of backoff time based on the rate of resource change. All of these backoff strategies can reduce scheduler pressure and avoid high-frequency failure retries.

[0195] In S812, in response to the fact that the amount of idle resources in the federated cluster is greater than the maximum number of replicas or the current remaining demand, the resource set corresponding to the maximum number of replicas or the current remaining demand is scheduled to the offline data refresh task.

[0196] The federated scheduler obtains the amount of idle resources in the federated cluster and compares it with the maximum number of replicas for the offline data refresh task or the current remaining demand. Specifically, if the offline data refresh task is being scheduled for resource allocation for the first time, the idle resources are compared with the maximum number of replicas for the offline data refresh task; if the offline data refresh task is not being scheduled for resource allocation for the first time, the idle resources are compared with the remaining demand for the offline data refresh task.

[0197] If the amount of idle resources is greater than the maximum number of replicas, it indicates that the current federated cluster has sufficient idle resources to meet the maximum number of replicas for the offline data refresh task. At this time, the federated scheduler will allocate all GPU cards indicated by the maximum number of replicas to the offline data refresh task, so that the offline data refresh task can be expanded to the maximum number of replicas at once.

[0198] If the amount of idle resources is less than the maximum number of replicas, the resources to be allocated are determined according to step S806, and the remaining demand corresponding to the offline data refresh task is calculated. In response to the resource release event, the amount of idle resources in the federated cluster is obtained. When the amount of idle resources is greater than the current remaining demand, the federated scheduler allocates the GPU card indicated by the remaining demand to the offline data refresh task.

[0199] When online inference traffic decreases, some GPUs are released back into the cluster. For offline data refresh tasks, this embodiment no longer uses the method of allocating resources only once upon admission. Instead, it continuously retains the remaining demand after the task starts and continuously scales the task from the current number of replicas to the maximum number of replicas through a scheduling loop. By continuously scaling up the running offline data refresh tasks, GPUs released from online inference can be quickly converted into offline computing throughput, rather than remaining idle for a long time, thus improving the efficiency of GPU reclamation during low-peak periods.

[0200] The resource locking mechanism deducts allocated resources in advance after the federal scheduling decision, so that subsequent placements will not consume the same batch of resources repeatedly, reducing the risk of overselling resources at the federal level.

[0201] The resource scheduling method provided in this disclosure can bring the following beneficial effects: 1. Improve resource guarantee capabilities for high-priority training tasks during the high-priority phase. Through the judgment of reclaimable resources at the federation layer and event-driven preemption at the sub-cluster layer, high-priority training tasks are no longer blocked by low-priority offline tasks for extended periods.

[0202] 2. Reduce preemption costs. The federation layer selects the set of sub-clusters with the lowest preemption costs, and the sub-cluster layer selects the set of Pods with the fewest preempted Pods and the lowest priority, reducing the impact on low-priority services.

[0203] 3. Improve GPU recycling efficiency during low-peak periods. By continuously scaling up running offline data refresh tasks, GPUs released from online inference can be quickly converted into offline computing throughput, rather than remaining idle for extended periods.

[0204] 4. Reduce the risk of overselling resources at the federal level. The resource locking mechanism deducts allocated resources in advance after the federal scheduling decision, ensuring that subsequent placements do not consume the same batch of resources repeatedly.

[0205] 5. Reduce GPU fragmentation in pipeline parallel training. Resource scheduling allocates GPUs in integer multiples of the batch size, ensuring that the allocation results can be actually used by the training task.

[0206] 6. Improve the overall GPU utilization of the cluster. For example... Figure 9As shown, based on production cluster observation data, after deployment, the average GPU utilization increased from 31.82% to 43.38%, an increase of 11.56 percentage points; the Streaming-multiprocessor (SM) activity increased from 20.41% to 29.77%, an increase of 9.36 percentage points; and SM occupancy increased from 5.28% to 8.16%, an increase of 2.88 percentage points. GPU memory usage decreased from 54.98% to 53.28%, a decrease of 1.70 percentage points.

[0207] 7. Reduce redundant GPU requirements. Reduce redundant GPU requirements caused by misaligned pipeline granularity from approximately 5% to approximately 1%. In a production cluster with 50,000 GPUs, this is equivalent to reducing the invalid usage or fragmentation of approximately 2,000 GPUs.

[0208] 8. Low intrusion on upper-layer business processes. This solution is mainly achieved by extending the Placement protocol, the federated scheduler, and the sub-cluster scheduler, without requiring large-scale modifications to the upper-layer business framework. By extending the fields and states in federated scheduling objects such as FedJob, Placement, and Work, preemption, continuous scaling, and step-aware scheduling capabilities are incorporated into the existing federated scheduling process without requiring large-scale modifications to the upper-layer business framework.

[0209] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0210] Figure 10 This is a block diagram illustrating a resource scheduling apparatus according to an exemplary embodiment. The resource scheduling apparatus is configured in a federated scheduler, as shown below. Figure 10 The device may include: a task determination module 1010 for determining the task to be processed; a priority determination module 1020 for obtaining the priority of the online inference task in response to the task being an online inference task and the idle resources of the federated cluster not meeting the start conditions of the online inference task; a resource preemption module 1030 for performing a preemption operation on the resources currently occupied by the target task in response to the priority of the online inference task being higher than the priority of the target task; a scheduling step size determination module 1040 for obtaining the scheduling step size of the offline task in response to the task being an offline task; and a resource scheduling module 1050 for performing cyclic expansion according to the scheduling step size of the offline task until the amount of resources occupied by the offline task reaches the target value.

[0211] In some embodiments, the offline task includes an offline training task. The resource scheduling module 1050 includes: a current replica number determination unit, used to determine the current replica number based on the amount of idle resources in the federated cluster and the scheduling step size of the offline training task; wherein the current replica number and the scheduling step size of the offline training task satisfy a multiple constraint; and a resource scheduling unit, used to schedule the resources corresponding to the current replica number to the offline training task; if the current replica number is less than a target value, in response to a resource release message, the step of determining the current replica number based on the amount of idle resources in the federated cluster and the scheduling step size of the offline training task is returned to be executed until the amount of resources occupied by the offline task reaches the target value.

[0212] In some embodiments, the current replica number determination unit is specifically used to determine, for each candidate sub-cluster of the federated cluster, the number of sub-replicas that satisfies the multiple constraint of the scheduling step size based on the amount of idle resources of the candidate sub-cluster; and to sum the number of sub-replicas of each candidate sub-cluster to obtain the current replica number.

[0213] In some embodiments, the resource scheduling unit is further configured to schedule the resources corresponding to the current number of replicas to the offline training task in response to the current number of replicas meeting the start conditions of the offline training task; and to enqueue the offline training task into the backoff queue in response to the current number of replicas not meeting the start conditions of the offline training task.

[0214] In some embodiments, the resource scheduling unit is specifically used to schedule the resources corresponding to the number of sub-replicas in the candidate sub-cluster to the offline training task in descending order of the number of sub-replicas.

[0215] In some embodiments, the offline task includes an offline data refresh task; the resource scheduling module 1050 includes: a parameter acquisition unit, used to acquire the minimum number of replicas and the maximum number of replicas corresponding to the offline data refresh task; acquire the amount of idle resources in the federated cluster; a resource to be allocated determination unit, used to determine the resource to be allocated based on at least one of the idle resources in the federated cluster, the resource pool, the GPU type, the label constraint, and the scheduling step size of the offline data refresh task in response to the idle resources in the federated cluster being greater than the minimum number of replicas and less than the maximum number of replicas; the resource scheduling unit is also used to schedule the resource to be allocated to the offline data refresh task.

[0216] In some embodiments, the resource scheduling unit is further configured to enqueue the offline data refresh task into a backoff queue in response to the idle resource quantity of the federated cluster being less than the minimum number of replicas; and to schedule the resources corresponding to the maximum number of replicas or the current remaining demand to the offline data refresh task in response to the idle resource quantity of the federated cluster being greater than the maximum number of replicas or greater than the current remaining demand.

[0217] In some embodiments, the resource preemption module 1030 includes an available resource quantity acquisition module, used to acquire the available resource quantity of each candidate sub-cluster, the available resource quantity including idle resource quantity and preemptible resource quantity; a target sub-cluster selection module, used to select a target sub-cluster from multiple candidate sub-clusters based on the available resource quantity; and a resource scheduling unit, used to send a scheduling instruction to the local scheduler of the target sub-cluster, the scheduling instruction being used to instruct the local scheduler to select a target node from multiple candidate nodes of the target sub-cluster and perform a preemption operation on the resources occupied by the target task in the target node.

[0218] In some embodiments, the target sub-cluster selection module is specifically used to determine whether the available resources of each candidate sub-cluster meet the resource requirements of the online inference task; in response to the availability of resources of multiple candidate sub-clusters meeting the resource requirements of the online inference task, to determine the preemption cost of each candidate sub-cluster; and to select the candidate sub-cluster with the lowest preemption cost as the target sub-cluster.

[0219] In some embodiments, the scheduling instruction is used to instruct the local scheduler to determine the target preempted Pod (container group) set corresponding to each candidate node, and to determine the candidate node corresponding to the target preempted Pod set as the target node; the target preempted Pod set includes: the Pod set with the fewest preempted Pods, or the Pod set with the lowest priority preempted Pods.

[0220] In some embodiments, the scheduling instruction is further used to instruct the local scheduler to record the correspondence between the target preempted Pod set and the Pods deploying online inference tasks on the target node, and to schedule the Pods deploying online inference tasks to the target node in response to the resource release event corresponding to the target preempted Pod set.

[0221] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0222] The following reference Figure 11 To describe an electronic device 1100 according to such an embodiment of the present disclosure. Figure 11 The electronic device 1100 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0223] like Figure 11 As shown, the electronic device 1100 is manifested in the form of a general-purpose computing device. The components of the electronic device 1100 may include, but are not limited to: at least one processing unit 1110, at least one storage unit 1120, a bus 1130 connecting different system components (including storage unit 1120 and processing unit 1110), and a display unit 1140.

[0224] The storage unit stores program code, which can be executed by the processing unit 1110 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 1110 can perform actions such as... Figure 3 The steps shown. For example, electronic devices can achieve, for instance, the following steps. Figure 3 The steps shown.

[0225] Storage unit 1120 may include readable media in the form of volatile storage units, such as random access memory (RAM) 1121 and / or cache memory 1122, and may further include read-only memory (ROM) 1123.

[0226] Storage unit 1120 may also include a program / utility 1124 having a set (at least one) program module 1125, such program module 1125 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0227] Bus 1130 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0228] Electronic device 1100 can also communicate with one or more external devices 1170 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 1100, and / or any device that enables electronic device 1100 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1150. Furthermore, electronic device 1100 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1160. As shown, network adapter 1160 communicates with other modules of electronic device 1100 via bus 1130. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1100, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0229] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0230] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions that can be executed by a processor of the device to perform the described method. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0231] In an exemplary embodiment, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods described above.

[0232] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0233] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A resource scheduling method, characterized in that, Applied to the federated scheduler, including: Identify tasks to be processed; In response to the fact that the task to be processed is an online inference task, and the idle resources of the federated cluster do not meet the start conditions of the online inference task, the priority of the online inference task is obtained; In response to the fact that the priority of the online inference task is higher than that of the target task, a preemption operation is performed on the resources currently occupied by the target task; In response to the fact that the task to be processed is an offline task, the scheduling step size of the offline task is obtained; The system performs cyclic expansion based on the scheduling step size of the offline tasks until the amount of resources occupied by the offline tasks reaches the target value.

2. The resource scheduling method according to claim 1, characterized in that, The offline tasks include offline training tasks, and the step of performing cyclic expansion according to the scheduling step size of the offline tasks until the resource consumption of the offline tasks reaches the target value includes: The current number of replicas is determined based on the amount of idle resources in the federated cluster and the scheduling step size of the offline training task; wherein the current number of replicas and the scheduling step size of the offline training task satisfy a multiple constraint. The resources corresponding to the current number of replicas are allocated to the offline training task; If the current number of replicas is less than the target value, in response to the resource release message, return to the step of determining the current number of replicas based on the amount of idle resources in the federated cluster and the scheduling step size of the offline training task, until the amount of resources occupied by the offline task reaches the target value.

3. The resource scheduling method according to claim 2, characterized in that, The step of determining the current number of replicas based on the amount of idle resources in the federated cluster and the scheduling step size of the offline training task includes: For each candidate sub-cluster of the federated cluster, the number of sub-replicas that satisfy the multiple constraint of the scheduling step size is determined based on the amount of idle resources of the candidate sub-cluster. The current number of replicas is obtained by summing the number of sub-replicas in each of the candidate sub-clusters.

4. The resource scheduling method according to claim 2, wherein scheduling the resources corresponding to the current number of replicas to the offline training task includes: In response to the current number of replicas meeting the start conditions of the offline training task, the resources corresponding to the current number of replicas are scheduled to the offline training task; The method further includes: In response to the current number of replicas not meeting the start conditions of the offline training task, the offline training task is enqueued into a backoff queue.

5. The resource scheduling method according to claim 3, wherein scheduling the resources corresponding to the current number of replicas to the offline training task includes: Resources corresponding to the number of sub-replicas in the candidate sub-clusters are scheduled to the offline training task in descending order of the number of sub-replicas.

6. The resource scheduling method according to claim 1, characterized in that, The offline tasks include offline data refresh tasks; the step of performing cyclic expansion according to the scheduling step size of the offline tasks until the resource consumption of the offline tasks reaches the target value includes: Obtain the minimum and maximum number of replicas corresponding to the offline data refresh task; Obtain the amount of free resources in the federated cluster; In response to the fact that the amount of idle resources in the federated cluster is greater than the minimum number of replicas and less than the maximum number of replicas, the resources to be allocated are determined based on at least one of the idle resources in the federated cluster, the resource pool, the graphics processor (GPU) type, the tag constraint, and the scheduling step size of the offline data refresh task. The resources to be allocated are scheduled to the offline data refresh task.

7. The resource scheduling method according to claim 6, characterized in that, The method further includes: In response to the fact that the amount of idle resources in the federated cluster is less than the minimum number of replicas, the offline data refresh task is enqueued into the backoff queue; In response to the fact that the amount of idle resources in the federated cluster is greater than the maximum number of replicas or greater than the current remaining demand, the resources corresponding to the maximum number of replicas or the current remaining demand are scheduled to the offline data refresh task.

8. The resource scheduling method according to claim 1, characterized in that, The step of preempting the resources currently occupied by the target task in response to the online inference task having a higher priority than the target task includes: Obtain the available resource quantity of each candidate sub-cluster, wherein the available resource quantity includes the idle resource quantity and the preemptible resource quantity; Select the target sub-cluster from the multiple candidate sub-clusters based on the available resource quantity; A scheduling instruction is sent to the local scheduler of the target sub-cluster. The scheduling instruction is used to instruct the local scheduler to select a target node from multiple candidate nodes of the target sub-cluster and to perform a preemption operation on the resources occupied by the target task in the target node.

9. The resource scheduling method according to claim 8, characterized in that, The step of selecting the target sub-cluster from the multiple candidate sub-clusters based on the available resource quantity includes: Determine whether the available resources of each candidate sub-cluster meet the startup conditions of the online inference task; In response to the availability of resources in multiple candidate sub-clusters satisfying the launch conditions of the online inference task, the preemption cost of each candidate sub-cluster is determined. The candidate sub-cluster with the lowest preemption cost is selected as the target sub-cluster.

10. The resource scheduling method according to claim 8, characterized in that, The scheduling instruction is used to instruct the local scheduler to determine the target preemptible container group Pod set corresponding to each candidate node, and to determine the candidate node corresponding to the target preemptible Pod set as the target node; The target set of preempted Pods includes: the set of Pods with the fewest preempted Pods, or the set of Pods with the lowest priority among the preempted Pods.

11. The resource scheduling method according to claim 10, characterized in that, The scheduling instruction is also used to instruct the local scheduler to record the correspondence between the target preempted Pod set and the Pods deploying the online inference task on the target node, and in response to the resource release event corresponding to the target preempted Pod set, to schedule the Pods deploying the online inference task to the target node.

12. A resource scheduling device, characterized in that, Configured in the federated scheduler, including: The task determination module is used to determine the tasks to be processed. The priority determination module is used to obtain the priority of the online inference task in response to the fact that the task to be processed is an online inference task and the idle resources of the federated cluster do not meet the start conditions of the online inference task; The resource preemption module is used to preempt the resources currently occupied by the target task in response to the fact that the priority of the online inference task is higher than the priority of the target task. The scheduling step size determination module is used to obtain the scheduling step size of the offline task in response to the task to be processed being an offline task. The resource scheduling module is used to perform cyclic expansion according to the scheduling step size of the offline task until the amount of resources occupied by the offline task reaches the target value.

13. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the executable instructions to implement the resource scheduling method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the resource scheduling method as described in any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the resource scheduling method as described in any one of claims 1 to 11.

Images