Resource scheduling method and device

Abstract

Embodiments of the present specification provide a resource scheduling method and device. In the resource scheduling method, a plurality of tasks to be scheduled are obtained. For any first task in the plurality of tasks, an estimated execution duration of the first task is determined by using a duration prediction model based on a task feature of the first task and a current resource state related to a GPU card in a target cluster. A task priority of the first task is determined based on a task type of the first task and historical card usage quality data of a user submitting the first task. In a simulation environment, different scheduling strategies are respectively generated by using a plurality of multi-objective optimization algorithms based on the estimated execution duration, the task priority, and the current resource state of each of the plurality of tasks. An execution cost of each scheduling strategy is evaluated, and a target scheduling strategy is determined according to the execution cost. The plurality of tasks are scheduled to the target cluster for execution according to the target scheduling strategy.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of computer technology, and more particularly to a resource scheduling method and apparatus. Background Technology

[0002] In the field of AI cluster resource scheduling, for a queue of tasks to be scheduled, the scheduling system typically generates a single scheduling strategy based on preset rules (such as task priority and submission time) and directly issues the strategy to the cluster for execution. To improve decision-making accuracy, some systems also dynamically adjust priorities by combining historical task execution data or user profiles. The common logic of these methods is "decision first, execution later," that is, the scheduling strategy is implemented as soon as it is determined.

[0003] However, the above mechanisms still have room for improvement in practical applications. Due to the lack of pre-evaluation capabilities for the execution effect of scheduling strategies, the system struggles to predict the specific impact of the strategy on cluster performance metrics (such as average waiting time, resource utilization, and task completion rate) before its implementation. This easily leads to two problems: First, different scheduling strategies often have inherent conflicts between multiple optimization objectives (such as reducing large task blocking and shortening small task waiting time), and existing methods struggle to find a good balance without practical verification; second, if the generated scheduling strategy is not ideal, it will directly cause resource waste or task backlog, and this trial-and-error cost is particularly significant in large-scale clusters with tens of thousands of kilowatts.

[0004] Therefore, how to predict the execution effect of different scheduling strategies and select the best one without introducing the risks of real cluster operation is a problem that still needs to be solved in current scheduling technology. Summary of the Invention

[0005] This specification describes one or more embodiments of a resource scheduling method and apparatus that can significantly reduce the scheduling trial and error costs of large-scale clusters and improve the accuracy and stability of scheduling decisions.

[0006] Firstly, a resource scheduling method is provided, executed by a scheduling system, including:

[0007] Get multiple tasks to be scheduled;

[0008] For any first task among the multiple tasks, the estimated execution time of the first task is determined using a duration prediction model based on the task characteristics of the first task and the current resource status related to the GPU card in the target cluster.

[0009] Based on the task type of the first task and the historical card usage quality data of the user who submitted the first task, the task priority of the first task is determined.

[0010] In the simulation environment, based on the estimated execution time, task priority and current resource status of each of the multiple tasks, different scheduling strategies are generated by various multi-objective optimization algorithms.

[0011] Evaluate the execution cost of each scheduling strategy and determine the target scheduling strategy based on the execution cost;

[0012] According to the target scheduling strategy, the multiple tasks are scheduled to be executed in the target cluster.

[0013] Secondly, a resource scheduling device is provided, comprising:

[0014] The acquisition unit is used to acquire multiple tasks to be scheduled.

[0015] The determining unit is used to determine the estimated execution time of any first task among the plurality of tasks by using a duration prediction model, based on the task characteristics of the first task and the current resource status related to the GPU card in the target cluster.

[0016] The determining unit is further configured to determine the task priority of the first task based on the task type of the first task and the historical card usage quality data of the user who submitted the first task;

[0017] The generation unit is used to generate different scheduling strategies in a simulation environment based on the estimated execution time, task priority, and current resource status of each of the multiple tasks, using a variety of multi-objective optimization algorithms.

[0018] The evaluation unit is used to evaluate the execution cost of each scheduling strategy and determine the target scheduling strategy based on the execution cost.

[0019] The scheduling unit is used to schedule the multiple tasks to the target cluster for execution according to the target scheduling strategy.

[0020] Thirdly, a computer storage medium is provided on which a computer program is stored, which, when executed in a computer, causes the computer to perform the method of the first aspect.

[0021] Fourthly, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method of the first aspect.

[0022] The resource scheduling method and apparatus provided in one or more embodiments of this specification firstly, estimate the execution time of each task using a duration prediction model, and determine the task priority by combining the task type and the user's historical card usage quality data. This provides a dual quantitative basis of time and value for scheduling decisions, making priority settings more scientific and reasonable. Secondly, in a simulation environment, based on the estimated duration, priority, and current resource status of each task, various multi-objective optimization algorithms are used to generate different scheduling strategies. This enables offline verification of multiple scheduling schemes without affecting the operation of the real cluster, effectively avoiding the trial-and-error risks in the production environment. Finally, by evaluating the execution cost of each strategy and selecting the target scheduling strategy for execution, a quantitative trade-off can be made among multiple objectives, selecting the scheme with the least interference to existing tasks and the best overall cost. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram illustrating an implementation scenario of one embodiment disclosed in this specification;

[0025] Figure 2 A schematic diagram of a scheduling system is shown in one example of this specification;

[0026] Figure 3 This diagram illustrates a resource scheduling method according to one embodiment of the present specification.

[0027] Figure 4 This document illustrates a diagram of the sub-clusters that are divided for the target cluster in one example.

[0028] Figure 5 This specification illustrates a resource scheduling mechanism for different task types in one example.

[0029] Figure 6 A schematic diagram illustrating a method for scheduling computing tasks is shown in one example of this specification;

[0030] Figure 7 A schematic diagram of a resource scheduling apparatus according to an embodiment of this specification is shown. Detailed Implementation

[0031] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0032] Figure 1This is a schematic diagram illustrating an implementation scenario of one of the embodiments disclosed in this specification. Figure 1 First, multiple tasks to be scheduled are acquired. For each task, a duration prediction model is used to estimate its execution time by combining the task's characteristics (e.g., model structure, dataset size) with the current resource status of the cluster (e.g., cluster load, network bandwidth, resource fragmentation). Simultaneously, task priority is determined based on task type (e.g., key tasks tagged with important projects, trial runs, ordinary tasks, or valley filling tasks) and historical user card usage quality data (including GPU utilization and / or historical success rate). Based on this, in a simulation environment, various multi-objective optimization algorithms are used to generate different scheduling strategies by integrating the estimated duration, priority, and resource status of each task. The strategy with the lowest cost is selected as the optimal scheduling strategy, and the task is scheduled to the target cluster for execution accordingly. These multi-objectives include, but are not limited to, maximizing cluster resource utilization, minimizing average task queuing time, and ensuring timely scheduling of high-priority tasks. This method provides a quantitative basis for scheduling by estimating duration and prioritizing, uses a simulation environment to realize offline verification and selection of multiple strategies, and achieves quantitative trade-offs among multiple conflicting objectives based on execution cost, thereby reducing the scheduling trial and error cost of large-scale clusters (e.g., 90%+), improving resource utilization and achieving second-level scheduling decisions.

[0033] It should be noted that the above Figure 1 The scheduling methods for the multiple tasks shown can be executed by a scheduling system.

[0034] Figure 2 A schematic diagram of a scheduling system is shown in one example of this specification. Figure 2 As shown, this scheduling system is deployed at the management and control layer of a 10,000-GPU cluster, responsible for resource allocation and scheduling of multiple tasks to be executed within the cluster. Figure 2 As shown, the scheduling system consists of six core modules: cluster scheduling, task scheduling, time-based scheduling, intelligent scheduling and simulation engine, fairness mechanism, and transparent service. Adhering to the three principles of fairness, impartiality, and openness, it constructs an intelligent resource scheduling system for ultra-large-scale domestic GPU clusters. The goal is to efficiently utilize tens of thousands of GPUs of computing power, ensure the stability of core business operations, and improve the fairness of resource allocation and user experience. The functions and system logic of each module are explained in detail below.

[0035] First, the core idea of ​​cluster operations is to divide the original cluster into two parts: cluster 1 and cluster 2. Cluster 1 is used to generate training or inference tasks and cluster 2 is used for development and debugging tasks, thereby realizing resource isolation and dynamic reuse between the R&D and production environments.

[0036] Furthermore, cluster 1 can be divided into multiple sub-clusters: sub-cluster 11, sub-cluster 12, and sub-cluster 13. Sub-cluster 11 is used to execute pure CPU tasks, sub-cluster 12 is used to execute GPU tasks with no more than a preset threshold of GPU cards, and sub-cluster 13 is used to execute GPU tasks with more than the preset threshold of GPU cards. Each sub-cluster is configured with a corresponding runtime limit, and the runtime limit of sub-cluster 13 is less than that of sub-cluster 12, and the runtime limit of sub-cluster 12 is less than that of sub-cluster 11. When the runtime allocated to any sub-cluster exceeds the corresponding limit, the sub-task is terminated. Simultaneously, the system monitors the resource idle status of cluster 2. When cluster 2 has idle resources, the idle resources are dynamically allocated to cluster 1; when a new development and debugging task is submitted by a user corresponding to cluster 2, the corresponding amount of resources are reclaimed from cluster 1 and returned to cluster 2.

[0037] It should be noted that this solution, through the resource pool architecture of cluster partitioning and hierarchical sub-clusters, can achieve the governance goal of "large tasks not blocking small tasks"; through dynamic resource allocation between R&D and production clusters, it can ensure the availability of development and debugging resources while avoiding the idle waste of R&D resources and improving the overall resource utilization rate.

[0038] Secondly, the core idea of ​​task management is to determine task priority and perform differentiated scheduling based on task type and the historical card usage quality data of the users who submit tasks.

[0039] Specifically, the scheduling system first determines the initial priority of tasks based on task type, which includes key tasks tagged with "key project," trial tasks, ordinary tasks, and valley filling tasks. Then, a target score is determined based on the user's historical success rate and / or GPU utilization, which is positively correlated with historical success rate and / or GPU utilization. Further, the base score can be adjusted based on user behavior events within a predetermined time period to obtain the final target score. The final task priority is determined based on the target score and initial priority. For key tasks, it is determined whether the user's task concurrency meets the hard limit. If so, available resources are checked. When resources are insufficient, a resource preemption process is triggered: candidate tasks with lower priority or target scores below a threshold are filtered, and tasks to be reclaimed are identified and reclaimed differentially—tasks with high target scores that have not been forcibly terminated are saved and re-added to the waiting queue, while tasks with low target scores or those that have been forcibly terminated are directly terminated and resources are reclaimed. For ordinary tasks, when resources are insufficient, they are added to the waiting queue, where tasks are sorted by target score, with tasks with higher target scores having higher scheduling priority. For trial runs, allocate a preset minimum resource specification and set a maximum runtime. For valley filling tasks, configure them to start only when idle resources meet preset conditions and have the attribute of being preempted by priority tasks.

[0040] It should be noted that this solution, through task management, can avoid the resource waste caused by a single first-come, first-served strategy, realize the principle of resource allocation and scheduling with differentiated configuration for different task types, and improve the overall cluster efficiency.

[0041] Next, the core idea of ​​time-based scheduling is to detect idle fragmented computing resources in the cluster, divide valley filling tasks or ordinary tasks into time-based subtasks for scheduling, or break down computing resource-intensive tasks into subtasks and schedule them to be executed during idle periods.

[0042] Specifically, for valley filling tasks or ordinary tasks, the system detects the idle fragmented computing resources of the target cluster and uses these resources to schedule time-limited subtasks on a per-card-hour basis. The resources used by these time-limited subtasks are preemptible. For computing resource-intensive tasks, the task is broken down into multiple subtask slices, each corresponding to a preset computing power consumption unit. The system obtains the idle time slots of GPU resources in the target cluster and schedules multiple subtask slices to execute during these idle time slots. Each subtask slice occupies GPU resources during its corresponding idle time slot and releases them after execution. These tasks are interruptible and resumable pluggable tasks. The system saves the task's state information during execution and resumes execution based on the state information after a subtask slice is interrupted.

[0043] It should be noted that this solution can solve the problem of uneven computing power utilization by using time-based resource allocation. By reusing fragmented computing power in both time and space dimensions, the overall utilization rate of the cluster can be improved to a high level, reducing the cost of idle computing power.

[0044] Secondly, the core idea of ​​intelligent scheduling and simulation engines is to generate different scheduling strategies based on multi-objective optimization algorithms in a simulation environment, evaluate the execution cost of each strategy, and then determine the target scheduling strategy.

[0045] Specifically, a simulation environment can be built into the scheduling system. In this environment, based on the estimated execution time, task priority, and current resource status of multiple tasks, various multi-objective optimization algorithms are used to generate different scheduling strategies. The system evaluates the execution cost of each scheduling strategy, determines the target scheduling strategy based on the execution cost, and then applies the target scheduling strategy to the production environment.

[0046] It should be noted that this solution, by building a simulation environment into the scheduling system, can perform offline verification of various scheduling strategies without affecting the actual operation of the production environment. This avoids the high risk and high cost of trial and error in the production environment, while ensuring a reasonable trade-off between multiple optimization objectives in the scheduling strategy.

[0047] In addition, the core idea of ​​the fairness mechanism is to dynamically reduce the priority of users who exceed the task volume within a preset rolling time window by statistically analyzing the task volume of users, thereby balancing short-term efficiency and long-term fairness.

[0048] Specifically, the scheduling system counts the total GPU time or total number of tasks successfully scheduled and run by each user who submitted a task within a preset rolling time window. When the total GPU time or total number of tasks for any user exceeds the corresponding preset threshold, the priority of newly submitted tasks by that user is dynamically reduced, and the reduction is positively correlated with the amount of excess.

[0049] It should be understood that this fair practice can prevent large teams from monopolizing resources, make resource allocation opportunities more equal, and improve user satisfaction.

[0050] Finally, the aforementioned transparent service can also be called a multi-agent natural language query system. This system, through a multi-agent architecture, allows users to query task status and scheduling logic using natural language, thereby making scheduling information transparent.

[0051] Specifically, the scheduling system receives a natural language query request from the target user, including the task identifier of the target task to be queried; the main agent parses the request and identifies the user's intent; based on the user's intent, it determines the target sub-agent from multiple specialized sub-agents; based on the task identifier, it calls the target sub-agent to retrieve factor data related to the target task's running status from the scheduling system; and based on the factor data, it generates a query result in natural language and returns it to the user. These multiple specialized sub-agents include: a task information query agent, used to query task attribute information and queue status; a queuing analysis agent, used to analyze the task's position in the queue and its estimated waiting time; a priority interpretation agent, used to explain the decision logic of task priority; an identity conversion agent, used to convert the user's identity identifier into a system-recognizable identity identifier; a statistics agent, used to statistically analyze resource usage according to preset dimensions; and a resource statistics agent, used to provide a global perspective on resource usage.

[0052] It should be understood that this solution lowers the barrier for users to obtain scheduling information through self-service intelligent Q&A, presents the scheduling decision-making logic transparently, and enhances users' trust in the scheduling system.

[0053] Furthermore, the aforementioned fairness principle is achieved through task pool partitioning and a dynamic demotion mechanism, ensuring relative fairness in resource acquisition for users / teams with varying resource usage, and achieving fair scheduling with demotion for higher usage. This impartiality principle ensures that all scheduling strategies are based on objective data, including task type, historical success rate, GPU utilization, and behavioral events, automatically calculating task priorities and avoiding human intervention bias. The principle of transparency is achieved through a multi-agent query system, which transparently displays scheduling information such as task status, queue position, and priority explanation to users, eliminating information asymmetry.

[0054] In summary, this solution can significantly reduce the scheduling trial and error costs of 10,000-card-level clusters, ensuring the supply of resources for critical tasks while taking into account the overall resource utilization and scheduling fairness of the cluster, and providing a highly reliable, efficient, and user-friendly computing power scheduling platform for large-scale AI model training and inference tasks.

[0055] Figure 3 This diagram illustrates a resource scheduling method according to an embodiment of this specification, which can... Figure 2 The scheduling system shown executes, such as Figure 3 As shown, the method may include the following steps:

[0056] Step S302: Obtain multiple tasks to be scheduled.

[0057] The scheduling system obtains multiple tasks to be scheduled from the task submission entry.

[0058] For these multiple tasks, from a technical process perspective, the task type of a single task can include any of the following: large model training task, inference task, or data preprocessing task. From a business priority perspective, the task type of a single task can include any of the following: key task with a key project tag, trial run task, ordinary task, and valley filling task.

[0059] In some embodiments, the scheduling system can receive natural language instructions input by the user, which specify the training objective, such as "Perform SFT training on a risk control model for a certain scenario to achieve an AUC of 80%." The scheduling system then uses a large language model to perform semantic analysis on the natural language instructions and, combined with built-in expert experience and historical task workflow templates, decomposes the training objective into a series of quantifiable and executable sub-tasks. For example, it could be decomposed into: Sub-task 1: Adjust the learning rate range to [0.001, 0.01]; Sub-task 2: Try three optimizer combinations; Sub-task 3: Increase the batch size to 256. These sub-tasks may have logical dependencies; for example, the execution of sub-task 2 depends on the completion result of sub-task 1. Afterward, the decomposed sub-tasks can be included in the task set to be scheduled.

[0060] It should be understood that through this goal decomposition process, vague business requirements of users can be transformed into structured scheduling inputs, while the system's built-in scientific experimentation capabilities can reduce the large number of inefficient experimental tasks submitted blindly by users.

[0061] Step S304: For any task T1 among multiple tasks, use the duration prediction model to determine the estimated execution duration of task T1 based on the task characteristics of task T1 and the current resource status related to the GPU card in the target cluster.

[0062] The aforementioned duration prediction model is an AI model trained on historical task data. Its core function is to accurately estimate the actual running time of a task. The task runtime mentioned above is the estimated time from task submission to completion.

[0063] The task characteristics mentioned above are static and unchanging. Taking task T1 as a large model training task as an example, its task characteristics may include, but are not limited to, at least one of the following: model structure, size of training dataset, and GPU model. The current resource status mentioned above may include, but is not limited to, at least one of the following: cluster load, network bandwidth, resource fragmentation level, number of idle GPUs, network throughput, and memory usage.

[0064] Unlike traditional scheduling methods that rely on users to declare task execution times, the duration prediction model comprehensively considers multiple factors affecting task execution, and its prediction error rate can be controlled within 10%. Accurate duration prediction provides a reliable input basis for the generation of subsequent scheduling strategies.

[0065] Additionally, this solution can continuously optimize the aforementioned duration prediction model and continuously compare the performance of the old and new models. For example, after each of the above tasks is completed, the actual execution data of each task (such as actual training time, GPU utilization, and reasons for recycling) can be written into the historical task and resource usage database. Then, the duration prediction model can be retrained periodically using the newly added data to improve the prediction accuracy.

[0066] Step S306: Determine the task priority of task T1 based on the task type of task T1 and the historical card usage quality data of the user who submitted task T1.

[0067] The task types here can be categorized according to business priority, and may include, but are limited to, key tasks tagged with key projects, trial tasks, ordinary tasks, or valley filling tasks. In this solution, different task types correspond to different resource guarantee strategies. For example, for key tasks, the corresponding resource guarantee strategy is: a single task quota is not lower than a preset threshold (e.g., 128 cards), a total quota upper limit can be set, the task has the ability to preempt resources from other tasks, and the number of concurrent tasks and the total number of tasks for the same user in the same business system are limited. For trial tasks, the corresponding resource guarantee strategy is: time-limited operation (e.g., 15 minutes), allocated with the minimum resource specifications. For ordinary tasks, the corresponding resource guarantee strategy is: support for heterogeneous resource scheduling, and queuing based on user task objectives. For valley filling tasks, the corresponding resource guarantee strategy is: automatic triggering of startup when system idle resources meet preset conditions, and the ability to be preempted by key tasks.

[0068] The historical card usage quality data of the aforementioned users may include historical success rate and / or GPU utilization, etc.

[0069] Specifically, the initial priority is determined based on the task type of task T1. In this scheme, different task types correspond to different initial priorities; for example, key tasks have a higher initial priority than ordinary tasks. Then, the target score for task T1 is determined based on historical success rate and / or GPU utilization. This target score is positively correlated with historical success rate and / or GPU utilization; that is, tasks with higher historical success rates and higher GPU utilization have higher target scores. The task priority of task T1 is then determined based on the target score and the initial priority.

[0070] In some embodiments, the target score is determined through the following steps: Calculating a base score based on historical success rate and / or GPU utilization; Obtaining behavioral events of the user submitting task T1 within a predetermined time period; Adding or subtracting the base score based on the event type of the behavioral event to obtain the target score.

[0071] The event types here include positive incentive events and negative constraint events. Positive incentive events can include events where users launch models. For example, if a user launches a model within the past 30 days, they can add 10 points to their base score to incentivize task completion and the creation of business value.

[0072] The aforementioned negative constraint events can include any of the following: failure to complete an experimental sharing task, discrepancies between submitted task resource requirements and actual usage, and prolonged GPU usage with utilization below a threshold. For example, failing to complete an experimental sharing task deducts 5 points, submitting task resource requirements that do not match actual usage deducts 10 points, and prolonged GPU usage with utilization below a threshold deducts 50 points. Through this refined behavioral reward and punishment mechanism, resource allocation is strongly correlated with user behavior, guiding users to proactively optimize task quality.

[0073] In one example, the target score can be calculated using the following formula:

[0074] Target score = Task success rate × GPU utilization + Behavioral bonus / minus.

[0075] In other embodiments, the base score can be used directly as the target score, and this specification does not limit this to that.

[0076] In addition, in practice, the above target score may not be calculated. Instead, the initial priority determined based on the task type of task T1 may be adjusted directly based on expert experience and combined with the user's historical card usage quality data to obtain the task priority of task T1. This manual does not limit this.

[0077] It should be noted that, in the process of determining task priorities, the scheduling system also ensures the fairness of resource allocation when multiple users share a cluster through the following methods.

[0078] The scheduling system calculates the total GPU time or number of tasks successfully scheduled and run by each user who submitted the above multiple tasks within a preset rolling time window. This rolling time window is a real-time rolling 24-hour window, rather than a fixed calendar day, to avoid scheduling jitter caused by "sudden changes at midnight".

[0079] In this scheme, for users whose registration duration is less than the preset protection duration, the total card time or total number of tasks for that user will be considered as zero.

[0080] In one example, the above statistical scope can be limited to ordinary tasks, excluding tasks in high-priority queues such as key tasks and trial runs.

[0081] When determining the final task priority, the system employs a weighted decreasing mechanism based on the number of tasks submitted. Specifically, it counts the total number of tasks successfully scheduled and run by each user within a preset rolling time window (such as a real-time rolling 24-hour window). When the total number of tasks submitted by any user exceeds a preset threshold, the dynamic priority of its newly submitted tasks is calculated using the following weighted decreasing formula.

[0082] Final priority = Initial priority × 1 / (1 + α × number of tasks) + Adjustment item

[0083] Where α is the decay coefficient, and the number of tasks is the total number of tasks successfully executed by the user within the rolling time window. When the total number of tasks a user has within the rolling time window reaches the threshold a (e.g., 25), the priority of newly submitted tasks decreases by one level; when it reaches the threshold b (e.g., 50), it decreases by two levels; and when it reaches the threshold c (e.g., 100), it decreases by three levels.

[0084] Furthermore, when a user's total number of tasks within the rolling time window exceeds a preset cooling threshold (e.g., 100), a forced cooling mechanism is triggered. New tasks submitted by this user will be subject to queuing restrictions and will only be added to the scheduling queue after the user's existing tasks have reached a preset completion rate (e.g., 20%), thus preventing extreme resource monopolization. It should be noted that if a user's target score is extremely high, this can partially offset the attenuation, reflecting the scheduling principle that efficient users should be rewarded. In addition, for new users whose registration and usage time is less than the preset protection period (e.g., 3 days), their total number of tasks within the rolling time window is forcibly considered zero to avoid scheduling disadvantages due to insufficient target score accumulation.

[0085] The above-mentioned system uses a rolling window to count user task volume and a weighted decreasing mechanism to dynamically reduce the priority of users who submit tasks frequently, thus avoiding resource monopolies caused by the "Matthew effect." At the same time, it balances the scheduling rights of both high-efficiency users and new users through target scoring hedging and cold start protection.

[0086] In addition, this solution combines task type and user's historical card usage quality data (historical success rate, GPU utilization, and behavioral event bonuses and deductions) to determine task priority, so that resource allocation not only considers the value of the task itself, but also relates to the user's usage behavior, forming a positive cycle of "resource efficiency and priority".

[0087] Step S308: In the simulation environment, based on the estimated execution time, task priority, and current resource status of each of the multiple tasks, different scheduling strategies are generated using various multi-objective optimization algorithms.

[0088] The simulation environment described above is a digital twin sandbox environment built into the scheduling system. It constructs a virtual image based on the real state of the target cluster, which can perform pre-simulation and verification of scheduling strategies without affecting the actual operation of the production environment.

[0089] The optimization objectives of the aforementioned multi-objective optimization algorithms include at least: maximizing cluster resource utilization, minimizing average task queuing time, and ensuring timely scheduling of high-priority tasks. In the simulation environment, various multi-objective optimization algorithms (such as ant colony optimization and Pareto optimization) can be executed in parallel, generating corresponding scheduling strategies. Each scheduling strategy reflects the trade-offs between multiple optimization objectives achieved by different algorithms.

[0090] In one embodiment, the aforementioned multiple scheduling strategies may include: a pure priority scheduling strategy, a weighted fair scheduling strategy, and a hybrid scheduling strategy. The pure priority scheduling strategy can ensure that key tasks are executed quickly, but the waiting time of other tasks increases significantly (e.g., by 200%). The weighted fair scheduling strategy can ensure that the latency of key tasks increases slightly (e.g., by 15%), but the waiting time of other tasks is greatly reduced (e.g., reduced by 60%). The hybrid scheduling strategy can ensure that the latency of key tasks is further reduced (e.g., by 5%), and the waiting time of other tasks is reduced by 40%.

[0091] It should be noted that this solution pre-tests the effects of various scheduling strategies in a simulation sandbox before actual scheduling. Based on a multi-objective optimization algorithm, candidate solutions are automatically generated and their execution costs are quantitatively compared. The optimal scheduling strategy is then selected for execution, avoiding the high risk and high cost of trial and error in the production environment.

[0092] Step S310: Evaluate the execution cost of each scheduling strategy and determine the target scheduling strategy based on the execution cost.

[0093] In one example, the evaluation of execution cost comprehensively considers multiple indicators such as cluster throughput, average task latency, response latency of high-priority tasks, and resource fragmentation. By quantitatively comparing the comprehensive execution cost of various scheduling strategies, the scheduling strategy with the lowest cost is selected as the target scheduling strategy for this scheduling, thereby avoiding the shortcomings of selecting a scheduling scheme based solely on experience.

[0094] Step S312: According to the target scheduling strategy, multiple tasks are scheduled to be executed on the target cluster.

[0095] In one example, the target cluster mentioned above is cluster 1, one of the two clusters divided from the original cluster, used to perform generative training or inference tasks. This cluster also includes cluster 2, which is used for development and debugging tasks. Cluster 2, also known as the development pool, is configured to allocate at least a preset number of GPU cards to each development user to ensure basic resource needs for development and debugging. Cluster 2 is further divided into a secdev local GPU development pool and a cpudev CPU development pool. The secdev pool statically reserves a preset number (e.g., 4) of GPU cards for each development user to ensure basic debugging resources; the cpudev pool supports trial runs and production tasks in pure CPU scenarios.

[0096] Figure 4 This document illustrates a diagram of the sub-clusters that are divided for the target cluster in one example of this specification. Figure 4 In this system, sub-clusters are divided according to task size. Sub-cluster 11 (cputask) is used to execute pure CPU tasks (e.g., trial runs and computation tasks without GPUs), with a maximum runtime of 15 days. Sub-cluster 12 (sectask) is used to execute GPU tasks with no more than a preset threshold (e.g., 128 GPUs), i.e., small to medium-sized GPU training or inference tasks, with a maximum runtime of 15 days and a resource utilization threshold (e.g., 70%). When the GPU utilization of a task falls below this threshold within a preset duration (e.g., 2 hours), resource reclamation is triggered. Sub-cluster 13 (seclargetask) is used to execute very large-scale tasks with more than a preset threshold of GPUs, with a maximum runtime of 7 days and a resource utilization threshold (e.g., 50%). When the GPU utilization of a task falls below this threshold within a preset duration, resource reclamation is triggered.

[0097] In summary, each sub-cluster is configured with differentiated runtime limits, following a gradient strategy where the runtime limit for sub-cluster 13 is less than that for sub-cluster 12, and the runtime limit for sub-cluster 12 is less than that for sub-cluster 11. When the runtime of a task assigned to any sub-cluster exceeds the runtime limit corresponding to that sub-cluster, the task is terminated. Furthermore, each sub-cluster can be configured with differentiated resource utilization thresholds. This allows for stricter criteria for reclaiming inefficient tasks on extremely large-scale tasks, preventing large tasks from consuming excessive resources and slowing down the overall cluster scheduling efficiency. Finally, through these differentiated configurations, the blocking of smaller tasks caused by large tasks consuming resources for extended periods is prevented.

[0098] Of course, in practice, a dedicated R&D sub-cluster can be divided from cluster 1, that is, a preset number of GPU computing cards can be statically reserved for each registered R&D user for model debugging and trial operation.

[0099] It should be noted that in this solution, a dynamic resource allocation mechanism is configured between Cluster 1 and Cluster 2. Specifically, the scheduling system can detect the idle resource status of Cluster 2. When idle resources are detected in Cluster 2, they are dynamically allocated to Cluster 1 for use in generative training or inference tasks. When a new development and debugging task is submitted by a user corresponding to Cluster 2, the corresponding amount of resources are reclaimed from Cluster 1 and returned to Cluster 2. Through this dynamic allocation mechanism, while ensuring that debugging resources are available to developers, waste caused by long-term idle development resources is avoided.

[0100] It should also be noted that the above-mentioned division of the target cluster into multiple sub-clusters according to task scale, with each sub-cluster configured with differentiated runtime and utilization control strategies, achieves the governance goal of "large tasks not blocking small tasks". At the same time, the dynamic resource allocation mechanism between the R&D cluster and the production cluster takes into account both development experience and production efficiency.

[0101] In step S312, the scheduling system applies the target scheduling strategy selected during the simulation verification phase to the production environment, allocating multiple tasks to be scheduled to the corresponding GPU nodes in the target cluster for execution according to the target scheduling strategy. Through this method, the scheduling system pre-simulates the effects of different scheduling strategies in a simulation sandbox before actual scheduling, automatically generating multiple candidate solutions based on a multi-objective optimization algorithm and selecting the optimal one. This avoids the high risk and cost of trial and error directly in the production environment, while ensuring a reasonable trade-off between multiple optimization objectives in the scheduling strategy.

[0102] Additionally, during the scheduling process, the scheduling system employs differentiated processing mechanisms for different task types:

[0103] For example, for trial runs, the scheduling system allocates a preset minimum resource specification and sets a maximum runtime (e.g., 15 minutes). This mechanism allows trial runs to quickly complete verification with minimal resources, avoiding prolonged occupation of high-specification resources. For valley filling tasks, the scheduling system configures them to start only when system idle resources meet preset conditions and allows them to be preempted by key tasks. Valley filling tasks utilize the cluster's idle, fragmented computing resources for execution. When key tasks require resources, they can be preempted and resources released at any time, thereby improving the overall cluster utilization without affecting core business operations.

[0104] Furthermore, in this scheme, different task types can correspond to different resource scheduling mechanisms.

[0105] Figure 5 This specification illustrates a resource scheduling mechanism for different task types in one example. Figure 5 In the process, for key tasks, the scheduling system first determines whether the number of concurrent tasks submitted by the user meets preset hard limits, such as no more than one concurrent task per user in the same business system and no more than three total tasks for that user. If these hard limits are met, the system further determines whether the available resources meet the resource requirements of the key task. If the available resources meet the resource requirements, the task is directly scheduled for execution. If the available resources do not meet the resource requirements, a resource preemption process is triggered. This resource preemption process includes: identifying a task with a lower priority or a target score lower than a preset threshold from at least one currently running task, reclaiming the resources occupied by the task to be reclaimed, and allocating the reclaimed resources to the key task. If no task to be reclaimed is identified, the key task is added to a waiting queue.

[0106] Specifically, when recycling resources for a task, if the target score of the task is higher than the preset threshold and it has not been forcibly terminated, the task will be saved and added back to the waiting queue; if the target score of the task is lower than or equal to the preset threshold, or it has been forcibly terminated, the task will be terminated directly.

[0107] For example, for regular tasks, the scheduling system can determine whether the currently available resources meet the resource requirements of the task. If the available resources do not meet the requirements, the regular task is added to a waiting queue. Tasks in this waiting queue are sorted according to their respective target scores, with tasks having higher target scores having higher scheduling priority. This queuing mechanism guides users to improve task quality, creating a positive cycle between resource efficiency and priority.

[0108] Finally, during the scheduling of multiple tasks to the target cluster, the scheduling system monitors the resource utilization and running status of each task in real time. If any task's resource utilization is detected to be below the utilization threshold or to be inactive, it is determined to be an inefficient task. The scheduling system terminates the execution of the task and releases the cluster resources it occupies.

[0109] The aforementioned utilization thresholds can be configured differently based on the sub-cluster where the task is located. For small-to-medium-sized GPU tasks assigned to sub-cluster 12, resource reclamation is triggered when the GPU utilization rate remains below 70% for a preset duration (e.g., within 2 hours). For very large-scale GPU tasks assigned to sub-cluster 13, resource reclamation is triggered when the GPU utilization rate remains below the resource utilization threshold of 50% for a preset duration.

[0110] In addition to resource utilization detection, the scheduling system also identifies and reclaims inefficient tasks through the following mechanisms: First, an early ROI stopping mechanism, whereby the task is automatically terminated and intermediate models are saved when the training loss decreases at a rate below a preset threshold (e.g., 0.01 / hour), thus reducing invalid training and saving computing power. Second, zombie task identification and reclamation, whereby a task is considered to be in a dead state if there are no log updates within a preset time period (e.g., 1 hour), the task is killed and its occupied GPU memory and other resources are reclaimed to prevent task deadlock from causing resource stagnation.

[0111] It should be noted that in this scheme, the scheduling system also detects idle fragmented computing resources in the target cluster and utilizes these resources to schedule time-limited subtasks for valley filling tasks or ordinary tasks. These time-limited subtasks are subtasks based on a single hour, and the resources they use are preemptible.

[0112] It should also be noted that this solution provides corresponding scheduling methods for computing resource-intensive tasks (hereinafter referred to as computing tasks).

[0113] Figure 6 This paper illustrates a method for scheduling computing tasks in one example of this specification. Figure 6 First, the computing power task is broken down into multiple subtasks, each corresponding to a preset computing power consumption unit. This breakdown can be done in one of two ways: Method 1: Based on the total computing power requirement of the computing power task and the preset single-shard computing power consumption, the computing power task is broken down into multiple subtasks; Method 2: Based on the time window of the computing power task, the executable time range is determined, and the executable time range is divided into multiple time intervals based on idle periods. The offline task is then broken down into multiple subtasks corresponding one-to-one with these time intervals, where each time interval corresponds to one or more of the aforementioned idle periods.

[0114] Secondly, identify idle periods for GPU resources in the target cluster, such as off-peak business hours (e.g., 00:00 to 06:00 at night) or resource gaps after key tasks are completed.

[0115] Finally, multiple subtasks are scheduled to execute during idle periods. Each subtask slice occupies GPU resources during its corresponding idle period and releases those resources upon completion. For any given subtask slice, the scheduling system starts execution at the beginning of the corresponding idle period and releases the occupied GPU resources at the end of the idle period or when the subtask slice completes execution. Different subtask slices can be scheduled to execute on different idle periods or on different GPU cards within the same idle period, achieving spatial and temporal reuse of fragmented computing resources.

[0116] In a specific example, the aforementioned computing task is an interruptible and recoverable pluggable computing task. The scheduling system saves the state information of the computing task during execution. After a subtask fragment of the computing task is interrupted, the execution of the computing task is resumed based on the saved state information. The saving and restoration of task state is achieved through technologies such as containerization isolation and distributed storage, ensuring that subtask fragments can continue execution from the breakpoint after an interruption.

[0117] This solution breaks down offline tasks into time-based subtasks and utilizes idle time periods of the cluster for off-peak scheduling, achieving resource reuse in both time and space dimensions, thereby increasing the average daily utilization rate of the cluster from approximately 50% to over 85%.

[0118] It should be noted that, based on the scheduling method described above, this specification further provides a task and resource query method based on a multi-Agent architecture, which is executed through a multi-Agent interaction system and is used to provide users with natural language interactive query services for scheduling information.

[0119] Specifically, the multi-agent interaction system can receive natural language query requests input by the target user, parse the query request through the main agent, and identify the user's intent. Based on the user's intent, the target sub-agent is determined from multiple specialized sub-agents. Based on the task identifier, the target sub-agent is invoked to obtain factor data related to the running status of the target task from the scheduling system. Based on this factor data, a query result in natural language is generated and returned to the target user.

[0120] The aforementioned specialized sub-Agents include at least one of the following: Task Information Query Agent, used to query task attribute information and queue status, such as task ID, number of GPUs, submission time, and other basic attributes. Queue Analysis Agent, used to analyze the task's position in the queue and estimated waiting time, for example, returning "Your task is 15th in the queue, there are 50 tasks in total, and the estimated waiting time is approximately 30 minutes." Priority Explanation Agent, used to explain the decision logic of task priority; for example, when a user asks "Why is my task priority low?", this agent calls the scheduling rule tool and returns "Your task priority is low because: ① You have executed 50 tasks in the past 24 hours, triggering the fairness decay mechanism; ② Your current task's GPU utilization is only 40%, resulting in a deduction of health points." Identity Conversion Agent, used to convert user identity identifiers (such as nicknames) into system-recognizable identity identifiers (such as employee IDs), resolving the issue of inconsistent identity identifiers. Dimensional statistics agents are used to support resource usage statistics by user and business system dimensions, answering questions such as "How many resources am I using?" or "Total resource usage of a project?". Resource statistics agents provide a global view of resource usage, identifying top users or subsystems using resources.

[0121] Specifically, when the system detects that a user's intent is to query the reason for task queuing or priority details, it identifies the Priority Explanation Agent as the target sub-Agent. The Priority Explanation Agent is invoked, and by connecting to the priority decision tool in the scheduling system, it obtains the target data affecting the current task's priority as the aforementioned factor data. This target data includes at least one of the following: the cumulative number of tasks submitted by the user within a preset time window, the user's target score, and the real-time resource quota usage of the business system to which the task belongs. Based on the target data, a natural language explanation text is generated, such as "Your task currently has a low queue priority for the following reasons: 1. You have successfully run 50 tasks in the past 24 hours, triggering the fairness decay mechanism; 2. Your current task's health score is 75, lower than the queue average of 85."

[0122] Through a multi-agent interaction system, users can directly obtain multi-dimensional scheduling information such as task status, queuing information, and priority explanations via natural language, eliminating the need to memorize complex query commands or manually query multiple systems. This also makes the previously "black box" scheduling decision-making logic transparent to users, reducing their concerns about unfair resource allocation. Furthermore, the multi-agent architecture provides users with natural language query services, making the scheduling logic transparent and lowering the barrier to obtaining scheduling information. Features such as priority explanation also enhance user trust in the scheduling system.

[0123] It should be noted that after adopting this solution, significant technical effects were achieved: the normalized utilization rate of the cluster GPU increased from the original 44% to 92%, with an average daily benefit of 37,000 GPU hours; through a fairness-weighted diminishing returns mechanism, the resource utilization rate of the top 5% of high-frequency users decreased from 80% to 45%, and the average waiting time for small user tasks decreased by 40%; through GPU-level idle scheduling, the overall cluster utilization rate increased from the industry average of 50% to over 85%; at the same time, the SLA guarantee rate for key project tasks reached 99.9%, effectively solving the problems of resource conflicts and inefficiency under a 10,000-GPU cluster.

[0124] Corresponding to the resource scheduling method described above, one embodiment of this specification also provides a resource scheduling apparatus, such as... Figure 7 As shown, the device may include:

[0125] The acquisition unit 702 is used to acquire multiple tasks to be scheduled. The determination unit 704 is used to determine the estimated execution time of any first task among the multiple tasks, based on the task characteristics of the first task and the current resource status related to the GPU card in the target cluster, using a duration prediction model. The determination unit 704 is also used to determine the task priority of the first task based on its task type and the historical GPU quality data of the user who submitted the first task. The generation unit 706 is used to generate different scheduling strategies in the simulation environment based on the estimated execution time, task priority, and current resource status of each of the multiple tasks, using various multi-objective optimization algorithms. The evaluation unit 708 is used to evaluate the execution cost of each scheduling strategy and determine the target scheduling strategy based on the execution cost. The scheduling unit 710 is used to schedule the multiple tasks to the target cluster for execution according to the target scheduling strategy.

[0126] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 3 The method described.

[0127] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements a combination... Figure 3 The method described.

[0128] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the medium or device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0129] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0130] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this specification. It should be understood that the above description is only a specific embodiment of this specification and is not intended to limit the scope of protection of this specification. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of this specification should be included within the scope of protection of this specification.

Claims

1. A resource scheduling method, executed by a scheduling system, comprising: Get multiple tasks to be scheduled; For any first task among the multiple tasks, the estimated execution time of the first task is determined using a duration prediction model based on the task characteristics of the first task and the current resource status related to the GPU card in the target cluster. Based on the task type of the first task and the historical card usage quality data of the user who submitted the first task, the task priority of the first task is determined. In the simulation environment, based on the estimated execution time, task priority and current resource status of each of the multiple tasks, different scheduling strategies are generated by various multi-objective optimization algorithms. Evaluate the execution cost of each scheduling strategy and determine the target scheduling strategy based on the execution cost; According to the target scheduling strategy, the multiple tasks are scheduled to be executed in the target cluster.

2. The method according to claim 1, wherein, The optimization objectives of the multi-objective optimization algorithm include at least: maximizing cluster resource utilization, minimizing the average queuing time of tasks, and ensuring timely scheduling of high-priority tasks.

3. The method according to claim 1, wherein, The process of obtaining multiple tasks to be scheduled includes: Receive natural language instructions input by the user, the natural language instructions indicating the training objective; The natural language instructions are semantically analyzed using a large language model and matched with historical task workflow templates to decompose the training objective into multiple sub-tasks with logical dependencies. The multiple subtasks are grouped into the multiple tasks to be scheduled.

4. The method according to claim 1, further comprising: Detect the resource utilization and operating status of the multiple tasks; If any second task is detected to have a resource utilization rate below the utilization threshold or to be inactive, the execution of the second task is terminated and the cluster resources it occupies are released.

5. The method according to claim 1, wherein, The historical card quality data includes historical success rate and / or GPU utilization. Determining the task priority of the first task includes: Determine the initial priority based on the task type of the first task; The target score for the first task is determined based on the historical success rate and / or GPU utilization; the target score is positively correlated with the historical success rate and / or GPU utilization. The task priority of the first task is determined based on the target score and the initial priority.

6. The method according to claim 5, wherein, Determining the target score for the first task based on the historical success rate and / or GPU utilization includes: Calculate the base score based on the historical success rate and / or GPU utilization. Obtain the user's behavioral events within a predetermined time period; Based on the event type of the behavioral event, the base score is added or subtracted to obtain the target score; the event type includes positive incentive events and negative constraint events.

7. The method according to claim 6, wherein, The positive incentive events include: events where users log into the model; The negative constraint events include any of the following: failure to complete the experimental sharing task, discrepancies between the submitted task resource requirements and actual usage, and long-term GPU occupation with utilization below the threshold.

8. The method according to claim 1, wherein, The task type of the first task includes any of the following: key task with a key project tag, trial run task, ordinary task, and valley filling task.

9. The method according to claim 8, wherein, The first task is either a valley-filling task or a regular task; the method further includes: Detect the idle fragmented computing resources of the target cluster; Utilizing the idle fragmented computing resources, schedule time-based subtasks for the valley filling task or ordinary task; the time-based subtasks are subtasks with a unit of one hour per card, and the resources they use are preemptible.

10. The method according to claim 8, wherein, The first task is a priority task; the method further includes: Determine whether the user's task concurrency meets the preset hard limit conditions; If the aforementioned hard constraints are met, then determine whether the currently available resources meet the resource requirements of the key task; If the currently available resources do not meet the resource requirements, a resource preemption process is triggered. The resource preemption process includes: selecting tasks with a lower priority than the first task or a target score lower than a preset threshold from at least one currently running task, reclaiming the resources occupied by the task to be reclaimed, and allocating the reclaimed resources to the key task. Specifically, when recycling resources for the task to be recycled, if the target score of the task to be recycled is higher than a preset threshold and has not been forcibly terminated, the task to be recycled is saved and added back to the waiting queue; if the target score of the task to be recycled is lower than or equal to the preset threshold, or has been forcibly terminated, the task to be recycled is terminated directly.

11. The method according to claim 8, wherein, The first task is a normal task; the method further includes: Determine whether the currently available resources meet the resource requirements of the ordinary task; If the available resources do not meet the resource requirements, the ordinary task is added to the waiting queue; wherein, each task in the waiting queue is sorted according to its corresponding target score, and the task with the higher target score has a higher scheduling priority in the waiting queue.

12. The method according to claim 8, wherein, Before scheduling the plurality of tasks to the target cluster for execution, the method further includes: If the first task is a test run task, then a preset minimum resource specification is allocated to the test run task, and its maximum runtime is set; or; If the first task is a valley filling task, then the valley filling task is configured to start only when idle resources meet preset conditions, and it is configured to have the attribute of being preempted by key tasks.

13. The method according to claim 1, further comprising: The total amount of GPU time or the total number of tasks successfully scheduled and run by each user who submitted the multiple tasks within a preset rolling time window is counted. When any user's total card time or total number of tasks exceeds the corresponding preset threshold, the priority of newly submitted tasks by that user will be dynamically reduced, and the reduction will be positively correlated with the amount of excess.

14. The method according to claim 1, wherein, The target cluster includes multiple sub-clusters, each configured with a corresponding runtime limit; the method further includes: When the runtime allocated to any sub-cluster exceeds the runtime limit corresponding to that sub-cluster, the subtask is terminated.

15. The method according to claim 14, wherein, The plurality of sub-clusters include: The first sub-cluster is used to execute pure CPU tasks; The second sub-cluster is used to execute GPU tasks with a GPU card count not exceeding a first preset threshold. The third sub-cluster is used to execute GPU tasks where the number of GPU cards exceeds a first preset threshold; The third runtime limit configured for the third sub-cluster is less than the second runtime limit configured for the second sub-cluster; the second runtime limit is less than the first runtime limit configured for the first sub-cluster.

16. The method according to claim 1, wherein, The target cluster is the first cluster among the two types of clusters divided from the original cluster, used to perform generative training or inference tasks; The two clusters also include a second cluster for performing development and debugging tasks; the method further includes: Detect the resource idle status of the second cluster; When idle resources are detected in the second cluster, the idle resources are dynamically allocated to the first cluster; When a new development and debugging task is submitted by a user corresponding to the second cluster, the corresponding amount of resources are reclaimed from the first cluster and returned to the second cluster.

17. The method according to claim 1, further comprising: Acquire a second task that consumes computing resources; The second task is broken down into multiple subtasks, each subtask corresponding to a preset computing power consumption unit; Obtain the idle time periods of GPU resources in the target cluster; The multiple subtasks are scheduled to be executed during the idle time period, wherein each subtask occupies GPU resources during the corresponding idle time period and releases the GPU resources after execution.

18. The method according to claim 17, wherein, The step of breaking down the second task into multiple subtask fragments includes: Based on the total computing power requirement of the second task and the preset single-chip computing power consumption, the second task is broken down into multiple sub-task fragments; or, The executable time range is determined based on the time window of the second task. The executable time range is divided into multiple time intervals based on the idle time period. The second task is then decomposed into multiple sub-task fragments that correspond one-to-one with the multiple time intervals, wherein each time interval corresponds to one or more of the idle time periods.

19. The method according to claim 17, wherein, The second task is an interruptible and resumable pluggable task; the method further includes: Save the status information of the second task during its execution; After a subtask fragment of the first task is interrupted, the execution of the second task is resumed based on the status information.

20. The method of claim 17, wherein, The step of scheduling the multiple subtasks to be executed during the idle time slot includes: For any first subtask slice, execution begins at the start of the corresponding first idle period and the occupied GPU resources are released at the end of the first idle period or when the first subtask slice is completed. Different subtasks are scheduled to be executed on different idle time slots or on different GPU cards within the same idle time slot.

21. The method according to claim 1, further comprising: Receive a natural language query request from the target user, which includes the task identifier of the target task to be queried; The main agent parses the natural language query request to identify the user's intent; Based on the user's intent, the target sub-Agent is determined from multiple specialized sub-Agents; Based on the task identifier, the target sub-Agent is invoked to obtain factor data related to the running status of the target task from the scheduling system; The query results are generated in natural language based on the factor data and returned to the target user.

22. The method according to claim 21, wherein, The multiple specialized sub-Agents include: a task information query agent, used to query task attribute information and queue status; a queue analysis agent, used to analyze the task's position in the queue and the estimated waiting time; a priority interpretation agent, used to interpret the decision logic of task priority; an identity conversion agent, used to convert user identity identifiers into system-recognizable identity identifiers; a statistics agent, used to statistically analyze resource usage according to preset dimensions; and a resource statistics agent, used to provide a global perspective on resource usage.

23. The method according to claim 22, wherein, The step of determining the target sub-Agent from multiple professional sub-Agents based on the user intent includes: If the user intent is identified as querying the reason for task queuing or priority details, then the priority interpretation sub-Agent is identified as the target sub-Agent; The step of obtaining factor data affecting the running status of the target task from the scheduling system includes: The priority interpretation agent is invoked to obtain data affecting the priority of the target task from the scheduling system as the factor data.

24. A resource scheduling device, comprising: The acquisition unit is used to acquire multiple tasks to be scheduled. The determining unit is used to determine the estimated execution time of any first task among the plurality of tasks by using a duration prediction model, based on the task characteristics of the first task and the current resource status related to the GPU card in the target cluster. The determining unit is further configured to determine the task priority of the first task based on the task type of the first task and the historical card usage quality data of the user who submitted the first task; The generation unit is used to generate different scheduling strategies in a simulation environment based on the estimated execution time, task priority, and current resource status of each of the multiple tasks, using a variety of multi-objective optimization algorithms. The evaluation unit is used to evaluate the execution cost of each scheduling strategy and determine the target scheduling strategy based on the execution cost. The scheduling unit is used to schedule the multiple tasks to the target cluster for execution according to the target scheduling strategy.

25. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed in a computer, it causes the computer to perform the method of any one of claims 1-23.

26. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-23.

Images