A multi-tenant GPU cluster elasticity quota scheduling method and system

Abstract

This invention implements a method and system for elastic quota scheduling of multi-tenant GPU clusters. First, data collection and preprocessing steps are performed, extracting key indicators affecting task execution time from the task history database and inputting the data into a prediction model module. The prediction model module constructs and trains a task duration prediction model for each user, and then inputs the trained task duration prediction model to obtain the time prediction result. Finally, the established model is used to predict the execution time of newly submitted tasks, and the accuracy of the model is evaluated using mean absolute error (MAE), mean squared error (MSE), and coefficient of determination (R²). This solution addresses the issues of low resource utilization and fairness and efficiency in resource use in multi-tenant scenarios currently existing in GPU clusters trained with deep learning.

Description

Technical Field

[0001] This invention relates to the field of cloud computing technology, and in particular to a method and system for elastic quota scheduling of multi-tenant GPU clusters. Background Technology

[0002] In the field of multi-tenant GPU cluster resource management and scheduling, several related technologies and methods already exist. These include:

[0003] Task Packaging and Shared GPU Scheduling: To improve GPU resource utilization, many studies have explored task packaging and scheduling onto a single GPU for resource sharing. Gandiva utilizes online analytics to reflectively determine whether to share resources on accelerators. AntMan achieves finer-grained GPU sharing through dynamic scaling. Salus implements two primitives for fast task switching and memory sharing to achieve more efficient GPU sharing. Horus converts the user model into an ONNX graph representation and extracts workload features to determine task packaging. Lucid supports task packaging using a non-intrusive scheduling paradigm. Muri leverages multi-resource interleaving to improve resource utilization and reduce resource consumption. However, these techniques cannot accurately predict interference between tasks to guide scheduling decisions.

[0004] Queue-based or fixed-quota scheduling systems: Currently, there are fixed-quota scheduling systems that allocate a fixed resource quota to each tenant in a GPU cluster, such as the Slurm cluster management system. These systems are typically based on static resource allocation strategies and cannot dynamically adjust according to real-time demand and resource utilization. Therefore, such fixed-quota scheduling systems may lead to resource waste or insufficient resources.

[0005] Cluster management tools: Kubernetes is a popular open-source orchestration solution suitable for container-based workloads. With Kubernetes, these workloads can be managed effectively in a manner similar to traditional HPC clusters. However, Kubernetes itself does not provide all of Slurm's scheduling capabilities—such as batch scheduling and swarm scheduling—so some extensible solutions are needed to implement these functionalities.

[0006] GPU Cluster Monitoring Technology: Existing technologies include several methods for monitoring GPU clusters, such as Prometheus and DCGM Exporter, used to collect GPU resource utilization and performance data. These tools can monitor hardware metrics such as GPU device temperature, power consumption, and memory usage, as well as job execution status and performance metrics. However, current resource managers and monitors typically collect metrics at the GPU hardware device level, lacking the collection and analysis of process-level GPU utilization information.

[0007] The cluster managed by the above technology may have the following problems:

[0008] Low resource utilization: Users often fail to accurately predict the resource requirements of their tasks, resulting in a mismatch between requested and actual resource usage, leading to low GPU resource utilization. This not only means that expensive hardware resources are not being fully utilized, but also that other users who might need these resources cannot obtain them in a timely manner.

[0009] Inadequate quota management mechanisms: Traditional quota management mechanisms are typically fixed and lack flexibility, failing to adjust resource allocation based on the real-time status of the cluster. This mechanism cannot be effectively utilized when resources are idle, and may lead to long task queues when resources are scarce, affecting cluster throughput and user experience.

[0010] Insufficient user feature mining: Existing scheduling work often fails to fully mine user usage patterns and behavioral characteristics. If this information is effectively utilized, it can predict user resource needs, dynamically adjust resource allocation strategies, and thus optimize the overall cluster operating efficiency.

[0011] To address the aforementioned shortcomings, this application aims to resolve the need for more intelligent quota management and resource scheduling mechanisms in GPU cluster scheduling systems under multi-tenant environments. This will enable efficient resource utilization, ensure fairness and service level quality, and mine user behavior data to further optimize scheduling strategies. Summary of the Invention

[0012] To address this, this invention first proposes a multi-tenant GPU cluster elastic quota scheduling method. The method begins with data collection and preprocessing. Key indicators affecting task execution time are extracted from the task history database. After cleaning and standardization through a data preprocessing workflow, text feature extraction is performed. TF-IDF transformation and NMF dimensionality reduction methods are used to extract key information representing user task characteristics and convert the text into numerical vectors that the model can process. For the remaining numerical features, standardization is performed before the data is input into the prediction model module. The prediction model module employs a method that combines user queues and cluster queues to optimize... The task scheduling process builds and trains a task duration prediction model for each user. When the user's historical information is insufficient to train an independent model, it reverts to using a general model trained on all tasks in the cluster for prediction. For the data collected and preprocessed after feature extraction, feature processing is first performed through text vectorization, data normalization, and feature aggregation. Then, the data is input into the trained task duration prediction model to obtain the time prediction result. Finally, the established model is used to predict the execution time of newly submitted tasks, and the accuracy of the model is evaluated by the mean absolute error (MAE), mean squared error (MSE), and coefficient of determination (R²).

[0013] The task duration prediction model needs to be updated periodically. Specifically, as the number of historical tasks gradually increases, user habits change, or when there are hardware upgrades, system software updates, or changes in workload, the model will be updated regularly to ensure the accuracy and real-time nature of the data.

[0014] This invention also proposes a multi-tenant GPU cluster elastic quota scheduling system, including a task management module, a scheduling module, a performance analysis module, a monitoring module, and a historical load database. Each module communicates with the other via gRPC through interface calls and information transmission.

[0015] The task management module is responsible for managing user-submitted tasks, maintaining task queues for each user, supporting queuing based on task priority, managing elastic quotas to ensure fairness, and collaborating with the cluster queue manager to ensure that tasks can be submitted to the scheduling module efficiently and in an orderly manner.

[0016] The scheduling module integrates a performance interference mixed model and a task time prediction model. It determines the scheduling order and node placement of tasks based on their resource requirements, priorities, and predicted running times. When a high-priority task arrives, it triggers the preemption logic of low-priority tasks to ensure the service level of critical tasks.

[0017] The monitoring module obtains real-time running information of tasks through process-level monitoring;

[0018] The performance analysis module uses monitoring data to analyze the performance indicators of various tasks and records the performance indicators in the resource object field of the task, providing input for the performance interference model.

[0019] The historical load database stores historical monitoring data, providing historical data support for the performance analysis module and the task time prediction module.

[0020] The specific implementation of the task management module is as follows: the user submits a task to the server, the server collects the task's metadata and writes it into the database. The task format has been defined in the previous section. The submitted task first goes through the task duration prediction module. The task's metadata is used as input to predict the task's duration. If the user does not have historical data, the system will wait for the task's performance analysis data. When the performance analysis is completed, the task's utilization information is input into the prediction model to estimate the task's duration.

[0021] The task is then submitted to the user queue for queuing. After being checked by the task queue and quota module, the specific AIJob resources are created through the cluster interaction module and submitted to the Kubernetes cluster.

[0022] Once the task is completed, the cluster interaction module will detect the end of the task and request performance data. Then, it will write the performance data of the task runtime into the historical load database for future prediction and analysis.

[0023] The workflow of the task queue and quota module is as follows: First, the queues are initialized by creating an independent queue for each user to store task requests submitted by the user. These queues are organized according to the user's quota and the task's priority. Next, quotas are set: each user has a fixed resource quota. Then, performance analysis is performed: while tasks are waiting in the queue, they are submitted to the performance analysis module for utilization data collection, and the utilization information is then written back to the task. Next, a polling strategy is implemented: the controller polls the user queues, retrieves all high-priority tasks that do not exceed the user's resource quota, queues them, sorts them according to quota utilization and submission time, and submits them to the cluster queue. If high-priority tasks are empty and the cluster queue is empty, low-priority tasks can be retrieved according to quota utilization and submitted to the cluster queue. Finally, tasks are submitted: the controller module is requested to check the cluster queue's workload, AIJobs are created one by one, and submitted to the cluster. If the submission is successful, the user's quota is updated, and the task is deleted from the queue.

[0024] The scheduling module is built on the Kubernetes scheduler framework and adopts a plug-in design.

[0025] The monitoring module deploys multiple Exporters on nodes to expose monitoring metrics, which are then aggregated and persisted by Prometheus, and visualized by Grafana.

[0026] The specific implementation method of the performance analysis module is as follows:

[0027] When a user task is queued in the cluster, it is first submitted to the performance analysis module for analysis. This module submits the user's Pod to the specified performance analysis node, and periodically reads the monitoring metrics of Promethues, collects the utilization metrics of the task, interacts with the Kubernetes cluster, and returns the results to the client.

[0028] The performance analysis module provides four interfaces: analysis task submission interface, task cancellation analysis interface, queue information retrieval interface, and analysis result request interface.

[0029] The specific workflow is as follows: Step 1, Performance Analysis Request Reception: The interface receives performance analysis requests from users and parses the task information contained in the requests; Step 2, Pod Task Submission: Based on the parsed task information, a corresponding Pod is created in Kubernetes, the Pod is configured using the Job field, and a performance analysis node is specified; Step 3, PodController Startup: The Controller listens to the status of the Pod in Kubernetes and waits for the Pod to enter the Running state; Step 4, Data Monitoring and Collection: After the Pod runs, the Controller periodically sends requests to Prometheus to obtain monitoring data; Step 5, Monitoring Data Steady-State Judgment: When the rate of change of the average data of the Pod performance is lower than the threshold for several consecutive reads, and the steady-state condition is met, the performance data is considered reliable; Step 6, Pod Stopping and Resource Release: The Pod is stopped from running, and the resources it occupies are released; Step 7, Utilization Information Recording: The collected utilization information is written into the Pod's annotations; Step 8, Result Feedback: The analysis results are returned to the user through the interface and stored in the database for subsequent queries.

[0030] The technical effects to be achieved by this invention are as follows:

[0031] To address the issues of low resource utilization and fairness and efficiency in resource use in multi-tenant deep learning training GPU clusters, specifically:

[0032] Improve resource utilization: Enable users to accurately predict the resource requirements of their tasks, thereby improving the utilization of GPU resources.

[0033] Improve the performance of the quota management mechanism: Adjust resource allocation according to the real-time status of the cluster, effectively utilize resources when they are idle, and improve the cluster's throughput and user experience.

[0034] Enhance user feature mining capabilities: Fully explore user usage patterns and behavioral characteristics, predict user resource needs, dynamically adjust resource allocation strategies, and thereby optimize the overall cluster operating efficiency. Attached Figure Description

[0035] Figure 1 Schematic diagram of the task duration prediction algorithm;

[0036] Figure 2 Task management module workflow;

[0037] Figure 3 Scheduling module workflow;

[0038] Figure 4 Schematic diagram of a multi-tenant GPU cluster elastic quota scheduling system;

[0039] Figure 5 Schematic diagram of the elastic quota scheduling algorithm; Detailed Implementation

[0040] The following are preferred embodiments of the present invention, which are described in conjunction with the accompanying drawings. However, the present invention is not limited to these embodiments.

[0041] This invention proposes a flexible quota scheduling method for multi-tenant GPU clusters. It employs machine learning, building and training a separate model for each user, enabling the model to more accurately reflect the user's task characteristics and execution behavior. When user historical information is insufficient to train an independent model, the system reverts to using a general model trained on all tasks across the cluster for prediction. This two-layer prediction mechanism ensures both broad applicability and personalized accuracy in prediction.

[0042] The main algorithm flow is as follows:

[0043] Data Collection and Preprocessing: Key metrics affecting task execution time are extracted from the task history database, such as the amount of resources requested, and other finer-grained performance metrics (e.g., sm_active, cpu_usage, mem_copy_util, gpu_mem_max). This data undergoes cleaning and standardization through a data preprocessing workflow to ensure model accuracy. For text feature extraction, TF-IDF transformation and NMF dimensionality reduction methods are used to extract key information that represents the user's task characteristics. These methods transform text into numerical vectors that the model can process. For the remaining numerical features, Z-score normalization is used to avoid the model being excessively affected by features with large numerical ranges during training.

[0044] Model Building: In the user history information layer, TF-IDF transformation and NMF dimensionality reduction methods are applied to the task history data of each user to extract key information that can represent the user's task characteristics. If the amount of user history data is sufficient, a targeted prediction model will be built based on this information. For new users or users with insufficient history data, the cluster history information layer is used to train a general prediction model based on the historical performance data of the entire cluster.

[0045] Prediction and Evaluation: The established model is used to predict the execution time of newly submitted tasks, and the accuracy of the model is evaluated using metrics such as MAE (mean absolute error), MSE (mean squared error), and R2 (coefficient of determination).

[0046] Periodic Update Model: As the number of historical tasks gradually increases, user habits may also change. Furthermore, cluster hardware upgrades, system software updates, or changes in workload can all affect task execution time. Therefore, this paper will update the model periodically to ensure data accuracy and real-time performance.

[0047] Elastic management mechanism for GPU resource quotas among multiple tenants:

[0048] In modern multi-tenant GPU cluster environments, resource management and scheduling are crucial for ensuring efficiency and fairness. To address the conflict between resource utilization and fairness inherent in traditional fixed quota allocation mechanisms, this invention proposes an elastic management mechanism for GPU resource quotas among multiple tenants. The core of this mechanism lies in two aspects: First, when idle resources exist in the cluster, the system can elastically allocate these resources to queued tasks, thereby improving resource utilization. Second, when a task arrives for the original user, the system can ensure the protection of the user's fixed quota through a refined preemption strategy, while considering factors such as task size and the number of times preemption occurs to avoid excessive preemption and maintain fairness.

[0049] In the elastic management mechanism for GPU resource quotas among multiple tenants proposed in this paper, the design of queues is crucial for achieving efficient resource utilization and fair allocation. The mechanism involves a two-level queue system: user queues and cluster queues, which work together to optimize the task scheduling process.

[0050] The user queue is the first-level queue, with each user having their own queue. When a user submits a task, these tasks first enter their respective user's queue. Tasks within a user queue are ordered according to their specified priority and arrival time (following the first-come, first-served, i.e., FIFO principle). This design ensures fair scheduling of tasks within a user's queue and provides ordered input for subsequent cluster-level scheduling. The quota management module polls users' task queues to check if there are any tasks available for entry into the cluster queue, whether users have exceeded their fixed quotas, and whether there are idle resources in the cluster. Furthermore, when selecting low-priority tasks, it considers the user's already used quota resources to ensure relatively fair allocation.

[0051] The cluster queue is the second-level queue. It receives tasks from various user queues and further schedules them based on task priority and the current resource status of the cluster. The cluster queue is divided into a high-priority task queue and a low-priority task queue. When idle resources exist in the cluster, tasks in the high-priority task queue will be scheduled first to ensure timely processing of important tasks. Furthermore, when there are high-priority tasks in the cluster queue but cluster resources are occupied by low-priority tasks, the system will preempt resources according to a set strategy to ensure the execution of the high-priority tasks. This two-level queue design fully considers the resource contention problem in a multi-tenant environment and achieves a balance between resource utilization and fairness through reasonable queuing and scheduling strategies.

[0052] The scheduling algorithm is described as follows:

[0053] First, the quota management module polls the user queue to determine if there are any high-priority tasks that do not exceed the user's fixed quota, and submits them to the cluster queue. If the cluster still has spare resources at this time, it sorts the user queues by their used quotas from smallest to largest, continues to poll the user queues, and retrieves low-priority tasks to submit to the cluster queue in the scheduler.

[0054] Next, the scheduler places tasks, prioritizing tasks with higher priority.

[0055] When there are high-priority tasks waiting in the cluster queue and low-priority tasks running in the cluster, a preemption mechanism will be triggered. The preempted task will be evaluated according to the following metrics:

[0056] 1. The resource request volume should be matched with the request volume of high-priority tasks to avoid preempting too many tasks.

[0057] 2. Select the task with the largest user quota over-allocation for preemption. If there are multiple over-allocation tasks, select the task that has been running for a longer time and has been preempted less often to avoid frequent preemption.

[0058] Performance interference-aware training task hybrid placement strategy:

[0059] To establish an accurate model of interference in deep learning training tasks, this invention executes multiple tasks in parallel under the same computing environment, obtaining the performance degradation rate after mixed placement. This allows for the construction of a predictive model of inter-task performance interference, capable of predicting the degree of performance interference between different training tasks when placed in parallel. This predictive model not only provides a theoretical basis for the efficient allocation of GPU cluster resources but also provides empirical support for the optimization of multi-task scheduling systems.

[0060] To achieve efficient cluster resource management, this invention proposes a comprehensive task placement algorithm that aims to maximize cluster utilization, increase throughput, and minimize the impact of resource fragmentation. Furthermore, by carefully balancing resource sharing among tasks, this algorithm further reduces the performance degradation of tasks co-located on the same GPU.

[0061] First, the dataset was constructed. This paper selected 16 common deep learning models, and different batch sizes and mixed precisions were chosen for each model. (See table below.)

[0062]

[0063] First, performance and utilization data for each model are collected individually. Tracking is performed at the start and end of the training phase in each model's code file to return the start and end times. The average speed of each iteration is calculated, and then the utilization data collected by Prometheus based on the start and end times is written to a data table. A total of 90 data points were collected, and the individual performance data table is shown below:

[0064]

[0065] Next, a mixed-task run was performed. For all task combinations, the total GPU memory potentially required by the selected model was estimated based on previously collected individual performance data. If the total memory usage exceeded a preset threshold (30GB), performance evaluation was not performed, as this would cause an OutOfMemoryError (OOM). An independent process was created for each selected model, and a Barrier from the multiprocessing library was used to synchronize processes, ensuring they were all warmed up before performance data collection began. Once all models were warmed up, the start time of data collection was recorded. After each subprocess finished, the main process collected GPU-related metrics by accessing Prometheus. Finally, the collected performance data and GPU metrics were integrated to calculate the speed decay rate and other relevant metrics for each model. Through mixed deployments of two and three tasks, 7466 data points were collected. Compared to the previous dataset on performance utilization of exclusive tasks, the dataset on mixed tasks adds columns starting with cosum_, which represent the sum of utilization data when multiple tasks are run in parallel, reflecting the resource sharing efficiency when different models are running in parallel. It also adds a co_speed column, which represents the speed after mixed running, and decay, which is calculated by co_speed / speed, to represent the speed decay during training.

[0066] To establish a reliable interference prediction model, this invention explores several algorithms:

[0067] Mathematical formulaic models, such as linear regression, second-order linear regression, and Bayesian networks, which are based on statistical methods to predict dependencies, can provide an intuitive interpretation of the model.

[0068] Decision tree models, including gradient boosting decision trees and random forests, utilize an ensemble of multiple decision trees to improve the accuracy and robustness of predictions.

[0069] Traditional machine learning methods, such as deep neural networks (DNN), support vector machines (SVM), convolutional neural networks (CNN), and recurrent neural networks (RNN), are suitable for handling complex nonlinear relationships and are able to learn complex feature representations on large-scale datasets.

[0070] After testing these representative models, this paper compares the mean squared error (MSE) and the coefficient of determination (R²). 2 Two evaluation metrics, mean squared error (MSE) and coefficient of determination (R²), are used to assess their performance. The MSE measures the difference between the model's predictions and the actual values, while the R² indicates the model's ability to explain data variability. In this study, the random forest regression model exhibited the best performance, demonstrating the lowest MSE and the highest R². 2The score indicates that it outperforms other models in both accuracy and interpretability. Therefore, the random forest regression model was selected as the final model to predict performance degradation in hybrid deployment environments.

[0071] Regarding the accuracy of the interference prediction model, the model of this invention achieved an accuracy of up to 92%. Compared with other scheduling algorithms, it used the scheduling process of a real cluster on the scheduling simulator, resulting in performance improvements in metrics such as queuing time delay.

[0072] Meanwhile, multi-tenant quotas are guaranteed 99% of the time.

[0073]

[0074] This invention also provides a multi-tenant GPU cluster elastic quota scheduling system, developed based on Kubernetes, the most popular resource management framework in the cloud-native field. It defines customized resources for user training tasks, manages tasks through Kubernetes' controller mechanism, extends the Kubernetes scheduling plugin framework, and develops a process-level fine-grained monitoring exporter for resource utilization monitoring. The overall system architecture is shown below. It mainly consists of the following modules: task queue module, scheduling module, performance analysis module, and online monitoring module. Each module communicates with other modules via gRPC for interface calls and information transmission. The functions of each module are as follows:

[0075] Task Management Module: Responsible for managing user-submitted tasks. This module maintains task queues for each user, supports queuing based on task priority, and manages elastic quotas to ensure fairness. It collaborates with the cluster queue manager to ensure tasks are submitted to the scheduling module efficiently and in an orderly manner.

[0076] The scheduling module is the core decision engine, integrating a performance interference mixed deployment model and a task time prediction model. It is responsible for determining the scheduling order and node placement of tasks based on their resource requirements, priorities, and predicted runtime. When a high-priority task arrives, this module can also trigger preemption logic for low-priority tasks to ensure the service level of critical tasks.

[0077] Monitoring module: Obtains real-time task execution information through process-level monitoring. This information is crucial for administrators to have transparency regarding the cluster's operational status, provides data support for task performance analysis, and is periodically stored in the historical load database.

[0078] The performance analysis module uses monitoring data to analyze performance metrics for various tasks, such as GPU utilization, SM utilization, memory usage, and video memory usage, and records these metrics in the task's resource object field. These analysis results provide input to the performance interference model, helping the scheduling module to perform more accurate resource management and task scheduling.

[0079] Historical load database: A database that stores historical monitoring data, providing historical data support for the performance analysis module and task time prediction module, enabling the prediction model to be trained and optimized based on actual operating data.

[0080] Through the organic combination and collaboration of the above modules, GPUSched enables more intelligent and efficient GPU resource scheduling and management in complex and ever-changing multi-tenant environments. This not only improves overall resource utilization but also enhances user experience, ultimately boosting the overall cluster's operational efficiency by reducing task waiting time and improving computing performance.

[0081] Monitoring module:

[0082] The monitoring module serves as the foundation for performance analysis and cluster utilization status, providing multi-dimensional monitoring metric collection. Multiple exporters are deployed on nodes to expose monitoring metrics, which are then aggregated and persisted by Prometheus, and finally visualized by Grafana.

[0083] Performance Analysis Module:

[0084] When a user task is queued in the cluster, it is first submitted to the performance analysis module for analysis. This module submits the user's Pod to the specified performance analysis node and periodically reads Promethues monitoring metrics to collect task utilization metrics, which are then returned to the client.

[0085] The performance analysis module provides four interfaces: analysis task submission interface, task cancellation analysis interface, queue information retrieval interface, and analysis result request interface.

[0086] The client submits tasks for analysis via the task submission interface, and this module creates the corresponding analysis Pod. If cancellation is needed, the task cancellation interface can be called, and the module will delete the corresponding analysis Pod. It can also retrieve information about the queue of tasks currently being analyzed to determine if the performance analysis nodes are overloaded.

[0087] The performance module needs to interact with the Kubernetes cluster, create workload Pods for performance analysis, start a controller to monitor the Pod status, and exchange data with the monitoring module. The specific workflow is as follows:

[0088] 1. Performance Analysis Request Reception: The interface receives performance analysis requests from users and parses the task information contained in the requests.

[0089] 2. Pod Task Submission: Based on the parsed task information, create the corresponding Pod in Kubernetes, configure the Pod using the Job field, and specify the performance analysis node.

[0090] 3. Pod Controller Startup: The controller monitors the status of Pods in Kubernetes and waits for the Pods to start.

[0091] Running status.

[0092] 4. Data monitoring and collection: After the Pod starts running, the Controller periodically sends requests to Prometheus to obtain monitoring data.

[0093] 5. Steady-state assessment of monitoring data: When the rate of change of the average data for Pod performance is lower than a threshold for several consecutive reads, the performance data is considered reliable.

[0094] 6. Stopping and Releasing Resources: Stop the Pod from running and release the resources it occupies.

[0095] 7. Utilization Information Recording: The collected utilization information is written into the Pod's annotations.

[0096] 8. Results Feedback: The analysis results will be returned to the user via the interface and stored in the database for subsequent queries.

[0097] Task Management Module:

[0098] Users submit tasks to the server, which collects the task's metadata and writes it to the database. The task format was defined in the previous section. The submitted task then first passes through the task duration prediction module. Here, the task's metadata is used as input to predict the task's duration. If the user has no historical data, the system will wait for performance analysis data. Once the performance analysis is complete, the task's utilization information is input into the prediction model to estimate the task's duration.

[0099] The task is then submitted to the user queue for queuing. After being checked by the quota management module, it is submitted to the Kubernetes cluster through the cluster interaction module to create specific AIJob resources.

[0100] Once the task is completed, the cluster interaction module will detect the end of the task and request performance data. Then, it will write the performance data of the task runtime into the historical load database for future prediction and analysis.

[0101] The workflow of the task queue and quota module:

[0102] 1. Initialize queues: The system creates an independent queue for each user to store the task requests submitted by the user. These queues are organized according to the user's quota and the task's priority.

[0103] 2. Set quotas: Each user has a fixed resource quota to ensure that a single user does not consume too many cluster resources and affect the tasks of other users.

[0104] 3. Performance Analysis: While a task is waiting in the queue, it is submitted to the performance analysis module to collect utilization data, and then the utilization information is written back to the task.

[0105] 4. Polling Strategy: The controller polls the user queue, retrieves all high-priority tasks that do not exceed the user's resource quota, and queues them. These tasks are then sorted by quota usage and submission time before being submitted to the cluster queue. If both the high-priority and cluster queues are empty, lower-priority tasks can be retrieved based on quota usage and submitted to the cluster queue.

[0106] 5. Submit Task: Request the controller module to check the cluster queue's workload, create AIJobs one by one, and submit them to the cluster. If the submission is successful, update the user's quota and delete the task from the queue.

[0107] Through the above strategies, the introduction of the quota system avoids excessive resource consumption by individual users, ensures that users can fairly obtain system resources within the quota limit, and can schedule low-priority tasks when system resources are sufficient, thereby improving the utilization rate of the cluster.

[0108] Scheduling module:

[0109] The system's scheduling module is built on the Kubernetes scheduler framework. It employs a pluggable design, allowing for flexible expansion and adaptation to diverse scheduling needs. Leveraging the pluggable nature of the Kubernetes scheduler, this system introduces a series of custom scheduling plugins to achieve fine-grained control and optimization of Pod load balancing.

Claims

1. A multi-tenant GPU cluster elastic quota scheduling method, characterized in that: First, data collection and preprocessing steps are performed. Key indicators affecting task execution time are extracted from the task history database, including: the number of resources requested, sm_active, cpu_usage, mem_copy_util, and gpu_mem_max. After cleaning and standardization through the data preprocessing workflow, text feature extraction is performed. TF-IDF transformation and NMF dimensionality reduction methods are used to extract information that can represent the user's task characteristics and convert the text into numerical vectors that the model can process. For the numerical features in the key indicators, standardization is performed before the data is input into the prediction model module. The prediction model module uses a user queue and The method of using a cluster queue to optimize the task scheduling process involves building and training a task duration prediction model for each user. When the user's historical information is insufficient to train an independent model, the method reverts to using a general model trained on all tasks in the cluster for prediction. For the data collected and preprocessed after feature extraction, feature processing is first performed through text vectorization, data normalization, and feature aggregation. Then, the data is input into the trained task duration prediction model to obtain the time prediction result. Finally, the established model is used to predict the execution time of newly submitted tasks, and the accuracy of the model is evaluated using the mean absolute error (MAE), mean squared error (MSE), and coefficient of determination (R²). The prediction model module employs a combination of user queues and cluster queues as a flexible management mechanism for GPU resource quotas among multiple tenants. The user queue is the first-level queue. Each user has their own user queue. When a user submits a task, the task first enters the user's queue. The tasks in the user queue will be sorted according to their specified priority and arrival time, following the first-come, first-served principle. The quota management module polls the user's task queue to check if there are any tasks that can enter the cluster queue, whether the user's fixed quota has been exceeded, and whether there are any idle resources in the cluster. When selecting low-priority tasks, the module will take into account the quota resources already used by the user to ensure relatively fair allocation. The cluster queue is a second-level queue that receives tasks from various user queues and performs further scheduling based on task priority and the current resource status of the cluster. The cluster queue is divided into a high-priority task queue and a low-priority task queue. When there are idle resources in the cluster, tasks in the high-priority task queue will be scheduled first to ensure the timely processing of important tasks. When there are high-priority tasks in the cluster queue but cluster resources are occupied by low-priority tasks, the system will preempt resources according to the set strategy to ensure the execution of high-priority tasks. The specific method for resource scheduling is as follows: First, the quota management module polls the user queue to determine if there are any high-priority tasks that do not exceed the user's fixed quota, and submits them to the cluster queue. If the cluster still has spare resources at this time, it sorts the user's used quota from smallest to largest, continues to poll the user queue, and takes out low-priority tasks to submit to the cluster queue in the scheduler. Next, the scheduler places tasks, prioritizing the placement of high-priority tasks. When there are high-priority tasks waiting in the cluster queue and low-priority tasks running in the cluster, a preemption mechanism will be triggered. The preempted task will be evaluated according to the following metrics: First, the resource request volume should be matched with the request volume of high-priority tasks to avoid preempting too many tasks; Secondly, select the task with the largest user quota over-allocation for preemption. If there are multiple over-allocation tasks, select the task that has been running for a longer time and has been preempted less often to avoid frequent preemption.

2. The method for elastic quota scheduling of a multi-tenant GPU cluster as described in claim 1, characterized in that: The task duration prediction model needs to be updated periodically, specifically by regularly updating the model to ensure the accuracy and real-time nature of the data.

3. A multi-tenant GPU cluster elastic quota scheduling system, characterized in that, It includes a task management module, a scheduling module, a performance analysis module, a monitoring module, and a historical load database. Each module communicates with the others via gRPC through interface calls and information transmission. The task management module is responsible for managing user-submitted tasks, maintaining task queues for each user, supporting queuing based on task priority, managing elastic quotas to ensure fairness, and collaborating with the cluster queue manager to ensure that tasks can be submitted to the scheduling module efficiently and in an orderly manner. The scheduling module integrates a performance interference mixed model and a task time prediction model, and applies the multi-tenant GPU cluster elastic quota scheduling method described in claim 1 or 2 to determine the scheduling order and node placement of tasks based on the resource requirements, priorities and predicted running times of the tasks. The monitoring module obtains real-time running information of tasks through process-level monitoring; The performance analysis module uses monitoring data to analyze the performance indicators of various tasks and records the performance indicators in the resource object field of the task, providing input for the performance interference model. The historical load database stores historical monitoring data, providing historical data support for the performance analysis module and the task time prediction module.

4. The multi-tenant GPU cluster elastic quota scheduling system as described in claim 3, characterized in that, The specific implementation of the task management module is as follows: the user submits a task to the server, the server collects the task's metadata and writes it into the database. The task format has been defined in the previous section. The submitted task first goes through the task duration prediction module. The task's metadata is used as input to predict the task's duration. If the user does not have historical data, the system will wait for the task's performance analysis data. When the performance analysis is completed, the task's utilization information is input into the prediction model to estimate the task's duration. The task is then submitted to the user queue for queuing. After being checked by the task queue and quota module, the specific AIJob resources are created through the cluster interaction module and submitted to the Kubernetes cluster. Once the task is completed, the cluster interaction module will detect the end of the task and request performance data. Then, it will write the performance data of the task runtime into the historical load database for future prediction and analysis.

5. The multi-tenant GPU cluster elastic quota scheduling system as described in claim 4, characterized in that, The workflow of the task queue and quota module is as follows: First, the queue is initialized by creating an independent queue for each user to store the task requests submitted by the user. These queues are organized according to the user's quota and the priority of the task. Then, the quota is set so that each user has a fixed resource quota. Next, performance analysis is performed: while the task is waiting in the queue, it is submitted to the performance analysis module to collect utilization data, and then the utilization information is written back to the task. Then, a polling strategy is implemented: the controller polls the user queue, retrieves all high-priority tasks that do not exceed the user's resource quota, queues them, sorts them according to the quota usage and submission time, and submits them to the cluster queue; If high-priority tasks are empty and the cluster queue is empty, low-priority tasks are retrieved according to quota usage and submitted to the cluster queue. Finally, to submit the task: request the controller module to check the cluster queue's workload, create AIJobs one by one, and submit them to the cluster. If the submission is successful, update the user's quota and delete the task from the queue.

6. The multi-tenant GPU cluster elastic quota scheduling system as described in claim 5, characterized in that, The scheduling module is built on the Kubernetes scheduler framework and adopts a plug-in design.

7. The multi-tenant GPU cluster elastic quota scheduling system as described in claim 6, characterized in that, The monitoring module deploys multiple Exporters on nodes to expose monitoring metrics, which are then aggregated and persisted by Prometheus, and visualized by Grafana.

8. The multi-tenant GPU cluster elastic quota scheduling system as described in claim 7, characterized in that, The specific implementation method of the performance analysis module is as follows: When a user task is queued in the cluster, it is first submitted to the performance analysis module for analysis. This module submits the user's Pod to the specified performance analysis node, and periodically reads the monitoring metrics of Promethues, collects the utilization metrics of the task, interacts with the Kubernetes cluster, and returns the results to the client. The performance analysis module provides four interfaces: analysis task submission interface, task cancellation analysis interface, queue information retrieval interface, and analysis result request interface. The specific workflow is as follows: Step 1, Performance Analysis Request Reception: The interface receives performance analysis requests from users and parses the task information contained in the requests; Step 2, Pod Task Submission: Based on the parsed task information, a corresponding Pod is created in Kubernetes, the Pod is configured using the Job field, and a performance analysis node is specified; Step 3, PodController Startup: The Controller listens to the status of the Pod in Kubernetes and waits for the Pod to enter the Running state; Step 4, Data Monitoring and Collection: After the Pod runs, the Controller periodically sends requests to Prometheus to obtain monitoring data; Step 5, Monitoring Data Steady-State Judgment: When the rate of change of the average data of the Pod performance is lower than the threshold for several consecutive reads, and the steady-state condition is met, the performance data is considered reliable; Step 6, Pod Stopping and Resource Release: The Pod is stopped from running, and the resources it occupies are released; Step 7, Utilization Information Recording: The collected utilization information is written into the Pod's annotations; Step 8, Result Feedback: The analysis results are returned to the user through the interface and stored in the database for subsequent queries.

Images