A heterogeneous GPU resource management and scheduling method

Abstract

This invention proposes a heterogeneous GPU resource management and scheduling method, relating to the field of GPU resource allocation technology. It addresses heterogeneous device management and unified abstract modeling, registering GPU resources of different architectures to a container orchestration platform and constructing a unified abstraction layer to shield underlying hardware differences. The method optimizes GPU cluster management based on multi-dimensional real-time monitoring and intelligent scheduling strategies, collecting GPU performance metrics, dynamically prioritizing tasks, and constructing task performance profiles. It makes scheduling decisions through multi-strategy collaboration to allocate optimal GPU resources to tasks. Fine-grained resource allocation is performed, partitioning GPUs spatially or temporally and dynamically adjusting resource allocation based on load status. A unified runtime library enables cross-architecture task API conversion and collects task execution data to optimize the scheduling model. Finally, it performs automatic GPU fault detection, isolation, and task migration, providing unified monitoring and alerting.

Description

Technical Field

[0001] This invention belongs to the field of GPU resource allocation technology, specifically relating to a method for managing and scheduling heterogeneous GPU resources. Background Technology

[0002] With the rapid development of artificial intelligence computing, modern data centers commonly deploy NVIDIA GPUs alongside domestically produced AI chips such as Ascend and Cambricon, forming heterogeneous computing environments. This heterogeneous architecture presents significant technical challenges. Different vendors employ CUDA and their own computing architectures, resulting in incompatible programming models and software stacks. This leads to difficulties in resource isolation, low scheduling efficiency, and consequently, severely underutilized resources, exponentially increased development and maintenance complexity, and difficulty in achieving load balancing and automatically increasing or decreasing GPU resources allocated to tasks based on real-time load conditions.

[0003] Existing technologies have significant limitations. First, while Kubernetes-based multi-device plugin solutions can manage various GPUs through independent plugins, resource pools are severely fragmented, the scheduler lacks a unified view, and it cannot intelligently match the optimal chip based on task characteristics. Furthermore, the maintenance cost of multiple plugins is high. Second, while vGPU and MxGPU-based virtualization resource pool solutions can achieve fine-grained allocation, they are limited to homogeneous environments, have poor cross-architecture compatibility, suffer significant performance loss at the virtualization layer, and cannot fully utilize the unique computing capabilities of domestically produced chips. In addition, traditional solutions such as Slurm are geared towards HPC scenarios and lack sufficient support for AI workloads; the management tools provided by each chip create scheduling silos, hindering collaboration.

[0004] In summary, there is an urgent need for a unified resource abstraction and intelligent scheduling mechanism across architectures to solve core problems such as hybrid pooling of NVIDIA and domestic chips, intelligent task-device matching, fine-grained resource isolation, and high development complexity. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a heterogeneous GPU resource management and scheduling method, comprising the following steps:

[0006] S1. Adopt the daemon set mode of the container orchestration platform to deploy heterogeneous device plugins on each node, build a device adaptation layer, abstract physical GPU resources into logical resource objects, and deploy a topology discovery service to build a GPU communication topology graph.

[0007] S2. Deploy a monitoring system to collect GPU computing power utilization, memory usage and power consumption indicators, develop a priority controller to calculate the dynamic priority of tasks, and build a machine learning performance analysis model to classify tasks and predict execution time.

[0008] S3. Develop a topology scoring plugin to calculate node scores, start an interference detector to monitor memory bandwidth usage to identify task co-location interference, implement a dynamic priority preemption mechanism, and build an intelligent matching engine to comprehensively score the matching degree of computing resources.

[0009] S4. Configure the GPU space partitioning granularity through device plug-in parameters, enable time-slice round-robin scheduling for lightweight tasks, and develop an automatic scaler controller to trigger vertical or horizontal scaling based on GPU utilization thresholds.

[0010] S5. A unified runtime dynamic link library is pre-installed in the user container to intercept native API calls and convert them into target hardware interfaces, collect task-level performance indicators, and drive incremental training of the prediction model and adaptive optimization of scheduling strategy weights.

[0011] S6. Configure multi-layer fault detection rules to trigger the isolation of abnormal devices and automatic task migration, listen for node status change events to trigger incremental updates of the topology graph, and deploy a unified monitoring panel to achieve visualization.

[0012] Optionally, in step S1, the daemon set mode of the container orchestration platform is adopted, and device plugins are deployed on each node. Each device plugin registers with the container runtime through a local socket and supports the allocation of remote call interfaces with container pre-startup.

[0013] A device adaptation layer is built as an extension component of the scheduler. The adapter design pattern is adopted to define a unified abstract interface specification that includes computing interface, memory interface and communication interface. The adapter dynamic library is injected through a preloading mechanism to intercept native API calls and convert them into standard interface calls.

[0014] Create custom resource definitions in the container orchestration platform application programming interface server to abstract physical GPU resources into logical resource objects; deploy an independent topology discovery service to collect bus interconnect topology, high-speed chip interconnect topology and server physical location information, convert the collected data into graph data structure and store it in a distributed graph database, and add topology affinity tags to nodes.

[0015] Optionally, in step S2, an open-source monitoring system stack is deployed to collect GPU computing power utilization, video memory usage, power consumption, temperature and ECC error rate through the performance indicator interface exposed by the device plug-in, and store them in a time-series database.

[0016] Develop a priority controller to implement service quality mapping rules, define three service quality levels for training tasks, inference services, and debugging jobs, and implement a priority inheritance mechanism;

[0017] A classification and labeling algorithm based on gradient boosting decision tree is constructed. The average computing power requirement, peak memory and communication frequency characteristics are input, and three types of labels are output: computing type, memory type and communication type. A performance prediction model based on long short-term memory network is constructed to predict the task execution time.

[0018] Optionally, in step S3, the topology scoring plugin identifies distributed training tasks by analyzing task metadata, scores candidate nodes, and prioritizes GPUs within the same non-uniform memory access architecture node or within the same rack; the interference detector monitors GPU memory bandwidth usage through a packet filter program, and determines interference when task bandwidth usage exceeds 80% and there are latency-sensitive tasks on the same device.

[0019] Optionally, in step S4, the granularity of the partitioning is configured through the parameters of the device plugin to support multi-instance GPU partitioning; an automatic scaler controller is developed to trigger scaling down when the graphics processor utilization is below a threshold for a continuous period of time, and to trigger scaling up when the task queuing time exceeds the service level target threshold.

[0020] Vertical scaling prioritizes adjusting the instance size on the current node of the task and uses hot migration technology to upgrade from a small size to a large size; horizontal scaling prioritizes scheduling new replicas to the node with the highest idle rate and adding compute nodes as needed through the cluster autoscaler.

[0021] Optionally, in step S5, multi-layer detection rules are configured: GPUXid errors trigger immediate isolation; if the error detection and correction error rate is greater than the detection threshold, it is marked as a predictive fault; if the temperature exceeds the threshold for a set period of time, frequency reduction protection is triggered.

[0022] Optionally, anomaly detection is performed by a node problem detector. After a fault is detected, a taint is automatically added to the node. The eviction manager evicts the container group on the faulty node based on the taint, and the scheduler reschedules the task to a healthy node. Node resource change events are monitored. When a node's status becomes unready for more than a set time, or when a new GPU node is added and its resources are reported, a topology refactoring is triggered.

[0023] This invention also proposes a heterogeneous GPU management and scheduling system, comprising:

[0024] The heterogeneous device management module is used to achieve pooled management of heterogeneous AI chips through a multi-device plug-in subsystem, a unified abstract adapter, and a topology discovery service.

[0025] The full-domain monitoring and profiling module is used to achieve fine-grained perception and intelligent modeling of resource status through the indicator collection engine, priority controller and profiling training service;

[0026] The intelligent scheduling decision module is used to achieve multi-strategy collaborative scheduling with topology awareness, interference awareness, dynamic priority preemption, and intelligent matching by extending the scheduler;

[0027] The fine-grained resource allocation module is used to dynamically allocate resources and automatically increase or decrease the GPU resources allocated to tasks based on real-time load conditions through the partition manager and automatic scaling controller.

[0028] The unified runtime support module is used to enable cross-architecture task execution and feedback through a unified runtime library and feedback agent;

[0029] The fault self-healing and observability module is used through a fault detection engine, topology reconstruction service, and unified monitoring dashboard.

[0030] Optionally, in the heterogeneous device management module, the unified abstract adapter is deployed in the initialization container of the user container group. By setting environment variables, it intercepts native API calls and maps the interfaces to standardized abstract interfaces.

[0031] Optionally, in the intelligent scheduling decision module, the extended scheduler registers six scoring plugins: topology-aware scoring plugin, interference scoring plugin, priority scoring plugin, matching degree scoring plugin, balance degree scoring plugin, and utilization rate scoring plugin. A weighted score model is adopted, and the total score is the weighted sum of the scores of each plugin, with the weights dynamically configured.

[0032] Compared with the prior art, the present invention has the following beneficial technical effects:

[0033] This invention achieves unified pooling and intelligent scheduling of heterogeneous GPU resources, significantly improving the performance of AI computing infrastructure. By shielding the differences between NVIDIA and domestic chip architectures through a unified abstraction layer, application code can run across platforms with zero modifications, improving development efficiency by over 40%. Multi-strategy intelligent scheduling increases cluster resource utilization by 30%-50% and reduces average task latency by 60%. Topology awareness and interference awareness mechanisms optimize distributed training communication efficiency by 25%, ensuring a 99.9% SLO compliance rate for latency-sensitive services. Fine-grained resource allocation and second-level automatic increase or decrease of GPU resources allocated to tasks based on real-time load conditions support dynamic resource sharing, reducing resource fragmentation to below 5%. The fault self-healing system achieves minute-level GPU fault detection and automatic task migration, with an MTTR of less than 5 minutes and cluster availability of 99.95%. A unified monitoring dashboard provides end-to-end observability, improving operational efficiency by 50%. The system has been validated in kilocalorie-level heterogeneous clusters and is suitable for scenarios such as intelligent computing centers and cloud-native AI platforms, effectively reducing the complexity and operating costs of heterogeneous environments. Attached Figure Description

[0034] Figure 1This is a graph showing GPU temperature monitoring data from the present invention.

[0035] Figure 2 This is a graph showing GPU power consumption monitoring data from the present invention.

[0036] Figure 3 This is an architecture diagram of the multi-network adaptation module of the present invention;

[0037] Figure 4 This is a diagram illustrating the working architecture of the entry controller of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0039] In the accompanying drawings of specific embodiments of the present invention, in order to better and more clearly describe the working principle of each component in the system and show the connection relationship of each part in the device, only the relative positional relationship between each component is clearly distinguished. It does not constitute a limitation on the signal transmission direction, connection sequence, or size and shape of each part within the component or structure.

[0040] Example 1

[0041] This embodiment provides a heterogeneous GPU resource management and scheduling method, including the following steps:

[0042] Step 1: Integrating heterogeneous devices into a unified abstract model.

[0043] Step 1.1: Implement multi-plugin collaborative deployment.

[0044] A daemon set pattern is adopted from the container orchestration platform, deploying heterogeneous device plugins on each node to ensure automatic plugin instance scheduling when a new node is added. The specific deployment list includes:

[0045] NVIDIA Device Plugin: Deployed via NVIDIA device plugin configuration files, environment variables are set to achieve fault isolation, and device list policies are configured to support the exposure of multiple instance graphics processor devices.

[0046] Huawei Ascend Device Plug-in: Deploy the Ascend Device Plug-in Daemon set configuration file, and support virtual device splitting through the virtual device quantity parameter. A single card can be virtualized into a maximum of eight virtual neural network processor instances.

[0047] Cambricon Device Plug-in: Deploy the Cambricon device plug-in configuration file, configure the watchdog to monitor the health status of the machine learning unit chip, and mark abnormal devices by unhealthy devices.

[0048] Plugins are categorized and managed through node tags. Each plugin registers with the container runtime through a local socket, supporting remote call interfaces such as allocation and container pre-start, enabling device allocation and environment initialization before container startup.

[0049] Step 1.2: Unified resource abstraction.

[0050] A device adaptation layer is built as an extension component of the container orchestration platform scheduler. The adapter design pattern is used to realize interface conversion, interface standardization, and define a unified abstract interface specification, including computing interface, memory interface and communication interface. All interfaces adopt serialization protocol.

[0051] Adapter component development involves developing independent adapter dynamic libraries for each chip, injecting them through a preloading mechanism when the container starts, intercepting native application programming interface calls, and converting them into standard interface calls.

[0052] The resource registration mechanism creates custom resource definitions in the application programming interface server of the container orchestration platform. Fields include architecture, computing unit, memory size, topology, etc., which abstract physical graphics processor resources into logical resource objects that can be uniformly scheduled, thus shielding the differences in underlying drivers.

[0053] Step 1.3, Topology-aware modeling.

[0054] Deploy a standalone topology discovery service, which runs as a set of daemons.

[0055] Data acquisition frequency: every 30 seconds, the bus interconnect topology is obtained via command; the high-speed chip interconnect topology is obtained via command; and the physical location information of the server is collected via the management interface.

[0056] Topology graph construction transforms the collected data into a graph data structure, where nodes represent graphics processing units (GPUs), edge weights represent communication latency, and the data is stored in a distributed graph database.

[0057] Automatic labeling adds topology affinity tags to nodes, supporting the scheduler in scoring topology affinity.

[0058] Step 2: Optimize and manage GPU clusters based on multi-dimensional real-time monitoring and intelligent scheduling strategies.

[0059] Step 2.1, Real-time monitoring system.

[0060] Deploy an open-source monitoring system stack, with a data collection frequency set to every five seconds.

[0061] The metrics acquisition path involves collecting data such as GPU computing power utilization, video memory usage, and power consumption through the performance metrics interface exposed by the device plug-in; and collecting data such as streaming multiprocessor array activity and error detection and correction rate through the node performance analyzer.

[0062] Time-series storage uses a time-series database, sets a 15-day local data retention period, and pushes data to long-term storage via a remote write interface, supporting historical data analysis.

[0063] Anomaly detection is preset, and monitoring rules are configured to calculate the average value of the sliding window in real time. When the temperature exceeds 85 degrees Celsius or the power consumption exceeds 95% of the thermal design power, an alarm is triggered, and the device is automatically marked as unhealthy and isolated.

[0064] Step 2.2: Develop a priority controller within the container orchestration platform scheduling framework.

[0065] The service quality mapping rules define three service quality levels: a cardinality of 1,000 for training tasks, 500 for inference tasks, and 100 for debugging tasks. Urgent tasks are multiplied by a coefficient of 1.5, and ordinary tasks are multiplied by 1.0.

[0066] The priority calculation formula is as follows: the final priority equals the basic service quality multiplied by the urgency coefficient plus the queue waiting time multiplied by the time weight, where the time weight is set to 0.1 per minute to prevent task starvation.

[0067] Priority inversion is avoided by implementing a priority inheritance mechanism. When a low-priority task holds resources required by a high-priority task, the priority of the low-priority task is temporarily increased to the same level to avoid deadlock.

[0068] Step 2.3: Construct a performance analysis system based on machine learning.

[0069] Data acquisition sources include historical task resource time series obtained from the monitoring system, task metadata obtained from the container platform, and computation patterns extracted through log analysis.

[0070] The classification and labeling algorithm uses a gradient boosting decision tree model to classify tasks. Input features include average computing power requirement, peak memory usage, and communication frequency. The output consists of three categories: computational, memory-intensive, and communication-intensive. It is recommended that the training dataset contain at least 10,000 historical task records.

[0071] The performance prediction model is a regression model based on a long short-term memory network. It predicts the execution time of a task on different chips. The input is a task profile vector plus a chip specification vector. The output is the predicted execution time and resource utilization. The model is incrementally trained every 24 hours. The mean absolute error of the model evaluation metric must be less than 15%.

[0072] Step 3: Intelligent scheduling decision-making based on multi-strategy collaboration.

[0073] Step 3.1: Extend the default scheduler of the container orchestration platform and develop a topology scoring plugin.

[0074] Communication pattern recognition identifies the task type by analyzing metadata in the form of key-value pairs. If the annotation indicates that it is a distributed training task, the topology-aware scoring mechanism is activated.

[0075] The score for candidate nodes is calculated using the following formula:

[0076] ;

[0077] ;

[0078] ;

[0079] Where TotalHop is the total number of hops, which is the sum of the topological distance hops between all task instances; MaxAllowedAvgHop is the maximum allowed average number of hops, which is the upper limit of the average number of hops between GPU pairs allowed by the system; W hop _tw_penalty_weight_represents the penalty score per unit of hops;_CrossNodeCount_count_represents the number of physical nodes the task instance crosses;_W node This is the cross-node penalty weight, which is the penalty score for crossing each node.

[0080] Prioritize GPUs within the same non-uniform memory access architecture node (0 hops) or within the same rack (≤2 hops). Distributed training tasks require all workload instances to be scheduled to GPUs with a topology distance of less than 3 hops.

[0081] Step 3.2: Interference sensing and scheduling, and development of interference detector components.

[0082] The interference detection mechanism monitors GPU memory bandwidth usage through a packet filter program. When it detects that a task's bandwidth usage exceeds 80% and there are latency-sensitive tasks on the same device, it is determined to be interference.

[0083] Resource isolation strategy: Automatically allocate exclusive attributes to latency-sensitive tasks, ensuring that they exclusively occupy the GPU's SM partition or are allocated to a separate physical card during scheduling; allocate shared attributes to bandwidth-intensive tasks (such as large model training), allowing multiple tasks to co-locate but enabling QoS rate limiting, limiting their memory bandwidth to no more than 60%.

[0084] Step 3.3: Dynamic priority scheduling to achieve controller preemption.

[0085] The preemption trigger condition is that when a high-priority task (FinalPriority>800) is in a pending state for more than 5 minutes due to insufficient resources, and a low-priority task (FinalPriority<300) in the cluster occupies the target resource, preemption is triggered.

[0086] The resource release process sends a SIGTERM signal to low-priority tasks, providing a 30-second graceful exit period. If this timeout occurs, the container group is forcibly deleted. The state of preempted tasks is automatically saved to checkpoint storage (such as PV volumes) and restored from the breakpoint upon restart.

[0087] Step 3.4, Intelligent Matching Engine.

[0088] MatchScore:

[0089] ;

[0090] Where w1, w2, w3 are weighting coefficients, preferably 0.3, 0.3, 0.4; S compute ,S memory ,S perf It consists of three sub-ratings.

[0091] The scheduler prioritizes the node with the highest MatchScore. If the score difference between the top 3 nodes is less than 5%, the node with the lowest current load is selected to achieve load balancing.

[0092] Step 4: Fine-grained resource allocation, and automatically increase or decrease the GPU resources allocated to tasks based on real-time load conditions.

[0093] Step 4.1: Select resource partitioning mode, configure space partitioning, and configure the granularity of partitioning through device plugin parameters.

[0094] The NVIDIA A100 supports splitting a multi-instance GPU into up to 7 independent instances, each of which can be configured to a specified specification; the Huawei Ascend 910 supports splitting into 8 virtual neural network processors, each of which can be configured to a specified virtual specification.

[0095] The partitioning strategy is defined through custom resource definitions, allowing users to specify exclusive or shared modes.

[0096] Round-robin scheduling enables time-slice scheduling for lightweight tasks. It sets the time slice to 10 milliseconds using the GPU driver's time-slice scheduler and polls multiple container tasks. Configure the appropriate annotation to enable this mode.

[0097] Step 4.2: Dynamic resource adjustment and development of an automatic expander controller.

[0098] Scaling / scaling trigger conditions:

[0099] Reduced capacity: When the graphics processor utilization rate is below 20% for 5 consecutive minutes, it is considered as idle resources.

[0100] Scaling up: When the queuing time of a task exceeds the service level target threshold, for example, when the 99th percentile latency of an inference task is greater than 100 milliseconds, it is considered a resource hotspot.

[0101] Vertical scaling: For tasks that have been allocated multiple GPU instances, dynamically adjust the instance size, for example, upgrading from a small size to a large size. The adjustment process uses hot migration technology, and the task pause time is less than 1 second.

[0102] Horizontal scaling: For stateless inference services, a horizontal container group auto-scaler automatically increases or decreases the number of container group replicas based on a custom GPU utilization metric. During scaling, new replicas are prioritized for scheduling to the nodes with the highest idle rates.

[0103] Step 5: Unify runtime support and status feedback.

[0104] Step 5.1: Build a unified runtime library.

[0105] The Application Programming Interface (API) translation layer pre-loads a unified runtime dynamic link library within the user container and intercepts CUDA calls through a pre-loading mechanism. For example, after intercepting memory allocation calls, the adapter layer determines the target architecture based on environment variables and translates them into the corresponding hardware vendor's proprietary memory allocation interface.

[0106] This tool performs computational kernel transformation, developing an automated model transformation tool for mainstream deep learning frameworks. It maps native graphics processor operators to dedicated graph executors or operator libraries for the target hardware. Transformation rules are stored in a configuration file, and custom operators can be registered.

[0107] For performance optimization, performance monitoring probes are inserted into the adaptation layer to record the time taken for API calls. If the latency of a certain conversion path exceeds a preset threshold, the system will automatically switch to the pre-compiled optimized execution path.

[0108] Step 5.2: Deploy the feedback agent as a sidecar container.

[0109] Performance metrics are collected by using the hardware vendor's command-line tools to collect task-level GPU utilization and by using the container monitoring tool interface to collect container-level memory and CPU usage, with a sampling frequency of once every 10 seconds.

[0110] Incremental training of the model involves comparing the collected actual performance data with the previous predictions after completing a certain number of tasks (e.g., 100) and calculating the error. If the mean absolute error exceeds a set threshold (e.g., 15%), the incremental training process of the prediction model is triggered to update the model parameters.

[0111] The scheduling strategy is optimized by dynamically adjusting the weights of scheduling plugins based on feedback data. For example, if a task using a topology-aware scheduling strategy fails to achieve the expected improvement in communication efficiency within a week, the weight of that scheduling plugin is reduced accordingly, while the weights of other components such as the intelligent matching engine are increased to achieve adaptive optimization of the scheduling strategy.

[0112] Step 6: Fault self-healing and operation and maintenance observability assurance.

[0113] Step 6.1, Anomaly Detection and Migration.

[0114] Fault detection thresholds are configured with multi-layered detection rules. GPUXid errors trigger immediate isolation; ECC error rates > 1 per hour are marked as predictive faults; and temperatures > 90℃ for 30 seconds trigger frequency reduction protection.

[0115] The automated migration process involves anomaly detection handled by a node issue detector, which automatically adds a "gpu-unhealthy=true" taint to the node upon detecting a fault. The eviction manager removes the container group from the faulty node based on the taint, and the scheduler reschedules the task to a healthy node. The migration process persists checkpoint data via PVCs to ensure that training tasks can be recovered from the most recent cycle.

[0116] Step 6.2, Dynamic topology reconstruction.

[0117] The refactoring trigger condition is to listen for node resource change events. When a node's Ready state changes to Not Ready for more than 3 minutes, or when a new GPU node is added and its resources are reported, a topology refactoring is triggered.

[0118] When the graph database is updated, the topology collector re-executes the discovery process, updates the node and edge relationships in Neo4j, and sets the active=false flag for faulty nodes. The scheduler listens for graph database change events, refreshes the local topology cache (TTL=60 seconds), and reschedules affected tasks within 1 minute.

[0119] Step 6.3: Deploy a unified monitoring panel.

[0120] Monitoring metrics are collected through system monitoring and alarm toolkits. Table 1 is a detailed explanation of the scheduler monitoring metrics, Table 2 is a detailed analysis of the metrics, and Table 3 is an operation and maintenance monitoring configuration table.

[0121] Table 1. Detailed Explanation of Scheduler Monitoring Indicators

[0122]

[0123] Table 2 Detailed Analysis of Indicators

[0124]

[0125] Table 3 Operation and Maintenance Monitoring Configuration Table

[0126]

[0127] The Grafana Dashboard is configured with six visualizations to comprehensively display the system status.

[0128] The first is the resource overview view, which uses a GPU utilization heatmap to visually present the resource load of the entire cluster and each node.

[0129] The second is the task scheduling view, which displays the changes in task queuing time in the form of a trend chart to help analyze scheduling delays and bottlenecks.

[0130] The third is the topology status view, which graphically displays the physical interconnection between GPUs and clearly presents the hardware connection architecture of the cluster.

[0131] Fourth is the performance analysis view, which uses charts such as execution time distribution graphs to show the actual running efficiency of different types of tasks.

[0132] The fifth is the fault alarm view, which records and displays various abnormal events that occur in the system in a timeline format, making it easy to track and trace back.

[0133] The sixth is the cost analysis view, which provides data insights for resource optimization and cost control by comparing the power consumption and computing power output ratio of various types of chips.

[0134] Alarm policies allow for multi-level alarm configuration. A P2 alarm is triggered when the scheduling failure rate exceeds 5%, and a P1 alarm is triggered when the GPU failure rate exceeds 3 GPUs per hour. The system is integrated into WeChat Work or Slack via Webhook and supports automatic creation of maintenance work orders.

[0135] Example 2

[0136] This embodiment proposes a heterogeneous GPU management and scheduling system, which is a unified management and intelligent scheduling platform for heterogeneous GPUs built on an extension mechanism. Through a layered and decoupled architecture design, it achieves pooled management, intelligent scheduling, and operation and maintenance support for heterogeneous AI chips such as NVIDIA GPUs, Huawei Ascend, and Cambricon MLUs. The system adopts an architecture that separates the control plane and the data plane, and consists of six core functional modules: device management plane, global monitoring plane, intelligent scheduling plane, resource allocation plane, runtime support plane, and operation and maintenance support plane. Each plane is deeply integrated through Kubernetes native extension mechanisms (CRD, DevicePlugin, SchedulerExtender) to form a closed-loop resource management ecosystem.

[0137] Core functional modules:

[0138] (1) Heterogeneous device management module

[0139] This module serves as the system's resource management layer, responsible for the unified management and abstraction of heterogeneous computing devices. It consists of three parts: a multi-device plug-in subsystem, a unified abstract adapter, and a topology discovery service.

[0140] The multi-device plugin subsystem adopts the DaemonSet pattern, deploying vendor-native device plugins on each compute node in the cluster. Examples include NVIDIA Device Plugin v0.14.0 and above, Huawei CANN Device Plugin v5.0 and above, and Cambricon CNML Device Plugin v3.0 and above. These plugins use a standard plugin manager interface to register devices, perform health status checks, and report resource information. Each plugin independently manages its corresponding chip device, supporting device-level fault isolation and status maintenance.

[0141] The unified abstract adapter constructs a middleware abstraction layer, deployed within user container groups as an initialization container. It comprises three core dynamic libraries: a compute adapter library, a memory adapter library, and a communication adapter library. Through the LD_PRELOAD mechanism, this adapter layer intercepts API calls from applications within the container to the underlying chip's native runtime, such as CUDARuntime, CANN ACL, or CNRT interfaces, and maps them uniformly to standardized abstract interfaces, thereby shielding the differences in instruction sets and memory models between different chips.

[0142] The topology discovery service is deployed as a standalone deployment. It periodically collects physical interconnect topology information between GPUs or NPUs using vendor-specific tools, such as nvidia-smi topo-m or Huawei npu-smi info-t, including connection methods such as NVLink, PCIe, HCCS, and RoCEv2. Based on the collected data, the service constructs a weighted directed graph model, where weights are set according to information such as communication latency or bandwidth. This topology graph is stored in a designated key space in etcd and provides a topology query interface to components such as the scheduler through a dedicated API.

[0143] (2) Full-domain monitoring and profiling module

[0144] This module constitutes the system's monitoring plane, enabling fine-grained perception of resource status and intelligent behavior modeling. It consists of three parts: an indicator acquisition engine, a dynamic priority controller, and a profile training service.

[0145] The metrics collection engine is deployed based on Prometheus. It automatically discovers the monitoring metric endpoints exposed by various device plugins through service monitoring functions, and collects more than twenty GPU-related metrics every 5 seconds, including compute utilization, memory usage, power consumption, temperature, ECC error rate, etc. The collected metric data is stored in a long-term storage system, supporting historical data backtracking and analysis for up to 90 days.

[0146] The dynamic priority controller is deployed using an admission control callback mechanism. It intercepts container group creation requests and dynamically calculates a priority weight between 1 and 1000 based on the service quality level, service tier protocol level, and current task queuing time specified in the request annotations. The calculated weight value is then injected into a specific field of the container group. Simultaneously, the controller maintains a global priority queue for scheduling decisions.

[0147] The profiling training service is deployed on dedicated GPU training nodes. It pulls historical task performance data from the object storage system every morning at midnight and utilizes the Kubeflow Pipeline component to build an automated model training pipeline. The service employs a hybrid XGBoost and LSTM model to generate models for predicting task resource consumption and profiling chip performance characteristics. The generated models are version-managed using native Kubernetes resource objects and support A / B testing and canary release processes, ensuring the security and controllability of model updates.

[0148] (3) Intelligent scheduling decision module

[0149] This module is the core decision engine of the system. It realizes multi-strategy collaborative scheduling through the extended scheduler, and develops a second-generation scheduler heterogeneous-gpu-scheduler. It registers six scoring plugins: topology-aware scoring plugin, interference scoring plugin, priority scoring plugin, matching degree scoring plugin, balance degree scoring plugin, and utilization rate scoring plugin. It adopts a weighted score model, and the total score is the weighted sum of the scores of each plugin. The weights can be dynamically configured.

[0150] The topology-aware scheduling plugin implements filtering and scoring of expansion points. In the filtering expansion point stage, node combinations that cross NUMA domains are eliminated. In the scoring expansion point stage, affinity scores are calculated for candidate nodes based on the number of hops between GPUs. Fully reduced tasks are prioritized for scheduling to fully connected subgraphs.

[0151] An interference-aware scheduling plugin maintains an interference matrix and records historical performance degradation data for co-located tasks. During scheduling, the matrix is ​​queried. If it is detected that co-location of memory bandwidth-sensitive tasks and compute-intensive tasks may cause performance interference >10%, the node is directly rejected in the Filter phase, or a -50-point penalty is applied in the scoring extension phase.

[0152] The dynamic priority preemption plugin implements preemption expansion points, listens for high-priority tasks (>800 points) in the waiting queue, and when the waiting time exceeds 5 minutes, scans for low-priority tasks (<300 points) occupying resources, sends a preemption signal and gracefully terminates the low-priority container group. The status of the preempted task is automatically checked to distributed storage.

[0153] (4) Fine-grained resource allocation module

[0154] This module is responsible for dynamically allocating and adjusting GPU resources, automatically increasing or decreasing the amount of resources allocated to tasks based on real-time load conditions. The module mainly consists of a partition manager and an automatic scaling controller.

[0155] The partition manager is deployed as a device plug-in sidecar container. It performs physical GPU partitioning operations based on a predefined partitioning strategy. Specifically, different virtualization technologies are used for GPUs from different manufacturers: NVIDIA GPUs are partitioned into compute instance units using MIG technology, Huawei Ascend GPUs are partitioned into virtual NPUs using SOCA virtualization technology, and Cambricon GPUs are partitioned into virtual MLUs using MLU-VM technology. Supported partitioning granularities include full card, half card, and quarter card, and also support partitioning according to custom ratios, such as allocating 30% of the computing power and 6GB of video memory.

[0156] The auto-scaling controller is a custom HPA controller implementation. It continuously monitors GPU utilization metrics in the system. When a task's GPU utilization falls below 20% for five consecutive minutes, the controller triggers scaling down, reducing its GPU allocation by modifying the resource limits of the corresponding container group. When it detects that a task's queuing latency exceeds a set service level target threshold—for example, an inference task latency exceeding 50 milliseconds or a training task queue exceeding 10 minutes—the controller triggers scaling up. During scaling up, it first attempts to vertically scale up the node currently hosting the task. If that node lacks sufficient resources, it adds new compute nodes through the cluster auto-scaler.

[0157] (5) Unified runtime support module

[0158] This module constitutes the runtime plane, designed to achieve a closed loop of task execution and performance feedback across architectures. It consists of two parts: a unified runtime library and a feedback agent.

[0159] The unified runtime library is mounted to the user's container group as a container storage interface. It includes a runtime transformation engine and a performance probe. The transformation engine dynamically loads the corresponding chip adapter based on the environment variable TARGET_GPU_ARCH, such as adapters from NVIDIA, Ascend, or Cambricon, thereby translating standard CUDA API calls into the native instructions of the underlying chip. The performance probe is inserted in a pre-loaded manner, responsible for collecting latency and throughput data for each API call and generating trace data to push to the Jaeger system.

[0160] The feedback broker is deployed as a sidecar container. It collects task-level performance metrics in real time, querying data once per second using tools such as nvidia-smi or npu-smi. The broker aggregates the collected data every 30 seconds and pushes the aggregation results to a Kafka topic named gpu-metrics-feedback. The backend profiling training service continuously consumes data from this topic, enabling online incremental learning of the model. Every 500 performance feedback data points processed, the service triggers a model fine-tuning to continuously optimize task scheduling and resource allocation decisions.

[0161] (6) Fault self-healing and observability module

[0162] This module aims to ensure the high availability and operational efficiency of the system, and mainly consists of three parts: a fault detection engine, a topology reconstruction service, and a unified monitoring dashboard.

[0163] The fault detection engine extends the node problem detector and defines custom fault detection rules for GPUs, such as detecting Xid errors, ECC threshold exceeding limits, and temperature alarms. When the engine detects a GPU fault, it adds a GPU hardware fault taint to the corresponding node and removes all GPU tasks on that node. Simultaneously, the engine calls the cloud vendor's API, such as Alibaba Cloud's ECS API, to perform a power-down isolation operation on the faulty GPU and automatically creates a repair work order.

[0164] The topology refactoring service monitors lifecycle events for nodes and container groups. When a node's state becomes NotReady or a new GPU node is added to the cluster, the service triggers an incremental update of the topology graph. It notifies the scheduler to refresh its topology cache through a key-value store mechanism and ensures that the rescheduling of affected tasks is completed within ten seconds.

[0165] The unified monitoring dashboard is built on a Grafana, Loki, and Tempo technology stack, forming a complete observability platform. This platform provides six core monitoring views: a resource overview view, displaying the cluster's GPU pooling and fragmentation rates; a scheduling analysis view, displaying scheduling latency distribution and preemption event statistics; a performance insight view, displaying task-level GPU and communication efficiency; a fault management view, displaying average fault recovery time and fault topology impact; a cost optimization view, displaying power consumption per TFLOPS and chip cost-effectiveness; and an SLO compliance view, displaying the P99 percentile of task queuing time and resource request fulfillment rate. Furthermore, the platform supports role-based multi-tenant access control, ensuring that each team can only view resource information within its own namespace.

[0166] During the resource registration phase, after the device plugin is started, it registers the detected GPU resources to the core agent through the remote procedure call framework. The core agent then reports to the core management endpoint and gateway to create heterogeneous GPU custom resource instances.

[0167] During the task submission phase, the user submits a container group and specifies the amount of GPU resources requested. The priority controller calculates the priority and injects annotations, and the scheduler enters the decision-making process.

[0168] During the scheduling decision-making phase, the scheduler sequentially calls the topology-aware scoring plugin, the interference scoring plugin, and other plugins to comprehensively calculate the node scores, select the optimal node, and bind it.

[0169] During the resource allocation phase, the partition manager performs GPU partitioning based on the scheduling results, and the device plugin performs allocation operations, returning the device path and driver information.

[0170] During the task execution phase, the unified runtime library is loaded when the user container starts, API calls are dynamically transformed, and the feedback agent continuously collects performance data.

[0171] During the feedback and optimization phase, the profiling training service periodically consumes performance data, updates the prediction model, and dynamically adjusts the scheduling strategy parameters.

[0172] Example 3

[0173] like Figure 3 As shown, the overall architecture of the multi-network adaptation module is illustrated. Its core objective is to enable container groups within virtual machines to access both the primary and secondary networks simultaneously, thereby meeting the network requirements of different business scenarios.

[0174] The meanings of architectural layers and components are as follows:

[0175] Within the virtual machine's user space, multiple container groups run, each with two network interfaces mounted. The primary network interface is used for routine communication, such as scheduling instructions and log transmission; the secondary network interface is dedicated to high-performance scenarios, such as high-throughput data transmission required for GPU tasks.

[0176] At the virtual machine kernel level, the virtual network interface card (NIC) corresponding to the main network is managed through a network plugin, while an independent virtual NIC is allocated to the auxiliary network.

[0177] At the virtualization management layer, the underlying layer supports the main network through virtual distributed switches and the auxiliary network through virtual distributed switches or single-root IO virtualization technology, thereby achieving hardware-level network performance acceleration.

[0178] The functions of the multi-network adaptation module specifically include:

[0179] First, configure dual network interfaces for the container group. The primary network interface handles general traffic, while the secondary network interface is dedicated to carrying high-bandwidth GPU task data streams.

[0180] Secondly, the scheduling system can automatically match the appropriate network interface type to the GPU task based on the type of the task, thereby achieving coordinated scheduling of computing and network resources.

[0181] In addition, a full-stack telemetry acquisition module has been added to the architecture. This module collects data in real time, including the computing power utilization and memory usage of heterogeneous GPUs, as well as underlying network bandwidth, storage I / O performance, and CPU load. The collected data is transmitted to the scheduling decision module through a monitoring and alarm system or standardized interfaces such as Kafka, providing a basis for decision-making regarding the accurate allocation and load balancing of heterogeneous GPU resources.

[0182] This design references the heterogeneous computing power support logic of the training-puss fusion platform, aiming to comprehensively improve GPU resource utilization through software and hardware co-optimization.

[0183] The specific implementation steps are as follows:

[0184] Step one is resource initialization. Deploy the MultusCNI plugin and data acquisition device, configure hardware adaptation for the main and auxiliary network interfaces, such as completing SRIOV network card virtualization and DPDK driver installation, and register the network requirement characteristics of heterogeneous GPUs, such as specifying that training and inference tasks require more than 10Gbps of bandwidth.

[0185] Step two is data acquisition. The data acquisition device collects GPU computing power, network bandwidth, and storage I / O data in real time at a frequency of once per second, and uploads it to the monitoring and alarm database to form a global status view linking the GPU, network, and storage.

[0186] Step three is scheduling and execution. When an AI training or inference task is received, the scheduling system automatically matches the auxiliary network interface and allocates the corresponding heterogeneous GPU resources; when a lightweight inference task is received, the main network interface is used to improve resource reuse.

[0187] Step four involves dynamic adjustment. Based on the real-time data reported by the data acquisition unit, if insufficient network bandwidth is detected, the system can automatically switch the task to a better network interface or dynamically adjust the resource allocation ratio of the GPU task.

[0188] Example 4

[0189] Figure 4 The working architecture of the ingress controller is shown, the core of which is to enable unified external access and traffic routing for multiple services within the cluster.

[0190] The initiator of the access, including tenant administrators and cloud administrators, makes the request from the outside through the HTTPS access address of a specific service, such as https: / / www.example.com / svc1.

[0191] The unified ingress controller acts as the cluster's traffic gateway, receiving external HTTPS requests and routing them according to the path in the request. For example, a request with the path / svc1 will be forwarded to service 1, while a request with the path / svc2 will be forwarded to service 2.

[0192] The backend services and container groups are responsible for handling the forwarded requests. Requests for / svc1 will be handled by the service 1 container group deployed on different nodes; requests for / svc2 will be handled by the service 2 container group on the corresponding node.

[0193] The ingress controller acts as a unified reception desk for cluster services, routing external requests to the corresponding services and their container groups within the cluster according to predetermined rules. This mechanism not only simplifies access to multiple services but also ensures access security through the HTTPS protocol.

[0194] Specifically, an ingress controller is deployed in the Kubernetes cluster as the unified access point for heterogeneous GPU services. Corresponding routing rules are configured for different GPU-hosted services, such as Service1 for AI inference and Service2 for computing power interface services. Rules are typically based on a combination of paths and domain names, such as using paths / svc1 and / svc2 and the domain name www.example.com to distribute traffic. These routing rules support HTTPS, providing encrypted access to GPU services, and can interface with the cluster's certificate management component. When the scheduling system allocates GPU tasks to container groups on different nodes, the ingress controller can automatically associate the container group instances in the server's backend member table, thereby achieving dynamic load balancing of traffic. The specific steps are as follows:

[0195] Create servers for services hosted on the GPU and configure ingress controller rules to map external access addresses, such as https: / / www.example.com / svc1, to the corresponding servers, such as the AI ​​inference service.

[0196] External users or management systems initiate requests through the address of the ingress controller. The ingress controller forwards the requests to the GPU container group on the corresponding node for processing based on the request path. The entire process ensures access security through HTTPS.

[0197] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A heterogeneous GPU resource management and scheduling method, characterized in that, Includes the following steps: S1. Adopt the daemon set mode of the container orchestration platform to deploy heterogeneous device plugins on each node, build a device adaptation layer, abstract physical GPU resources into logical resource objects, and deploy a topology discovery service to build a GPU communication topology graph. S2. Deploy a monitoring system to collect GPU computing power utilization, video memory usage and power consumption indicators, develop a priority controller to dynamically prioritize computing tasks, and build a machine learning performance analysis model to classify tasks and output three types of labels: computing, memory and communication, as well as predict task execution time. S3. Develop a topology scoring plugin to calculate node scores, start an interference detector to monitor memory bandwidth usage to identify task co-location interference, implement a dynamic priority preemption mechanism, and build an intelligent matching engine to comprehensively score the matching degree of computing resources. S4. Configure the GPU space partitioning granularity through device plugin parameters. This includes configuring the partitioning granularity through resource partitioning mode selection, space partitioning configuration, and device plugin parameter configuration. Enable time-slice round-robin scheduling for lightweight tasks and develop an automatic scaler controller to trigger vertical or horizontal scaling based on GPU utilization thresholds. S5. A unified runtime dynamic link library is pre-installed in the user container to intercept native API calls and convert them into target hardware interfaces, collect task-level performance indicators, and drive incremental training of the prediction model and adaptive optimization of scheduling strategy weights. S6. Configure multi-layer fault detection rules to trigger the isolation of abnormal devices and automatic task migration, listen for node status change events to trigger incremental updates of the topology graph, and deploy a unified monitoring panel to achieve visualization. In step S1, the daemon set mode of the container orchestration platform is adopted to deploy device plugins on each node. Each device plugin registers with the container runtime through a local socket and supports the allocation of remote call interfaces with container pre-startup. A device adaptation layer is built as an extension component of the scheduler. The adapter design pattern is adopted to define a unified abstract interface specification that includes computing interface, memory interface and communication interface. The adapter dynamic library is injected through a preloading mechanism to intercept native API calls and convert them into standard interface calls. Create custom resource definitions in the container orchestration platform application programming interface server to abstract physical GPU resources into logical resource objects; deploy an independent topology discovery service to collect bus interconnect topology, high-speed chip interconnect topology and server physical location information, convert the collected data into graph data structure and store it in a distributed graph database, and add topology affinity tags to nodes.

2. The method according to claim 1, characterized in that, In step S2, an open-source monitoring system stack is deployed to collect GPU computing power utilization, video memory usage, power consumption, temperature and ECC error rate through the performance indicator interface exposed by the device plug-in, and store them in a time-series database. Develop a priority controller to implement service quality mapping rules, define three service quality levels for training tasks, inference services, and debugging jobs, and implement a priority inheritance mechanism; A classification and labeling algorithm based on gradient boosting decision tree is constructed. The average computing power requirement, peak memory and communication frequency characteristics are input, and a performance prediction model based on long short-term memory network is constructed to predict the task execution time.

3. The method according to claim 1, characterized in that, In step S3, the topology scoring plugin identifies distributed training tasks by analyzing task metadata, scores candidate nodes, and prioritizes GPUs within the same non-uniform memory access architecture node or within the same rack. The interference detector monitors GPU memory bandwidth usage through a packet filter program, and determines interference when task bandwidth usage exceeds 80% and there are latency-sensitive tasks on the same device.

4. The method according to claim 1, characterized in that, In step S4, the granularity of the partitioning is configured through the parameters of the device plugin to support multi-instance GPU partitioning; an automatic scaler controller is developed to trigger scaling down when the graphics processor utilization is below a threshold for a continuous period of time, and to trigger scaling up when the task queuing time exceeds the service level target threshold. Vertical scaling prioritizes adjusting the instance size on the current node of the task and uses hot migration technology to upgrade from a small size to a large size; horizontal scaling prioritizes scheduling new replicas to the node with the highest idle rate and adding compute nodes as needed through the cluster autoscaler.

5. The method according to claim 1, characterized in that, In step S5, multi-layer detection rules are configured. GPUXid errors trigger immediate isolation. Error detection and correction error rates greater than the detection threshold are marked as predictive faults. Temperature exceeding the threshold for a set duration triggers frequency reduction protection.

6. The method according to claim 1, characterized in that, Anomaly detection is performed by the node problem detector. Once a fault is detected, a taint is automatically added to the node. The eviction manager evicts the container group on the faulty node based on the taint, and the scheduler reschedules the task to a healthy node. Listen for node resource change events. When a node's status becomes "not ready" for more than a set time, or when a new GPU node is added and its resources are reported, trigger a topology refactoring.

7. A heterogeneous GPU management and scheduling system, characterized in that, For implementing the method as described in any one of claims 1-6, comprising: The heterogeneous device management module is used to achieve pooled management of heterogeneous AI chips through a multi-device plug-in subsystem, a unified abstract adapter, and a topology discovery service. The full-domain monitoring and profiling module is used to achieve fine-grained perception and intelligent modeling of resource status through the indicator collection engine, priority controller and profiling training service; The intelligent scheduling decision module is used to achieve multi-strategy collaborative scheduling with topology awareness, interference awareness, dynamic priority preemption, and intelligent matching by extending the scheduler; The fine-grained resource allocation module is used to dynamically allocate resources and automatically increase or decrease the GPU resources allocated to tasks based on real-time load conditions through the partition manager and automatic scaling controller. The unified runtime support module is used to enable cross-architecture task execution and feedback through a unified runtime library and feedback agent; The fault self-healing and observability module is used through the fault detection engine, topology reconstruction service, and unified monitoring dashboard. In the heterogeneous device management module, the unified abstract adapter is deployed in the initialization container of the user container group. It intercepts native API calls by setting environment variables and maps the interfaces to standardized abstract interfaces.

8. The system according to claim 7, characterized in that, In the intelligent scheduling decision module, the extended scheduler registers six scoring plugins: topology perception scoring plugin, interference scoring plugin, priority scoring plugin, matching degree scoring plugin, balance degree scoring plugin, and utilization rate scoring plugin. A weighted score model is adopted, and the total score is the weighted sum of the scores of each plugin, with the weights dynamically configured.

Images