A method and system for unified management and scheduling of heterogeneous computing power resources

Abstract

The application discloses a method and system for unified management and scheduling of heterogeneous computing power resources, and the method comprises the following steps: based on the Device Plugin mechanism, automatic perception and identification of heterogeneous intelligent computing resources are realized, resource registration is completed through communication between an RPC protocol and a Kubelet component, and a full life cycle management process of 'perception-monitoring-distribution' is established; in view of the heterogeneity of manufacturer interfaces, a unified Device Plugin adaptation platform is constructed, standardized service APIs are provided, and the differences between underlying hardware and manufacturer interfaces are shielded, so that the cross-platform adaptation cost is reduced; based on the chip type, computing power quota and other requirements of a task, a flexible resource scheduling strategy is designed, and on-demand calling and dynamic optimization of the heterogeneous intelligent computing resources are realized. The application can realize unified management and efficient scheduling of the heterogeneous intelligent computing resources, effectively shield the underlying differences of different manufacturer intelligent computing devices, and shorten the resource supply cycle of AI training and reasoning tasks.

Description

Technical Field

[0001] This invention relates to the field of cloud-native and computing power scheduling technologies, and in particular to a method and system for unified management and scheduling of heterogeneous computing power resources. Background Technology

[0002] With the deep integration of technologies such as artificial intelligence, big data, cloud computing, and metaverse, global computing power demand has entered a stage of exponential growth, and the form of computing tasks has undergone a fundamental transformation. First, traditional CPU-based general-purpose computing (oriented towards transaction processing and logical operations) can no longer meet the ultra-high computing power, parallelism, and low latency requirements of new workloads such as large-scale model training, multimodal inference, autonomous driving simulation, and scientific computing. Especially after the ChatGPT explosion in 2022, the scale of large model parameters jumped from tens of billions to trillions, forcing the computing power architecture to shift from "general-purpose-oriented" to "intelligent heterogeneous-oriented". Second, modern data centers, intelligent computing centers, and cloud platforms need to simultaneously support diverse and mixed workloads such as AI training / inference, edge computing, and high-performance computing (HPC). Different workloads have completely different requirements for computing power type, accuracy, bandwidth, and power consumption. Third, driven by policies and technologies such as "Eastern Data, Western Computing", computing power networks, and cloud-edge-device collaboration, computing resources have spread from centralized large-scale data centers to edge nodes, intelligent computing hubs, and terminal devices, forming a ubiquitous heterogeneous computing power pattern that crosses regions, levels, and vendors.

[0003] Simply increasing CPU clock speed and core count has hit a triple bottleneck of power consumption, memory, and interconnectivity. The computing power industry has fully entered the era of heterogeneous computing, with hardware architecture exhibiting high fragmentation: 1. Diversified computing chip types have formed a multi-level heterogeneous computing power system with CPUs as the general control core, GPUs as the main parallel acceleration, and NPUs / TPUs / MLUs for AI-specific acceleration; 2. NVIDIA CUDA, AMD ROCm, Huawei CANN, Cambricon, Intel oneAPI, etc., lack unified standards, interoperability interfaces, and a compatible ecosystem; 3. Multiple computing power carriers such as physical servers, virtual machines, containers, serverless functions, edge boxes, and smart terminals coexist, resulting in physically dispersed and logically isolated "chimney-style" deployments of computing power resources.

[0004] Traditional CPU-centric computing resource scheduling technologies that allocate resources statically or dynamically based on general resource metrics, along with software stacks and driver frameworks built around a single hardware vendor's ecosystem and lacking cross-platform compatibility, suffer from the following problems: 1. Resource discovery, status acquisition, driver interfaces, and performance metrics for different computing power architectures (x86 / ARM / GPU / NPU, etc.) are completely incompatible, and the scheduling system cannot uniformly identify, register, and monitor the true status of heterogeneous computing power; 2. Traditional scheduling uses the number of CPU cores, memory size, and the number of GPUs as core metrics, completely ignoring the essential differences in heterogeneous computing power and failing to perceive computing power performance. For example, the A100 and T4 GPUs have a 10-fold difference in computing power, but the scheduler treats them as "one GPU" and allocates them equally, failing to perceive task adaptation. Large model training requires high-memory GPUs, while inference requires low-power NPUs. Traditional scheduling randomly allocates resources, leading to task failures or performance crashes; 3. Integrating new computing power chips (such as domestic NPUs and DPUs) into existing systems requires deep customization, development, adaptation, and debugging, which can take several months and cannot quickly empower business operations. Summary of the Invention

[0005] Purpose of the invention: The purpose of this invention is to provide a method and system for unified management and scheduling of heterogeneous computing resources, shielding the differences in underlying hardware and realizing intelligent scheduling and load balancing.

[0006] Technical solution: The method for unified management and scheduling of heterogeneous computing resources described in this invention includes the following steps:

[0007] (1) Complete resource registration by communicating with the node agent component of the container orchestration system through the RPC protocol, and establish a full lifecycle management process of "perception-monitoring-allocation";

[0008] (2) Construct a device plug-in adaptation platform;

[0009] (3) Based on the chip type and computing power quota requirements of the task, design resource scheduling strategies to realize on-demand call and dynamic optimization of heterogeneous intelligent computing resources.

[0010] Further, step (1) includes:

[0011] Device plugins customize different devices through resource extension, enabling device identification and registration in the container orchestration system;

[0012] The device plugin registers with the node agent component, and the registration information includes the communication socket, interface version, and resource name;

[0013] The node proxy component calls the plugin to traverse the device directory to obtain the device list and continuously obtains and updates the device health status information;

[0014] The scheduler listens for changes in node resources, declares the type, quantity, and model of intelligent computing resources, finds nodes that meet the requirements, and subtracts the corresponding amount of resources to complete the intelligent computing resource scheduling.

[0015] The node's proxy component sends a request to the device plugin, carrying a list of intelligent computing resource IDs. The device plugin uses the ID list to find the device path, driver directory, and environment variables, and returns the information to the node proxy component to complete the allocation of intelligent computing resources.

[0016] Furthermore, in step (2), the device plug-in adaptation platform adopts a three-layer architecture design, including:

[0017] The interface layer defines core interfaces such as device list reporting, device allocation, and node proxy component registration, serving as a standardized entry point for interaction between device plugins and the container orchestration system.

[0018] The abstraction layer extracts the common management logic of various heterogeneous resources, including unified registration to node agent components, unified device status reporting, and unified device lifecycle management.

[0019] The implementation layer performs personalized operations such as device discovery and hardware status collection, and implements specific device management logic.

[0020] Furthermore, the implementation layer inherits the general management logic of the abstract layer, and implements device discovery and resource allocation functions in a differentiated manner.

[0021] Furthermore, heterogeneous resources include various computing resources such as GPUs, NPUs, and MLUs.

[0022] Furthermore, in step (3), the resource scheduling strategy is defined through the strategy structure, and the scheduling extender schedules the appropriate node according to the video memory, computing power, card type, and scheduling strategy.

[0023] Furthermore, the scheduling extender interacts with the cluster based on the official client library of the container orchestration system.

[0024] Furthermore, filtering nodes that meet the criteria based on the scheduling policy includes: parsing the user-defined scheduling policy; querying the heterogeneous device status of each node; and filtering nodes that meet the criteria.

[0025] The filtered nodes are sorted according to the scheduling strategy type: when the strategy type is compact scheduling, nodes with high resource utilization are selected first; when the strategy type is distributed scheduling, nodes with low resource utilization are selected first.

[0026] A system for unified management and scheduling of heterogeneous computing resources includes:

[0027] The heterogeneous computing power resource access layer is used to connect to computing power devices of different architectures and types, including CPUs, GPUs, and NPUs, to achieve standardized access and status collection of various heterogeneous computing powers.

[0028] The unified resource management module is used to centrally register, manage topology, pool resources and monitor status of all access heterogeneous computing power, build a globally unified view of computing power resources and shield the differences in underlying hardware.

[0029] The resource scheduling decision module is used to make matching decisions between tasks and heterogeneous computing power based on task requirements, computing load, business priority affinity strategies, etc., to achieve intelligent scheduling and load balancing.

[0030] A computer device includes: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the programs, when executed by the processors, implement a method for unified management and scheduling of heterogeneous computing resources.

[0031] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: 1. It realizes automatic perception, unified registration, flexible scheduling and dynamic allocation of heterogeneous intelligent computing devices; 2. It shortens the adaptation and deployment cycle of heterogeneous intelligent computing resources, and provides efficient and flexible resource supply support for model training and inference of artificial intelligence tasks. Attached Figure Description

[0032] Figure 1 This is the business process of the present invention;

[0033] Figure 2 This is a structural diagram of the "sensing-monitoring-allocation" system of the present invention;

[0034] Figure 3 This is a schematic diagram illustrating the unified management of heterogeneous computing resources according to the present invention;

[0035] Figure 4 This is a schematic diagram illustrating the unified scheduling of heterogeneous computing resources according to the present invention;

[0036] Figure 5 This is the cluster resource monitoring interface of the present invention;

[0037] Figure 6 This is a diagram showing the operational verification results of the present invention;

[0038] Figure 7 This is a graph showing the resource usage verification results of the present invention. Detailed Implementation

[0039] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0040] like Figure 1 As shown, the method for unified management and scheduling of heterogeneous computing resources according to the present invention includes the following steps:

[0041] (1) Complete resource registration by communicating with the Kubelet component through the RPC protocol and establish a full lifecycle management process of "perception-monitoring-allocation";

[0042] The method for unified management and scheduling of heterogeneous computing resources is built on a cloud-native Kubernetes system, enabling unified management and scheduling of heterogeneous intelligent computing resources from multiple vendors, such as NVIDIA GPUs, Ascend NPUs, and Cambricon MLUs. Its overall structure is as follows: Figure 2 As shown.

[0043] Different devices are customized through Extended Resources, and device plugins are used to manage the intelligent computing resource devices throughout their entire lifecycle. The device plugins mainly complete the identification and registration of devices in Kubernetes, and then complete the allocation and scheduling of device resources.

[0044] A device plugin platform is built at the device plugin layer to connect to heterogeneous intelligent computing resources from different manufacturers such as NVIDIA, NPU, and MLU, shielding the differences between heterogeneous intelligent computing resources and providing standard interfaces such as HTTP-based JSON to facilitate the development, debugging, and management of heterogeneous devices.

[0045] The device plugin platform is deployed in DaemonSet mode. After startup, it uses the gRPC protocol to call the kubelet.sock file to register with kubelet. The registration information includes the Unix socket, API Version, and ResourceName. After registration, kubelet starts the intelligent computing resource reporting process. kubelet calls the plugin's ListAndWatch() method to traverse the files in the device directory to obtain the device list and continuously obtain the latest device health status information, updating it to the Kubernetes APIServer. This completes the reporting of intelligent computing resources in the Kubernetes system. The Kubernetes scheduler listens for changes in node resources and uses extended resource requirements when deciding on Pod scheduling.

[0046] When a Pod for training or inference needs intelligent computing resources, the type, quantity, and specific model of the intelligent computing resources are declared in the `limits` field under `Resource` in the Pod's YAML file. The Kubernetes scheduler finds a node that meets the demand and subtracts the corresponding number of GPUs from that node, completing the scheduling of the Pod with intelligent computing resources. Then, the kubelet on that node sends an `Allocate()` request to the device plugin, carrying a list of intelligent computing resource IDs. The device plugin looks up the device path, driver directory, and environment variables based on the ID list and returns the response to the kubelet via `AllocateResponse()`. The kubelet uses this information to allocate GPUs to the container and mounts the required driver directories for the GPUs, thus completing the allocation of intelligent computing resources by Kubernetes for the Pod with intelligent computing services.

[0047] (2) Design and build a unified Device Plugin adaptation platform;

[0048] To achieve efficient and unified management of heterogeneous intelligent computing resources from multiple vendors, the platform adopts a three-layer architecture design, such as... Figure 3 As shown, the core objective is to shield the differences in heterogeneous computing power and achieve standardized management of various intelligent computing resources.

[0049] Interface Layer (DevicePluginServer): Following the official Kubernetes device plugin interface specification, it defines core interfaces such as device list reporting, device allocation, and kubelet registration. It serves as a standardized entry point for device plugins to interact with the Kubernetes system, providing a unified interaction specification for heterogeneous computing power management.

[0050] The abstract layer (HeteroDevicePlugin): As the core of heterogeneous computing power management, it is an abstract class for general heterogeneous device plugins. It defines the general logic for device management. GPUs, NPUs, MLUs, etc., can quickly adapt to heterogeneous devices from any manufacturer by inheriting from it and only needing to override DiscoverDevices() and AllocateResource(). Its core function is to extract the general management logic of various heterogeneous resources such as GPUs, NPUs, and MLUs, including unified registration to kubelet, unified device status reporting, and unified device lifecycle management, realizing "develop once, reuse across multiple chips". It effectively shields the common differences between heterogeneous computing power from different manufacturers and reduces the complexity of management.

[0051] Implementation layer (GPUPlugin / NPUPlugin, etc.): This layer differs only in implementing device discovery and resource allocation; all other aspects inherit from the parent class, offering extremely high extensibility. It implements specific device management logic tailored to the unique characteristics of devices from different manufacturers. This is the core layer that directly interacts with the hardware, responsible for manufacturer-specific operations such as device discovery and hardware status acquisition.

[0052] Meanwhile, the Device class is designed to uniformly encapsulate the core attributes of heterogeneous devices (such as device ID, health status, video memory, model, manufacturer, etc.), shielding the attribute differences of devices from different manufacturers. This allows kubelet and scheduler to only process standardized device information, further simplifying the heterogeneous computing power management logic and ensuring that the platform has efficient and universal heterogeneous computing power management capabilities.

[0053] (3) Based on the chip type and computing power quota requirements of the task, design resource scheduling strategies to realize on-demand access and dynamic optimization of heterogeneous intelligent computing resources, such as... Figure 4 As shown.

[0054] Resource scheduling policies are defined through `SchedulerPolicy`. `HeteroSchedulerExtender` schedules resources to suitable nodes based on memory, computing power, and card type, according to the scheduling policy: parsing the user-defined scheduling policy; querying the heterogeneous device status of each node; filtering nodes that meet the criteria; and sorting the filtered nodes according to the scheduling policy type: when the policy type is `binpack`, nodes with high resource utilization are prioritized; when the policy type is `spread`, nodes with low resource utilization are prioritized. `HeteroSchedulerExtender` interacts with the Kubernetes cluster based on `k8s.io / client-go`, without intruding on Kubernetes, conforming to cloud-native design principles. Key methods:

[0055] ParseAnnotation: Parses the scheduler-policy annotation of a Pod and is the entry point for scheduling policies;

[0056] FilterNodes: The core logic, which combines strategies (binpack / spread) and node device status to filter nodes;

[0057] HandleHttpRequest: Adapts to the HTTP request format of the Kubernetes scheduler and serves as the entry point for providing services to the outside world.

[0058] This invention also provides a system for unified management and scheduling of heterogeneous computing resources, comprising:

[0059] The heterogeneous computing power resource access layer is used to connect to computing power devices of different architectures and types, including CPUs, GPUs, NPUs, etc., to achieve standardized access and status collection of various heterogeneous computing power.

[0060] The unified resource management module is used to centrally register, manage topology, pool resources and monitor status of all access heterogeneous computing power, build a globally unified view of computing power resources and shield the differences in underlying hardware.

[0061] The resource scheduling decision module is used to make matching decisions between tasks and heterogeneous computing power based on task requirements, computing load, business priority affinity strategies, etc., to achieve intelligent scheduling and load balancing.

[0062] To verify the effectiveness of this method, an experiment was conducted. The hardware environment for the experiment is shown in Table 1, and the software environment is shown in Table 2.

[0063] Table 1 Hardware Environment

[0064] Table 2 Software Environment

[0065] This test requires three servers: one with NVIDIA GPU resources, one with Huawei Ascend NPU resources, and one with Baidu Kunlun XPU. The heterogeneous computing resources are managed, allocated, and scheduled through a unified management and scheduling method.

[0066] Heterogeneous scheduling and flattening verification:

[0067] (1) Verification target

[0068] The verification platform's ability to automatically identify, uniformly register, and monitor the status of heterogeneous intelligent computing resources from multiple vendors ensures that different types of resources can be standardized and managed.

[0069] (2) Verification process

[0070] Resource access configuration: Deploy the corresponding device plugins (DaemonSet mode) for Huawei Ascend 310P3, NVIDIA A10, and Kunlun Core R200 through HeteroDevicePlugin;

[0071] Checking the status of cluster management: This can be done through the cluster resource monitoring interface, such as... Figure 5As shown: Huawei Ascend 310P3, node address 192.168.3.7, status "unused", vDevice allocation 0 / 7, computing power allocation 0 / 8; NVIDIA A10, node address 192.168.3.5, status "used", vDevice allocation 1 / 4, computing power allocation 60 / 100.

[0072] Through the above process, the platform can achieve unified management of Huawei Ascend 310P3 and NVIDIA A10, support visualized monitoring of resource status and quotas, and provide a foundation for targeted scheduling.

[0073] Heterogeneous scheduling platform scheduling verification:

[0074] (1) Verification target

[0075] Verify the platform's ability to accurately schedule specified types of heterogeneous resources, ensuring that intelligent computing services can be bound to target resources and operate normally.

[0076] (2) Verification process

[0077] Business Configuration and Deployment: Write the `ascend-demo-vnpu.yaml` file, declare the Huawei Ascend 310P3 resource requirements using `resources.limits`, and execute the deployment: `kubectl apply -f ascend-demo-vnpu.yaml`

[0078] Business operation verification: Enter the container to check the execution status of the image classification business: `kubectl exec -itresnetinfer1-1-1usoc-vnpu-6gkrt bash`. The business loops to identify dog ​​images, and the log output shows the recognition result as "beagle" with a confidence score of 0.902203. Figure 6 As shown.

[0079] Resource usage verification, such as Figure 7 As shown:

[0080] The AI ​​Core utilization rate of the Huawei Ascend 310P3 is 2%, with memory usage of 74MB / 5381MB; the business process (Python 3.7) is bound to the NPU, and the resources are successfully utilized.

[0081] Through the above process, the platform can achieve targeted scheduling of heterogeneous resources, and services can be accurately bound to the Huawei Ascend310P3 and run stably, verifying the scheduling accuracy and service compatibility.

Claims

1. A method for unified management and scheduling of heterogeneous computing resources, characterized in that, Includes the following steps: (1) Complete resource registration by communicating with the node agent component of the container orchestration system through the RPC protocol, and establish a full lifecycle management process of "perception-monitoring-allocation"; (2) Construct a device plug-in adaptation platform; (3) Based on the chip type and computing power quota requirements of the task, design resource scheduling strategies to realize on-demand call and dynamic optimization of heterogeneous intelligent computing resources.

2. The method for unified management and scheduling of heterogeneous computing resources according to claim 1, characterized in that, Step (1) includes: Device plugins customize different devices through resource extension, enabling device identification and registration in the container orchestration system; The device plugin registers with the node agent component, and the registration information includes the communication socket, interface version, and resource name; The node proxy component calls the plugin to traverse the device directory to obtain the device list and continuously obtains and updates the device health status information; The scheduler listens for changes in node resources, declares the type, quantity, and model of intelligent computing resources, finds nodes that meet the requirements, and subtracts the corresponding amount of resources to complete the intelligent computing resource scheduling. The node's proxy component sends a request to the device plugin, carrying a list of intelligent computing resource IDs. The device plugin uses the ID list to find the device path, driver directory, and environment variables, and returns the information to the node proxy component to complete the allocation of intelligent computing resources.

3. The method for unified management and scheduling of heterogeneous computing resources according to claim 1, characterized in that, In step (2), the device plug-in adaptation platform adopts a three-layer architecture design, including: The interface layer defines core interfaces such as device list reporting, device allocation, and node proxy component registration, serving as a standardized entry point for interaction between device plugins and the container orchestration system. The abstraction layer extracts the common management logic of various heterogeneous resources, including unified registration to node agent components, unified device status reporting, and unified device lifecycle management. The implementation layer performs personalized operations such as device discovery and hardware status collection, and implements specific device management logic.

4. The method for unified management and scheduling of heterogeneous computing resources according to claim 3 implements the general management logic of the abstract layer by layer inheritance, and differentiates the functions of device discovery and resource allocation.

5. The method according to claim 1, characterized in that, Heterogeneous resources include: GPU, NPU, MLU and other computing resources.

6. The method according to claim 1, characterized in that, In step (3), the resource scheduling strategy is defined through the strategy structure. The scheduling extender schedules the appropriate node according to the video memory, computing power, card type, and scheduling strategy.

7. The method according to claim 6, characterized in that, The scheduling extender interacts with the cluster based on the official client library of the container orchestration system.

8. The method according to claim 6, characterized in that, Filtering nodes that meet the criteria based on the scheduling policy includes: parsing the user-defined scheduling policy; querying the heterogeneous device status of each node; and filtering nodes that meet the criteria. The filtered nodes are sorted according to the scheduling strategy type: when the strategy type is compact scheduling, nodes with high resource utilization are selected first; when the strategy type is distributed scheduling, nodes with low resource utilization are selected first.

9. A system for unified management and scheduling of heterogeneous computing resources, characterized in that, include: The heterogeneous computing power resource access layer is used to connect to computing power devices of different architectures and types, including CPUs, GPUs, and NPUs, to achieve standardized access and status collection of various heterogeneous computing powers. The unified resource management module is used to centrally register, manage topology, pool resources and monitor status of all access heterogeneous computing power, build a globally unified view of computing power resources and shield the differences in underlying hardware. The resource scheduling decision module is used to make matching decisions between tasks and heterogeneous computing power based on task requirements, computing load, business priority affinity strategies, etc., to achieve intelligent scheduling and load balancing.

10. A computer device, characterized in that, include: One or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein the programs, when executed by the processors, implement the methods as described in claims 1-8.

Images