A multi-intelligent center-oriented heterogeneous computing power unified scheduling method and system

Abstract

This invention discloses a unified scheduling method and system for heterogeneous computing power across multiple intelligent computing centers. It includes: abstracting the physical heterogeneous resources of multiple intelligent computing centers into Kubernetes virtual nodes using a four-tuple mapping model to form a unified resource view; establishing a multi-level resource preference constraint expression and matching mechanism based on a combination of intelligent computing center identifiers and computing pool identifiers; defining a unified processor type enumeration and structured capability description framework; employing a multi-stage plug-in scheduling framework to decouple the scheduling process into three stages: filtering, scoring, and pre-binding, and achieving flexible expansion of scheduling strategies through standardized plug-in interfaces; and performing capacity gating and specification availability verification during the admission stage for interactive jobs. This invention solves problems such as fragmented cross-center resource views, inconsistent descriptions of heterogeneous devices, rigid scheduling strategies, and poor interactive job experience, achieving unified, intelligent, and efficient scheduling of heterogeneous computing power resources across multiple intelligent computing centers.

Description

Technical Field

[0001] This invention relates to the fields of cloud computing and computing power scheduling technology, specifically to a method and system for unified scheduling of heterogeneous computing power for multiple intelligent computing centers. Background Technology

[0002] With the rapid development of artificial intelligence and high-performance computing applications, centralized single computing centers can no longer meet the ever-increasing demand for large-scale, distributed computing. Against this backdrop, intelligent computing center networks composed of multiple geographically distributed, heterogeneous devices have emerged, aiming to improve the overall utilization efficiency of computing resources and the reliability of services through pooling and collaborative scheduling. However, under this new multi-center, highly heterogeneous computing architecture, traditional resource scheduling technologies face severe challenges, and existing mainstream solutions have the following significant shortcomings:

[0003] 1) The abstraction and description of heterogeneous computing resources lack a unified standard. In existing technologies, cloud-native orchestration systems, represented by Kubernetes, have become the de facto standard for data center resource management. Their default schedulers are mainly designed for homogeneous general-purpose computing resources (such as CPUs and memory). For heterogeneous accelerators such as GPUs, NPUs, and DPUs, although they can be declared through the "extended resource" mechanism, they essentially only provide simple quantity identifiers and lack standardized descriptions of key attributes such as device model, architecture generation, computing power, memory bandwidth, and interconnect topology. Device plugins provided by different vendors (such as NVIDIA, Huawei Ascend, Cambricon, etc.) use their own proprietary resource naming and tagging systems, making it impossible for upper-layer scheduling systems to understand and match diverse computing power demands and supplies from a unified and standardized perspective. This results in complex and fragmented scheduling logic, and high costs for system maintenance and cross-platform porting.

[0004] 2) Insufficient ability to express fine-grained strategies for multi-center scheduling; existing scheduling frameworks (such as the Kubernetes native scheduler and its enhancements like Volcano) primarily optimize resource contention within a single cluster. Their scheduling strategies (such as node selectors and affinity / anti-affinity) did not fully consider multi-layered, geographically dispersed resource pool models like "intelligent computing center - computing power pool" in their initial design. Therefore, users struggle to accurately express complex cross-center preferences and constraints, such as "prioritizing the A100 computing power pool in the Beijing center, then downgrading to a similar pool in the Shanghai center, while avoiding centers currently under maintenance." While multi-cluster management projects (such as Karmada) can distribute applications across multiple clusters, their strategies have a high level of abstraction, resulting in weak perception and decision support for fine-grained scheduling factors such as computing power type, resource pool level, cross-center network quality, and cost. This makes it impossible to achieve business-aware, intelligent cross-center computing power optimization and load balancing.

[0005] 3) Interactive job scheduling feedback is lagging, resulting in low resource matching efficiency. For interactive jobs such as Notebook jobs that require rapid startup, existing scheduling systems generally adopt a passive "submit first, queue later, match later" model. Users cannot predict resource availability when submitting jobs, often resulting in long queues and potential failures due to insufficient resources. This leads to a poor user experience and causes the scheduling system to make numerous ineffective matching attempts, increasing system overhead and reducing overall resource utilization efficiency. The root cause lies in the lack of a mechanism for real-time capacity gating and intelligent instant feedback during the job submission admission phase.

[0006] 4) The scheduling system architecture is tightly coupled, resulting in poor scalability and maintainability. Existing scheduling logic is often deeply embedded in the scheduler's core code. When a new scheduling strategy needs to be added (e.g., scheduling based on real-time cost or scheduling based on specific topology affinity), it is usually necessary to directly modify the scheduler's source code. This tightly coupled architecture makes the system difficult to scale, and the development, testing, and deployment of new strategies have long cycles, high risks, and an inability to adapt to rapidly changing business needs and diverse scheduling strategies. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a unified scheduling method and system for heterogeneous computing power across multiple intelligent computing centers. This overcomes the deficiencies of existing technologies, achieves unified abstraction, precise scheduling, and efficient governance of heterogeneous computing power across multiple centers, and improves system scalability, scheduling flexibility, and interactive work experience.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A method for unified scheduling of heterogeneous computing power across multiple intelligent computing centers includes the following steps:

[0010] S1: Virtual Node Abstraction; The physical heterogeneous computing power resources of multiple intelligent computing centers are mapped to virtual nodes in the Kubernetes cluster according to the four-tuple mapping model. Each virtual node represents a resource combination that is uniquely determined by the intelligent computing center identifier, computing power pool identifier, processor type and specification model, and carries metadata through tags. The real-time resource status of each center is synchronized by the virtual node controller.

[0011] S2: Preference Constraint Expression and Matching: Receive job requests submitted by users, the requests containing multi-level resource preference constraints defined based on a binary combination of intelligent computing center identifier and computing power pool identifier; the multi-level resource preference constraints include at least hard constraints and soft preferences;

[0012] S3: Unified processor type management: Based on a predefined unified processor type enumeration and processor capability description framework, it performs compatibility verification between the resource requirements of job requests and the processor capabilities represented by virtual nodes.

[0013] S4: Multi-stage plug-in scheduling; a multi-stage plug-in scheduling framework is used to make scheduling decisions for the job. The multi-stage plug-in scheduling framework includes at least a filtering stage, a scoring stage, and a pre-binding stage executed sequentially. Each stage performs corresponding filtering, scoring, and pre-binding operations through plug-ins that conform to a unified interface specification to determine the target virtual node.

[0014] S5: Interactive Job Gating: In response to the recognition that a job is interactive, capacity gating and specification availability verification are performed during the job submission admission phase, and feedback is provided in real time based on the verification results.

[0015] Preferably, the multi-level resource preference constraints include:

[0016] Hard constraints are used to specify the computing center and / or computing power pool to which a job must be scheduled;

[0017] Soft preferences are used to specify one or more weighted combinations of intelligent computing centers and computing power pools.

[0018] Extended constraints are used to specify at least one of the following constraints: minimum number of available nodes, maximum network latency, or cost budget.

[0019] Preferably, during the scoring phase, the rules for scoring virtual nodes based on the soft preference include:

[0020] If the intelligent computing center identifier and computing power pool identifier of a virtual node are both completely matched with the identifier specified by a certain soft preference item, then the node obtains the full weight score of that preference item.

[0021] If only the intelligent computing center identifier of a virtual node matches a certain soft preference item, then the node receives a first proportion weight score for that preference item that is lower than the full weight.

[0022] If a virtual node's computing power pool identifier matches a certain soft preference item, then the node receives a second-proportion weight score for that preference item that is lower than the first proportion.

[0023] Preferably, the unified processor type enumeration includes at least a graphics processor, a neural network processor, a deep computing unit, a machine learning unit, a general-purpose computing unit, and a central processing unit; the processor capability description framework is defined through a protocol buffer message format, and the message structure contains fields for describing basic attributes, computing power, memory specifications, interconnect topology, and software stack compatibility;

[0024] Step S3, unified processor type management, also includes: maintaining a processor specification registry, using a key-value mapping structure to store the capability description data of each processor, and registering detailed specifications and capability information of heterogeneous processors from different manufacturers and models using a unified structured capability description framework.

[0025] Preferably, in the multi-stage pluggable scheduling framework:

[0026] The filtering operations performed in the filtering phase include at least: central pool constraint filtering based on hard constraints, compatibility filtering based on the unified processor type, filtering based on resource capacity, and filtering based on node taints and job tolerance.

[0027] The scoring operations performed in the scoring phase include at least: central pool preference scoring based on the soft preference level, scoring based on resource utilization, scoring based on topology distribution, and scoring based on resource cost.

[0028] The operations performed during the pre-binding phase include at least: reserving resources in the target intelligent computing center, verifying user or project quotas, and writing scheduling binding metadata, and the operations support rollback in case of failure.

[0029] Preferably, all plugins in the filtering stage, scoring stage, and pre-binding stage implement a predefined programming language interface;

[0030] The execution order and priority of plugins are orchestrated and controlled through configuration files. Plugins in the filtering phase are executed serially, while plugins in the scoring phase support parallel execution.

[0031] Preferably, step S5: interactive job gating specifically includes:

[0032] S501: Automatically identifies Notebook-type interactive jobs by checking one or more of the following: preset job type tags, container image name keywords, exposed port numbers, and resource request patterns.

[0033] S502: For the identified interactive jobs, perform capacity gating on the verification dimensions, which include: whitelist of specifications and models, central concurrent quantity limit, real-time resource availability and user quota;

[0034] S503: Based on the verification results, return four types of results: immediately available, insufficient quantity, temporarily unavailable, or permanently unavailable, and provide alternative specifications or waiting time suggestions when resources are insufficient.

[0035] This invention also discloses a heterogeneous computing power unified scheduling system for multiple intelligent computing centers, comprising:

[0036] The virtual node abstraction module is used to abstract the physical resources of each intelligent computing center into virtual nodes in the Kubernetes cluster according to the four-tuple mapping model, and to configure metadata tags for the virtual nodes, including intelligent computing center identifier, computing power pool identifier, processor type and specification model.

[0037] A resource synchronization controller is connected to the virtual node abstraction module and multiple intelligent computing centers, and is used to synchronize the real-time resource status of each center and update the resource information of the corresponding virtual nodes.

[0038] The scheduling request interface is used to receive job requests submitted by users.

[0039] The constraint parsing module, connected to the scheduling request interface, is used to parse out multi-level resource preference constraints defined by a binary combination of intelligent computing center identifier and computing power pool identifier from the job request;

[0040] A unified specification registry is used to store and manage heterogeneous processor specification information based on a unified processor type enumeration and processor capability description framework.

[0041] A multi-stage pluggable scheduling engine is connected to the constraint parsing module, the unified specification registration center, and the virtual node abstraction module, respectively. It is used to load and execute the filtering pluggable set, the scoring pluggable set, and the pre-bound pluggable set in sequence to determine the target node from the virtual nodes according to the multi-level resource preference constraints, processor compatibility, and resource capacity.

[0042] The job pre-verification module is connected to the scheduling request interface and is used to identify interactive jobs and perform capacity gating and specification availability verification on them before scheduling.

[0043] Preferably, the resource synchronization controller is specifically used to: communicate with the resource management systems of each intelligent computing center through a state transition application programming interface to periodically synchronize resource information to update the capacity and availability of virtual nodes; and to add a scheduling prohibition taint to virtual nodes from intelligent computing centers that are not accessible from the network.

[0044] Preferably, the pre-job verification module specifically includes:

[0045] The job identification unit is used to identify Notebook-type interactive jobs by checking one or more of the following methods: job type labels, container image name keywords, exposed port numbers, and resource request patterns.

[0046] The gating execution unit is used to perform capacity gating on the identified jobs, including specification and model whitelist verification, central concurrent number limit check, real-time resource availability query and user quota verification.

[0047] The feedback unit is used to generate and return feedback information containing resource availability status and alternative suggestions based on the gating results.

[0048] This invention provides a method and system for unified scheduling of heterogeneous computing power across multiple intelligent computing centers. It offers the following advantages:

[0049] By employing virtual node abstraction and real-time synchronization mechanisms, a unified global view and intelligent operation and maintenance of cross-center resources are achieved, laying a solid foundation for efficient and reliable scheduling. Through a four-tuple mapping model combined with the continuous resource synchronization and state maintenance of the virtual node controller, dispersed and heterogeneous physical resources are uniformly abstracted into standardized Kubernetes node objects. This significantly reduces the complexity of cross-center resource management; the scheduler only needs to interact with a unified virtual node interface, eliminating the need to adapt to numerous heterogeneous backend systems. Secondly, it ensures the accuracy and real-time nature of scheduling decisions, performing scheduling based on accurate real-time capacity information synchronized by the controller, significantly improving the success rate of the first round of scheduling and global resource utilization. Furthermore, by incorporating center-level intelligent fault tolerance and operation and maintenance capabilities, and using an automatic taint marking mechanism to isolate faulty or under-maintenance centers, seamless fault switching and canary releases for cross-center services are achieved, ensuring high availability of the scheduling service.

[0050] By establishing a standardized heterogeneous processor description system and a unified registry center, device-independent scheduling and forward-looking seamless scalability are achieved. By defining a unified processor type enumeration covering mainstream processors and a structured capability description framework based on protocol buffers, supplemented by a dynamically managed unified specification registry center, the scheduling logic is completely decoupled from specific device models. The scheduler only needs to identify the standardized "processor type" and query the structured capability attributes, without needing to understand and adapt to each vendor's proprietary interfaces, greatly simplifying the core scheduling logic and reducing code complexity and long-term maintenance costs. It also endows the system with easy scalability and ecosystem compatibility. Supporting new processors only requires adding description entries to the registry center without modifying any core scheduling code, enabling the system to smoothly and quickly adapt to the ever-emerging AI computing hardware, effectively avoiding technology lock-in.

[0051] By employing a multi-level preference constraint expression and a multi-stage pluggable scheduling architecture, the system achieves refined and intelligent scheduling strategies and sustainable system architecture evolution. At the user interface level, it provides a multi-level expression of hard constraints, soft preferences, and extended constraints based on a binary combination of "intelligent computing center identifier + computing pool identifier." At the system execution level, it constructs a three-stage pluggable scheduling engine that follows a standard interface, encompassing filtering, scoring, and pre-binding. Through this combination, at the business level, it provides users with precise and flexible cross-center scheduling strategy expression capabilities, allowing for customization of requirements from multiple dimensions such as cost, performance, location, and network, upgrading scheduling from resource allocation to intelligent decision-making driven by business objectives. At the system level, the pluggable architecture achieves complete decoupling of strategy logic from the core scheduling framework. Each strategy, as an independent pluggable, can be dynamically loaded, combined, and upgraded. This makes adding or optimizing scheduling strategies safe and convenient, supporting "hot-plugging" and rapid iteration of strategies, greatly improving the system's scalability, maintainability, and agile response to future business needs.

[0052] Through a pre-emptive interactive intelligent gating and verification mechanism, the developer experience and system resource utilization have been fundamentally improved. This is achieved by integrating a pre-emptive job verification module that integrates automatic identification, real-time capacity gating, and instant feedback. For users, this represents a leap from "blindly waiting after submission" to "instant awareness before submission." The system provides clear feedback within milliseconds, such as "immediate availability," "estimated waiting time," or "alternative solution suggestions," completely eliminating unnecessary waiting and significantly improving development efficiency and satisfaction. For the system, it intercepts a large number of requests destined to fail due to insufficient resources from the scheduling queue, significantly reducing the scheduler's unnecessary load and redundant queries in the backend system, thereby improving the overall system throughput and resource utilization efficiency. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in this invention or the prior art, the accompanying drawings used in the description of the prior art will be briefly introduced below.

[0054] Figure 1 The flowchart of the steps of the present invention is a method for unified scheduling of heterogeneous computing power in multiple intelligent computing centers;

[0055] Figure 2 The overall system architecture layer diagram of this invention;

[0056] Figure 3 The multi-stage plug-in scheduling flowchart in this invention;

[0057] Figure 4 The interactive job gating and verification flowchart in this invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0059] Example 1, as Figures 1 to 4 As shown, this embodiment provides a unified scheduling method for heterogeneous computing power across multiple intelligent computing centers. This method is executed by a scheduling system deployed on a unified management plane, which is interconnected with multiple intelligent computing centers (such as those in Beijing, Shanghai, and Shenzhen) located in different geographical locations. Each intelligent computing center contains one or more computing power pools equipped with heterogeneous accelerators from different manufacturers or models (such as NVIDIA GPUs, Huawei NPUs, Cambricon MLUs, etc.). The method includes the following steps:

[0060] S1: Virtual node abstraction;

[0061] The goal of this step is to map physical resources scattered across multiple intelligent computing centers into a logically unified Kubernetes cluster resource view. In practice:

[0062] S101: Define the quadruple mapping model: The model is (AiCenterId, PoolId, ProcessorType, SKU) → Virtual Node. Here, AiCenterId is the unique identifier of the intelligent computing center (e.g., center-bj), PoolId is the identifier of the computing pool (e.g., pool-gpu-a100), ProcessorType is a unified processor type enumeration value (e.g., GPU), and SKU is the specific specification model (e.g., a100-80g).

[0063] S102: Create and Tag Virtual Nodes: Based on the four-tuple mapping model described above, create a virtual node in the Kubernetes management cluster for each unique four-tuple combination. Virtual node naming follows a convention, for example, vnode-center-bj-pool-gpu-a100-gpu-a100-80g. Simultaneously, assign a set of standardized tags to this virtual node to carry its metadata:

[0064] grampus.openi.org.cn / aiCenterId=center-bj

[0065] grampus.openi.org.cn / poolId=pool-gpu-a100

[0066] grampus.openi.org.cn / processorType=GPU

[0067] grampus.openi.org.cn / sku=a100-80g

[0068] Optional tags include grampus.openi.org.cn / generation=ampere (architectural generation).

[0069] S103: Synchronizing Resource Status: A separate virtual node controller component runs in the management cluster. It periodically (e.g., every 30 seconds) polls the resource management systems of each computing center via a REST API (Representation State Transition Application Programming Interface) to obtain real-time information such as the total resource volume, allocated resources, and health status of each computing pool. The controller compares the cached data and dynamically executes the creation, updating (refreshing capacity information), or deletion of virtual nodes. If a center's network becomes temporarily unavailable, the controller adds a NoSchedule taint to all corresponding virtual nodes to prevent new jobs from being scheduled there, thus achieving fault isolation.

[0070] S2: Preference Constraint Expression and Matching

[0071] This step addresses the complex scheduling requirements when users submit jobs. Users declare their resource preferences in the job's YAML configuration file via annotations or specific API fields. Specifically, this includes:

[0072] S201: Define multi-level constraints:

[0073] Hard constraints are conditions that must be met. They are used to specify the intelligent computing center and / or computing pool to which the job must be scheduled; for example, centerId: ["center-bj"] means that the job must be deployed in the Beijing center; poolId: ["pool-gpu-a100"] means that the A100 computing pool must be used; they can also be combined as (center-bj, pool-gpu-a100).

[0074] Soft preference, a weighted bias selection. Used to specify one or more weighted combinations of intelligent computing centers and computing pools; for example, a user can declare: preferred (center-bj, pool-gpu-a100), weight 90; secondary (center-sh, pool-gpu-v100), weight 60.

[0075] Extended constraints are used to specify at least one of the following constraints: minimum number of available nodes, maximum network latency, or cost budget. For example, a minimum of 2 available nodes, cross-center network latency < 50ms, or hourly cost < 50 yuan.

[0076] S202: Matching and Scoring Implementation: During the scheduler's scoring phase, the central pool scoring plugin executes the matching logic. Taking the soft preference "(center-bj, pool-gpu-a100): weight 90" as an example:

[0077] Exact match: If the virtual node labels exactly match center-bj and pool-gpu-a100, the score is 90 × 100% = 90 points.

[0078] Center matching: If a node only matches center-bj but not the pool, the score is 90 × 50% = 45 points.

[0079] Pool matching: If a node only matches pool-gpu-a100 but not the center, the score is 90 × 30% = 27 points.

[0080] The scores of all soft preferences are summed and normalized to form the final preference score for that node.

[0081] S3: Unified Processor Type Management: Based on a predefined unified processor type enumeration and processor capability description framework, it performs compatibility checks on the resource requirements of job requests and the processor capabilities represented by virtual nodes; specific implementation includes:

[0082] S301: Establishing an Enumeration and Description Framework: The system predefines processor type enumerations such as Graphics Processing Unit (GPU), Neural Processing Unit (NPU), Deep Computing Unit (DCU), Machine Learning Unit (MLU), General Purpose Computing Unit (GCU), and Central Processing Unit (CPU). Simultaneously, Protocol Buffers (protobuf) defines a structured message called "Processor Capability" as a unified capability description framework. This message contains:

[0083] Basic attributes (manufacturer, model, generation);

[0084] Computing power (supported precision, computing power unit: trillions of operations per second, dedicated instruction set);

[0085] Memory specifications (capacity, bandwidth, type);

[0086] Interconnect topology (NVIDIA link / high-speed peripheral component interconnect bandwidth, topology);

[0087] Software stack compatibility (supported drivers, deep learning frameworks, container images).

[0088] S302: Maintenance Specification Registry: The system maintains a processor specification registry (which can be based on a database or configuration center). When a new intelligent computing center or a new processor model comes online, operations and maintenance personnel only need to register one record. For example, the key is GPU-A100-80GB, and the value is a protobuf object populated with detailed "processor capabilities" of the A100. The scheduler can understand and compare the differences between different devices by querying this registry.

[0089] S4: Multi-stage pluggable scheduling; a multi-stage pluggable scheduling framework is used to make scheduling decisions for the job. The multi-stage pluggable scheduling framework includes at least a filtering stage, a scoring stage, and a pre-binding stage executed sequentially. Each stage performs corresponding filtering, scoring, and pre-binding operations through pluggins that conform to a unified interface specification to determine the target virtual node; specific implementation includes:

[0090] S401: Phase Division and Plugin Execution

[0091] (1) Filtering phase: The scheduler sequentially calls a series of filter plugins to quickly eliminate nodes that do not meet the hard conditions.

[0092] Central pool filter plugin: Hard constraint filtering based on step S2.

[0093] Processor type filter plugin: Checks whether the node processor type meets the job requirements (if the job requires an NPU, then GPU nodes are filtered out).

[0094] Resource filter plugin: Checks CPU, memory, and extended resource (such as nvidia.com / gpu) capacity.

[0095] Taint Tolerance Filter Plugin: Checks node taints and Pod tolerance.

[0096] (2) Scoring stage: For nodes that pass the filter, multiple scoring plugins calculate scores in parallel.

[0097] Central pool scoring plugin: Calculates scores based on soft preferences in step S2.

[0098] Resource utilization rating plugin: Prefers to select nodes with more balanced resource utilization (e.g., 60%-80%).

[0099] Topology Distribution Scoring Plugin: Promotes the even distribution of container groups among topological domains such as the center and regions.

[0100] Cost-aware scoring plugin: Based on the resource cost model, prioritize nodes with lower costs.

[0101] The scores of each plugin are weighted and combined, and the node with the highest total score wins.

[0102] (3) Pre-binding phase: Perform atomic operations before final binding.

[0103] Capacity reservation plugin: Calls the target intelligent computing center's application programming interface to temporarily reserve the required resources.

[0104] Quota verification plugin: Verifies whether a user's or project's quota has exceeded the limit.

[0105] Metadata binding plugin: Writes associated metadata on container groups and nodes.

[0106] If any operation fails, all operations are rolled back, and the scheduling cycle restarts.

[0107] S402: Plug-in Architecture: All plugins implement the Go language interfaces defined by the Kubernetes scheduler framework (such as the filtering plugin interface, scoring plugin interface, and pre-binding plugin interface). A configuration file determines the loading, activation order, and priority of plugins, enabling "hot-plugging" and flexible orchestration of scheduling strategies.

[0108] S5: Interactive Job Gating: In response to the identification of an interactive job, capacity gating and specification availability verification are performed during the job submission admission phase, and feedback is provided in real time based on the verification results. Specific implementation includes:

[0109] S501: Job Identification: When a user submits a container group, the job pre-validation module uses a multi-level strategy to identify whether it is a Notebook-type job.

[0110] Highest priority: Check the container group tag grampus.openi.org.cn / workload-type:notebook.

[0111] Secondly: Check if the container image name contains the keywords "jupyter" or "notebook".

[0112] Finally: Check if typical ports such as 8888 are exposed.

[0113] S502: Capacity Gating: Access control is triggered immediately once the device is identified as a Notebook.

[0114] The system verifies whether the requested specifications (e.g., a100-80g) are on the target center's "whitelist." It checks if the user's active Notebook instance count in the current center exceeds the limit. It queries the target computing pool's resource inventory in real time. It verifies the user's resource quota.

[0115] S503: Instant Feedback: Verification is completed in milliseconds and the results are returned via the Kubernetes verification admission control webhook; specifically, the returned results include:

[0116] (1) Immediately available: Access granted.

[0117] (2) Insufficient quantity: Rejected, prompting "Only X cards are available, Y cards are required".

[0118] (3) Temporarily unavailable: Rejected, with the message "Resources are expected to be released in Z minutes. You can wait in the queue".

[0119] (4) Permanently unavailable: Reject, prompt "Specification does not exist or has been discontinued", and recommend alternative specifications (such as "It is recommended to use v100-32g").

[0120] This mechanism enables "knowing before submission," avoiding long and fruitless waiting times for users.

[0121] This invention employs a four-tuple mapping model to abstract geographically dispersed and architecturally diverse physical resources into virtual nodes within a Kubernetes cluster, supplemented by a standardized tagging system to carry metadata. This completely breaks down resource barriers between different intelligent computing centers, constructing a logically unified and information-complete global resource view at the scheduling layer. The scheduler does not need to interact directly with multiple heterogeneous backend systems; it only needs to make decisions based on the standard interface of virtual nodes, greatly reducing the complexity of cross-center resource management and scheduling. The virtual node controller ensures near real-time availability of view information through a periodic synchronization mechanism, enabling scheduling decisions to be based on real and accurate resource capacity, thereby significantly improving scheduling success rate and resource utilization. Simultaneously, an automatic isolation mechanism for unreachable centers achieves center-level fault isolation and lossless canary release capabilities, ensuring high availability of the entire system.

[0122] A multi-level constraint system based on a binary combination of "intelligent computing center identifier + computing pool identifier" is proposed, including hard constraints, soft preferences, and extended constraints. This provides users with unprecedented and refined cross-center scheduling strategy expression capabilities. Users can flexibly specify resource preferences according to business needs (such as data localization, cost, and performance), from "must be satisfied" to "preferred selection," achieving precise control of the strategy. The scheduling system can intelligently parse and quantify these preferences, performing precise matching and weight calculation during the scoring phase. More importantly, when the optimal preference cannot be satisfied, the matching algorithm can achieve automated intelligent fallback, such as falling back from the designated pool of the preferred center to other pools within the same center or alternative centers. While ensuring business continuity, it achieves intelligent load balancing and disaster recovery switching across centers, significantly improving the system's flexibility and reliability.

[0123] Addressing the core challenge of fragmented descriptions of heterogeneous accelerators, this invention defines a unified processor type enumeration covering mainstream types and constructs a structured capability description framework based on a "protocol buffer message format," managed through a dynamic registry. This establishes a unified "technical language" and "capability model" for the complex and diverse heterogeneous computing power. The scheduler does not need to understand and adapt to various vendor-specific interfaces; it only needs to match and verify according to the standardized ProcessorType enumeration and capability description framework. This liberates the scheduling logic from the cumbersome work of processing device differences, making it simple and universal. This not only significantly reduces code complexity and maintenance costs, but also allows for "out-of-the-box" scalability by simply adding entries to the enumeration and registry when new processors need support, without modifying the core scheduling logic. This effectively avoids vendor lock-in and improves the long-term portability of the system.

[0124] By clearly deconstructing the scheduling process into three independent stages—filtering, scoring, and pre-binding—and defining standardized plugin interfaces for each stage, a complete decoupling of the core scheduling framework from specific scheduling strategies is achieved. Each scheduling strategy (such as filtering by center, scoring by cost, and quota verification) is encapsulated as an independent plugin, assembled and sorted through configuration files. This design makes adding, modifying, or replacing scheduling strategies extremely simple and secure, without touching the scheduler's core code, truly achieving "hot-swappable" strategies. This not only greatly improves the system's scalability, allowing for rapid response to new business needs, but also significantly enhances the system's maintainability and testability, as each plugin can be independently developed, tested, and deployed, reducing the overall system complexity and upgrade risks.

[0125] Addressing the unique characteristics of interactive jobs such as Notebook jobs, this invention designs a rapid response mechanism that integrates automatic identification, capacity gating, and real-time verification during the admission control phase. This fundamentally changes the resource supply experience for interactive jobs, achieving a paradigm shift from "blind waiting" to "knowledge before submission." Interactive jobs are intelligently identified through multi-dimensional features, and multiple verifications of resource availability, quotas, and concurrency limits are completed within milliseconds. When resources are insufficient, the system immediately provides users with clear feedback (such as "available immediately," "insufficient quantity," and estimated waiting time) and intelligent alternative suggestions. This avoids the inefficient scenario of users submitting jobs and then waiting in long queues only to ultimately fail, greatly improving the efficiency and user experience for developers. Simultaneously, by intercepting invalid scheduling attempts at the submission stage, it also reduces the invalid load on the scheduling system, improving overall resource utilization efficiency.

[0126] Example 2, as Figure 2As shown in the figure, this embodiment of the invention also provides a unified scheduling system for heterogeneous computing power across multiple intelligent computing centers. This system is deployed in a unified Kubernetes management cluster, serving as the brain and central hub for resource scheduling across multiple intelligent computing centers. It interacts with multiple downstream heterogeneous intelligent computing centers through standardized interfaces, forming a logically centralized but physically distributed scheduling governance plane. The functions, internal structure, and interaction relationships of each core module are described in detail below.

[0127] The Virtual Node Abstraction Module (VNode Module) is the cornerstone of the entire system, responsible for transforming physical, distributed computing resources into logically unified resource objects recognizable by the standard Kubernetes scheduler. Its core function is to create and maintain a corresponding virtual node in the management cluster for each unique resource combination based on a four-tuple mapping model (Intelligent Computing Center Identifier, Computing Pool Identifier, Processor Type, Specifications). The VNode Module includes a mapping engine and a resource synchronization controller. The mapping engine executes the mapping logic, generates virtual node names conforming to naming conventions, and configures a set of standardized metadata tags for each virtual node (such as grampus.openi.org.cn / aiCenterId, grampus.openi.org.cn / poolId, etc.). These tags are crucial for all subsequent scheduling decisions. The resource synchronization controller, as an active component of the VNode Module, runs as an independent controller in the cluster. It periodically queries the resource management systems of each intelligent computing center through a REST API (Representation State Transition Application Programming Interface). Its core responsibilities include data synchronization, lifecycle management, and fault handling. Specifically, the resource synchronization controller obtains the real-time capacity and status information of each computing pool and updates the resource fields of the corresponding virtual nodes accordingly; it dynamically manages the creation, updating and deletion of virtual nodes based on the synchronization results; and when it detects that a certain intelligent computing center network is unreachable, it automatically adds a NoSchedule taint to all its virtual nodes to prevent new jobs from being scheduled to the faulty area, thereby achieving fault isolation and high system availability.

[0128] The Unified Specification Registry provides standardized data support for a unified abstraction. As a centralized storage and service component, it persists and manages standardized descriptions of all heterogeneous processors. Its data model is based on the ProcessorCapability message structure defined by Protocol Buffers, with each record corresponding to a specific processor specification (e.g., "NVIDIA A100-80GB"), indexed by a key consisting of type, model, and memory. The center provides complete CRUD APIs, allowing operations personnel to input detailed information about new processor models and providing real-time query services to the scheduling engine. This enables the scheduler to efficiently determine the compatibility between job requirements and the actual capabilities of nodes based on a unified, structured capability description.

[0129] The scheduling request interface and constraint parsing module form the front end for system-user interaction, responsible for receiving and understanding user scheduling intentions. The scheduling request interface provides a high-level API compatible with Kubernetes API extensions or standalone interfaces to receive job requests submitted by users in formats such as YAML or JSON. Closely connected to this interface is the constraint parsing module, which, upon receiving a job request, performs deep parsing of annotations or specific fields in the request, extracting user-defined, multi-level resource preference constraints based on a binary combination of intelligent computing center identifiers and computing pool identifiers. This module transforms unstructured user business requirements into structured constraint objects that can be precisely processed within the system, clearly defining hard constraints, soft preferences, and extended constraints, providing crucial input for the subsequent accurate decision-making of the scheduling engine.

[0130] The multi-stage pluggable scheduling engine is the system's "decision-making core," employing a highly scalable pluggable architecture to execute complex scheduling logic. Essentially, it's a strengthened and deeply extended Kubernetes scheduler, strictly adhering to the Kubernetes Scheduler Framework specification. The engine's built-in pluggable manager dynamically loads, instantiates, and manages three main types of pluggable sets from configuration files: filtering pluggable sets, scoring pluggable sets, and pre-bound pluggable sets, precisely controlling the execution order, priority, and parallel / serial strategies of each pluggable. Its scheduling decisions follow a rigorous workflow: First, in the filtering phase, the engine sequentially calls filtering pluggables, such as the central pool filter pluggable, which uses the constraint parsing module's output for hard constraint filtering; and the processor type filter pluggable, which queries the unified specification registry center for device compatibility filtering, quickly eliminating candidate nodes that do not meet basic conditions. Next, in the scoring phase, for the filtered candidate virtual nodes, multiple scoring pluggables run in parallel, such as the central pool scoring pluggable, which calculates preference scores based on soft preferences; and the resource utilization scoring pluggable, which calculates scores for other dimensions such as resource utilization. Finally, the pluggable manager normalizes each score according to preset weights, selecting the optimal node with the highest overall score. Finally, in the pre-binding phase, pre-binding plugins are executed sequentially before final binding. For example, the capacity reservation plugin calls the underlying API of the intelligent computing center to reserve resources. This phase is designed to be transactional; if any plugin fails, it will trigger a rollback of all operations, ensuring the consistency of resource status.

[0131] The job pre-validation module performs rapid interception and precise validation of jobs before they enter the scheduling queue. It is typically deployed as a Kubernetes Validating Admission Webhook within the API server's admission chain. This module comprises three coordinating units: a job identification unit, a gating execution unit, and a feedback unit. When a job request passes through the API server, the job identification unit is triggered. It employs a multi-level hybrid strategy to intelligently identify interactive jobs like Notebooks, sequentially checking Pod explicit tags, scanning container image name keywords, analyzing exposed ports and resource request patterns to ensure accuracy. Once identification is successful, the gating execution unit immediately and synchronously executes a series of rigorous capacity gating checks, including specification whitelist verification, user-level / central-level concurrency limit checks, availability queries based on real-time resource views, and user quota verification. Finally, the feedback unit generates a clear response within milliseconds based on the gating results, categorized as "immediately available" (admission approved), "insufficient quantity," "temporarily unavailable," and "permanently unavailable." For the latter three rejection scenarios, the feedback information will precisely include the specific reason, the estimated waiting time for resource release, or the intelligently recommended alternative specifications, thereby directly realizing the functions of fast failure and intelligent suggestions, greatly optimizing the user experience.

[0132] The system's collaborative workflow is as follows: After a user submits a job, the job pre-verification module first performs rapid gating; if it is an interactive job, it provides immediate and explicit feedback. After the job passes the admission test, the constraint parsing module analyzes its resource preferences. Subsequently, the multi-stage pluggable scheduling engine is activated. During the scheduling process, it coordinates with the device capability standards provided by the unified specification registration center and the global real-time resource view maintained by the virtual node abstraction module to execute multi-stage, pluggable intelligent scheduling decisions. Throughout the system's operation, the resource synchronization controller continuously and proactively synchronizes the resource status of each intelligent computing center in the background to ensure the freshness and accuracy of the global view. Finally, the job is bound to the optimal virtual node, and the system's underlying driver maps it to the actual physical intelligent computing center for execution. The overall design of this system fully embodies the architectural principles of separation of concerns and high cohesion and low coupling. Each module has clearly defined responsibilities and boundaries, and they collaborate efficiently through well-defined interfaces, collectively forming a flexible and intelligent unified scheduling platform across intelligent computing centers.

[0133] The virtual node abstraction module standardizes geographically dispersed physical computing power pools with varying management interfaces into unified virtual nodes within a Kubernetes cluster. This eliminates the need for the scheduler to adapt to numerous heterogeneous backend systems, requiring only interaction with a single standard virtual node interface, significantly simplifying the complexity of cross-center resource operations. Secondly, this module achieves near real-time consistency of resource status. The resource synchronization controller, through proactive fetching and differential synchronization mechanisms, ensures the accuracy and timeliness of the global resource view, ensuring that every scheduling decision is based on reliable capacity information, thereby significantly improving scheduling success rate and overall resource utilization. Furthermore, the module's built-in fault detection and taint marking mechanisms provide the system with center-level automatic fault isolation and lossless operation and maintenance capabilities. When a single intelligent computing center fails or undergoes maintenance, its impact is strictly limited to that center, preventing global scheduling disruption and effectively guaranteeing high availability and business continuity for cross-center scheduling services.

[0134] The unified specification registry center achieves the abstraction and unification of technical details for heterogeneous accelerators by defining standardized processor type enumerations and a structured capability description framework. The scheduling logic is completely freed from identifying and handling the differences between devices from different vendors, instead making decisions based on a unified "processor type" and standardized capability attributes (such as computing power and memory bandwidth). This makes the core scheduling code concise and universal, greatly reducing code maintenance costs. Furthermore, this system architecture provides unprecedented scalability. When new processors (such as next-generation NPUs or dedicated AI chips) need to be integrated, operations personnel only need to add the corresponding specification description entry to the registry center, without modifying the core code or processes of the scheduling engine.

[0135] The collaborative operation of the scheduling request interface, constraint parsing module, and multi-stage pluggable scheduling engine provides industry-leading, business-aware, and granular scheduling strategy expression capabilities on the user side through a multi-level constraint expression mechanism (hard constraints, soft preferences, and extended constraints). This elevates scheduling from simple resource allocation to intelligent decision-making that meets business objectives. On the system side, the pluggable architecture based on standard interfaces achieves complete decoupling of strategy logic from the scheduling framework. Each filtering, scoring, or binding strategy is encapsulated as an independent pluggable and assembled via configuration files. This makes adding, optimizing, or replacing scheduling strategies safe, simple, and efficient. It significantly improves the system's scalability, maintainability, and ability to quickly respond to changes in business requirements.

[0136] The pre-job verification module delivers direct and significant user value and operational benefits by bringing capacity gating and specification verification to the moment of job submission. It achieves a paradigm shift in the interactive job experience from "blind waiting" to "instant awareness." Through rapid identification and synchronous verification, the system can provide clear feedback such as "immediately available," "estimated waiting time," or "alternative specification suggestions" the instant a user submits a Notebook-type job. This completely eliminates the pain of prolonged, ineffective waiting due to insufficient resources, greatly improving the productivity and satisfaction of R&D personnel. From a system operation perspective, this mechanism intercepts a large number of scheduling attempts destined to fail due to insufficient resources at the entry point. This prevents invalid jobs from consuming scheduling queue resources, reduces unnecessary load on the scheduler and backend systems, thereby improving the throughput and processing efficiency of the entire scheduling system, achieving a dual optimization of user experience and system efficiency.

[0137] Through the modular and standardized architecture design described above, not only were specific technical challenges such as inconsistent resource abstraction, imprecise strategy expression, rigid system architecture, and poor user experience overcome one by one, but more importantly, a flexible, intelligent, reliable, and sustainably evolving cross-computing center scheduling platform was built. This makes it possible to efficiently and precisely manage and schedule large-scale, widely distributed heterogeneous computing resources, providing key supporting technologies for building national or enterprise-level integrated computing networks.

[0138] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for unified scheduling of heterogeneous computing power across multiple intelligent computing centers, characterized in that, Includes the following steps: S1: Virtual Node Abstraction; The physical heterogeneous computing power resources of multiple intelligent computing centers are mapped to virtual nodes in the Kubernetes cluster according to the four-tuple mapping model. Each virtual node represents a resource combination that is uniquely determined by the intelligent computing center identifier, computing power pool identifier, processor type and specification model, and carries metadata through tags. The real-time resource status of each center is synchronized by the virtual node controller. S2: Preference Constraint Expression and Matching: Receives job requests submitted by users, the requests containing multi-level resource preference constraints defined based on a binary combination of intelligent computing center identifiers and computing pool identifiers; the multi-level resource preference constraints include hard constraints, soft preferences, and extended constraints; the hard constraints specify the intelligent computing center and / or computing pool to which the job must be scheduled; the soft preferences specify one or more weighted combinations of intelligent computing center and computing pool preferences; the extended constraints specify at least one of the following constraints: minimum number of available nodes, maximum network latency, or cost budget. S3: Unified processor type management: Based on a predefined unified processor type enumeration and processor capability description framework, it performs compatibility verification between the resource requirements of job requests and the processor capabilities represented by virtual nodes. S4: Multi-stage plug-in scheduling; a multi-stage plug-in scheduling framework is used to make scheduling decisions for the job. The multi-stage plug-in scheduling framework includes at least a filtering stage, a scoring stage, and a pre-binding stage executed sequentially. Each stage performs corresponding filtering, scoring, and pre-binding operations through plug-ins that conform to a unified interface specification to determine the target virtual node. In the scoring phase, the rules for scoring virtual nodes based on the soft preferences include: if both the intelligent computing center identifier and the computing power pool identifier of a virtual node completely match the identifier specified by a certain soft preference item, then the node receives the full weight score of that preference item; if only the intelligent computing center identifier of a virtual node matches a certain soft preference item, then the node receives a first proportion weight score of that preference item, which is lower than the full weight; if only the computing power pool identifier of a virtual node matches a certain soft preference item, then the node receives a second proportion weight score of that preference item, which is lower than the first proportion. S5: Interactive Job Gating: In response to the recognition that a job is interactive, capacity gating and specification availability verification are performed during the job submission admission phase, and feedback is provided in real time based on the verification results.

2. The heterogeneous computing power unified scheduling method for multiple intelligent computing centers according to claim 1, characterized in that: The unified processor type enumeration includes at least graphics processors, neural network processors, deep computing units, machine learning units, general-purpose computing units, and central processing units; the processor capability description framework is defined through a protocol buffer message format, and the message structure contains fields for describing basic attributes, computing power, memory specifications, interconnect topology, and software stack compatibility; Step S3, unified processor type management, also includes: maintaining a processor specification registry, using a key-value mapping structure to store the capability description data of each processor, and registering detailed specifications and capability information of heterogeneous processors from different manufacturers and models using a unified structured capability description framework.

3. The heterogeneous computing power unified scheduling method for multiple intelligent computing centers according to claim 1, characterized in that: In the multi-stage pluggable scheduling framework: The filtering operations performed in the filtering phase include at least: central pool constraint filtering based on hard constraints, compatibility filtering based on the unified processor type, filtering based on resource capacity, and filtering based on node taints and job tolerance. The scoring operations performed in the scoring phase include at least: central pool preference scoring based on the soft preference level, scoring based on resource utilization, scoring based on topology distribution, and scoring based on resource cost. The operations performed during the pre-binding phase include at least: reserving resources in the target intelligent computing center, verifying user or project quotas, and writing scheduling binding metadata, and the operations support rollback in case of failure.

4. The heterogeneous computing power unified scheduling method for multiple intelligent computing centers according to claim 1, characterized in that: All plugins in the filtering, scoring, and pre-binding stages implement predefined programming language interfaces. The execution order and priority of plugins are orchestrated and controlled through configuration files. Plugins in the filtering phase are executed serially, while plugins in the scoring phase support parallel execution.

5. The heterogeneous computing power unified scheduling method for multiple intelligent computing centers according to claim 1, characterized in that: Step S5: Interactive job gating specifically includes: S501: Automatically identifies Notebook-type interactive jobs by checking one or more of the following: preset job type tags, container image name keywords, exposed port numbers, and resource request patterns. S502: For the identified interactive jobs, perform capacity gating on the verification dimensions, which include: whitelist of specifications and models, central concurrent quantity limit, real-time resource availability and user quota; S503: Based on the verification results, return four types of results: immediately available, insufficient quantity, temporarily unavailable, or permanently unavailable, and provide alternative specifications or waiting time suggestions when resources are insufficient.

6. A unified scheduling system for heterogeneous computing power across multiple intelligent computing centers, characterized in that, include: The virtual node abstraction module is used to abstract the physical resources of each intelligent computing center into virtual nodes in the Kubernetes cluster according to the four-tuple mapping model, and to configure metadata tags for the virtual nodes, including intelligent computing center identifier, computing power pool identifier, processor type and specification model. A resource synchronization controller is connected to the virtual node abstraction module and multiple intelligent computing centers, and is used to synchronize the real-time resource status of each center and update the resource information of the corresponding virtual nodes. The scheduling request interface is used to receive job requests submitted by users. The constraint parsing module, connected to the scheduling request interface, is used to parse multi-level resource preference constraints defined by a binary combination of intelligent computing center identifier and computing pool identifier from the job request. The multi-level resource preference constraints include hard constraints, soft preferences, and extended constraints. The hard constraints specify the intelligent computing center and / or computing pool to which the job must be scheduled. The soft preferences specify one or more weighted combinations of intelligent computing center and computing pool preferences. The extended constraints specify at least one of the following constraints: minimum number of available nodes, maximum network latency, or cost budget. A unified specification registry is used to store and manage heterogeneous processor specification information based on a unified processor type enumeration and processor capability description framework. A multi-stage pluggable scheduling engine, connected to the constraint parsing module, the unified specification registration center, and the virtual node abstraction module, loads and sequentially executes the filtering pluggable set, the scoring pluggable set, and the pre-binding pluggable set to determine the target node from the virtual nodes based on the multi-level resource preference constraints, processor compatibility, and resource capacity. The scoring pluggable set scores the virtual nodes based on the soft preferences according to the following rules: if both the intelligent computing center identifier and the computing power pool identifier of the virtual node completely match the identifier specified by a certain soft preference item, then the node receives the full weight score for that preference item; if only the intelligent computing center identifier of the virtual node matches a certain soft preference item, then the node receives a first proportion weight score for that preference item, lower than the full weight; if only the computing power pool identifier of the virtual node matches a certain soft preference item, then the node receives a second proportion weight score for that preference item, lower than the first proportion. The job pre-verification module is connected to the scheduling request interface and is used to identify interactive jobs and perform capacity gating and specification availability verification on them before scheduling.

7. The system according to claim 6, characterized in that: The resource synchronization controller is specifically used to: communicate with the resource management systems of each intelligent computing center through a state transition application programming interface to periodically synchronize resource information to update the capacity and availability of virtual nodes; and to add a scheduling prohibition taint to virtual nodes from intelligent computing centers that are not accessible from the network.

8. The system according to claim 6, characterized in that: The pre-job verification module specifically includes: The job identification unit is used to identify Notebook-type interactive jobs by checking one or more of the following methods: job type labels, container image name keywords, exposed port numbers, and resource request patterns. The gating execution unit is used to perform capacity gating on the identified jobs, including specification and model whitelist verification, central concurrent number limit check, real-time resource availability query and user quota verification. The feedback unit is used to generate and return feedback information containing resource availability status and alternative suggestions based on the gating results.

Images