An AI computing power resource management system and method supporting cross-domain fine-grained scheduling

Abstract

This invention discloses an AI computing resource management system and method supporting fine-grained cross-domain scheduling, relating to the field of artificial intelligence technology. It addresses the background issues of propagation latency and jitter in the process of resource status collection, aggregation, and propagation to usage by the scheduler. This leads to inconsistencies across fields in the global view and time asynchrony across nodes, resulting in a sharp increase in communication and performance crashes. It also lacks engineering guarantees for atomic start / stop and parameter consistency of cross-node container groups. The proposed solution includes constructing a global topology resource view: a multi-regional coordinator cluster based on a distributed consensus protocol synchronizes and maintains a global resource view in real time. This invention reduces erroneous scheduling caused by cross-domain status lag, avoids misjudgments caused by cross-field inconsistencies, improves the robustness of cross-domain scheduling, reduces the risk of a sharp increase in communication, and avoids catastrophic communication degradation in cross-domain multi-GPU tasks.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an AI computing resource management system and method that supports cross-domain fine-grained scheduling. Background Technology

[0002] Artificial intelligence, abbreviated as AI, is a new technological science that studies and develops theories, methods, technologies, and application systems to simulate, extend, and expand human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems.

[0003] In AI computing power service platforms, scheduling granularity has gradually evolved from whole machine or whole card to GPU card level, and computing resources present a pattern of multiple regions, multiple operating domains, and multiple clusters coexisting. In cross-domain wide area network environments, the resource status from collection, aggregation, and propagation to being used by the scheduler has propagation latency and jitter, which leads to "cross-field inconsistency" and "cross-node time asynchrony" problems between global view fields, thus causing erroneous scheduling, such as making allocation decisions for GPU cards that have been occupied or degraded.

[0004] On the other hand, multi-GPU training or inference is highly sensitive to GPU interconnect topology and cross-node network quality. Simply splicing multiple GPU cards across domains can cause communication paths to fall into the worst-case links, leading to a sharp increase in communication load and causing performance crashes. Existing schedulers typically only consider resource quantity, load, or price, lacking a focus on "visualization." Figure 1 The mechanism for incorporating "consistency or freshness risk" into decision-making also lacks engineering guarantees for cross-node container group atomic start-up and shutdown and parameter consistency. Summary of the Invention

[0005] This invention provides an AI computing power resource management system and method that supports cross-domain fine-grained scheduling. It solves the problem in existing AI computing power service platforms where the scheduling of resource status from collection, aggregation, and propagation to usage by the scheduler suffers from propagation latency and jitter, leading to "cross-field inconsistency" and "cross-node time asynchrony" between global view fields. This results in a sharp increase in communication load and performance crashes. Furthermore, it lacks a mechanism to control the "view" field. Figure 1 There is a lack of mechanisms to incorporate "consistency or freshness risk" into decision-making, as well as a lack of engineering guarantees for cross-node container group atomic start-up and shutdown and parameter consistency.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A method for managing AI computing resources that supports fine-grained scheduling across domains includes the following steps: Step S1: Construct a global topology resource view: A global resource view is synchronized and maintained in real time through a multi-region coordinator cluster based on a distributed consensus protocol. This view not only records the basic resource status of each GPU card on each computing node, but also records the hardware interconnection topology between GPUs within the node and the network location attributes of the region where the node is located. At least some fields of the global resource view have a view version number, a commit timestamp, and a freshness flag, which are used to indicate the most recent consistent commit time and the most recent valid update time of the field. Step S2: Receive user computing power request, the request including specifying the required number and type of GPUs and the desired interconnect topology preference, and optionally carrying task type label and communication mode label; Step S3: Perform pre-screening of topology and state constraints: Based on the global topology resource view, perform the first layer of screening to exclude GPU cards that do not meet the constraints of resource quantity, type, basic health and load, forming an original candidate GPU resource pool. Screening for high-frequency dynamic states also meets the freshness threshold; when the freshness is not met, the corresponding GPU card or node will be downgraded, frozen or removed. Step S4: Construct and score cross-domain virtual computing units: For the original candidate GPU resource pool, perform a cross-node virtual combination exploration process, selecting a group of GPU cards from nodes that may be located in different regions to form a virtual computing unit; for each virtual computing unit candidate scheme, calculate a comprehensive score based on a unified cost model; the unified cost model simultaneously quantifies and evaluates the computational performance cost, cross-domain resource integration cost, and economic cost of the scheme, and includes a view staleness risk penalty item; the view staleness risk penalty item is related to the freshness flag and submission timestamp of the fields referenced by the candidate scheme; and perform version number consistency verification before scoring, if the verification fails, the candidate scheme is rolled back or eliminated; Step S5: Decision and Instantiation: Select the candidate virtual computing unit with the best score, and collaboratively create a group of secure container instances with GPU card-level binding on each of the physical computing nodes involved. These instances are configured to belong to the same logical task unit through a virtual network, and the atomic startup of the container group and the consistent configuration of communication parameters are completed through a two-phase consistent startup barrier during the instantiation phase. The two-phase consistent startup barrier includes at least a preparation-to-commit phase and a commit-to-start phase: resource locking and parameter distribution verification are completed in the preparation-to-commit phase, and simultaneous startup is completed in the commit-to-start phase. If any node fails in the preparation-to-commit phase, a global rollback is performed to release the locked resources.

[0007] Preferably, the multi-regional coordinator cluster employs incremental synchronization and lease mechanisms to maintain the view. Figure 1Consistency: For static or semi-static attributes, lease-based strong consistency synchronization is adopted, and the view version number is generated by consensus submission; for the dynamic state of the GPU card, high-frequency incremental synchronization is adopted, and submission timestamps and freshness markers are maintained; scheduling decisions use version number verification and freshness threshold verification simultaneously to achieve a balance between strong consistency of key attributes and eventual consistency of high-frequency states. The freshness marker includes the most recent update time, sampling source, and credibility level; the freshness threshold is adaptively set according to the task type label and communication mode label, and satisfies that the freshness threshold for AllReduce type training tasks is less than the freshness threshold for single-card inference tasks.

[0008] Preferably, the comprehensive score P of the unified cost model is weighted and synthesized from the following components: performance cost component, integration cost component, economic cost component, and view staleness risk penalty component; wherein the view staleness risk penalty component is a monotonically increasing function of "current time minus submission timestamp" with respect to the dynamic fields referenced by the candidate solution, and different risk weights are set for different fields; the economic cost component includes at least the GPU real-time unit price, cross-domain data transmission bandwidth cost, and cross-domain surcharge; wherein the cross-domain surcharge is dynamically adjusted according to real-time network latency and available bandwidth, and may optionally include a service level (SLA) penalty / refund item to constrain cross-domain network jitter risk.

[0009] Preferably, the cross-node virtual combination exploration process adopts a hierarchical pruning strategy: the first step is intra-node aggregation, which prioritizes finding GPU card groups that meet or partially meet the topology requirements within a single physical node to form local sub-units; the second step is cross-domain complementarity, which uses the local sub-unit as the core to search for complementary GPU cards from other regional nodes and evaluates the cross-domain virtual computing units formed after complementarity; the exploration process is limited by the cross-domain latency budget, which is dynamically determined according to the task type label and communication mode label.

[0010] Preferably, the performance cost component quantifies the degree of matching between the GPU interconnect topology within the virtual computing unit and the request preference through a topology matching function, and assigns different scores according to interconnect technology levels such as NVLink, PCIe, and RDMA; the topology matching function includes at least node resource sufficiency, real-time load, network quality, node health, and cross-domain cost, and the weight of each scoring dimension is dynamically adjusted according to the task type label. The node health is calculated by weighting at least three indicators among GPU error count, temperature over-limit duration, ECC error rate, power stability, and network packet loss rate, and when the health is lower than the health threshold, a freeze window is executed on the node, and the node does not participate in the candidate pool construction within the freeze window.

[0011] Preferably, the downgrading, freezing, or removal process includes increasing the risk penalty on the node score when the freshness tag exceeds a first threshold, but the node can still participate; freezing the node and putting it into a read-only observation state when it exceeds a second threshold; and removing the node and triggering view resynchronization when it exceeds a third threshold.

[0012] Preferably, the two-phase consistent startup barrier uses a distributed transaction ID associated with the container group startup request. During the preparation to commit phase, resource locking, image pull completion confirmation, GPU binding confirmation, and communication parameter verification are performed on each node. During the commit startup phase, a commit command is broadcast using the transaction ID as the key, and a commit receipt is recorded. If any node fails to return a receipt within the timeout window during the commit startup phase, a consistent termination process is triggered, and the system rolls back to a preset safe state. The consistent configuration of the communication parameters includes at least the topology-aware parameters of the distributed communication library, the hierarchical AllReduce parameters, the cross-domain bandwidth limit parameters, and the network congestion control parameters. These parameters are uniformly distributed by the coordinator during the preparation to commit phase and confirmed by the receipts from each node.

[0013] Preferably, the creation of the secure container instance includes GPU card-level binding and multi-layer isolation, and the security isolation mechanism includes GPU virtualization device-level isolation, container namespace isolation, virtual network isolation, and storage encryption and access control.

[0014] An AI computing resource management system supporting fine-grained cross-domain scheduling includes a user portal module, a cross-regional coordinator, a resource scheduling engine, an instance management module, a security isolation control module, an image and storage management module, a billing management module, and a multi-protocol access gateway. The cross-regional coordinator is a cluster of coordinator nodes deployed in different geographical regions, operating based on a distributed consensus protocol and maintaining a global resource view with version numbers, commit timestamps, and freshness markers. The resource scheduling engine outputs GPU-level scheduling results based on a unified cost model. The instance management module collaboratively creates container instances on each node and completes atomic startup and failure rollback through a two-phase consistent startup barrier.

[0015] The beneficial effects of this invention are as follows: 1. By using version number, commit timestamp, freshness flag and freshness threshold to participate in decision-making, the erroneous scheduling caused by cross-domain status lag is reduced, and the rollback due to version consistency verification failure avoids misjudgment caused by cross-field inconsistency. 2. By using a unified cost model, performance, fragment integration complexity, and economic cost are incorporated into the same scoring framework, and a view obsolescence risk penalty is introduced to improve the robustness of cross-domain scheduling. 3. By using a two-stage consistent startup barrier, atomic start-stop and communication parameter consistency of logical task units are achieved, reducing the startup failure rate and performance jitter of multi-GPU cross-domain splicing, and achieving consistent rollback in failure scenarios.

[0016] 4. By using topology matching degree and worst link constraints, the risk of a sharp increase in communication ratio is reduced, and catastrophic communication degradation in cross-domain multi-GPU tasks is avoided. Attached Figure Description

[0017] Figure 1 This is a system architecture diagram of an AI computing resource management system and method that supports cross-domain fine-grained scheduling, as proposed in this invention.

[0018] Figure 2 This is a multi-layer security isolation architecture diagram of an AI computing resource management system and method that supports cross-domain fine-grained scheduling, as proposed in this invention. Detailed Implementation

[0019] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0020] Example 1, referring to Figure 1-2 A method for managing AI computing resources that supports fine-grained scheduling across domains includes the following steps: Step S1: Construct a global topology resource view: A global resource view is synchronized and maintained in real time through a multi-regional coordinator cluster based on a distributed consensus protocol. This view not only records the basic resource status of each GPU card on each computing node, but also records the hardware interconnection topology between GPUs within the node and the network location attributes of the node's region. At least some fields of the global resource view contain a view version number, commit timestamp, and freshness flag, used to indicate the most recent consistent commit time and the most recent valid update time of the field. The multi-regional coordinator cluster uses incremental synchronization and lease mechanisms to maintain the view. Figure 1 Consistency: For static or semi-static attributes, lease-based strong consistency synchronization is adopted, and the view version number is generated by consensus submission; for the dynamic state of GPU cards, high-frequency incremental synchronization is adopted, and submission timestamps and freshness flags are maintained; scheduling decisions use version number verification and freshness threshold verification simultaneously to achieve a balance between strong consistency of key attributes and eventual consistency of high-frequency states. The freshness flag includes the most recent update time, sampling source, and credibility level; the freshness threshold is adaptively set according to the task type label and communication mode label, and satisfies that the freshness threshold for AllReduce-type training tasks is less than the freshness threshold for single-card inference tasks. Step S2: Receive and parse topology-aware computing power requests: Receive user computing power requests, which include specifying the number and type of GPUs required and the desired interconnect topology preferences, and optionally carry task type tags and communication mode tags; Step S3: Perform pre-screening of topology and state constraints: Based on the global topology resource view, perform the first layer of screening to exclude GPU cards that do not meet the constraints of resource quantity, type, basic health, and load, forming an original candidate GPU resource pool. Screening for high-frequency dynamic states also meets the freshness threshold. When the freshness is not met, the corresponding GPU card or node will be downgraded, frozen, or removed. The downgraded, frozen, or removed process includes increasing the risk penalty for the node score when the freshness mark exceeds the first threshold, but it can still participate; freezing the node and putting it into a read-only observation state when it exceeds the second threshold; and removing the node and triggering view resynchronization when it exceeds the third threshold. Step S4: Construct and score cross-domain virtual computing units: For the original candidate GPU resource pool, perform a cross-node virtual combination exploration process, selecting a group of GPU cards from nodes that may be located in different regions to form a virtual computing unit; for each virtual computing unit candidate scheme, calculate a comprehensive score based on a unified cost model; the unified cost model simultaneously quantifies and evaluates the computational performance cost, cross-domain resource integration cost, and economic cost of the scheme, and includes a view staleness risk penalty item; the view staleness risk penalty item is related to the freshness flag and submission timestamp of the fields referenced by the candidate scheme; and execute a version control before scoring. If the consistency check fails, the candidate solution will be rolled back or eliminated. The comprehensive score P of the unified cost model is weighted and synthesized from the following components: performance cost component, integration cost component, economic cost component, and view staleness risk penalty component. The view staleness risk penalty component is a monotonically increasing function of "current time minus submission timestamp" with respect to the dynamic fields referenced by the candidate solution, and different risk weights are set for different fields. The economic cost component includes at least the GPU real-time unit price, cross-domain data transmission bandwidth cost, and cross-domain surcharge. The cross-domain surcharge is dynamically adjusted according to real-time network latency and available bandwidth, and may optionally include... Service Level Agreement (SLA) penalties / refunds are used to constrain cross-domain network jitter risks. The cross-node virtual combination exploration process adopts a hierarchical pruning strategy: the first step is intra-node aggregation, prioritizing the search for GPU decks that meet or partially meet topology requirements within a single physical node to form local sub-units; the second step is cross-domain complementarity, using the local sub-unit as the core to search for complementary GPU cards from other regional nodes and evaluate the cross-domain virtual computing units formed after complementarity; the exploration process is constrained by the cross-domain latency budget, which is dynamically determined based on task type labels and communication mode labels, and the performance cost component is quantized by the topology matching degree function. The degree of matching between the GPU interconnect topology within the computing unit and the request preference is evaluated, and different scores are assigned based on interconnect technology levels such as NVLink, PCIe, and RDMA. The topology matching function includes at least node resource sufficiency, real-time load, network quality, node health, and cross-domain cost. The weights of each scoring dimension are dynamically adjusted according to the task type label. The node health is calculated by weighting at least three indicators among GPU error count, temperature over-limit duration, ECC error rate, power stability, and network packet loss rate. When the health is lower than the health threshold, a freeze window is executed on the node, and the node does not participate in the candidate pool construction during the freeze window. Step S5: Decision and Instantiation: Select the candidate virtual computing unit with the best score, and collaboratively create a group of secure container instances with GPU card-level binding on each of the physical computing nodes involved. These instances are configured to belong to the same logical task unit through a virtual network, and the atomic startup of the container group and the consistent configuration of communication parameters are completed through a two-phase consistent startup barrier during the instantiation phase. The two-phase consistent startup barrier includes at least a preparation-to-commit phase and a commit-to-start phase: resource locking and parameter distribution verification are completed in the preparation-to-commit phase, and simultaneous startup is completed in the commit-to-start phase; if any node fails in the preparation-to-commit phase, a global rollback is performed to release the locked resources. The two-phase consistent startup barrier uses a distributed transaction ID to associate the container group startup request, and in the preparation-to-commit phase, it performs a global rollback to release the locked resources for each node. The process involves resource locking, image pull completion confirmation, GPU binding confirmation, and communication parameter verification. During the commit startup phase, a commit command is broadcast using the transaction ID as the key, and a commit receipt is recorded. If any node fails to return a receipt within the timeout window during the commit startup phase, a consistency termination process is triggered, and the system rolls back to a preset safe state. The consistent configuration of the communication parameters includes at least the topology-aware parameters of the distributed communication library, the hierarchical AllReduce parameters, the cross-domain bandwidth limit parameters, and the network congestion control parameters. These parameters are uniformly distributed by the coordinator during the commit preparation phase and confirmed by each node's receipt. The creation of the secure container instance includes GPU card-level binding and multi-layer isolation. The security isolation mechanism includes GPU virtualization device-level isolation, container namespace isolation, virtual network isolation, and storage encryption and access control.

[0021] Reference Figure 1 An AI computing resource management system supporting fine-grained cross-domain scheduling is disclosed, comprising a user portal module, a cross-regional coordinator, a resource scheduling engine, an instance management module, a security isolation control module, an image and storage management module, a billing management module, and a multi-protocol access gateway. The cross-regional coordinator is a cluster of coordinator nodes deployed in different geographical regions, operating based on a distributed consensus protocol and maintaining a global resource view with version numbers, commit timestamps, and freshness markers. The resource scheduling engine outputs GPU-level scheduling results based on a unified cost model. The instance management module collaboratively creates container instances on each node and completes atomic startup and failure rollback through a two-phase consistent startup barrier.

[0022] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0023] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0024] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for managing AI computing resources that supports fine-grained cross-domain scheduling, characterized in that, Includes the following steps: Step S1: Construct a global topology resource view: A global resource view is synchronized and maintained in real time through a multi-regional coordinator cluster based on a distributed consensus protocol; This view not only records the basic resource status of each GPU card on each computing node, but also records the hardware interconnection topology between GPUs within the node and the network location attributes of the region where the node is located. At least some fields of the global resource view have a view version number, a commit timestamp, and a freshness flag, which are used to indicate the most recent consistent commit time and the most recent valid update time of the field. Step S2: Receive and parse topology-aware computing power requests: Receive user computing power requests, which include specifying the number and type of GPUs required and the desired interconnect topology preferences, and optionally carry task type tags and communication mode tags; Step S3: Perform pre-screening of topology and state constraints: Based on the global topology resource view, perform the first layer of screening to exclude GPU cards that do not meet the constraints of resource quantity, type, basic health and load, forming an original candidate GPU resource pool. Screening for high-frequency dynamic states also meets the freshness threshold; when the freshness is not met, the corresponding GPU card or node will be downgraded, frozen or removed. Step S4: Construct and score cross-domain virtual computing units: For the original candidate GPU resource pool, perform a cross-node virtual combination exploration process to select a group of GPU cards from nodes that may be located in different regions to form a virtual computing unit; For each candidate scheme of virtual computing unit, a comprehensive score is calculated based on a unified cost model; The unified cost model simultaneously quantifies the computational performance cost, cross-domain resource integration cost, and economic cost of the evaluation scheme, and includes a view obsolescence risk penalty item. The view obsolescence risk penalty item is related to the freshness flag and submission timestamp of the fields referenced by the candidate solution; and a version number consistency check is performed before scoring. If the check fails, the candidate solution is rolled back or eliminated. Step S5: Decision and Instantiation: Select the candidate scheme of virtual computing unit with the best score, and collaboratively create a group of secure container instances with GPU card-level binding on each physical computing node involved. These instances are configured to belong to the same logical task unit through the virtual network, and the atomic startup of the container group and the consistent configuration of communication parameters are completed through a two-phase consistent startup barrier during the instantiation phase. The two-phase consistent startup barrier includes at least a preparation to commit phase and a commit to start phase: resource locking and parameter distribution verification are completed in the preparation to commit phase, and simultaneous startup is completed in the commit to start phase; If any node fails during the commit preparation phase, a global rollback is executed to release locked resources.

2. The AI ​​computing resource management method supporting cross-domain fine-grained scheduling according to claim 1, characterized in that, The multi-region coordinator cluster uses incremental synchronization and lease mechanisms to maintain view consistency: lease-based strong consistency synchronization is used for static or semi-static attributes, and a consensus-based version number is generated by committing; high-frequency incremental synchronization is used for the dynamic state of GPU cards, and commit timestamps and freshness markers are maintained. The scheduling decision uses both version number verification and freshness threshold verification to achieve a balance between strong consistency of key attributes and eventual consistency of high-frequency states. The freshness marker includes the most recent update time, sampling source, and confidence level. The freshness threshold is adaptively set based on the task type label and communication mode label, and satisfies the requirement that the freshness threshold for AllReduce type training tasks is less than the freshness threshold for single-card inference tasks.

3. The AI ​​computing resource management method supporting cross-domain fine-grained scheduling according to claim 1, characterized in that, The comprehensive score P of the unified cost model is weighted and synthesized from the following components: performance cost component, integration cost component, economic cost component, and view staleness risk penalty component; wherein the view staleness risk penalty component is a monotonically increasing function of "current time minus submission timestamp" with respect to the dynamic fields referenced by the candidate solution, and different risk weights are set for different fields; the economic cost component includes at least the GPU real-time unit price, cross-domain data transmission bandwidth cost, and cross-domain surcharge; wherein the cross-domain surcharge is dynamically adjusted according to real-time network latency and available bandwidth, and may optionally include a Service Level Agreement (SLA) penalty / refund item to constrain cross-domain network jitter risk.

4. The AI ​​computing resource management method supporting cross-domain fine-grained scheduling according to claim 1, characterized in that, The cross-node virtual combination exploration process adopts a hierarchical pruning strategy: the first step is intra-node aggregation, which prioritizes finding GPU card groups that meet or partially meet the topology requirements within a single physical node to form local sub-units; the second step is cross-domain complementarity, which uses the local sub-unit as the core to search for complementary GPU cards from other regional nodes and evaluates the cross-domain virtual computing units formed after complementarity; the exploration process is limited by the cross-domain latency budget, which is dynamically determined according to the task type label and communication mode label.

5. The AI ​​computing resource management method supporting cross-domain fine-grained scheduling according to claim 1, characterized in that, The performance cost component quantifies the degree of matching between the GPU interconnect topology within the virtual computing unit and the request preference through a topology matching function, and assigns different scores based on interconnect technology levels such as NVLink, PCIe, and RDMA. The topology matching function includes at least node resource sufficiency, real-time load, network quality, node health, and cross-domain cost, and the weights of each scoring dimension are dynamically adjusted according to the task type label. The node health is calculated by weighting at least three indicators among GPU error count, temperature over-limit duration, ECC error rate, power stability, and network packet loss rate. When the health is lower than the health threshold, a freeze window is executed on the node, and the node does not participate in the candidate pool construction during the freeze window.

6. The AI ​​computing resource management method supporting cross-domain fine-grained scheduling according to claim 1, characterized in that, The downgrading, freezing, or removal process includes increasing the risk penalty on the node score when the freshness tag exceeds a first threshold, but the node can still participate; when it exceeds a second threshold, the node is frozen and put into a read-only observation state. When the third threshold is exceeded, the node is removed and view resynchronization is triggered.

7. The AI ​​computing resource management method supporting cross-domain fine-grained scheduling according to claim 1, characterized in that, The two-phase consistent startup barrier uses a distributed transaction ID to associate container group startup request. During the preparation and commit phase, resource locking, image pull completion confirmation, GPU binding confirmation, and communication parameter verification are performed on each node. During the commit initiation phase, a commit command is broadcast using the transaction ID as the key, and a commit receipt is recorded. If any node fails to return an acknowledgment within the timeout window during the commit initiation phase, the consistency termination process is triggered and the system rolls back to the preset safe state. The consistent configuration of the communication parameters includes at least the topology-aware parameters of the distributed communication library, the hierarchical AllReduce parameters, the cross-domain bandwidth limit parameters, and the network congestion control parameters. These parameters are uniformly issued by the coordinator during the commit preparation phase and confirmed by the acknowledgments of each node.

8. The AI ​​computing resource management method supporting cross-domain fine-grained scheduling according to claim 1, characterized in that, The creation of the secure container instance includes GPU card-level binding and multi-layer isolation. The security isolation mechanism includes GPU virtualization device-level isolation, container namespace isolation, virtual network isolation, and storage encryption and access control.

9. An AI computing resource management system supporting cross-domain fine-grained scheduling according to claim 1, characterized in that, It includes a user portal module, a cross-regional coordinator, a resource scheduling engine, an instance management module, a security isolation control module, an image and storage management module, a billing management module, and a multi-protocol access gateway; The cross-regional coordinator is a cluster of coordinator nodes deployed in different geographical regions. It works based on a distributed consensus protocol and maintains a global resource view with version numbers, commit timestamps, and freshness markers. The resource scheduling engine outputs GPU card-level scheduling results based on a unified cost model; The instance management module collaboratively creates container instances on each node and completes atomic startup and failure rollback through a two-phase consistent startup barrier.

Images