A dynamic load balancing and automatic operation and maintenance management system of an enterprise-level AI resource pool

Abstract

The application is specifically a dynamic load balancing and automatic operation and maintenance management system of an enterprise-level AI resource pool, relates to the technical field of cloud computing and intelligent computing resource scheduling, and comprises: a fragment increment accounting module; an access double-arch gate module; an absorption area collection and scheduling module; a reservation reverse arrangement module; and a linkage protection verification module.In the application, the combination mechanism of the access double-arch gate and the reservation reverse arrangement module changes the queuing and waiting of large resource demand tasks from unpredictable to promised and redeemable; when the tasks enter, three-arch judgment is performed according to the comprehensive fragment increment, the placement request that may cause over-limit fragments is intercepted in the access stage, and the scattered tasks with the fragment increment in the middle gear are guided to the absorption area for centralized placement, so that the continuous idle resources in the ordinary area are kept complete for a long time, the success rate of instant scheduling when the large task arrives is improved, and the generation of new fragments is inhibited at the source.

Description

Technical Field

[0001] This invention relates to the field of cloud computing and intelligent computing resource scheduling technology, and in particular to a dynamic load balancing and automated operation and maintenance management system for an enterprise-level AI resource pool. Background Technology

[0002] Enterprise-level AI resource pools, with accelerator cards such as GPUs as core resources, support differentiated tasks such as model training and inference. Dynamic load balancing and automated operation and maintenance management are crucial for ensuring resource utilization efficiency and task continuity. As multi-tenant deployments expand and task resource specifications become more discrete, idle resources exhibit a fragmented and intertwined pattern in both spatial distribution and expiration time.

[0003] On the one hand, scattered idle cards are dispersed across different nodes, making it difficult to place tasks with high resource requirements in a timely manner;

[0004] On the other hand, the natural dispersal of resources upon task expiration makes it easy for aggregated resources to be fragmented again. How to establish a coordinated mechanism among task admission, fragmentation management, and operational protection remains a technical challenge in this field.

[0005] For example, in the prior art:

[0006] The patent application with application number CN120179045A discloses a GPU computing power dynamic scheduling system. It adjusts task scheduling based on GPU core thermal response and instruction path behavior through modules such as thermal expansion monitoring, path disturbance identification, load migration, scheduling label revision and node priority sorting, thereby enhancing the load balance between computing units. However, it focuses on selecting task placement locations based on hardware operating status and does not quantify or intercept the incremental resource fragmentation caused by the placement action itself. There is still room for improvement in the fragmentation source control during the admission stage.

[0007] The patent application with application number CN121501512A discloses a GPU memory fragmentation optimization scheduling method and system. It calculates a fragmentation score that reflects the continuity of memory, sorts the candidate GPU list in ascending order, and selects the first GPU after sorting to assign a task. However, it uses fragmentation measurement for the selection of points in a single scheduling, and the measurement dimension is concentrated at the spatial level. It does not cover the impact of the task's expiration time on fragmentation evolution, nor does it involve the arrangement of the cleanup time limit after fragmentation exceeds the limit.

[0008] Patent application CN121722529A discloses a method and system for scheduling computing resources in heterogeneous computing systems based on intelligent prediction. It divides tasks into different categories and divides them into reserved partitions and fragmented partitions. In the reserved partitions, continuous resource blocks that match the resource requirements of high resource-demand tasks are reserved. However, the reserved partitions are statically divided based on prediction results. How to prevent newly arrived tasks from re-occupying the continuous resources in the partitions during the consolidation and aggregation process, and how to ensure that the consolidation plan is carried out within the promised time limit, have not yet been provided at the mechanism level.

[0009] The aforementioned technologies have made contributions to hardware state-aware scheduling, fragment quantization and point selection, and partition reservation, providing diverse implementation ideas for computing resource scheduling.

[0010] Based on this, how to further integrate the fragment increment interception in the access phase, the time-limited sorting and fulfillment of over-limit tasks, and the dynamic protection of aggregated resources during the sorting and execution period into a mutually supportive overall mechanism, so that the waiting time for large resource demand tasks is predictable, the sorting results are free from secondary fragmentation before fulfillment, and new fragments are diverted at the source, is a technical problem that the existing technology has not yet solved, but that this application aims to solve. Summary of the Invention

[0011] The purpose of this invention is to provide an enterprise-level AI resource pool dynamic load balancing and automated operation and maintenance management system in order to solve the above problems.

[0012] To achieve the above objectives, the present invention adopts the following technical solution:

[0013] A dynamic load balancing and automated operation and maintenance management system for an enterprise-level AI resource pool includes:

[0014] The fragment increment calculation module is used to collect and register the running data of nodes and rented tasks within the resource domain, calculate the spatial and temporal fragmentation index, perform hypothetical allocation on candidate placements, and output the comprehensive fragment increment and calculation version number.

[0015] The dual-level access gate module is used to make three-level judgments based on the comprehensive fragment increment and the accounting version number, according to the upper limit of access increment and the consolidation trigger increment: if it does not exceed the upper limit of access increment, it is allowed; if it is between the two, those that meet the threshold of small tasks are reassigned to the absorption area, and the rest are placed in the ordinary area and counted as fragment debt; if it exceeds the consolidation trigger increment, a pending order is generated, transferred to the reservation queue, and targeted consolidation is triggered.

[0016] The absorption area collection and scheduling module is used to accept tasks that meet the threshold for small tasks and those on the migration list that are reassigned according to the three-level judgment and are designated to the absorption area. The tasks are collected and placed according to the compact principle and can be flexibly expanded and contracted according to the capacity level of the absorption area.

[0017] The appointment scheduling module is used to receive pending orders, calculate the fulfillment deadline based on the ledger data, select the target layout, generate a migration list and set the acceptance time, and promote the migration by setting milestones in reverse order.

[0018] The linkage protection verification module is used to establish shadow locks on the target area according to the target layout and notify strict judgment. At the time of acceptance, it verifies whether the continuous resources meet the requirements. If they do, it binds the scheduled task and removes the shadow lock to restore access.

[0019] Preferably, the method by which the fragmentation increment calculation module calculates the spatial fragmentation index is as follows:

[0020] Full Card Level Space Fragmentation Index ,in, This represents the total number of idle cards within the resource domain. The maximum number of idle integer cards per node within the resource domain, when hour Set the empty state flag of the entire card;

[0021] Memory fragmentation index ,in, This represents the total amount of free video memory within the resource domain. This represents the maximum free video memory on a single card.

[0022] The task of applying for a full card is to use The task of requesting video memory allocation adopts Hybrid tasks that combine the needs of full card and memory partitioning and The larger of the two.

[0023] Preferably, the method for calculating the time fragmentation index and the comprehensive fragmentation increment is as follows:

[0024] Time Fragmentation Index ,in, As a time perspective window, For the width of the anchor slot, This refers to the number of tasks that fall within the time lookout window at their due date. The number of non-empty anchor slots occupied at the expiration time;

[0025] Assume the allocation is performed as follows:

[0026] Remove the candidate cards from the free set, deduct the corresponding card's free memory according to the single card's memory requirement, and set the task's estimated expiration time. Included in the set of maturity times, where, For the current moment, For the duration of the lease application;

[0027] Recalculate the increase in space debris before and after placement With the increase of time fragments Calculate the overall fragmentation increment ,in, and As weight, and .

[0028] Preferably, in the three-level determination:

[0029] The threshold for the mini-task is the required number of whole cards. And single-card video memory requirements ,in, This refers to the total memory capacity of a single graphics card.

[0030] Large tasks whose increments fall between the upper limit of the entry increment and the increment triggered by consolidation are placed in the normal zone and marked with an observation tag; their increments are included in fragment debt. When a task marked with an observation exits or expires, a space fragmentation index recalculation is triggered, and its original occupied location is listed as a priority merging object in the next background cleanup.

[0031] when The cumulative amount exceeds the fragment debt limit. Background consolidation is triggered at any time. Background consolidation selects objects to merge in descending order of their contribution to fragmented debt. Migration execution reuses the migration executor and runs with low bandwidth priority, without setting shadow locks or redemption time limits. Upon completion, [the following will occur]. Reset to zero.

[0032] Preferably, the method further includes:

[0033] Cards marked as reserved in the target layout establish shadow locks, and the placement of cards involving shadow locks is considered infeasible.

[0034] In determining the placement of remaining cards within the target area, both the upper limit of the admission increment and the increment triggered by the reorganization are multiplied by a tightening coefficient. ,in, A positive number less than 1;

[0035] All new tasks arriving during the protection period that meet the requirements for minor tasks will be reassigned to the absorption area.

[0036] Each determination also references the calculated version number. With layout version number If any version changes, it will be deemed invalid and re-evaluated.

[0037] Preferably, after receiving a reassigned small task, the absorption area aggregation and scheduling module aggregates and arranges the tasks within the absorption area according to the principle of compactness:

[0038] Choose the node with the most occupied cards in the absorption area for placement, and within the same node, prioritize the card with the smallest difference between free video memory and demand.

[0039] Initially, nodes are selected in descending order of the number of small tasks running on each node and assigned to the absorption zone. Nodes that leave the zone return to the normal zone after all their tasks have ended naturally or have been moved out. Tasks that have not yet expired are not expelled.

[0040] Absorption region occupancy rate ,in, This represents the total number of cards already occupied within the absorption area. The total capacity of the absorption area; when The water level is higher than the upper limit of the absorption zone capacity and the duration exceeds the hysteresis time. When performing expansion, The water level is below the absorption zone capacity and has remained below it for more than [time period missing]. Scaling down is performed at that time.

[0041] Preferably, in the migration list generated by the appointment reverse sorting module, the destination of the migration task that meets the small task threshold is designated as the absorption area;

[0042] When the migration scale exceeds the current reserve of the absorption area, the absorption area collection and scheduling module performs emergency expansion according to the migration gap capacity. The number of nodes is allocated according to the amount of the gap capacity. If the gap is still insufficient after expansion, the difference is migrated to the card slots in the ordinary area that have not been reserved by the target layout and is counted as fragment debt.

[0043] After the completion of the sorting and acceptance process, the small tasks that were temporarily placed in the general area will be gradually returned to the absorption area during low-load periods.

[0044] Preferably, the method for calculating the redemption period is as follows:

[0045] Delivery time limit Promised duration ,in, For the time of suspension, To ensure the duration, This serves as the baseline duration for planning and conservation initiation. To estimate the number of tasks to be migrated based on the initial target layout, the initial target layout is obtained by pre-scanning the candidate enumeration to select the candidate set with the fewest tasks to be migrated out. To transfer parallelism, The average migration time for a single task is calculated by classifying the amount of memory data migrated in a single task and taking the average migration time of completed tasks in the same category.

[0046] The promised duration shall not exceed the maximum promised duration. If the calculation result exceeds the limit, then... Commit to and enforce the upgrade mechanism.

[0047] Preferably, the appointment scheduling module is set with an acceptance time. ,in, Allow sufficient lead time for acceptance;

[0048] Starting from the acceptance time and working backwards, the latest completion time of the last migration action is the acceptance time minus the migration margin. The migration margin is the average migration time of a single task. Each preceding action is arranged forward according to the migration list order and the degree of migration parallelism to determine its own latest completion time.

[0049] The latest start time of the first action should not be earlier than the planned issuance time; otherwise, the migration parallelism will be increased or an upgrade will be triggered. If a single migration fails, it will be automatically retried. If it still fails, a local replanning will be triggered to select an alternative destination or adjust the order of subsequent actions.

[0050] The progress of the plan is verified at the preset checkpoints. If the progress is behind schedule, it is upgraded step by step, in the following order: expanding the scope of the cleanup, increasing the parallelism of the migration, and borrowing adjacent resource domains.

[0051] Preferably, in the linkage protection verification module:

[0052] The shadow lock card is shown as occupied, and the space fragmentation index calculation shows it as free but unallocable;

[0053] During acceptance, the actual number of consecutive idle cards in the target area is checked and organized based on the shadow lock list.

[0054] After acceptance, the linkage protection verification module binds the reservation task to the reserved card slot in the target layout. The bound card slot is directly converted to formal occupation without going through the idle state, the residual shadow lock is released and the standard threshold is restored, and a progress announcement is sent to the subsequent tasks in the reservation queue. If there are reusable remaining continuous resources, the subsequent task reuse acceptance conclusion is directly entered.

[0055] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0056] 1. This invention transforms the unpredictable queuing process for large resource-demanding tasks into a commensurable and fulfillable one through a combination of a dual-level access gate and a reservation-based reverse scheduling mechanism. When a task enters the system, a three-level judgment is performed based on the overall fragment increment. Placement requests that may cause excessive fragments are intercepted during the access phase. Scattered tasks with fragment increments in the middle level are guided to the absorption zone for centralized placement. As a result, the continuous idle resources in the ordinary zone remain intact for a long time, improving the success rate of real-time scheduling when large tasks arrive, and suppressing the generation of new fragments at the source.

[0057] 2. This invention promotes progress by setting milestones backward from the acceptance date of the organization plan. When progress falls behind, it is upgraded step by step, ensuring the fulfillment of commitments. Tenants can know the expected entry time when submitting tasks, and the scheduling order of the resource pool and the queuing experience of tenants are improved at the same time. Meanwhile, the organization target area is protected by shadow locks during the organization execution period, and the access judgment is tightened. Newly arrived scattered tasks are continuously diverted to the absorption area, and the continuous resources that are gathering are not broken up again before the fulfillment is completed. The organization action does not need to be frequently repeated. Attached Figure Description

[0058] Further details, features, and advantages of this application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0059] Figure 1 This is a system structure diagram of the present invention. Detailed Implementation

[0060] Several embodiments of this application will now be described in more detail with reference to the accompanying drawings to enable those skilled in the art to implement this application. This application may be embodied in many different forms and for various purposes and should not be limited to the embodiments set forth herein. These embodiments are provided to make this application thorough and complete, and to fully convey the scope of this application to those skilled in the art. The embodiments described do not limit this application.

[0061] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having a meaning consistent with their meaning in the relevant field and / or the context of this specification, and shall not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0062] Example 1

[0063] Its specific implementation method is combined with the appendix Figure 1 Please provide a detailed explanation.

[0064] In this embodiment, it includes:

[0065] Module 1, Fragment Incremental Accounting Module:

[0066] This module is responsible for the unified measurement and accounting of all fragmented states, and serves as the common data source for the three modules: access gate, absorption zone scheduling, and reverse sorting.

[0067] This module itself does not directly generate any scheduling decisions; it only answers two questions: what is the current fragmentation level of each resource domain; and how much will any hypothetical placement action change the fragmentation level. All downstream modules must include the calculation version number when referencing the calculation results from this module. This ensures that multiple modules work together on the same data snapshot.

[0068] The system divides the enterprise-level AI resource pool into several resource domains based on network topology and accelerator card model. The division rule is as follows: nodes under the same access layer switch, with the same acceleration card model and compatible scheduling tags are grouped into the same resource domain. The calculation is based on the resource domain as the basic unit, and cross-domain calculations are not merged.

[0069] Each node within the domain Total number of registered accelerator cards Number of free whole cards Number of idle processor cores Free memory capacity Each acceleration card (hereinafter, "card" is short for acceleration card) Registered total video memory capacity With free video memory capacity .Task The resource requirement is denoted as a quadruple. ,in The required number of cards. This refers to the required video memory capacity for a single card. For the required number of processor cores, The required memory capacity.

[0070] The spatial dimension is divided into two levels: the whole card level and the memory level. Whole card level: the total number of idle whole cards within the domain. Maximum number of idle whole cards per node ;when At that time, the whole card-level space fragmentation index ;when hour, Simultaneously, a flag indicating that no full card is available for allocation within the domain is set. Memory level: Total amount of free video memory within the domain. Maximum free video memory per card ;when At that time, the memory fragmentation index ;when hour, The selection rules for the two-tier index are as follows: the task of applying for a full card adopts... The task of requesting video memory allocation adopts Hybrid tasks that combine the needs of full card and memory partitioning and The larger of the two values ​​will be used in subsequent accounting in a stricter manner.

[0071] For example:

[0072] If the number of idle integer cards in 5 nodes of a certain resource domain are 4, 1, 1, 2, and 0 respectively, then... , , .

[0073] Let the time outlook window be (Default is 24 hours), Anchor slot width is (Default time is 30 minutes), the window is divided into Anchor slots. The expiration dates of all leased tasks within the statistics domain. The number of tasks falling into the window is The number of non-empty anchor slots occupied at the aforementioned expiration time is .when Time fragmentation index ;when hour, . The value of is between 0 and 1, and the more concentrated the expiration time, the lower the value.

[0074] Example: Inside the window The expiration times of these tasks occupy a total of 6 anchor slots. The system operates on a lease basis; all tasks must include an application for the lease duration upon entry. The set at the expiration time only counts the tasks that are currently being leased.

[0075] Place any candidate That is, to put the task Place to node Card collection First, this module performs a hypothesis assignment on a copy of the data snapshot: Remove each card from the free set, and press Deduct the corresponding card's free video memory and assign the task. Expected expiration time ( For the current moment, The duration of the lease application is included in the set of expiry times;

[0076] The index was then recalculated to obtain the space fragmentation index after placement. With time fragmentation index Compared with the spatial fragmentation index before placement Time Fragmentation Index By comparison, the increase in space debris is obtained. With the increase of time fragments Comprehensive fragment increment The weight , and Default value , The resource pool, primarily for full-card demand, can be increased. .

[0077] A negative increment indicates that placement causes fragmentation to converge; this type of placement is encouraged. Continuing the previous example: placing a single-card task on a node with 4 idle cards results in the number of idle integer cards on each node becoming 3, 1, 1, 2, and 0 respectively. , , Therefore .

[0078] The accounting process employs a dual-channel approach: event-driven and periodic verification.

[0079] The system will recalculate the results immediately upon triggering six types of events: task entry, task exit, task expiration, migration completion, node online, and node offline; and every [time period]... (Default 60 seconds) Perform a full recalculation of the ledger and compare it with the event-driven result. If they are inconsistent, the full recalculation result shall prevail and the exception event shall be recorded to prevent ledger drift caused by event loss. Each recalculation changes the accounting version number. The algorithm is monotonically increasing, and each decision made by the downstream module must specify the referenced component. .

[0080] Domain-level ledger records: , , , , , , The count of due tasks for each anchor point slot and the calculation version number. Empty card status flag; node-level ledger records: , , Each card Zone markers (general zone, absorption zone, target zone for processing), protected zone markers; task-level ledger records: Entry time, expiry time Task priority. All three types of ledgers are persistently stored and support sorting by accounting version number. Replaying fragmented states at any historical moment provides a data foundation for auditing and review.

[0081] Full card empty state ( When a large task's access request is made, it is directly transferred to the reservation reverse scheduling process; when a new node goes online, a full recalculation is performed before access is opened; when a task is renewed, it is first removed from the expiration set according to the original expiration time, and then re-entered according to the new expiration time to avoid double counting; the domain clock is based on the unified time service, and all expiration times are based on the time service to avoid the expiration time being misassigned to the anchor point slot due to the clock drift of each node.

[0082] Module 2, Access Dual-Gate Module:

[0083] This module is located on the task entry path. For each entry request, this module generates candidate placement, calls module one to calculate the fragment increment, performs release, reassignment, and suspension three-branch processing according to the dual threshold, and cooperates with module five to implement virtual occupation judgment during the reverse schedule execution period.

[0084] Entry requests are first filtered by pre-scheduling constraints (affinity rules, anti-affinity rules, label constraints, and tenant quota reserves) to obtain a list of feasible nodes. Candidate nodes are then enumerated and placed on the feasible node list, with the enumeration order being nodes in the normal zone first, followed by nodes in the absorption zone. Within the same node, a set of occupancy cards is generated according to the compact principle of "prioritizing filling the high-occupancy nodes". Each feasible placement is sent to a module for comprehensive fragment increment calculation. (The "increment" below refers to the total fragment increment.) (abbreviation).

[0085] Upper limit of access increment Incremental triggering of consolidation satisfy Default value , It supports configuration by resource domain, and each adjustment must be logged and the effective date noted. Historical judgments will still be replayed based on the thresholds at that time. Thresholds and They are dimensionless numbers between 0 and 1, and have the same dimension.

[0086] First priority, proceed with permission:

[0087] Existence satisfies When placing it, take one of them. The node with the lowest load level is allowed to enter; if there are multiple nodes with the same minimum value, the node with the lowest load level is selected.

[0088] The second stage involves reassignment and bookkeeping:

[0089] All feasible placements are satisfied. First, determine whether the task meets the threshold for a small task. The threshold requirement is the number of whole cards required. And single-card video memory requirements ;

[0090] Tasks that meet the threshold are reassigned to the absorption area and processed in Module 3; large tasks that do not meet the threshold can be placed in... The smallest ordinary area location, but at the same time, the increment of this placement is included in the domain-level fragment debt. Furthermore, an observation mark is attached to the task. When a task with an observation mark exits or expires, a spatial fragmentation index recalculation is immediately triggered, and its original occupied location is listed as a priority merging object in the next background reorganization.

[0091] when The cumulative amount exceeds the fragment debt limit. (Default value is 0.3) A low-priority background cleanup is triggered. The background cleanup selects objects to be merged in descending order of their contribution to the fragment debt at each placement location, migrating scattered occupancy to the absorption area or high-occupancy nodes for merging. The migration execution is performed by the migration executor of the reuse module four and runs with low bandwidth priority. No shadow locks or redemption time limits are set. After the background cleanup is completed... The process of clearing and organizing the background information has no promised timeframe, distinguishing it from the targeted organization in Module Four.

[0092] The low-bandwidth priority is implemented as follows: When the migration executor initiates a video memory copy, it marks a low priority level (value of 0, default value of 3 for service traffic, and 5 for control signaling) in the service category field of the RDMA send queue. The network switch performs weighted round-robin scheduling based on the IEEE 802.1p priority label. Low-priority traffic only uses idle bandwidth (i.e., the remaining bandwidth after service traffic has occupied it) and does not participate in bandwidth preemption during link congestion, ensuring that background processing does not interfere with the normal I / O of online training and inference tasks.

[0093] Third gear, suspended:

[0094] The minimum increment for large tasks among all feasible placements still satisfies When this happens, the gate will not allow passage, a pending order will be generated, and the task will be transferred to the reservation queue in Module 4. Simultaneously, a directional sorting trigger signal will be sent to Module 4, including an incremental list of all candidate placements, which will serve as the basis for the reverse scheduling plan. Continuing from the example in Module 1: If the placement... , (If the task expires at the exact moment and falls into an existing dense anchor point slot), then... The decision was made to allow passage.

[0095] Suspended order persistence record:

[0096] Task identifier, resource requirements Lease duration Task priority, suspension time (Suspension time is the same as the scheduled entry time), referencing the calculated version number. The list includes all candidate placements and their incremental lists, as well as trigger reason codes. The pending list is a direct input for module four in creating a reverse schedule and also serves as evidence for subsequent auditing and tracing.

[0097] When an inverted index plan is in progress within the domain, Module 5 will organize the target area node set. Set to protected state and set the target layout. Cards marked as reserved at the gate are shown as occupied, i.e., shadow locks: any placement of cards involving shadow locks is considered infeasible and will no longer participate in enumeration and incremental calculation; placement of other cards in the target area will still be calculated for incremental increases as usual, but both thresholds will be multiplied by a tightening coefficient. (Default value is 0.5) Strict judgment;

[0098] During the protection period, all newly arrived small tasks that meet the threshold will be reassigned to the absorption area. The shadow lock mechanism ensures that continuous resource blocks that are being moved and gathered will not be fragmented again by newly arrived tasks before their realization is completed. During the protection period, each judgment in Module 2, except for referencing the calculated version number, will be performed accordingly. In addition, the layout version number must also be cited. If any version changes, the current judgment will be invalidated and a new judgment will be made based on the latest version.

[0099] When a high-priority task falls into the third tier, a preemption evaluation can be requested: enumerate combinations of low-priority tasks that can be preempted, and execute preemption only when all three conditions are met simultaneously. After preemption, place the task that meets the conditions... The preempted mission supports checkpoint relocation, and the relocation time shall not exceed half of the remaining lease duration of the preempted mission. The relocation time is calculated as follows: The estimated value is taken from the same range; if any condition is not met, the task will still be transferred to the reservation queue. Tasks already in the reservation queue will not participate in taking over other tasks, nor will they be taken over by other tasks, to ensure the certainty of the fulfillment time limit.

[0100] Serialization of gate-level determinations for the same resource domain:

[0101] The calculation version number is read at the start of the determination. Review before placement If changes have occurred, the current judgment is invalidated and recalculated based on the new snapshot; judgments between different resource domains can be executed in parallel. This avoids resource collisions caused by two concurrent inbound requests passing through the gate simultaneously based on expired snapshots.

[0102] In addition to incremental conditions, the release decision must also pass the node load level verification: after placement, the processor utilization, memory utilization, and network outgoing bandwidth utilization of the node must not exceed the node load level limit (default is 85%). The node load level value is the maximum value among the processor utilization, memory utilization, and network outgoing bandwidth utilization. The comparison of load levels between nodes and the determination of whether the load level is compliant are based on the maximum value.

[0103] When the recommended low-load node increment exceeds the limit, while the load of nodes with acceptable increments is too high, the decision-making order of "fragmentation first, balancing second" is executed, that is, the node with the lowest load is selected from the nodes with acceptable increments; if neither of these conditions can be met, the process switches to the second or third tier. This rule ensures that the fragmentation management of this system does not conflict with the existing dynamic load balancing objectives and that the order is clear.

[0104] Each judgment records the input snapshot, incremental list, hit gear, reason for the judgment, referenced accounting version number and processing time, forming a replayable judgment log; during auditing, the complete reasoning process of the previous historical judgment can be reproduced based on the log as is, ensuring that the system behavior is explainable, verifiable and verifiable.

[0105] Module 3, Absorption Area Collection and Scheduling Module:

[0106] This module is the successor to the "reassignment" branch in the overall design concept. It is responsible for absorbing small tasks that are judged to have excessive incremental capacity by the gate into the designated absorption area, so that scattered needs are kept away from the continuous resources of the ordinary area, thereby reducing the generation of new fragments from the source. At the same time, it serves as a temporary resettlement area for migration tasks during the reverse scheduling and sorting process, ensuring the smooth progress of the sorting plan.

[0107] An absorption region is a set of nodes within a resource domain that are marked specifically to undertake scattered small tasks. The operating principle of the absorption zone is "tight rather than loose within the zone, and organized rather than scattered outside the zone": small tasks are stacked as compactly as possible within the zone, and continuous idle resources in the ordinary zone are kept intact for use by large tasks and consolidation. The absorption zone, ordinary zone, and consolidation target zone are mutually exclusive; any node can belong to only one zone at a time.

[0108] Nodes enter the absorption zone according to the following rules: Initially, the system determines entry based on the small tasks running on each node (those that meet the threshold). The nodes with the most sub-tasks and no more than one idle integer card are sorted in descending order and then assigned to the appropriate nodes. , to be included in the scale The capacity should be 1.2 times the normal demand for small tasks within the domain. The normal demand for small tasks within the domain is calculated based on the peak number of whole cards occupied by small tasks simultaneously online over the past 7 natural days. During operation, when the capacity level of the absorption area exceeds the limit and expansion is required, priority will be given to selecting the node with the fewest conflicts with the affinity tags of large tasks in the ordinary area and the fewest idle whole cards. ;

[0109] Newly added nodes will undergo a full recalculation before being granted access. When a node enters the zone, its existing tasks will not be forcibly migrated; only the assignment of new large tasks to the node will be stopped. Once the existing large tasks have expired naturally, the node's status will be completely transferred to the absorption zone.

[0110] When the absorption zone has excess capacity and needs to be reduced, the outgoing nodes are selected in the order of "least number of small tasks first". The outgoing node first stops accepting new tasks, and waits until all tasks on it end naturally or are reorganized and moved out, then the absorption zone mark is removed and it returns to the normal zone. Before returning, a full recalculation is performed. The outgoing process does not allow the eviction of tasks that have not yet expired to avoid affecting tenants.

[0111] Define absorption region occupancy rate ,in for The total number of cards already in use. for Total card capacity. Set the water level below the absorption zone capacity. (Default value is 0.35) and the water level above the absorption zone capacity. (Default value is 0.8). When And the duration exceeds the hysteresis duration. (Default is 10 minutes) A ​​capacity expansion is performed once, adding 1 to 2 nodes each time; when And lasting longer than A reduction in capacity is performed once per cycle, releasing one node each time. No action is taken when the capacity level is between two levels or the duration is insufficient. Asymmetric hysteresis is used for expansion and reduction, with expansion responding faster than reduction to prevent frequent fluctuations during demand decline periods.

[0112] The task to enter the absorption zone must meet a threshold. That is, the number of whole cards required. And single-card video memory requirements Tasks that do not meet the threshold are not allowed to enter the absorption zone. The placement within the zone follows a compact principle: priority is given to nodes with the most occupied cards within the zone; within the same node, priority is given to cards with the smallest difference between free memory and demand, ensuring that scattered usage on individual nodes is filled as quickly as possible, while other nodes maintain a state of free full-card usage, leaving room for potential full-card demand within the zone. Tasks within the zone are also subject to node load level verification; after placement, the node's processor utilization, memory utilization, and outgoing bandwidth utilization must not exceed the node's load level limit.

[0113] During the execution of the reverse scheduling plan, organize the smaller tasks that need to be migrated out of the target area, with the default migration destination being the absorption area:

[0114] When generating the migration list in Module 4, the threshold will be met. The destination of the relocation mission is uniformly designated as This module will arrange specific locations within the area according to the principle of compactness; if the relocation scale exceeds... Based on the current remaining capacity, this module will first trigger an emergency expansion (regardless of hysteresis time). (Restrictions) The number of nodes to be expanded in an emergency is determined by the migration gap capacity. The number of nodes is equal to the total card capacity required to fill the gap. Only the remaining capacity after expansion is allowed to be migrated into the ordinary area that has not been targeted for deployment. Reserved card slots will be counted as fragment debt according to the second tier rules. .

[0115] After the reorganization plan is accepted, the small tasks that were temporarily moved into the general area due to the reorganization will be gradually taken back to the absorption area by this module during subsequent low-load periods. Low-load periods refer to the periods when the node load level is continuously lower than half of the node load level limit.

[0116] Request lease duration for tasks entering the absorption zone The lease term shall not exceed the maximum limit for leases within the district. (Default is 72 hours). Expired tasks can be renewed, but the threshold must be met again. Verification with node load water level;

[0117] Tasks within the zone are not exempt from preemption. High-priority, large tasks, after being evaluated by Module 2, can preempt low-priority tasks within the zone that support checkpoint relocation, according to the rules. Preempted tasks are then moved to other locations within the zone or suspended and awaiting completion. The lease cap ensures that the absorption zone maintains a structure primarily composed of short, quick tasks, preventing long-duration, large-occupancy tasks from solidifying and forming new structural fragments within the zone.

[0118] The absorption region is empty. If the branch has not yet been assigned to any node, the reassignment will automatically degenerate into being placed in the normal zone according to the second tier rules and counted as fragment debt. This also triggers the initialization of the absorption region;

[0119] When the absorption zone capacity is full and there are no available nodes for emergency expansion, the reassignment request is rolled back to Module 2 for reassessment, and if necessary, transferred to the reservation queue. When a node fails and goes offline, its tasks are handled according to the fault tolerance policy configured by the tenant, the absorption zone capacity is automatically recalculated, and expansion is triggered based on the absorption zone capacity level. Every entry, exit, expansion, reduction, admission, and revocation action in this module is recorded in a ledger snapshot and the triggering basis for audit playback.

[0120] Module 4, Reverse Scheduling Module:

[0121] This module is the core of the "time-limited fulfillment" execution in the overall concept. It is responsible for managing the queue of reservation tasks that are suspended by the gate, calculating the fulfillment time limit for each reservation task, reorganizing and advancing the plan according to the fulfillment time limit, ensuring the fulfillment of commitments through checkpoints and upgrade mechanisms, and finally completing the acceptance and notifying the reservation task to enter the site.

[0122] After receiving the pending order and targeted organization trigger signal from Module 2, this module executes nine steps in sequence: queueing up scheduled tasks, calculating fulfillment deadlines, selecting target layouts, generating migration lists, scheduling milestones, issuing and executing plans, advancing checkpoints, fulfillment acceptance, and notifying entry. Any status change occurring in any step is recorded in the plan ledger and simultaneously notifies Module 5 to activate or update protection.

[0123] Appointment queue Maintained separately for each resource domain. The sorting rule within the queue is: those with earlier fulfillment deadlines are ranked first; for those with the same fulfillment deadline, resource requirements are ranked first. The number of whole cards The larger ones are listed first; if they are the same, the time of hanging is determined. Those who arrive earlier are placed at the front.

[0124] Multiple reusable scheduled tasks within the same organizational target area can be merged into a single reverse schedule, provided that the resource requirements of each task can be incorporated into the same target layout. Furthermore, the planned completion time must not be later than the earliest of the individual member's task completion deadlines. Consolidation can reduce migration costs, but the deadlines for any member's tasks must not be exceeded.

[0125] Scheduled Task The redemption period ,in The promised duration is the time to suspend. In the formula The base duration for planning and protection initiation (default is 10 minutes). To estimate the number of tasks to be migrated based on the initial target layout, the initial target layout undergoes a pre-scan according to the candidate enumeration rules of this module. The candidate set with the fewest tasks to be migrated is selected as the initial target layout, and the number of tasks to be migrated is then calculated. Pre-scan only counts, without issuing any migration actions; To determine the degree of parallelism during migration (default value is 3). The average migration time for a single task is divided into three categories based on the amount of video memory data migrated in a single task: less than 8GB, 8GB to 40GB, and greater than 40GB. The average migration time of each category is the average time of the same category that has been completed within the last 30 natural days.

[0126] Supplement to the conversion logic between preliminary target layout and final target layout:

[0127] It is important to clearly distinguish the positioning and transformation relationship between "preliminary target layout" and "final target layout" here: the preliminary target layout only applies to the committed timeframe. Used in the estimation phase, its function is to quickly (within 500ms) obtain an upper bound estimate of the number of tasks that need to be migrated out. To support the calculation of the committed duration, the layout results of this stage are neither written into the planning ledger nor distributed to the migration executor.

[0128] Final Target Layout It is compiled after the committed duration is determined, and it takes into account three additional constraints on the basis of the initial target layout:

[0129] To resolve migration conflicts, check if there are two migration tasks that are mutually dependent as source and destination. If so, adjust the destination to break the circular dependency.

[0130] Load balancing fine-tuning, in Among the same candidate layouts, prioritize the one that has been sorted out. The layout with the minimum internal node load variance is used to avoid any node load exceeding the 85% limit after optimization.

[0131] Fault tolerance redundancy, with an extra card slot reserved as a shadow redundancy slot (not involved in migration, only used as an emergency switch slot in case of target card failure).

[0132] The initial layout's migration task list is marked as "estimated state," while the final layout's migration task list is marked as "execution state." The two are switched at the moment the plan is issued, after which only the final layout takes effect. If the final layout becomes infeasible due to an execution-side anomaly (such as a node going offline midway), the system automatically rolls back to the pre-scanning stage to regenerate the initial layout and triggers the plan upgrade mechanism.

[0133] When the system has no historical data during a cold start, the three default values ​​are 5 minutes, 8 minutes, and 15 minutes respectively. Maximum commitment duration. (Default time is 4 hours), calculation results exceed Press at time A commitment is made, and the upgrade mechanism is mandated during the planning process. Example: estimation. , , If it is 8 minutes, then minute.

[0134] Target layout This refers to the specific distribution within the domain of the contiguous resource blocks used to accommodate scheduled tasks after processing. The selection rule is: enumerate the domain's capacity... The candidate node set must have a minimum sum of free integer cards and available integer cards. The number of vacant full cards in a node refers to the number of full cards currently occupied by tasks that support checkpoint migration and that have available destination card slots within the domain, and each node satisfies the task's affinity and label constraints.

[0135] Supplementary explanation of the target layout enumeration algorithm:

[0136] The enumeration algorithm employs a backtracking search strategy with pruning to control computational complexity in large-scale resource domains. The algorithm input consists of a list of all nodes within the resource domain and the scheduled task requirements. The output includes the current node occupancy log and the optimal target layout. .

[0137] The specific implementation involves three steps:

[0138] The first step is initial screening of candidate nodes: After removing unqualified nodes based on task affinity tags and anti-affinity constraints, the remaining nodes are sorted in descending order by "number of available full cards + number of currently available full cards", retaining only the top-ranked nodes. Each node participates in the subsequent enumeration ( Take the smaller of 10 and the total number of nodes in the domain to ensure that the upper bound of the calculation is controllable.

[0139] The second step, combination generation and cost evaluation: from... Try selecting from each node in turn. to Given a combination of nodes, the sum of the number of idle integer cards and the number of integer cards that can be moved within the combination is required. Each feasible combination is assigned a comprehensive cost score. ,in This represents the total number of tasks requiring relocation. The number of tasks that do not support checkpoint migration. Take 1.0, Set the weight to 10.0 (meaning the cost weight of a non-transferable task is equivalent to that of 10 transferable tasks).

[0140] Step 3: Optimal Output-Protection Relationship Mapping: Selecting... The smallest combination of nodes is used as the set of nodes in the target area. and in Internally, according to the principle of "uniform distribution across nodes", Each card slot is marked as Reserved slots in the system (avoid concentrating all reserved slots on the same node to prevent single-node failures from affecting scheduled tasks). The pre-scanning stage is for estimation. The enumeration performed reuses the above algorithm but only outputs... Instead of storing layout details, it prioritizes data and ensures that pre-scanning overhead remains within a reasonable range. When the number of nodes in the domain exceeds 500, a greedy approximation mode is automatically activated: starting with the node with the most currently available whole cards, adjacent nodes are added incrementally until the condition is met. This addresses the need to replace global enumeration and ensure the real-time performance of the algorithm in ultra-large clusters.

[0141] The set with the fewest required migration tasks and the fewest migration tasks that do not support checkpoint migration is selected from the candidate set as the target area node set. ; The internal plan reserves space for writing appointment tasks This serves as the basis for setting shadow locks in Module 5. The selection should prioritize minimizing the number of migrations rather than node load, because the goal at this stage is to achieve time-limited fulfillment rather than load balancing.

[0142] The migration list is generated according to the principle of "easy first, difficult later; small first, large later": the first priority for migration out is... Internally meet the threshold The second priority is to relocate out of the absorption zone. The internal support for large-scale checkpoint migration tasks targets qualified new checkpoints in the general area that have not been [relocated / relocated]. Reserved card slots;

[0143] The third priority is to handle tasks that do not support checkpoint migration. These tasks are not forced to migrate; instead, the target layout is adjusted to bypass checkpoint occupation. If bypassing fails, an escalation mechanism is activated. Each migration action records the source checkpoint, destination checkpoint, estimated migration duration, and latest completion time. After the inventory is generated, the total number of checkpoints released by all migration actions is reviewed. The sum of the number of available whole cards must be no less than Otherwise, select a new target layout.

[0144] Set the acceptance time ,in Allow 5 minutes for acceptance testing (default value). Starting from the front and working backwards: The latest completion time of the last migration action is... Subtract the migration margin, which is the average migration time per task. Each preceding action is ordered according to the list and its degree of parallelism. Arrange the latest completion times of each item in order. The latest start time of the first action must not be earlier than the time the plan is issued; otherwise, the migration parallelism should be increased. The value is used to compress the total execution time of the plan or trigger an upgrade. The completed inverted plan is then sent to the migration executor, and... and Synchronize to Module 5 to start protection.

[0145] The migration executor proceeds according to a milestone timetable, with the number of migrations executed concurrently at any given time not exceeding [a certain threshold]. Automatic retry is performed when a single migration fails, with a maximum of 2 retries. If the migration still fails, the task is marked as a migration obstruction and a local replanning is triggered immediately.

[0146] Choose an alternative destination for the task, or adjust the order of subsequent actions without breaking the milestone; escalate the process if the blocked task cannot be resolved locally. The duration of business interruption for the migrated task during the migration is controlled by the checkpoint switching window, and the window duration is pre-registered according to the task specifications and made public to the tenant.

[0147] The checkpoint migration and video memory data copying are implemented using techniques known in the field, and this solution uses them as a basic approach.

[0148] Additional information on the internal implementation of the migration executor:

[0149] The migration executor adopts a state machine-driven architecture. The lifecycle of each migration task is divided into six states: pending deployment, source-end freezing, GPU memory copying, destination-end recovery, consistency verification, and source-end release. The system embeds a checkpoint agent in the source-end task process. This agent is based on the native checkpoint interface of the deep learning framework (PyTorch or TensorFlow) that the task depends on. After receiving the freeze command, it serializes the model parameters, optimizer state, and training iteration count in the current GPU memory into a contiguous memory block, and calculates the Mason slew checksum of the memory block as a data consistency benchmark.

[0150] The video memory copy uses the RDMA (Remote Direct Memory Access) protocol for data transmission via the InfiniBand network, with the maximum transmission bandwidth determined by the current tenant's quality of service level. After copying, the destination recovery agent loads the data blocks into the target card's video memory and performs checksum comparison. If the comparison matches, the source card's video memory is cleared and returned to the free pool; if the comparison fails, incremental retransmission is triggered, retransmitting only the data fragments that failed the checksum comparison. The duration of task service interruptions during the entire migration process is controlled within the checkpoint switching window, which is pre-registered according to the task specifications and defaults to no more than 30 seconds. If the estimated interruption duration exceeds the window, the task is not included in the migration list and is instead handled by adjusting the target layout to avoid its occupancy. During execution, the migration executor continuously reports the current status and progress to Module 4, supporting subsequent milestone scheduling and upgrade determination.

[0151] The progress of the plan is verified at two checkpoints: 50% and 80% of the promised time has been consumed. The number of migrations to be completed is compared with the number of migrations actually completed. If the number is behind, it is upgraded step by step.

[0152] Level 1 upgrade, expanding the scope of the reorganization: adding vacant slots in adjacent nodes to relax the node constraints of the target layout;

[0153] Second-level upgrade to improve migration intensity: parallelism Temporarily increase (default increase to 6), and open a bandwidth priority channel for migrated traffic;

[0154] When a Level 2 upgrade is triggered, the system sends a temporary bandwidth request to the network control plane, temporarily increasing the CoS label of RDMA traffic involved in the target area migration task to 4 (higher than 3 for ordinary service traffic). After confirming that the current link load has not exceeded the warning threshold (port occupancy ≤ 90%), the control plane allocates a minimum bandwidth guarantee (30% of the total link bandwidth) for this type of traffic and enables the Explicit Congestion Notification (ECN) mechanism: once the queue depth exceeds the threshold, the switch marks the ECN bit in the forwarding packet. Upon receiving the mark, the migration executor actively reduces the sending window instead of directly dropping packets, thereby ensuring the migration progress while avoiding severe congestion of online services; after the upgrade is completed, the CoS label is restored to the default low priority level.

[0155] Level 3 upgrade, borrowing adjacent resource domains: Select a temporary takeover area from adjacent resource domains with compatible tags, agree on a return time with the neighboring domain and register the borrowing in the ledger, and complete the recovery before the return time. Level 3 upgrades do not change the commitment to the tenant's fulfillment time limit.

[0156] arrive Acceptance will be performed on time.

[0157] Verification Internal Press Is the number of reserved consecutive idle whole cards not less than [amount]? Are the available video memory slots of each card sufficient? Whether the node load level is compliant;

[0158] Upon successful acceptance, the shadow lock is released, the scheduled task is notified to enter and resources are bound, and management follows the normal lease process after task entry. If acceptance fails, a fallback measure is implemented: the remaining workload is re-estimated and a new fulfillment deadline is given; tenants are offered four options: extend the waiting period, reduce resource specifications, transfer to an adjacent resource domain, or cancel the reservation; the tenant's choice is recorded in the ledger; if the original deadline is exceeded due to system issues... The duration of the deviation is recorded in the performance deviation record and used as the basis for subsequent parameter calibration and capacity planning.

[0159] The complete plan ledger records: original suspension order, target layout candidate evaluation process, migration list and execution receipts for each action, checkpoint verification conclusions, upgrade actions and basis, acceptance conclusions, and fallback disposal results. The judgment log in Module 2 and the protection record in Module 5 are mutually indexed to ensure that the entire process of any appointment task from suspension to entry is traceable and reproducible.

[0160] Module 5, Linkage Protection Verification Module:

[0161] This module serves as the gatekeeper and confirmation stage for the execution of the reverse scheduling process, undertaking four responsibilities: protecting the target area during the execution period, maintaining the shadow lock and linking it with the gate access control; organizing the acceptance process; restoring normal access to the scheduled tasks and ordinary areas after acceptance; and configuring, managing, and auditing all process parameters, ledgers, and logs.

[0162] This module runs in parallel with Module 4: Module 4 is responsible for freeing up resources, while this module is responsible for ensuring that the freed-up resources are not taken away and that the freeing-up results can be verified.

[0163] The protection state is divided into three phases: activation, maintenance, and deactivation.

[0164] Start-up phase:

[0165] Received the set of nodes for the sorted target area from Module 4 Target layout Afterwards, All internal nodes are set to the protected state flag, according to A shadow lock is established for the reserved card slot, and a protection notification is pushed to Module 2. The notification includes... Shadow lock slot list, expected protection expiration time (take the acceptance time) Tightening coefficient With layout version number ;

[0166] From the effective date of this announcement, Module 2 will be judged strictly according to the rules for determining virtual occupancy during the protection period.

[0167] Maintenance phase:

[0168] During the protection period, three types of changes will be continuously monitored, namely... Internal node state changes, migration execution receipts, and planned upgrade announcements; when the target layout is adjusted due to the upgrade mechanism, shadow locks are added or deleted synchronously, and an incremental announcement is pushed to Module 2, carrying the new layout version number. The judgment of referencing the old layout version in Module 2 is invalidated and re-judged.

[0169] Relief Phase:

[0170] Protection is lifted in three scenarios: acceptance passed, plan suspended, or transfer to an adjacent resource domain as a fallback. The lifting order is as follows: first, release the shadow lock slots that are not occupied by the scheduled task; then, clear the protection status flag; and finally, push the lifting notice to Module 2. Slots that have been bound to scheduled tasks are directly converted to the officially occupied state without going through the idle state transition, to prevent other entry requests from preempting them the moment they are lifted.

[0171] Shadow locks operate at the card position level, with each lock recording its associated node, card number, and locking criteria (layout version number). The maintenance rules consist of five parts: locking time, expected unlocking time, and four other items.

[0172] Firstly, the shadow lock slot is presented as occupied for Module 2, but as free but unallocable for the space fragmentation index calculation of Module 1. That is, the index calculation is based on physical free status, while the admission determination is based on logical occupied status; these two methods are separate to ensure... Continuity and authenticity;

[0173] Secondly, shadow locks are not automatically released due to routine events such as task expiration or changes in node load levels; they are only released when protection is lifted or the layout is adjusted.

[0174] Third, the shadow lock slots are prohibited from being temporarily placed in any form, including intermediate placement for background tidying;

[0175] Fourth, if the protection period exceeds the expected release time and the acceptance has not yet started, the shadow lock will automatically enter the delayed confirmation state. This module will request the plan status from module four. If the plan has been terminated, it will be released immediately. If the plan is still being executed, it will be extended according to the new milestone time.

[0176] arrive At that time, this module organizes verification according to the acceptance criteria specified in Module 4, and adds two independent verifications: one is to verify the shadow lock list stored in this module as a benchmark. The verification process involves two main steps: first, checking the consistency between the actual number of consecutive idle cards and the lock record to prevent false alarms on the execution side; second, randomly checking the actual occupancy of the destination card slots after the migration task has been completed to confirm the true placement of the migrated cards. Verification results are categorized into three levels: passed, conditionally passed, and failed.

[0177] If the process passes, the access recovery process will begin; if the process passes under certain conditions (continuous resource blocks meet the standards but individual indicators deviate from the node load level threshold by less than 5%), the scheduled task will be notified to enter the site first, and the deviations will be registered and resolved by the background process within a specified period; if the process fails, the fallback process of module four will be triggered.

[0178] After acceptance, perform a three-step recovery:

[0179] The first step is to combine the appointment task with... Once a reserved slot is bound, the task status changes from reservation to entry, and the lease commencement time is the binding completion time.

[0180] The second step is to release the remaining shadow lock and unlock it. The protection state, the ordinary area access judgment restoration standard threshold and No longer multiplied by the tightening factor ;

[0181] The third step is to add the reservation queue. Subsequent tasks will receive progress notifications. If there are reusable continuous resources remaining after this cleanup, the next task in the queue that meets the conditions can directly reuse the acceptance conclusion and skip the cleanup phase to enter the process. If a scheduled task is not confirmed to enter within the confirmation time (default 15 minutes) after the binding notification is issued, its reserved slot will be released and it will be moved to the end of the queue for reordering.

[0182] During the protection period Internal node failure:

[0183] Immediately freeze migration actions involving the affected node. Shadow locks on the faulty node are retained but marked as unavailable. The notification module performs a four-way replacement node selection and reorders milestones. If the fault prevents the target layout from being achieved, proceed directly to the fallback process. If a migrated task abnormally exits during migration: release its source and destination slots, and incorporate the source slot into the aggregation resources. If a tenant actively cancels a reservation: remove its shadow lock share, and fill the reservation queue sequentially. If the time synchronization service malfunctions: suspend milestone timing and protection timing. After service recovery, align timings once based on actual elapsed time to avoid misjudgments caused by timing drift.

[0184] This module centrally manages all operating parameters:

[0185] Dual threshold and Tightening coefficient Fragmented debt cap Small task threshold Node load water level upper limit, absorption zone capacity water level and lag time Maximum number of leases within the district Transfer Parallelism Basic duration Maximum commitment duration , acceptance lead time .

[0186] The default values ​​for the above parameters are based on the following calibration logic and industry experience:

[0187] Upper limit of access increment Incremental triggering of consolidation :

[0188] The values ​​for both were determined through simulation playback of 30 days of historical scheduling logs from an enterprise-level AI resource pool (containing 200 A100 GPUs, with an average daily task throughput of approximately 1200). At that time, the average holding time of continuous resource blocks (≥8 whole cards) in the ordinary area increased from 4.2 hours before simulation to 7.8 hours; At that time, the average number of times targeted consolidation is triggered is approximately 3 to 5 times per day, and the consolidation overhead accounts for less than 5% of the total migration bandwidth. A value below 0.1 indicates that the cleanup is too frequent, while a value above 0.2 indicates that the waiting time for large tasks increases significantly.

[0189] Fragment Debt Limit Based on the background, the tolerance setting for interference with business traffic is determined. Simulation shows that when... When the accumulated value reaches 0.3, background reorganization is triggered. During the migration, the impact on the latency of the online inference task P99 is less than 3%. If the upper limit is set to 0.5, the fragmentation of the ordinary area has been severely worsened, and the migration amount when reorganization is triggered is about twice that at 0.3.

[0190] Absorption zone capacity water level and and latency Minutes: Set based on the average lifecycle distribution of small tasks within the absorption zone. Simulation statistics show that the average lease duration of small tasks is 47 minutes. A 10-minute hysteresis duration can filter out short-term burst peaks (peaks lasting less than 5 minutes), and asymmetric hysteresis (reduced to...) Ensure that capacity is released smoothly after demand declines, and avoid expanding or contracting capacity more than 5 times a day.

[0191] Transfer Parallelism Limited by the total bandwidth of the InfiniBand link between the source and destination (default 200Gbps), the throughput of a single task reaches its peak when there are 3 concurrent migration tasks. When there are more than 3, the total throughput no longer increases linearly due to link congestion, and the service interruption time is extended.

[0192] Promised duration limit Hours: In accordance with industry SLA practices, the acceptable waiting window for tenants to enter a training task is generally no more than 4 hours. If this threshold is exceeded, tenants tend to cancel the task or switch to an alternative resource domain.

[0193] Parameters are distributed in the form of versioned configuration sets. Each change generates a new configuration version and records the person making the change, the time of the change, and the scope of effect. Historical judgments and historical plans are all replayed according to the configuration version at that time to ensure that the system behavior is reproducible.

[0194] This module will index five types of records in a unified manner:

[0195] Module 1 contains the versioned accounting ledger; Module 2 contains the judgment log; Module 3 contains the zoning adjustment records; Module 4 contains the planning ledger; and this module contains the protection and acceptance records. Any scheduled task can be retrieved using the task identifier, providing a complete record from suspension, plan creation, protection initiation, migration execution, fulfillment acceptance, to access restoration; any historical moment can be retrieved using the accounting version number. The fragmentation state and judgment logic at the time of replaying the configuration version. The retention period for trace data shall not be less than the longer of the longest lease period and the audit requirements.

[0196] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A dynamic load balancing and automated operation and maintenance management system for an enterprise-level AI resource pool, characterized in that, include: The fragment increment calculation module is used to collect and register the running data of nodes and rented tasks within the resource domain, calculate the spatial and temporal fragmentation index, perform hypothetical allocation on candidate placements, and output the comprehensive fragment increment and calculation version number. The dual-level access gate module is used to make three-level judgments based on the comprehensive fragment increment and the accounting version number, according to the upper limit of access increment and the consolidation trigger increment: if it does not exceed the upper limit of access increment, it is allowed; if it is between the two, those that meet the threshold of small tasks are reassigned to the absorption area, and the rest are placed in the ordinary area and counted as fragment debt; if it exceeds the consolidation trigger increment, a pending order is generated, transferred to the reservation queue, and targeted consolidation is triggered. The absorption area collection and scheduling module is used to accept tasks that meet the threshold for small tasks and those on the migration list that are reassigned according to the three-level judgment and are designated to the absorption area. The tasks are collected and placed according to the compact principle and can be flexibly expanded and contracted according to the capacity level of the absorption area. The appointment scheduling module is used to receive pending orders, calculate the fulfillment deadline based on the ledger data, select the target layout, generate a migration list and set the acceptance time, and promote the migration by setting milestones in reverse order. The linkage protection verification module is used to establish shadow locks on the target area according to the target layout and notify strict judgment. At the time of acceptance, it verifies whether the continuous resources meet the requirements. If they do, it binds the scheduled task and removes the shadow lock to restore access.

2. The dynamic load balancing and automated operation and maintenance management system for an enterprise-level AI resource pool according to claim 1, characterized in that, The method by which the fragment incremental accounting module calculates the spatial fragmentation index is as follows: Full Card Level Space Fragmentation Index ,in, This represents the total number of idle cards within the resource domain. The maximum number of idle integer cards per node within the resource domain, when hour Set the empty state flag of the entire card; Memory fragmentation index ,in, This represents the total amount of free video memory within the resource domain. This represents the maximum free video memory on a single card. The task of applying for a full card is to use The task of requesting video memory allocation adopts Hybrid tasks that combine the needs of full card and memory partitioning and The larger of the two.

3. The dynamic load balancing and automated operation and maintenance management system for an enterprise-level AI resource pool according to claim 2, characterized in that, The calculation methods for time fragmentation index and comprehensive fragmentation increment are as follows: Time Fragmentation Index ,in, As a time perspective window, For the width of the anchor slot, This refers to the number of tasks that fall within the time lookout window at their due date. The number of non-empty anchor slots occupied at the expiration time; Assume the allocation is performed as follows: Remove the candidate cards from the free set, deduct the corresponding card's free memory according to the single card's memory requirement, and set the task's estimated expiration time. Included in the set of maturity times, where, For the current moment, For the duration of the lease application; Recalculate the increase in space debris before and after placement With the increase of time fragments Calculate the overall fragmentation increment ,in, and As weight, and .

4. The dynamic load balancing and automated operation and maintenance management system for an enterprise-level AI resource pool according to claim 1, characterized in that, In the three-level judgment: The threshold for the mini-task is the required number of whole cards. And single-card video memory requirements ,in, This refers to the total memory capacity of a single graphics card. Large tasks whose increments fall between the upper limit of the entry increment and the increment triggered by consolidation are placed in the normal zone and marked with an observation tag; their increments are included in fragment debt. When a task marked with an observation exits or expires, a space fragmentation index recalculation is triggered, and its original occupied location is listed as a priority merging object in the next background cleanup. when The cumulative amount exceeds the fragment debt limit. Background consolidation is triggered at any time. Background consolidation selects objects to merge in descending order of their contribution to fragmented debt. Migration execution reuses the migration executor and runs with low bandwidth priority, without setting shadow locks or redemption time limits. Upon completion, [the following will occur]. Reset to zero.

5. The dynamic load balancing and automated operation and maintenance management system for an enterprise-level AI resource pool according to claim 4, characterized in that, Also includes: Cards marked as reserved in the target layout establish shadow locks, and the placement of cards involving shadow locks is considered infeasible. In determining the placement of remaining cards within the target area, both the upper limit of the admission increment and the increment triggered by the reorganization are multiplied by a tightening coefficient. ,in, A positive number less than 1; All new tasks arriving during the protection period that meet the requirements for minor tasks will be reassigned to the absorption area. Each determination also references the calculated version number. With layout version number If any version changes, it will be deemed invalid and re-evaluated.

6. The dynamic load balancing and automated operation and maintenance management system for an enterprise-level AI resource pool according to claim 1, characterized in that, After receiving reassigned small tasks, the absorption area aggregation and scheduling module aggregates and places them within the absorption area according to the principle of compactness: Choose the node with the most occupied cards in the absorption area for placement, and within the same node, prioritize the card with the smallest difference between free video memory and demand. Initially, nodes are selected in descending order of the number of small tasks running on each node and assigned to the absorption zone. Nodes that leave the zone return to the normal zone after all their tasks have ended naturally or have been moved out. Tasks that have not yet expired are not expelled. Absorption region occupancy rate ,in, This represents the total number of cards already occupied within the absorption area. The total capacity of the absorption area; when The water level is higher than the upper limit of the absorption zone capacity and the duration exceeds the hysteresis time. When performing expansion, The water level is below the absorption zone capacity and has remained below it for more than [time period missing]. Scaling down is performed at that time.

7. The dynamic load balancing and automated operation and maintenance management system for an enterprise-level AI resource pool according to claim 1, characterized in that, In the migration list generated by the appointment inverted sorting module, the destination of the migrated tasks that meet the small task threshold is designated as the absorption area. When the migration scale exceeds the current reserve of the absorption area, the absorption area collection and scheduling module performs emergency expansion according to the migration gap capacity. The number of nodes is allocated according to the amount of the gap capacity. If the gap is still insufficient after expansion, the difference is migrated to the card slots in the ordinary area that have not been reserved by the target layout and is counted as fragment debt. After the completion of the sorting and acceptance process, the small tasks that were temporarily placed in the general area will be gradually returned to the absorption area during low-load periods.

8. The dynamic load balancing and automated operation and maintenance management system for an enterprise-level AI resource pool according to claim 7, characterized in that, The method for calculating the redemption period is as follows: Delivery time limit Promised duration ,in, For the time of suspension, To ensure the duration, This serves as the baseline duration for planning and conservation initiation. To estimate the number of tasks to be migrated based on the initial target layout, the initial target layout is obtained by pre-scanning the candidate enumeration to select the candidate set with the fewest tasks to be migrated out. To transfer parallelism, The average migration time for a single task is calculated by classifying the amount of memory data migrated in a single task and taking the average migration time of completed tasks in the same category. The promised duration shall not exceed the maximum promised duration. If the calculation result exceeds the limit, then... Commit to and enforce the upgrade mechanism.

9. The dynamic load balancing and automated operation and maintenance management system for an enterprise-level AI resource pool according to claim 8, characterized in that, The scheduled reverse scheduling module sets the acceptance time. ,in, Allow sufficient lead time for acceptance; Starting from the acceptance time and working backwards, the latest completion time of the last migration action is the acceptance time minus the migration margin. The migration margin is the average migration time of a single task. Each preceding action is arranged forward according to the migration list order and the degree of migration parallelism to determine its own latest completion time. The latest start time of the first action should not be earlier than the planned issuance time; otherwise, the migration parallelism will be increased or an upgrade will be triggered. If a single migration fails, it will be automatically retried. If it still fails, a local replanning will be triggered to select an alternative destination or adjust the order of subsequent actions. The progress of the plan is verified at the preset checkpoints. If the progress is behind schedule, it is upgraded step by step, in the following order: expanding the scope of the cleanup, increasing the parallelism of the migration, and borrowing adjacent resource domains.

10. The dynamic load balancing and automated operation and maintenance management system for an enterprise-level AI resource pool according to claim 1, characterized in that, In the linkage protection verification module: The shadow lock card is shown as occupied, and the space fragmentation index calculation shows it as free but unallocable; During acceptance, the actual number of consecutive idle cards in the target area is checked and organized based on the shadow lock list. After acceptance, the linkage protection verification module binds the reservation task to the reserved card slot in the target layout. The bound card slot is directly converted to formal occupation without going through the idle state, the residual shadow lock is released and the standard threshold is restored, and a progress announcement is sent to the subsequent tasks in the reservation queue. If there are reusable remaining continuous resources, the subsequent task reuse acceptance conclusion is directly entered.

Images