A pipeline idle awareness and communication-coordinated scheduling method for heterogeneous GPU clusters

By implementing pipeline idle awareness and communication-coordinated scheduling on heterogeneous GPU clusters, resource layout and scheduling strategies are optimized, solving the problems of low computing power utilization and communication conflicts in heterogeneous clusters, and achieving efficient parallel training.

CN123123438APending Publication Date: 2026-11-17CHONGQING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611267772.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-20
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing heterogeneous GPU clusters suffer from low computing power utilization, multi-task communication conflicts, and difficulty in adapting to dynamic loads during deep learning training, leading to pipeline bubbles and network congestion.

Method used

By collecting fine-grained features of single tasks and cluster network status, normalized dimensionless pipeline bubble loss and network contention loss are established, a joint scoring function is constructed, resource layout optimization and scheduling strategy adjustment are performed, and two-layer adaptive collaborative scheduling is achieved.

Benefits of technology

While ensuring the safety of GPU memory, we reduce transmission latency and congestion rate to achieve high throughput and low idle collaborative parallel training, thereby improving the utilization rate of cluster computing power.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123438A_ABST
    Figure CN123123438A_ABST
Patent Text Reader

Abstract

This application relates to the field of resource allocation technology, and in particular to a pipeline idle awareness and communication cooperative scheduling method for heterogeneous GPU clusters. The method includes: calculating normalized dimensionless pipeline bubble loss and network contention loss respectively; constructing a joint scoring function to measure the comprehensive cost of task combinations on a specific shared link; and constructing a candidate resource layout set by combining a closed-loop conflict filtering mechanism; selecting the optimal resource layout from the candidate resource layout set; calculating the global communication offset; and performing two-layer distribution and closed-loop adaptive adjustment of the scheduling strategy. This application optimizes pipeline execution and cluster communication resources through a two-layer adaptive scheduling strategy, reducing computing power idle loss and network transmission latency while ensuring memory operation safety. It can adapt to heterogeneous hardware and varied training tasks without manual parameter configuration, effectively improving the overall training efficiency and resource utilization stability of the cluster.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of resource allocation technology, specifically to a pipeline idle awareness and communication-coordinated scheduling method for heterogeneous GPU clusters. Background Technology

[0002] As the number of parameters in deep learning models reaches hundreds of billions or even trillions, and the scale of training data explodes, the computing power and physical memory limits of a single GPU node in a distributed parallel computing architecture based on GPU clusters are insufficient to support the storage of parameters, gradient updates, and activation value temporary storage for the entire model. To address this, the industry commonly employs pipeline parallelism (PP) technology, which vertically partitions the model computational graph of a single training task and deploys it across multiple GPU nodes, achieving data and gradient synchronization through cross-node network communication. However, when this technology is deployed in a real, complex cluster environment, the combined effects of heterogeneous hardware and multi-task concurrency present two major challenges: first, node computing power mismatch leads to severe computing power idleness within a single task, known as a pipeline bubble; second, the communication windows of multiple tasks on the shared network link overlap in the time domain, causing severe bandwidth contention and network congestion.

[0003] To address the aforementioned performance bottlenecks, existing improvement solutions mainly fall into the following two categories:

[0004] (1) Single-task static pipeline scheduling scheme based on micro-batch overlap. This type of scheme attempts to mask communication overhead by fixing the micro-batch launch timing. However, this type of scheme is limited by the ideal assumption of hardware homogeneity and network conflict-free, and lacks fine-grained perception and adaptive overlap capability for computational deviations and margins of heterogeneous nodes, resulting in a large number of bubbles in non-bottleneck nodes, which limits the utilization rate of cluster computing power.

[0005] (2) Multi-task cluster resource scheduling scheme based on coarse-grained allocation. This type of scheme attempts to achieve multi-task isolation and resource allocation through static allocation of GPU node quotas and video memory capacity during the task submission phase. However, most of these methods abstract the physical network as a conflict-free ideal medium, lacking both time-domain awareness of shared link conflicts and peak-shifting coordination capabilities, and ignoring the dynamic occupation of activated video memory by the pipeline warm-up mechanism, which can easily induce node video memory out-of-memory (OOM) crashes. Summary of the Invention

[0006] In view of this, this application discloses a pipeline idle awareness and communication-coordinated scheduling method for heterogeneous GPU clusters to solve the problems of low computing power utilization, multi-task communication conflicts, and difficulty in adapting to dynamic loads in existing heterogeneous cluster deep learning scheduling methods, including:

[0007] Step 1: Collect fine-grained features of a single task and establish a normalized dimensionless pipeline bubble loss.

[0008] Step 2: Collect cluster network status and calculate link network congestion loss, and establish normalized dimensionless network contention loss.

[0009] Step 3: Jointly model pipeline bubble loss and network contention loss, and construct a joint scoring function to measure the comprehensive cost of task combination on a specific shared link;

[0010] Step 4: Based on the joint scoring function and closed-loop conflict filtering mechanism, construct a set of candidate GPU resource layouts;

[0011] Step 5: Select the optimal resource layout from the candidate GPU resource layout set and calculate the global communication offset of the optimal resource layout;

[0012] Step 6: Based on the global optimal resource layout and the communication offset of all tasks, perform two-layer distribution and closed-loop adaptive adjustment of the scheduling strategy.

[0013] This application provides a pipeline idle awareness and communication cooperative scheduling method for heterogeneous GPU clusters. Through the design of a two-layer adaptive cooperative mechanism of cluster-level macro-network peak shaving and task-level micro-pipeline overlap, pipeline bubble loss between heterogeneous nodes is eliminated while ensuring the safety of GPU physical memory. This can reduce the transmission latency and congestion rate of shared links and achieve high throughput, low idle time and high stability cooperative parallel training under dynamic multi-task load. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating the pipeline idle awareness and communication-coordinated scheduling method for heterogeneous GPU clusters in an embodiment of this application.

[0015] Figure 2 This is a schematic diagram of the system implementation in the embodiments of this application;

[0016] Figure 3 This is a flowchart illustrating the algorithm implementation in the embodiments of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, features, and advantages of this application clearer and to facilitate a better understanding of the technical solutions of this application by those skilled in the art, the following detailed description of this application is provided in conjunction with the accompanying drawings and embodiments.

[0018] This embodiment includes a pipeline idle awareness and communication-coordinated scheduling method for heterogeneous GPU clusters, such as... Figure 1 As shown, it includes:

[0019] Step 1: Collect fine-grained features of a single task and establish a normalized dimensionless pipeline bubble loss.

[0020] Step 1.1: Establish the training task set, model computation graph, and GPU node set.

[0021] For a system with M concurrent training tasks and N GPU nodes, establish a set of concurrent training tasks. GPU node set ,use Indicates task The model calculation diagram. For Perform topological sorting, Divided into several pipeline stages, forming a computational graph stage set. ,in Indicates task The number of stages in the production line.

[0022] Furthermore, a mapping set from training tasks to GPU nodes is established. and the set of mappings from pipeline stages to GPU nodes. This is used to characterize the deployment affiliation of tasks and phases on physical hardware nodes.

[0023] Step 1.2: Average computation time and task iteration cycle during the data acquisition phase, and calculate the native communication window of the task.

[0024] The average calculation time for the aforementioned stage is calculated using the following formula: ,in, This represents the average calculation time for each stage. Indicates task The Average computation time for each stage Indicates task The number of pipeline stages. Furthermore, the computational stages. Absolute deviation relative to the average stage computation time of the task The formula is: .

[0025] The task iteration cycle is determined through runtime observation. At the beginning of a complete training iteration and the end time Confirmed, the formula is .

[0026] The native communication window is obtained by finding the union of the communication time intervals of all micro-batches and pipeline stages, and then mapping them uniformly to a single training iteration cycle. The formula is as follows:

[0027]

[0028] in, Indicates task The native communication window set, Indicates the micro-batch number. Indicates the stage number of the production line. Indicates task In micro-batch and stages The half-open physical time interval during which the next communication event lasts, i.e., the phase communication time. This means mapping absolute time to a single training iteration period. The relative time periods within; the union is used to eliminate overlapping overhead and characterize the task. Total native communication usage window within a single cycle.

[0029] Step 1.3: Calculate and normalize the pipeline bubble loss to obtain the normalized dimensionless pipeline bubble loss. .

[0030] After obtaining the effective computation time and hardware computing power at each stage, the bubble loss of the task in pipeline parallelism is further quantified. For the task... Let the effective computation time of its slowest bottleneck pipeline stage be... By analyzing the idle time of each stage and the effective computing power of the GPU nodes deployed in each stage... Perform weighted summation to construct the task. Absolute pipeline bubble loss The formula is:

[0031]

[0032] Furthermore, the absolute pipeline bubble loss is normalized by range to eliminate the impact of scale differences between different tasks on global optimization, thus obtaining the task... Normalized pipeline bubble loss The formula is:

[0033]

[0034] in, and Representing task sets respectively The maximum and minimum absolute bubble losses in the middle, when At that time, the normalization result is preset to 0.

[0035] Furthermore, all the task-related feature data in step 1 together constitute a concurrent task model, which is used to describe the task concurrent behavior features of the system.

[0036] Step 2: Collect cluster network status and calculate link network congestion loss, and establish normalized dimensionless network contention loss.

[0037] Step 2.1: Establish a network topology model for the heterogeneous cluster.

[0038] Let the physical network link set be... The connection mapping relationship between nodes and network links is as follows: The cluster network topology is abstractly described as a triple. ;in, Indicates the cluster network structure, This indicates the mapping relationship between the routes and physical links that data packets take when they are transmitted between GPU nodes.

[0039] Step 2.2: Collect the link capacity and actual traffic carried by each physical link in the network topology model, and calculate the link utilization rate. .

[0040] For physical links Let its effective transmission capacity be... The total demand for all transmitted data within the set observation period is , compute link Link utilization The formula is: .

[0041] Step 2.3: Based on the GPU nodes deployed at each stage of the task pipeline and their data transmission routes, establish a mapping relationship between tasks and network links. Calculate the communication window .

[0042] The mapping relationship between tasks and network links is given by the task... The set of network links traversed during execution is as follows: , will the task In shared links Map all communication events on the pipeline and determine the task based on the GPU nodes deployed at each stage of the task pipeline and their data transmission routes. Which physical links will the communication flow specifically pass through? Establish a mapping set from task to physical link. ; Computational task In shared links The communication window for all communication events on the device. The formula is:

[0043]

[0044] Where k=(m,s) is the communication event sequence number corresponding to the combination of micro-batch m and pipeline stage s. Indicates task In micro-batch and stages The half-open physical time interval during which a communication event lasts, i.e., the phase communication time.

[0045] Step 2.4, according to the communication window Calculate the communication overlap duration of the shared link .

[0046] For those sharing the same physical link Any two concurrent tasks and , and ,right and exist Find the intersection of the communication time windows and calculate the overlap duration of the communication windows in the time dimension. The formula is:

[0047]

[0048] Step 2.5: Quantify and normalize the link network congestion loss to obtain the normalized dimensionless network contention loss. .

[0049] Calculate the shared link by combining communication overlap duration and link utilization. The above task and Original network congestion losses caused by competing for bandwidth :

[0050]

[0051] in, This represents the preset link congestion threshold, preferably... , This indicates the link over-utilization rate, which occurs only when the total data demand of the link exceeds the effective capacity of the link (i.e., The excess portion is penalized only when the link utilization is below the threshold; no congestion penalty is generated when the link utilization is below the threshold.

[0052] To eliminate the influence of dimensions and make the network contention loss comparable to the pipeline bubble loss in step 1, a unified reference time period is introduced. Normalizing the original network congestion loss yields the dimensionless network contention loss. The formula is:

[0053]

[0054] in, This represents a unified reference time period, which can be taken as the maximum value of the iteration periods of all tasks within the cluster. The link observation period, or a specified baseline time period, is used. Normalized metrics can intuitively quantify the severity of network contention on shared links.

[0055] Step 3: Jointly model pipeline bubble loss and network contention loss, and construct a joint scoring function to measure the comprehensive cost of task combination on a specific shared link.

[0056] Step 3.1: Calculate the communication window occupancy ratio of the link. .

[0057] To quantify the time-domain occupancy of a single task on a specific shared link, the communication time window calculated in step 2.3 is used. Mapping to physical link On the timeline, construct a time-domain mapping set from the task communication time window to the link. Further computational tasks In shared links Normalized communication window occupancy ratio The formula is:

[0058]

[0059] in, Indicates task In shared links The first One communication time window, Indicates the physical duration of the window. Indicates task The actual training iteration cycle; Intuitively reflects the task Link occupied The higher the proportion of time occupied, the greater the possibility of time-domain conflicts with other tasks.

[0060] Step 3.2: Adjust the communication window occupancy ratio As a weight for pipeline bubble loss, and combined with network contention loss, a joint scoring function is constructed, the formula is:

[0061]

[0062] in, Indicates task and In shared links Network contention losses and Representing tasks and Production line bubble loss, This indicates the weight of network communication competition loss. This represents the idle loss weight of the pipeline, and the weight relationship satisfies... ; Indicates task The proportion of communication window occupied. Indicates task The proportion of communication window space occupied.

[0063] In a physical sense, The larger the value, the more likely the task combination is on a shared link. The higher the overall resource waste resulting from the combination of online competition and production line bubbles, the greater the waste. The smaller the value, the more suitable the task combination is for sharing the corresponding link in terms of time domain and computing power matching. For any candidate resource layout... By accumulating the joint score on its task-link mapping edges The global joint cost of the candidate layout can be calculated. In the scheduling decision, the resource layout scheme with the smaller global cost and the temporal closed-loop congruence constraint is retained first to realize the guidance of candidate GPU resource layout.

[0064] Furthermore, considering the dynamic switching between network congestion and GPU computing power bottlenecks at different stages during training, an adaptive feedback mechanism is introduced to weight the network communication contention loss. Weight of pipeline idle loss Perform cross-cycle adjustments, the cross-cycle adjustments including: Let the first... The network congestion observations within each scheduling cycle are as follows: GPU computing power idle time and pipeline bubble observations are as follows Then the first The weights for each scheduling cycle are updated as follows:

[0065]

[0066]

[0067] in, This indicates the preset weight update step size. This represents an interval clipping function, used to clip the independent variable... Limited to the range [0,1]. When the cluster observes increased network congestion (i.e., ... The system automatically increases the network weight. Suppress mapping schemes that suffer from severe network contention or excessive bandwidth consumption; when an increase in pipeline bubbles is observed (i.e. When ), the computing power weight will be adjusted. Updated to Prioritize ensuring the balance of computing power in the pipeline.

[0068] Step 4: Based on the joint scoring function and closed-loop conflict filtering mechanism, construct a set of candidate GPU resource layouts.

[0069] Step 4.1: Generate candidate GPU resource layout schemes based on the network topology model, cluster physical topology, and concurrent task model. .

[0070] The network topology model is the heterogeneous cluster network topology model established in step 2.1, and the concurrent task model is established in step 1. A complete candidate layout scheme. Defined as a set of quaternions, the formula is:

[0071]

[0072] in, This refers to the set of mappings from training tasks to GPU nodes established in step 1.1. This refers to the set of pipeline stages to GPU nodes established in step 1.1. This is the set of mappings from task communication flows to physical network links established in step 2.3. The time-domain mapping set from the task communication time window to the link established in step 3.1.

[0073] In the candidate layout scheme Under the mapping framework, according to Identify concurrent task pairs sharing the same physical link, and based on Extract the degree of overlap of communication windows of each task in the link time domain; then, apply the joint scoring function established in step 3. This is applied to the mapping relationship, providing a data foundation for the subsequent bipartite graph edge weight assignment in step 4.2 and the global cost accumulation in step 5.2.

[0074] Step 4.2: Combine the scoring function As a fundamental metric for calculating edge weights, a task-link bipartite graph is constructed to characterize the mapping relationship between tasks and network links. The formula is:

[0075]

[0076] by As a basis for edge connection Represents the set of concurrent task nodes ( ), Represents the set of shared network link nodes ( ), This represents the set of mapping edges between tasks and links, derived from the set of physical network links. get, Let represent the set of edge weights.

[0077] The set of edge weights, when the task... Use shared links in the communication path In a bipartite graph, for connection tasks... With Link edge , border rights The calculation formula is:

[0078]

[0079] in, Indicates task With all other concurrent tasks on this link The cumulative joint cost generated by (i.e., competing tasks sharing the link); Represented as a shared link The above task and The joint cost score arising from joint occupancy.

[0080] Step 4.3, Calculation The relative communication offset of the internal shared link.

[0081] When the task With the task Shared link In order to achieve time-domain peak shifting of the communication window, the two are defined in the link. Relative communication time offset as follows:

[0082]

[0083] in, and Representing tasks With the task Global communication time offset, Indicates that by link The relative phase difference is determined by the communication window boundary, data transmission direction, and local execution phase.

[0084] Step 4.4: Use Breadth-First Search (BFS) to traverse the map. Detect the closed-loop path in the graph. For each closed loop, based on the relative communication offset Perform congruence consistency check; if the congruence constraint is not met, discard the current candidate resource layout; all candidate resource layouts that pass the congruence constraint constitute the candidate GPU resource layout set.

[0085] Loop closure detection and congruence consistency constraint filtering are used to avoid conflicting communication offset requirements for the same task from different shared links; congruence consistency verification traverses all links along the closed loop, accumulating the relative communication offsets to ensure that the total offset increment is 0, as shown in the formula:

[0086]

[0087] in, This represents the relative offset obtained after local time-series solution. This represents the verification period of the closed loop, which can be taken as the least common multiple of the iteration periods of related tasks within the closed loop or a unified reference period. The failure of congruence constraints indicates a candidate layout. If there is an irreconcilable timing conflict, the candidate resource allocation should be discarded directly.

[0088] Furthermore, during the traversal of the bipartite graph, when the same task is repeatedly visited due to traversing multiple different shared links, an offset mapping table for the task is maintained. When the task node is visited again, the newly calculated offset is applied to the task's training iteration cycle. The modulo operation is performed and compared with the offset already recorded in the table; the process continues downward only if the two are consistent, thus strictly ensuring that the same task will not obtain multiple conflicting communication offsets in the global scheduling.

[0089] Step 5: Select the optimal resource layout from the candidate GPU resource layout set and calculate the global communication offset of the optimal resource layout.

[0090] Step 5.1: Construct multi-dimensional hardware resource constraints, filter the candidate resource layout set, and obtain the feasible resource layout set. .

[0091] Building upon the closed-loop congruence consistency filtering in step 4.4, this step further introduces multi-dimensional physical hardware capacity constraints. Candidate Layout To be included in the feasible set, the following conditions must be met simultaneously. :

[0092] (1) GPU computing power resource capacity constraint: any GPU node The total computation time for a single iteration of all pipeline stages deployed on the node must not exceed the effective computing power budget of that node. That is... ,in This represents the average calculation time for each stage (obtained in step 1.3). Indicates a unified scheduling cycle. Indicates the available computing power coefficient for a node (value range is...). This is used to reserve time overhead for underlying scheduling and context switching.

[0093] (2) Network link bandwidth capacity constraint: any physical link The total amount of communication data from all concurrent tasks carried on the link must not exceed the effective transmission capacity of the link. That is... ,in Indicates task In the link Total communication volume in a single iteration Indicate the effective capacity of the link (step 2.2).

[0094] (3) Closed-loop timing congruence consistency constraint (inherited from step 4.4): Candidate layouts inherit the congruence verification results of step 4.4, and any closed-loop path Must meet That is, the global communication offset calculation result does not have any timing contradictions; since conflicting layouts have been calculated and filtered out in step 4.4, all candidates in the candidate resource layout set naturally satisfy the congruence equation. The inherited constraint, together with other constraints, jointly defines the feasible set. .

[0095] Step 5.2: Calculate the set of feasible layouts Global joint total cost for each candidate layout in To minimize For the target By filtering, the optimal resource allocation can be obtained. .

[0096] For the set of feasible layouts Each candidate layout in Let the set of edges of its corresponding bipartite graph be... , Mapping edges The layout cost edge weight is Global Joint Total Cost Defined as the cumulative sum of all layout cost edge weights, the formula is:

[0097]

[0098] Based on this, with the goal of minimizing the total global joint cost, we start from the feasible set. The optimal global resource layout with the lowest overall resource waste and network contention cost is selected from the data. The formula is: .

[0099] Step 5.3: Based on optimal resource allocation Corresponding bipartite graph Construct a global communication time offset vector for the task cluster to guide global peak-shaving control of the communication time domain window for each task in the cluster.

[0100] In the set of concurrent task nodes Choose any benchmark task and will The reference communication offset is preset to zero, that is... Based on the baseline task node Starting from the optimal bipartite graph Perform a breadth-first search (BFS) graph traversal; to strictly distinguish between layout cost and edge weights. With time dimension parameters, define For the task via shared link The local time phase generated by the task; as the traversal process extends along the edge, the task... via shared link Reaching adjacent tasks At that time, the task is derived based on the local time phase. Global communication offset The formula is:

[0101]

[0102] in, Indicates task Global communication offset, and Representing tasks With the task In shared links Local time phase on, Indicates task The actual training iteration cycle; for task nodes that are repeatedly visited during graph traversal, the derived new offsets will be used to... The modulus is taken and a consistency check is performed with the offset already recorded by the node to ensure the uniqueness and consistency of the global communication offset calculation result.

[0103] Finally, traverse the bipartite graph. Solve the set The communication offsets of all concurrent tasks are calculated, and a global communication time offset vector for the task cluster is constructed. This guides the global peak-shifting control of the communication time domain window for each task in the cluster.

[0104] Step 6: Based on the global optimal resource layout and the communication offset of all tasks, the global optimal resource layout scheduling strategy is distributed in two layers and adaptively adjusted in a closed loop: The upper layer distributes the off-peak communication window to each task to achieve off-peak communication in the time domain across tasks; the lower layer adaptively adjusts the number of preheating micro-batches and the execution order of micro-batches in the H-1F1B pipeline within the task to maximize the overlap between computation and communication, and forms a closed-loop iterative optimization through runtime monitoring.

[0105] Step 6.1: Global communication offset distribution and off-peak communication window shifting execution.

[0106] Task execution agent receives tasks Global communication time offset , will the task native communication window collection The entire system is shifted along the time axis to construct a set of off-peak communication windows. The off-peak communication window is then distributed to the corresponding pipeline stages and the underlying communication scheduling queues to guide precise time-domain transmission of data. The formula for the set of off-peak communication windows is:

[0107]

[0108] Step 6.2: Calculate the hidden margin of stage communication and solve the preheating micro-batch increment based on the hidden margin of stage communication.

[0109] To eliminate pipeline bubbles caused by inter-stage communication waits, the H-1F1B pipeline scheduling strategy is adaptively adjusted within the task: Let the pipeline data collected in step 1.3 be... Each stage ( The average computation time for micro-batch processing is The time consumed for inter-node communication between stages is , Given the total number of pipeline stages for a task, the computation time of the bottleneck stage is defined as... .

[0110] stage The computational time margin available for hidden communication is The computational cost per micro-batch at this stage is... Calculation stage The relatively increased preheating micro-batch increment required to completely hide communication time :

[0111]

[0112] in, This represents the floor function. When communication time... When the computation time margin is exceeded, the system provides more forward computation overlap windows by increasing the number of preheating micro-batches, thereby achieving communication masking.

[0113] Step 6.3: Accumulate the preheating increments from back to front. Solve the first... The total number of forward micro-batches launched during the H-1F1B preheating phase of the pipeline stage The formula is:

[0114]

[0115] in, Representation phase The number of forward micro-batch launches required during the startup phase; This indicates that the number of preheated micro-batches in the last production line stage (end stage) is fixed at 1.

[0116] Step 6.4: Based on the number of preheating micro-batches at each stage The execution order of each micro-batch event within the pipeline is dynamically adjusted.

[0117] The micro-batch events include: micro-batch forward computation, activation value transmission, next-stage activation value reception, micro-batch reverse computation, gradient transmission, gradient reception, and communication waiting and synchronization events. By overlapping communication events as much as possible within the forward / reverse computation time of other pipeline stages, the computation and communication overlap rate is maximized, further reducing pipeline idle time caused by communication congestion.

[0118] Step 6.5: Perform a preheating configuration memory security check, and adaptively downgrade the number of preheating micro-batches that do not meet the check.

[0119] Adjusting the preheating micro-batch number Simultaneously, memory security checks are performed on the GPU nodes at each stage of the pipeline:

[0120]

[0121] in, Representation phase Using preheating Peak memory usage (including model parameters, gradients, and) (Activation value of each micro-batch of video memory); For the stage The maximum physical available video memory for GPU deployment. If the verification result shows that the video memory limit is exceeded, the system adopts an adaptive degradation strategy: decreasing the number of preheating micro-batches in increments of 1. And re-verify until the video memory safety constraints are met; when If the minimum value of 1 still exceeds the video memory limit, the activation checkpointing mechanism will be forcibly enabled to reduce the usage of active video memory and ensure the safe operation of the system.

[0122] Step 6.6: Monitor runtime status and re-execute candidate layout filtering to solve for the optimal layout. Global communication offset Preheating inside the mission This enables closed-loop adaptive iteration.

[0123] The task execution agent continuously collects data on the cluster's training throughput, GPU idle rate, stage computation time, stage communication time, link utilization, communication window overlap time, communication wait time, and GPU memory usage during runtime. At the end of each scheduling cycle, it updates the network and bubble weights based on real-time observations. and Absolute and normalized bubble losses and Network competition and losses Joint scoring Re-execute the candidate layout selection to solve for the optimal layout. Global communication offset And the preheating inside the mission This forms a two-layer closed-loop adaptive scheduling mechanism between cluster-level resource allocation and network peak shaving and micro-overlapping of task internal pipelines.

[0124] In some embodiments, when the pipeline idle awareness and communication-coordinated scheduling method for heterogeneous GPU clusters designed in this application is applied to a distributed parallel computing architecture, the system structure diagram is as follows: Figure 2 As shown; the flowchart of the algorithm for implementing the pipeline idle awareness and communication cooperative scheduling method for heterogeneous GPU clusters designed in this application is as follows. Figure 3 As shown.

[0125] Finally, it should be noted that the above description only depicts some embodiments of this application. For those skilled in the art, various changes, modifications, substitutions, and variations can be conceived of these embodiments without departing from the principles and spirit of this application. The scope of protection of this application is defined by the appended claims and their equivalents, and all the above-mentioned behaviors should be covered within the scope of protection of this application.

Claims

1. A pipeline idle awareness and communication-coordinated scheduling method for heterogeneous GPU clusters, characterized in that, include: Step 1: Collect fine-grained features of a single task and establish a normalized dimensionless pipeline bubble loss. Step 2: Collect cluster network status and calculate link network congestion loss, and establish normalized dimensionless network contention loss. Step 3: Jointly model pipeline bubble loss and network contention loss, and construct a joint scoring function to measure the comprehensive cost of task combination on a specific shared link; Step 4: Based on the joint scoring function and closed-loop conflict filtering mechanism, construct a set of candidate GPU resource layouts; Step 5: Select the optimal resource layout from the candidate GPU resource layout set and calculate the global communication offset of the optimal resource layout; Step 6: Based on the global optimal resource layout and the communication offset of all tasks, perform two-layer distribution and closed-loop adaptive adjustment of the optimal resource layout scheduling strategy.

2. The pipeline idle awareness and communication-coordinated scheduling method for heterogeneous GPU clusters according to claim 1, characterized in that, The normalized dimensionless flow line bubble loss is calculated using the following formula: ; ; in, This represents the normalized dimensionless flow line bubble loss. and Representing task sets respectively The maximum and minimum absolute bubble losses in the middle. Indicates task Absolute flow line bubble loss, Indicates task The number of stages in the production line. This represents the effective computation time of the slowest bottleneck pipeline stage. Indicates task The Average computation time for each stage This represents the idle time of each stage and the effective computing power of the GPU nodes deployed in each stage. At that time, the normalization result is preset to 0.

3. The pipeline idle awareness and communication-coordinated scheduling method for heterogeneous GPU clusters according to claim 1, characterized in that, The establishment of the normalized dimensionless network contention loss includes: Step 2.1: Establish a network topology model for the heterogeneous cluster; Step 2.2: Collect the link capacity and actual traffic carried by each physical link in the network topology model, and calculate the link utilization rate. ; Step 2.3: Based on the GPU nodes deployed at each stage of the task pipeline and their data transmission routes, establish a mapping relationship between tasks and network links, and calculate the communication window. ; Step 2.4, according to the communication window Calculate the communication overlap duration of the shared link ; Step 2.5: Quantify the link network congestion loss and normalize it to obtain the normalized dimensionless network contention loss.

4. The pipeline idle awareness and communication-coordinated scheduling method for heterogeneous GPU clusters according to claim 3, characterized in that, The quantized link network congestion loss is then normalized using the following formula: ; ; in, This represents the normalized dimensionless network contention loss. This represents the original network congestion loss. Indicates a unified reference time period. Indicates the duration of communication overlap. This represents the preset link congestion threshold. This indicates the link overutilization rate. Penalties are only imposed on the excess portion when the total data demand of the link exceeds the effective capacity of the link; no congestion penalty is imposed when the link utilization rate is below the threshold.

5. The pipeline idle awareness and communication-coordinated scheduling method for heterogeneous GPU clusters according to claim 1, characterized in that, The joint scoring function constructed to measure the combined cost of task combinations on a specific shared link is as follows: ; in, This represents the joint scoring function. Indicates task and In shared links Network contention losses and Representing tasks and Production line bubble loss, This indicates the weight of network communication competition loss. This represents the idle loss weight of the pipeline, and the weight relationship satisfies... ; Indicates task The proportion of communication window occupied. Indicates task The proportion of communication window space occupied.

6. The pipeline idle awareness and communication-coordinated scheduling method for heterogeneous GPU clusters according to claim 5, characterized in that, Adaptive feedback mechanism is used to weight network communication contention loss. Weight of pipeline idle loss To conduct cross-cycle adjustments, including: assuming the first The network congestion observations within each scheduling cycle are as follows: GPU computing power idle time and pipeline bubble observations are as follows , will the The weights for each scheduling cycle are updated as follows: ; ; in, This indicates the preset weight update step size. This represents an interval clipping function, used to clip the independent variable... Limited to the range [0,1].

7. The pipeline idle awareness and communication-coordinated scheduling method for heterogeneous GPU clusters according to claim 1, characterized in that, The construction of the candidate GPU resource layout set includes: Step 4.1: Generate candidate GPU resource layout schemes based on the network topology model, cluster physical topology, and concurrent task model. ; Step 4.2: Using the joint scoring function as the basic metric for calculating edge weights, construct a task-link bipartite graph to characterize the mapping relationship between tasks and network links. ; Step 4.3, Calculation The relative communication offset of the internal shared link; Step 4.4: Perform a breadth-first search traversal. Detect the closed-loop path in the graph. For each closed loop, based on the relative communication offset Perform congruence consistency check; if the congruence constraint is not met, discard the current candidate resource layout; all candidate resource layouts that pass the congruence constraint constitute the candidate GPU resource layout set.

8. The pipeline idle awareness and communication-coordinated scheduling method for heterogeneous GPU clusters according to claim 1, characterized in that, The calculation of the global communication offset for optimal resource layout includes: Step 5.1: Construct multi-dimensional hardware resource constraints, filter the candidate resource layout set, and obtain the feasible resource layout set. ; Step 5.2: Calculate the set of feasible layouts Global joint total cost for each candidate layout in To minimize For the target By filtering, the optimal resource allocation can be obtained. ; The global joint total cost For the set of feasible layouts Each candidate layout in Let the set of edges of its corresponding bipartite graph be... , Mapping edges The layout cost edge weight is Global Joint Total Cost Defined as the cost edge weight of all layouts The cumulative sum; Step 5.3: Based on optimal resource allocation Corresponding bipartite graph Construct a global communication time offset vector for the task cluster to guide global peak-shaving control of the communication time domain window for each task in the cluster.

9. The pipeline idle awareness and communication-coordinated scheduling method for heterogeneous GPU clusters according to claim 1, characterized in that, The scheduling strategy is implemented through a two-layer distribution and closed-loop adaptive adjustment. The upper layer distributes the off-peak communication window to each task; the lower layer adaptively adjusts the number of preheating micro-batches and the execution order of micro-batches within each task to maximize the overlap between computation and communication, and forms a closed-loop iterative optimization through runtime monitoring.

10. The pipeline idle awareness and communication-coordinated scheduling method for heterogeneous GPU clusters according to claim 9, characterized in that, Adaptive adjustment of H-1F1B production line preheating micro-batch quantity includes: Let the assembly line be numbered. The average computation time for each stage of the micro-batch is , Given the total number of pipeline stages for a task, the computation time of the bottleneck stage is defined as... ;stage The computational time margin available for hidden communication is Combined with the stage The computation time per micro-batch Calculation stage The relatively increased preheating micro-batch increment required to completely hide communication time The preheating increments at each stage are accumulated from back to front. Solve the first... The total number of forward micro-batches launched during the H-1F1B preheating phase of the pipeline stage The formula is: ; ; in, This represents the floor function. This indicates the time consumed for cross-node communication between stages. When the computation time margin is exceeded, the system provides a larger forward computation overlap window by increasing the number of preheating micro-batches. Representation phase The number of forward micro-batch launches required during the startup phase; This indicates that the number of preheated micro-batches in the last production line stage is fixed at 1.