A communication scheduling system for multi-distributed training tasks
By using dynamic priority allocation and coordinated communication scheduling through traffic interleaving modules, the communication contention problem in multiple distributed training tasks is solved, thereby improving training efficiency and resource utilization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- UNIV OF SCI & TECH OF CHINA
- Filing Date
- 2026-03-12
- Publication Date
- 2026-06-05
AI Technical Summary
In multi-distributed training tasks, communication contention increases training completion time by 40%, affecting cluster training efficiency. Existing technologies cannot simultaneously resolve the flow conflict problem within and between tasks.
A dynamic priority allocation module and a traffic interleaving module are introduced. The dynamic priority allocation module updates task priorities based on real-time cluster fluctuations, and the traffic interleaving module performs priority-based traffic interleaving on each link. The central controller module integrates the outputs of both to achieve priority-aware traffic interleaving.
It significantly shortens end-to-end training time, improves the overall resource utilization of the cluster, solves the problem of communication contention between tasks, and ensures training accuracy and efficient communication.
Smart Images
Figure CN122160436A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the technical field of distributed training and traffic scheduling, and in particular to a communication scheduling system for multiple distributed training tasks. Background Technology
[0002] As deep learning models rapidly expand to the level of hundreds of billions of parameters, a single GPU can no longer meet their training needs, and distributed parallel training has become the mainstream. However, while massive parallelism improves computational scalability, it also introduces communication overhead that increases with model growth, making communication a core bottleneck in deep learning training.
[0003] Real-world testing data shows that multiple training tasks sharing network resources leads to a significant problem of "communication contention between training tasks." Even with the most advanced single-training-task traffic scheduler, communication contention between training tasks still increases training completion time by an average of 40%, severely impacting cluster training efficiency. Therefore, there is an urgent need to provide a communication scheduling system for multiple distributed training tasks. Summary of the Invention
[0004] To address the problems existing in the prior art, this application provides a communication scheduling system for multiple distributed training tasks. As a new framework that can dynamically adapt to cluster fluctuations, globally coordinate scheduling objectives, and efficiently integrate priority and traffic interleaving mechanisms, it systematically solves the problem of inter-task communication competition in multi-tenant AI training clusters and improves overall training efficiency.
[0005] The objective of this application is achieved through the following technical solution:
[0006] This application provides a communication scheduling system for multiple distributed training tasks, including:
[0007] The dynamic priority allocation module is used to update task priority information based on real-time fluctuations in the cluster.
[0008] A traffic interleaving module is used to perform traffic interleaving based on the task priority information on each link;
[0009] The central controller module is connected to the dynamic priority allocation module and the traffic interleaving module, and is used to integrate the outputs of the dynamic priority allocation module and the traffic interleaving module to realize priority-aware traffic interleaving.
[0010] Preferably, the dynamic priority allocation module includes:
[0011] The priority scoring unit is used to score the priority based on the first indicator information of the training task and obtain the priority score. The first indicator information includes computational requirements, communication requirements and transmission overhead in the cluster.
[0012] The mapping relationship establishment unit is used to cluster the priority scores obtained by the priority scoring unit, establish a mapping relationship between the priority scores and the limited priorities of network control protocols and network interfaces, and use it as task priority information.
[0013] Preferably, the mapping relationship establishment unit is further used to cluster continuous priority scores using the DBSCAN density clustering algorithm, and map the clustering results to a finite discrete priority level supported by the network interface controller, thereby establishing a mapping relationship between priority scores and hardware priority queues.
[0014] Preferably, the traffic interleaving module includes:
[0015] The traffic modeling unit is used to model AI traffic based on the periodicity and burstiness characteristics of AI traffic using second indicator information; the second indicator information includes communication start time, communication period, and communication volume.
[0016] The single-link optimization unit is used to formalize the traffic conflict optimization problem in the single-link dimension based on the AI traffic modeling output by the traffic modeling unit, and to achieve traffic interleaving by adjusting the temporal offset of the training task; and to obtain a traffic interleaving scheme for optimizing traffic conflicts on a single link.
[0017] The cluster consensus unit is used to coordinate the timing offset of the training task on different links through the consensus algorithm, so that the timing offset of the same task on each link in the cluster remains consistent, and generates a total traffic interleaving scheme at the cluster level.
[0018] Preferably, the traffic modeling unit is used to model AI traffic using tuples (tj, Tj, Tj), where tj is the communication start time, Tj is the iteration period, and Tj is the set of communication phases on each link.
[0019] Preferably, the single-link optimization unit is used to solve the optimal timing offset in a single-link dimension by means of a formal optimization problem with the objective of minimizing the total conflict of all job pairs on the link. The objective is to minimize the total conflict of all job pairs on the link, and the constraint is that the optimal timing offset is not less than 0 and not greater than the iteration period.
[0020] Preferably, the cluster consistency unit is used to unify the time offset scheme of multiple links at the cluster level by constructing a weighted bipartite graph G=(J∪L,E,w), where J and L represent the job vertex set and the link vertex set, respectively, w represents the edge weight, and E represents the edge set; and at the cluster level, to construct a spanning tree that satisfies the time offset unification of the entire cluster and minimizes global conflicts by solving an integer programming problem.
[0021] Preferably, the central controller module includes: a coordination and conversion unit, used to convert the priority allocation information output by the dynamic priority allocation module into input parameters of the traffic interleaving module, used to guide the construction of the spanning tree in the cluster consistency unit.
[0022] Preferably, the central controller module further includes: a dynamic switching unit, used to enable the dynamic priority allocation module to reduce decision-making and control overhead when a highly dynamic event occurs, based on real-time monitoring of the cluster communication status; otherwise, to enable the traffic interleaving module.
[0023] Preferably, the highly dynamic events include at least one of node failure, link failure, and iteration time irregularity.
[0024] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:
[0025] A dynamic priority allocation mechanism is introduced. By analyzing the computation / communication / estimated completion time characteristics of tasks in real time and performing density clustering, it can adaptively assign priorities to newly arriving and running tasks, effectively addressing dynamic changes in the cluster and the uncertainty of task arrival, and avoiding the tail task delay problem caused by traditional static priority schemes. A priority-aware cluster-level traffic interleaving algorithm is designed. This algorithm, by constructing a weighted bipartite graph and solving for a spanning tree with priority constraints, can effectively solve the circular dependency problem that existing interleaving schemes cannot handle, and naturally possesses robustness to iteration time irregularities and node / link failures. A centralized unified optimization framework is adopted, with a central controller comprehensively coordinating the outputs of the priority module and the interleaving module, solving the scheduling target conflict and high coordination overhead problems that exist when the two are directly combined, ensuring that the two modules can complement and cooperate during both stable and fluctuating periods, achieving globally optimal communication scheduling.
[0026] In summary, this application, based on dynamic priority and traffic interleaving technology, is mainly committed to eliminating inter-task communication contention in multi-tenant deep learning training clusters through coordinated communication scheduling. While ensuring training accuracy, it significantly shortens end-to-end training time and improves the overall resource utilization of the cluster. Attached Figure Description
[0027] The present application will be further described below with reference to the accompanying drawings and embodiments.
[0028] Figure 1 This is a schematic diagram of the communication scheduling system provided in the embodiments of this application.
[0029] Figure 2 This is a schematic diagram of the average task completion time under hybrid deployment provided in the embodiments of this application.
[0030] Figure 3 This is a schematic diagram comparing the cluster congestion rate under different schemes provided in the embodiments of this application.
[0031] Figure 4 This is a schematic diagram illustrating the robustness comparison within a high-failure-rate cluster provided in an embodiment of this application.
[0032] Figure 5 This is a schematic diagram of bipartite graph modeling of link dependencies provided in an embodiment of this application.
[0033] Figure 6 This is a timing diagram illustrating the tuple-based modeling of AI training traffic provided in an embodiment of this application.
[0034] Figure 7 This is a schematic diagram of the architecture of the communication scheduling system provided in the embodiments of this application. Detailed Implementation
[0035] The present application will now be further described in conjunction with the accompanying drawings and specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments. The implementation process of the present application will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific implementation procedures, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application and not for limiting the scope of protection of the present application.
[0036] The following is a brief description of the technical field and related terms of the embodiments of this application, so as to facilitate understanding by those skilled in the art.
[0037] To address the issue that communication contention between training tasks still increases training completion time by an average of 40%, severely impacting cluster training efficiency, relevant technologies have proposed solutions such as traffic interleaving or priority allocation.
[0038] "Traffic interleaving" (e.g., Cassini) refers to avoiding bandwidth conflicts by centrally scheduling the communication and computation phases of different tasks, but it highly relies on accurate prediction of traffic patterns. In real production environments, iteration time fluctuations and frequent node / link failures can easily disrupt the interleaving state, forcing the system to frequently reschedule (high overhead) or tolerate inefficient interleaving (low performance).
[0039] "Priority allocation" (such as Crux) refers to assigning priorities to tasks based on the computation / communication ratio, prioritizing computationally intensive tasks. However, it uses decentralized link-level scheduling and lacks global coordination, resulting in inconsistent priority strategies for the same task on different links. Ultimately, communication latency is constrained by the most congested link, low-priority tasks are blocked for a long time, and throughput is severely reduced.
[0040] In other words, the relevant technologies cannot simultaneously resolve the issues of intra-task and inter-task flow conflicts, and cannot achieve the effects of reducing communication overhead and improving training efficiency.
[0041] Based on this, to address the limitation of related technologies that rely solely on accurate prediction of traffic patterns and lack cluster-wide coordination capabilities, a communication scheduling system for multiple distributed training tasks is provided. This system eliminates communication contention through a collaborative modular design. Specifically, a dynamic priority allocation module continuously adjusts task priorities based on real-time cluster fluctuations; a traffic interleaving module with fault recovery capabilities performs priority-based traffic interleaving on each link; and a central controller module integrates the outputs of the above two modules using a unified optimization framework, resolving scheduling conflicts while ensuring cluster-wide coordination. In summary, a collaborative modular design eliminates communication contention between training tasks. This new framework, which dynamically adapts to cluster fluctuations, globally coordinates scheduling objectives, and efficiently integrates priority and traffic interleaving mechanisms, systematically solves the problem of inter-task communication contention in multi-tenant AI training clusters, thereby improving overall training efficiency. The technical solution protected by this application will be described in detail below.
[0042] See Figure 1 This application provides a communication scheduling system for multiple distributed training tasks, comprising:
[0043] The dynamic priority allocation module is used to update task priority information based on real-time fluctuations in the cluster.
[0044] A traffic interleaving module is used to perform traffic interleaving based on the task priority information on each link;
[0045] The central controller module is connected to the dynamic priority allocation module and the traffic interleaving module, and is used to integrate the outputs of the dynamic priority allocation module and the traffic interleaving module to realize priority-aware traffic interleaving.
[0046] On each communication link, the traffic interleaving module does not independently perform equal traffic interleaving. Instead, it uses the task priority information output by the dynamic priority allocation module as input constraints. For example, it assigns better timing offset resources to high-priority tasks and implements avoidance scheduling for low-priority tasks, so that the link-level traffic conflict optimization has differentiated service capabilities and ensures the communication quality of key training tasks.
[0047] Therefore, the technical solution provided in this embodiment, based on the dynamic priority allocation module and the traffic interleaving module, primarily aims to eliminate inter-task communication contention in multi-tenant deep learning training clusters through coordinated communication scheduling. This significantly shortens end-to-end training time and improves overall cluster resource utilization while ensuring training correctness. Furthermore, a central controller unifies and optimizes the two core modules—dynamic priority allocation and fault-tolerant traffic interleaving—allowing for adaptive adjustment of task priorities based on real-time cluster status and globally consistent traffic interleaving through priority-aware algorithms. This fundamentally solves the performance bottlenecks caused by a lack of coordination or reliance on static assumptions in existing solutions, thereby improving cluster training efficiency.
[0048] In some embodiments, the dynamic priority allocation module includes:
[0049] The priority scoring unit is used to score the priority based on the first indicator information of the training task and obtain the priority score. The first indicator information includes computational requirements, communication requirements and transmission overhead in the cluster.
[0050] The mapping relationship establishment unit is used to cluster the priority scores obtained by the priority scoring unit, establish a mapping relationship between the priority scores and the limited priorities of network control protocols and network interfaces, and use it as task priority information.
[0051] In other words, the dynamic priority allocation module scores priorities based on metrics such as the computational and communication requirements of the training task and the transmission overhead within the cluster. The resulting priority scores are then clustered to establish a mapping between these scores and the finite priorities of network control protocols (such as PFC) and network interfaces (such as NICs).
[0052] As an example, the dynamic priority allocation module in the system determines its priority score based on the computational requirements (Rj), communication requirements (Bj), and estimated completion time (tj) of the training task, using the formula Sj = Rj / (Bj·tj). The estimated completion time serves as the transmission overhead within the cluster. j ∈ {1, 2, ..., N}, where N represents the total number of training tasks running simultaneously in the cluster, and j represents the j-th training task.
[0053] In some embodiments, the mapping relationship establishment unit is further configured to cluster continuous priority scores using the DBSCAN density clustering algorithm and map the clustering results to a finite number of discrete priority levels supported by the network interface controller, thereby establishing a mapping relationship between priority scores and hardware priority queues.
[0054] As an example, the mapping relationship establishment unit can use the DBSCAN density clustering algorithm to cluster continuous priority scores and map the clustering results to a finite discrete priority level (P=8) supported by the network interface controller (NIC), thereby establishing a mapping relationship between priority scores and hardware priority queues.
[0055] In some embodiments, the traffic interleaving module includes:
[0056] The traffic modeling unit is used to model AI traffic based on the periodicity and burstiness characteristics of AI traffic using second indicator information; the second indicator information includes communication start time, communication period, and communication volume.
[0057] The single-link optimization unit is used to formalize the traffic conflict optimization problem in the single-link dimension based on the AI traffic modeling output by the traffic modeling unit, and to achieve traffic interleaving by adjusting the temporal offset of the training task; and to obtain a traffic interleaving scheme for optimizing traffic conflicts on a single link.
[0058] The cluster consensus unit is used to coordinate the timing offset of the training task on different links through the consensus algorithm, so that the timing offset of the same task on each link in the cluster remains consistent, and generates a total traffic interleaving scheme at the cluster level.
[0059] As an example, the traffic interleaving module is used to model AI traffic based on its periodicity and burstiness characteristics, using metrics such as communication start time, communication cycle, and communication volume. Based on this AI traffic modeling, the traffic conflict optimization problem in a single-link dimension is formalized. Traffic interleaving is achieved by adjusting the temporal offset of the training tasks, thereby optimizing traffic conflicts in the training tasks. By integrating traffic interleaving schemes across multiple links in the cluster, a consistency algorithm is used to ensure the consistency of the temporal offset of the training tasks within the cluster, resulting in a cluster-level traffic interleaving scheme (the overall traffic interleaving scheme).
[0060] In some embodiments, the traffic modeling unit is used to model AI traffic using tuples (tj, Tj, Tj), where tj is the communication start time, Tj is the iteration period, and Tj is the set of communication phases on each link.
[0061] The purpose of this embodiment is to depict the periodicity and burstiness of AI traffic at this moment.
[0062] See Figure 6 This diagram illustrates the tuple-based modeling of AI training traffic. It shows the communication timing of jobs on two links (link l1 and link l2) in a distributed training environment. The horizontal axis represents time, and the vertical axis represents transmission bandwidth. Green rectangles represent the training traffic of a task on a specific link. This indicates the start and end times of the first communication activity of task j on link l1.
[0063] In some embodiments, the single-link optimization unit is used to solve the optimal timing offset in a single-link dimension by means of a formal optimization problem with the objective of minimizing the total conflict of all job pairs on the link. The objective is to minimize the total conflict of all job pairs on the link, and the constraint is that the optimal timing offset is not less than 0 and not greater than the iteration period.
[0064] As an example, the training traffic interleaving module is used to solve for the optimal timing offset τ^l_j in a single link dimension by a formal optimization problem minτ^l_jO^l. The goal of this problem is to minimize the total collisions of all job pairs on the link, with the constraint 0≤τ^l_j≤Tj.
[0065] In some embodiments, the cluster consistency unit is used to unify the time offset scheme of multiple links at the cluster level by constructing a weighted bipartite graph G=(J∪L,E,w), where J and L represent the job vertex set and the link vertex set, respectively, w represents the edge weight, and E represents the edge set of the bipartite graph: if the traffic of task j passes through link l, then (j,l)∈E; and at the cluster level, to construct a spanning tree that satisfies the time offset unification across the entire cluster and minimizes global conflicts by solving an integer programming problem.
[0066] See Figure 5 This is a schematic diagram of bipartite graph modeling for link dependencies. As an example, the training traffic interleaving module is used to unify the time-series offset scheme of multiple links at the cluster level by constructing a weighted bipartite graph G=(J∪L,E,w), where the vertex sets J and L represent jobs and links, respectively, and the edge weight w(j,l) represents the amount of communication conflict Ol of job j on link l.
[0067] Figure 5 The graph contains two subgraphs (a) and (b), which respectively illustrate the communication relationships between jobs (j, jobs) and links (l, links) in a distributed system, and how to optimize traffic interleaving and reduce communication conflicts by constructing a bipartite graph.
[0068] Jobs are represented by boxes of different colors in the diagram, labeled j1, j2, and j3; links are labeled l1, l2, and l3; arrows indicate the communication relationships between jobs. When two jobs need to communicate simultaneously through the same link, a conflict occurs, indicated by an X symbol in the diagram. Figure (a) shows the direct communication relationships between jobs and the resulting link conflicts without optimization; Figure (b) shows how to manage and optimize communication between jobs by constructing a bipartite graph, and further reduce link conflicts and improve communication efficiency through traffic interleaving and priority control.
[0069] Meanwhile, at the cluster level, a spanning tree is constructed by solving an integer programming problem. The goal of this problem is to minimize the total weight of the spanning tree (i.e., the total number of conflicts). A degree constraint ∑e∈δ(j)xe≤f(pj) based on job priority pj is introduced to achieve priority-aware temporal offset unification.
[0070] In some embodiments, the central controller module includes:
[0071] The coordination and conversion unit is used to convert the priority allocation information output by the dynamic priority allocation module into the input parameters of the traffic interleaving module, which is used to guide the construction of the spanning tree in the cluster consistency unit.
[0072] A dynamic switching unit is used to activate the dynamic priority allocation module to reduce decision-making and control overhead when a high-dynamic event occurs, based on real-time monitoring of the cluster communication status; otherwise, it activates the traffic interleaving module. Specifically, the high-dynamic event includes at least one of node failure, link failure, and irregular iteration time.
[0073] As an example, the central controller module coordinates the dynamic priority allocation module and the traffic interleaving module through a unified optimization framework. It takes the discrete priority pj output by the priority module as an input parameter to guide the construction of the spanning tree in the traffic interleaving module (i.e., the degree constraint f(pj)), thereby realizing priority-aware traffic interleaving.
[0074] Meanwhile, it will monitor the cluster status in real time (e.g., via INT telemetry) and rely primarily on the dynamic priority allocation module for rapid response when highly dynamic events (e.g., node / link failures, irregular iteration times) are detected; when the cluster status is relatively stable, the traffic interleaving module will be enabled to pursue higher communication efficiency and resource utilization.
[0075] Compared with the technical solutions of related technologies, the advantages of this application are:
[0076] By introducing a dynamic priority allocation mechanism, which analyzes the computation / communication / estimated completion time characteristics of tasks in real time and performs density clustering, it can adaptively allocate priorities to newly arriving and running tasks, effectively cope with dynamic changes in the cluster and the uncertainty of task arrival, and avoid the tail task delay problem caused by traditional static priority schemes.
[0077] A priority-aware cluster-level traffic interleaving algorithm was designed. This algorithm can effectively solve the circular dependency problem that existing interleaving schemes cannot handle by constructing a weighted bipartite graph and solving the spanning tree with priority constraints. It also has inherent robustness to iteration time irregularities and node / link failures.
[0078] A centralized unified optimization framework is adopted, in which the output of the priority module and the interleaving module are coordinated by the central controller. This solves the scheduling target conflict and high coordination overhead problems that exist when the two are directly combined. It ensures that the two modules can complement each other and cooperate during the stable period and the fluctuation period, so as to achieve the globally optimal communication scheduling.
[0079] As an example, see Figure 7 This paper presents a communication scheduling system for multiple distributed training tasks, with the following overall architecture:
[0080] Deep learning jobs are deep learning tasks submitted by tenants and need to be executed in the cluster.
[0081] The priority allocation engine (dynamic priority allocation module) is used to implement the analyzer and priority allocation functions. The analyzer analyzes the computational requirements, communication requirements, and estimated completion time of the deep learning job, and calculates the priority score. Priority allocation is used to cluster based on the priority score and establish a mapping relationship between the priority score and the limited priorities of network control protocols (such as PFC) and network interfaces (such as NICs).
[0082] The traffic interleaving engine (traffic interleaving module) is used to implement traffic interleaving at the link level (single link optimization unit) and at the cluster level (cluster consistency unit). Specifically, at the single link level, it solves the optimal time offset through a formal optimization problem to minimize the total conflict of all job pairs on the link. At the cluster level, it constructs a weighted bipartite graph and solves the spanning tree with priority constraints to achieve priority-aware time offset unification.
[0083] A cluster is a computing cluster consisting of multiple hosts (host1 to hostN), which can be connected by switches to form a network topology.
[0084] The Controller (Central Controller) coordinates the operation of the priority allocation module and the traffic interleaving module, triggers priority allocation, implements traffic interleaving with the cluster, triggers cluster-level traffic interleaving, and receives traffic interleaving schemes from the traffic interleaving engine.
[0085] Additionally, an option is available to set up a job scheduler to further optimize job execution.
[0086] The performance of the technical solution protected in this application will be verified through large-scale simulation and small-scale test bench experiments, and compared with existing solutions such as Cassini and Crux.
[0087] See Figure 2 , Figure 2 This diagram illustrates the average task completion time under hybrid deployment. It shows that the technical solution provided in this application (represented by CODA) significantly improves the average training speed: in experiments, the technical solution achieved a reduction of up to 26% in average training task time. The horizontal axis represents the number of ResNet models n (n=1, n=2, n=3). The vertical axis represents the normalized task completion time. Legend: Comparison of four strategies (Cassini, Crux, Cassini+Crux, CODA). The percentage at the top of the bar chart represents the JCT optimization ratio of the method. The specific hybrid scenario is BERT+ResNet. n.
[0088] To verify the significant reduction in cluster congestion, the fluctuation in the number of ECN tags within the cluster over a certain time window was statistically analyzed for different network topologies. This data reflects the degree to which different solutions alleviate network congestion caused by traffic conflicts. See also Figure 3 The CODA solution can achieve up to 7 times the mitigation of traffic conflicts.
[0089] Figure 3 In this diagram, ECN marks (1000 Plts / s) are a core indicator for measuring network congestion control efficiency, reflecting the ability of the traffic scheduling algorithm to adjust the congestion window. The left and right sides indicate the Fat-tree Topology and Rail-optimized Topology, respectively. 7.0X and 4.2X represent the multiples achieved by traffic conflict mitigation in the Fat-tree and Rail-optimized topologies, respectively.
[0090] See Figure 4The left and right sides indicate the fat-tree topology and the rail-optimized topology, respectively. The vertical axis represents the percentage reduction in JCT (job completion time reduction percentage), and the horizontal axis represents the network load level p. Different colors and markers in the legend represent different scheduling schemes. As can be seen, robustness tests were conducted in a high-failure-rate cluster environment. In this environment, as the failure rate increased, the performance of all the compared schemes deteriorated to varying degrees. The CODA scheme consistently maintained the highest task acceleration performance, demonstrating the robustness of the technical solution provided in this application within a high-failure-rate cluster.
[0091] As an example, an application environment for the communication scheduling system protected in this application is provided. The application environment includes a data center platform, which consists of a controller, a host, and a programmable switch. The communication scheduling system includes a dynamic priority allocation module, a traffic interleaving module, and a central controller module. The system may include the following steps during execution:
[0092] (1) Users submit AI training tasks to the data center platform, and the central controller module records the task's computing requirements, communication requirements, and current cluster status.
[0093] (2) The central controller module coordinates the dynamic priority allocation module and the traffic interleaving module to formulate a scheduling strategy and distribute the scheduling strategy to the relevant hosts;
[0094] (3) The dynamic priority allocation module calculates priority scores based on the task's computational requirements, communication requirements, and estimated completion time, and maps continuous scores to discrete priority levels.
[0095] (4) The training traffic interleaving module models the AI traffic, determines the temporal offset of each task, and ensures the consistency of temporal offset within the cluster.
[0096] (5) The central controller module monitors the cluster communication status in real time and selects priority allocation or traffic interleaving as the main scheduling mechanism based on the dynamic nature of the cluster.
[0097] Specifically, in step (1), the type of model training provided by the service provider to the central controller module includes the required gradient size. In this embodiment, a GPT-2 model training task is submitted to the platform. This task requires four GPU servers to work together, with a computational requirement of 1.2 × 10^12 FLOPs, a communication requirement of 500 MB, and an estimated completion time of 1200 seconds. The cluster has eight GPU servers, each equipped with an NVIDIA RTX 3090 GPU and a Mellanox ConnectX-6 100G dual-port network card. All servers are connected via a 100Gbps link, using a Spine-Leaf topology.
[0098] Specifically, in steps (2) and (3), the dynamic priority allocation module calculates the priority score as 1.2e12 / (500×1200)=2e9. The DBSCAN density clustering algorithm is used to automatically cluster the continuous priority scores into 8 discrete levels. In this embodiment, the task is assigned to priority level 3 (a priority range of 0-7). Simultaneously, the system periodically re-evaluates the priority of all tasks, and immediately updates the priority allocation when a new task is added to the cluster or the status of an existing task changes.
[0099] Specifically, in step (4), the training traffic interleaving module models the communication pattern of the task as (0, 100, {[0-20], [50-70]}), where 0 represents the communication start time, 100 represents the iteration period, and [0-20] and [50-70] represent the communication phases on each link. Through link-level traffic interleaving optimization, the timing offset of the task on link l is determined to be 15 seconds. The cluster-level traffic interleaving algorithm unifies the timing offsets on each link by constructing a weighted bipartite graph, ensuring that the timing offset of the task remains consistent across all links. When a circular dependency is detected in the cluster (such as three tasks competing with each other on different links), the algorithm resolves it through the constraint spanning tree method, ensuring that all tasks can obtain a reasonable timing offset.
[0100] Specifically, in step (5), the central controller module monitors the cluster communication status in real time through In-band Network Telemetry. When highly dynamic events such as irregular iteration times or node failures are detected, the system mainly relies on the dynamic priority allocation module for rapid response; when the cluster status is relatively stable, the system enables the traffic interleaving module to obtain higher communication efficiency. For example, in this embodiment, when a sudden high-priority task occurs in the cluster, the central controller module immediately triggers the priority allocation module to re-evaluate the priority of all tasks and adjusts the traffic interleaving plan to ensure that the high-priority task can complete the transmission quickly.
[0101] The above examples are merely illustrative of the technical concept and features of this invention, intended to enable those skilled in the art to understand the content of this invention and implement it accordingly, and should not be construed as limiting the scope of protection of this invention. All equivalent transformations or modifications made according to the spirit and essence of this invention should be included within the scope of protection of this invention.
[0102] It should be noted that in the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple. It is worth noting that "at least one" can also be interpreted as "one or more".
[0103] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are configured to distinguish similar objects and are not necessarily configured to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
Claims
1. A communication scheduling system for multiple distributed training tasks, characterized in that, include: The dynamic priority allocation module is used to update task priority information based on real-time fluctuations in the cluster. A traffic interleaving module is used to perform traffic interleaving based on the task priority information on each link; The central controller module is connected to the dynamic priority allocation module and the traffic interleaving module, and is used to integrate the outputs of the dynamic priority allocation module and the traffic interleaving module to realize priority-aware traffic interleaving.
2. The communication scheduling system according to claim 1, characterized in that, The dynamic priority allocation module includes: The priority scoring unit is used to score the priority based on the first indicator information of the training task and obtain the priority score. The first indicator information includes computational requirements, communication requirements and transmission overhead in the cluster. The mapping relationship establishment unit is used to cluster the priority scores obtained by the priority scoring unit, establish a mapping relationship between the priority scores and the limited priorities of network control protocols and network interfaces, and use it as task priority information.
3. The communication scheduling system according to claim 2, characterized in that, The mapping relationship establishment unit is also used to cluster continuous priority scores using the DBSCAN density clustering algorithm and map the clustering results to a finite discrete priority level supported by the network interface controller, thereby establishing a mapping relationship between priority scores and hardware priority queues.
4. The communication scheduling system according to claim 1, characterized in that, The flow interleaving module includes: The traffic modeling unit is used to model AI traffic based on the periodicity and burstiness characteristics of AI traffic using second indicator information; the second indicator information includes communication start time, communication period, and communication volume. The single-link optimization unit is used to formalize the traffic conflict optimization problem in the single-link dimension based on the AI traffic modeling output by the traffic modeling unit, and to achieve traffic interleaving by adjusting the temporal offset of the training task; and to obtain a traffic interleaving scheme for optimizing traffic conflicts on a single link. The cluster consensus unit is used to coordinate the timing offset of the training task on different links through the consensus algorithm, so that the timing offset of the same task on each link in the cluster remains consistent, and generates a total traffic interleaving scheme at the cluster level.
5. The communication scheduling system according to claim 4, characterized in that, The traffic modeling unit is used to model AI traffic using tuples (tj, Tj, Tj), where tj is the communication start time, Tj is the iteration period, and Tj is the set of communication phases on each link.
6. The communication scheduling system according to claim 5, characterized in that, The single-link optimization unit is used to solve the optimal timing offset in a single-link dimension by solving a formal optimization problem with the objective of minimizing the total conflict of all job pairs on the link. The objective is to minimize the total conflict of all job pairs on the link, and the constraint is that the optimal timing offset is not less than 0 and not greater than the iteration period.
7. The communication scheduling system according to claim 6, characterized in that, The cluster consistency unit is used to unify the time offset scheme of multiple links at the cluster level by constructing a weighted bipartite graph G=(J∪L,E,w), where J and L represent the job vertex set and the link vertex set, respectively, w represents the edge weight, and E represents the edge set. Furthermore, at the cluster level, a spanning tree that satisfies the uniformity of temporal offsets across the entire cluster and minimizes global conflicts is constructed by solving an integer programming problem.
8. The communication scheduling system according to claim 7, characterized in that, The central controller module includes a coordination and conversion unit, which converts the priority allocation information output by the dynamic priority allocation module into input parameters of the traffic interleaving module, and guides the construction of the spanning tree in the cluster consistency unit.
9. The communication scheduling system according to claim 8, characterized in that, The central controller module also includes: The dynamic switching unit is used to enable the dynamic priority allocation module to reduce decision-making and control overhead when a highly dynamic event occurs, based on real-time monitoring of the cluster communication status; otherwise, the traffic interleaving module is enabled.
10. The communication scheduling system according to claim 9, characterized in that, The highly dynamic events include at least one of node failure, link failure, and iteration time irregularity.