A switch-coordinated communication scheduler for distributed training

Abstract

The present application relates to the technical field of computer network flow control, and particularly relates to a switch cooperative communication scheduler for distributed training. The scheduler proposed by the present application comprises a priority manager, a freeze manager and an ECN threshold manager. The priority manager is used for monitoring data flow priority and judging whether a data packet needs an ECN mark, and comprises a task recorder, a priority tracker and an ECN marker. The freeze manager identifies and freezes tasks with inconsistent priorities across switches through a state machine, so as to avoid bandwidth competition caused by scheduling decision errors. The ECN threshold manager dynamically adjusts the ECN threshold, and fills the bandwidth not occupied by the highest priority task with low priority tasks. The three modules work cooperatively to realize fine communication scheduling in a multi-task concurrent scene without a large number of hardware queue resources. The present application breaks through the limitation of the number of hardware queues, eliminates the lag of end-side feedback, and significantly improves the link utilization and task iteration efficiency in the distributed training scene.

Description

Technical Field

[0001] This invention relates to the field of computer network traffic control technology, and more specifically to a switch collaborative communication scheduler for distributed training. Background Technology

[0002] Intelligent computing data centers typically comprise a compute node layer, a node interconnect layer, and a network switching layer to support the efficient execution of large-scale distributed artificial intelligence training tasks. At the compute node layer, intelligent computing centers usually deploy server clusters composed of high-performance GPUs, NPUs, or other AI accelerators. Within a single machine, accelerator cards communicate with each other via high-speed interconnect buses such as NVLink or NVSwitch, achieving ultra-low latency data exchange and shared GPU memory, thereby effectively improving model training performance and parallel computing efficiency within a single node. At the node interconnect layer, intelligent computing centers widely adopt high-speed network architectures based on RDMA (Remote Direct Memory Access) technology, with common implementations including InfiniBand or RoCE (RDMA over Converged Ethernet) Version 2. RDMA bypasses the CPU to directly access remote memory, achieving zero-copy data transmission and significantly reducing communication overhead and processing latency. This high-throughput, low-latency network interconnect mechanism provides crucial support for distributed deep learning training, enabling cross-node gradient aggregation and parameter synchronization to be completed with sub-millisecond latency. At the network switching layer, intelligent computing centers introduce programmable switches and are gradually exploring on-network computing technologies. By deploying customizable logical operations (such as aggregation, filtering, and compression) on switches, some communication and computing functions can be moved forward to the network layer for execution, thereby effectively reducing data transmission volume and synchronization latency, and improving the overall system's computing and communication overlap efficiency.

[0003] Intelligent computing centers primarily handle distributed AI model training tasks, generating communication traffic characterized by high concurrency, burstiness, and strong synchronization, resulting in highly unbalanced network load both temporally and spatially. The current ECMP routing mechanism used in intelligent computing networks suffers from a typical hash polarization problem, leading to link congestion and bandwidth contention when multiple bursty traffic flows are hashed to the same path. For example, Alibaba pointed out that approximately 36.3% of tasks (consuming about 51% of GPU resources) encountered communication contention issues during training. Achieving efficient traffic scheduling under network congestion has become a key bottleneck for the performance scaling of intelligent computing networks. Against this backdrop, coordinating computation and communication through an efficient traffic scheduling mechanism has become a core issue for optimizing the performance of intelligent computing centers.

[0004] Existing communication scheduling schemes for distributed training scenarios are mainly divided into two categories: centralized task schedulers and distributed congestion control algorithms, both of which have inherent limitations in their design.

[0005] Centralized task schedulers can globally perceive the communication and computation status of all tasks in the cluster and plan the communication sequence for each task based on this information to reduce bandwidth contention and improve overall system throughput. Cassini employs a static scheduling strategy, establishing a traffic characteristic model by offline analysis of the communication and computation phases of each task, and setting a start offset for tasks in the time dimension, thus allowing their communication phases to be interleaved in different time slices. This method can significantly reduce network congestion in environments with stable traffic patterns and simple task types. However, distributed training tasks are often highly dynamic, and the communication traffic between tasks is affected by iteration progress, batch size changes, model synchronization frequency, and interference from other tasks, resulting in inconsistencies between runtime traffic patterns and static analysis results. Since Cassini's scheduling mechanism lacks the ability to perceive and adapt to such dynamic characteristics, the fixed time offset cannot effectively eliminate real-time communication contention. Sincronia introduces the concept of coflow scheduling. It optimizes the overall task completion time by collecting global information about each flow within a task, grouping related flows into a single cooperative flow unit, and assigning a global priority to each cooperative flow based on the overall completion time (CCT). However, Sincronia's implementation relies on a centralized controller, requiring global collection of traffic information and the issuance of scheduling decisions. This process incurs significant control plane overhead and struggles to provide real-time responses in ultra-large-scale training clusters. Crux addresses the heterogeneous nature of GPU clusters by prioritizing tasks based on their GPU utilization (GPU intensity), allocating high-priority paths and queue resources to computationally intensive tasks. This method can reduce communication latency for high-load tasks to some extent in multi-task shared environments. However, both Sincronia and Crux are constrained by the scarcity of physical queue resources. In modern data center switches, the number of hardware priority queues is limited (typically 8). When hundreds or even thousands of training tasks exist simultaneously, the scheduler cannot independently allocate priority queues to each task, leading to priority mapping conflicts and performance degradation. Taking Crux as an example, its core idea is to map different tasks to different priority queues and achieve bandwidth isolation through strict priority scheduling on the switch side. When there are four tasks in the network, if the switch can provide four independent priority queues, Crux can allocate a priority queue to each task, achieving near-ideal bandwidth allocation. However, when the hardware only supports two queues, the four tasks must be compressed and mapped into two queues. Tasks sharing the same queue can no longer be isolated from each other through the priority mechanism, and their traffic will compete within the same queue.

[0006] Compared to centralized scheduling, distributed congestion control algorithms achieve adaptive adjustment of network resources through end-to-end feedback mechanisms, dynamically coordinating bandwidth usage across multiple tasks at runtime without global control. These algorithms typically detect link congestion signals (such as ECN markers or RTT fluctuations) and adjust transmission rates in a non-fair manner, thus enabling interleaved execution of different task communication phases. While PrioPlus wasn't initially designed for distributed training, its core idea can be applied to training traffic optimization. This method assigns different RTT thresholds to flows of different priorities, allowing low-priority flows to detect congestion more quickly and reduce their transmission rates, thereby freeing up more bandwidth resources for high-priority flows and achieving priority-aware non-fair bandwidth allocation. MLTCP is closer to distributed training scenarios. Based on the phased characteristics of the training iteration process, it uses the "number of bytes sent in the current iteration" as a congestion-aware indicator to dynamically adjust the transmission rate. In this way, MLTCP forms a non-fair but efficient bandwidth allocation pattern among different training tasks, allowing communication phases to be gradually separated in time. Experimental results show that MLTCP converges to a stable time-isolation state after approximately two iterations, thus alleviating bandwidth contention between tasks. However, despite the theoretically high flexibility of distributed congestion control, its deployment in large-scale clusters still faces numerous challenges. Modern distributed training systems typically run on hardware-offset RDMA networks to achieve extremely low transmission latency and high bandwidth utilization. Congestion control mechanisms in such networks (such as DCQCN, HPCC, and DCTCP) are usually integrated into the NIC firmware, resulting in very low software adjustability. Therefore, the process from algorithm design to actual deployment often requires a lengthy verification and standardization cycle, making it difficult to respond promptly to the dynamic demands of upper-layer training tasks. Although programmable congestion control implementations based on FPGAs, DPDKs, or DPUs have emerged in recent years, these solutions, while offering flexibility, also introduce additional software stack overhead and energy costs, reducing overall system efficiency. Furthermore, the decisions of schemes like MLTCP are entirely based on end-side observable information (such as RTT, ECN tags, and the number of bytes sent), lacking direct awareness of the global state. Therefore, it can only gradually adjust the transmission rate through continuous feedback loops, and its scheduling behavior exhibits a gradual convergence process. In situations where task communication modes are highly dynamic and have synchronous burst characteristics, this feedback-based adjustment often suffers from lag. By the time the system detects congestion and makes rate adjustments, contention has already occurred, and link bandwidth has been inefficiently used. Overall, existing distributed congestion control schemes have yet to achieve a good balance between adaptability, low latency, and deployability.

[0007] In summary, existing centralized scheduling schemes are limited by the real-time bottleneck of global state awareness and the physical constraints of hardware queue resources, making it difficult to achieve fine-grained adaptive scheduling in large-scale dynamic scenarios. Distributed congestion control schemes, on the other hand, are hampered by incomplete end-side information and deployment difficulties due to hardware rigidity, and their feedback-based adjustment mechanisms have inherent lag. Therefore, there is an urgent need for a collaborative communication scheduling scheme that can achieve real-time, fine-grained task priority awareness in the switch data plane without relying on large amounts of hardware queue resources, in order to systematically alleviate the communication contention problems caused by multi-task concurrency. Summary of the Invention

[0008] The purpose of this invention is to propose a switch-based collaborative communication scheduler for distributed training to solve the problem of low transmission efficiency caused by link bandwidth conflicts during distributed task training in AI data centers.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] A switch cooperative communication scheduler for distributed training includes: The priority manager is used to monitor and manage the priority of data streams, record tasks and their priority information, and determine whether packets need to be marked with an explicit congestion notification (ECN) tag based on their priority. The freeze manager is used to identify and freeze tasks that have the highest priority on the local switch but a low priority on the remote switch, in order to prevent the task from competing for bandwidth resources on the local switch. ECN Threshold Manager is used to dynamically adjust ECN thresholds to optimize bandwidth utilization; The freeze manager works in conjunction with the ECN threshold manager: The freeze manager avoids bandwidth waste caused by scheduling decision errors, and the ECN threshold manager uses the available bandwidth of low-priority tasks to compensate for bandwidth waste caused by the transmission rate of the highest-priority task sub-line.

[0011] Preferably, the priority manager includes: The task logger is used to receive notification packets sent when a task establishes a long connection via a lightweight distributed notification protocol, extract connection identifiers, task identifiers, and priority metadata, and maintain connection-task mapping tables and task-priority mapping tables. A priority tracker is used to determine the currently active highest priority task by monitoring the priority of recently arriving data packets and combining this with a frozen task set. An ECN marker is used to mark data packets with ECN when the queue length exceeds a threshold, the current task is not frozen, and the current task priority is lower than the highest priority.

[0012] Preferably, the priority tracker monitors the priority set of the most recent n packets passing through the switch, excludes the priority of frozen tasks, and determines the maximum value of the remaining priorities as the highest priority currently active, where n is the bandwidth-delay product (BDP).

[0013] Preferably, the freeze manager configures a state machine for each task, the state machine including: The "cannot be frozen" status indicates that the task has been marked by the local ECN. The freezeable state indicates that the task is the highest priority task active locally and has not been marked by the local ECN. A frozen state indicates that the task has not been marked with a local ECN and the ECN mark comes from a remote switch; The state machine performs state transitions based on the local ECN tag state and the remote ECN tag state.

[0014] Preferably, when a task is in a freezeable state and the task is detected to be marked with an ECN on a remote switch, the state machine transitions the task to a freeze state; when the remote ECN mark disappears or the local ECN mark becomes effective, the state machine returns the task from the freeze state to a non-freezeable state.

[0015] Preferably, the ECN threshold manager uses k min and k max Two ECN thresholds, where k max The ECN threshold manager dynamically adjusts k as a fixed value based on link utilization and maximum queue length. min value.

[0016] Preferably, the ECN threshold manager dynamically adjusts the kmin value in the following ways: When link utilization is lower than the target value, increase k min This is intended to provide more buffer space for low-priority data streams; When the link utilization exceeds the target value and the maximum queue length is lower than the set threshold, k will be... min Set the value to a very low value to prioritize the highest priority task. When the link utilization exceeds the target value and the maximum queue length is not lower than the set threshold, k is decreased step by step according to the BDP algorithm. min value.

[0017] This invention further protects a switch cooperative communication scheduling method for distributed training, comprising: Priority management steps: Monitor and manage the priority of data streams, record tasks and their priority information, and determine whether ECN tags need to be added to data packets based on their priority; Freeze management steps: Identify and freeze tasks that have the highest priority on the local switch but a low priority on the remote switch to prevent them from competing for bandwidth resources on the local switch; and ECN threshold management steps: Dynamically adjust ECN thresholds to optimize bandwidth utilization; The freeze management step and the ECN threshold management step are executed in concert: the freeze management step avoids bandwidth waste caused by scheduling decision errors, and the ECN threshold management step uses the available bandwidth of low-priority tasks to compensate for bandwidth waste caused by the transmission rate of the highest-priority task sub-line.

[0018] Preferably, the priority management step includes: The system receives notification packets sent when a task establishes a long connection via a lightweight distributed notification protocol, extracts the connection identifier, task identifier, and priority metadata, and maintains connection-task mapping tables and task-priority mapping tables. By monitoring the priority of recently arriving data packets and combining this with a frozen task set, the highest priority task currently active can be determined; and When the queue length exceeds the threshold, the current task is not frozen, and the current task priority is lower than the highest priority, the data packet is marked with ECN.

[0019] Preferably, the freeze management step configures a state machine for each task, the state machine including a non-freezeable state, a freezeable state, and a freeze state, and performs state transitions based on the local ECN tag status and the remote ECN tag status; when a task is in the freezeable state and it is detected that the task is tagged with an ECN on a remote switch, the task is transferred to the freeze state; when the remote ECN tag disappears or the local ECN tag becomes effective, the task is returned from the freeze state to the non-freezeable state.

[0020] Compared with the prior art, the present invention has the following beneficial effects: (1) By introducing a freeze manager and a priority tracker, this invention can achieve priority differentiation and scheduling of any number of tasks at the logical level without allocating an independent physical queue for each task. Specifically, the priority tracker dynamically identifies the currently active highest priority task by monitoring recent data packets, while the freeze manager eliminates the priority inconsistency problem across switches through a state machine mechanism. Thus, it supports large-scale multi-task concurrent scenarios with minimal hardware resource overhead, fundamentally breaking through the constraint of the number of queues on the scalability of the scheduler.

[0021] (2) This invention pushes priority awareness down to the switch data plane. The task logger completes the registration and mapping of task priorities during the connection establishment phase. The priority tracker infers the current highest priority active task in real time based on recent data packets, enabling ECN marking decisions to be executed immediately on the data packet forwarding path without waiting for end-side feedback convergence. This data plane-driven working mechanism ensures the synchronization of scheduling decisions and communication traffic, effectively avoiding bandwidth waste caused by feedback lag.

[0022] (3) This invention forms a dual-track optimization architecture of "avoiding contention" and "utilizing margin" through the collaborative design of the freeze manager and the ECN threshold manager. The freeze manager accurately identifies and freezes tasks with inconsistent priorities across switches to prevent bandwidth contention caused by scheduling decision errors; the ECN threshold manager dynamically adjusts the ECN marking threshold, allowing low-priority tasks to fill the remaining bandwidth when the highest priority task does not fully utilize the bandwidth. The two optimize "competitive bandwidth waste" and "non-competitive bandwidth waste" respectively, complementing each other. Experiments show that relying on either manager alone cannot achieve optimal performance, while the collaborative work of the two can maximize the overall link bandwidth utilization while ensuring the communication efficiency of high-priority tasks.

[0023] (4) This invention monitors the priority set of the most recent BDP (Bandwidth-Delay Product) data packets through a priority tracker and dynamically calculates the currently active highest priority by combining it with the frozen task set. This enables the scheduler to perceive the actual communication status of each task at the current moment in real time, rather than relying on offline analysis or historical statistics. This fine-grained dynamic perception capability ensures that priority arbitration always reflects the current real-time competition status of the system, thereby providing an accurate basis for ECN marking and freezing decisions.

[0024] (5) This invention adopts a lightweight switch collaboration scheme based on the standard ECN mechanism. It only needs to maintain two lightweight mapping tables (Conn-to-JobID and JobID-to-Priority) in the switch data plane and perform simple packet priority sampling and state machine transitions. It does not require modification of the end-side RDMA network card firmware, nor does it rely on the global coordination of a centralized controller. This design maintains high compatibility with existing RDMA network infrastructure, avoids the scalability bottleneck of centralized schemes and the deployment cost of programmable hardware, and achieves a good balance between adaptability, low latency and deployability.

[0025] In summary, this invention systematically solves the core problem of multi-task communication competition in distributed training scenarios through the organic collaboration of priority manager, freeze manager, and ECN threshold manager. It has achieved significant technical progress in overcoming hardware resource limitations, eliminating feedback lag, improving bandwidth utilization, achieving fine-grained perception, and ensuring deployability, providing a practical solution for efficient network scheduling of large-scale AI data centers. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings involved in the embodiments are now briefly described. Obviously, the drawings in the following description are merely illustrative of some embodiments of the present invention. For those skilled in the art, other forms of drawings can be constructed based on these drawings without creative effort.

[0027] Figure 1 This is an overall architecture diagram of the switch collaborative communication scheduler for distributed training proposed in this invention; Figure 2 This is a schematic diagram of the priority tracker proposed in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the frozen state machine proposed in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of the ECN threshold manager proposed in Embodiment 1 of the present invention; Figure 5 This is a diagram showing the overall performance of the scheduler proposed in Embodiment 2 of the present invention. Detailed Implementation

[0028] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0029] This invention proposes a switch-based collaborative communication scheduler for distributed training to systematically alleviate communication contention problems caused by multi-task concurrency. Specifically, this invention can accurately identify the tasks involved in contention and their priority relationships. In real-world systems, communication contention is often not globally uniform but exhibits significant spatiotemporal heterogeneity. Some tasks may generate bursts of traffic during specific communication phases (e.g., AllReduce or parameter synchronization), while others are in the computation phase and consume almost no network bandwidth. Therefore, the scheduler needs fine-grained observation capabilities, not only distinguishing the priorities of different tasks but also dynamically sensing whether they are currently in an active communication state and experiencing bottlenecks. Secondly, based on this, it leverages this fine-grained awareness to achieve near-optimal scheduling decisions while avoiding limitations imposed by hardware or application characteristics. Traditional solutions often rely on limited hardware resources (such as the number of switch queues) or specific application assumptions (such as predictable communication patterns and perfect task staggering), which are difficult to apply in real-world multi-tenant environments. Therefore, a more ideal scheduler should possess the following capabilities. On the one hand, it achieves near-global optimal bandwidth allocation logically (e.g., minimizing task completion time or reducing tail latency); on the other hand, it does not rely on scarce resources (e.g., a fixed number of priority queues) or strong assumptions (e.g., complete task compatibility) in its implementation.

[0030] The switch collaborative communication scheduler for distributed training proposed in this invention will be described below with reference to the relevant accompanying drawings and specific examples.

[0031] Example 1: Please see Figure 1 This embodiment provides a Switch Cooperative Communication Scheduler (SCCS) for distributed training, comprising three main modules: a priority manager, a freeze manager, and an ECN threshold manager. The priority manager monitors and manages the priority of data flows, records tasks and their priority information, and determines whether data packets need to be marked with an ECN tag based on priority. The freeze manager identifies and freezes tasks that have the highest priority on the local switch but a low priority on remote switches, preventing them from competing for bandwidth resources on the local switch. The ECN threshold manager dynamically adjusts the ECN threshold to optimize bandwidth utilization. These three modules work together to achieve fine-grained communication scheduling in multi-task concurrent scenarios. Details are as follows.

[0032] (a) Priority Manager like Figure 2 As shown, the priority manager includes three components: task logger, priority tracker, and ECN marker.

[0033] (1) Task recorder Each distributed training task involves multiple servers. Before training, each server establishes multiple RDMA reliable connection queues (QPs) for communication. When establishing a long-lived connection, SCCS uses a lightweight distributed notification protocol to send a customized notification packet to the switch. This packet contains a connection identifier (such as the target IP address and QPN), a task ID, and priority metadata. Upon receiving the notification, the switch forwards it to the control plane. The system maintains two tables in the control plane: the Conn-to-JobID table and the JobID-to-Priority table. The Conn-to-JobID table identifies the job corresponding to each incoming data packet, while the JobID-to-Priority table stores the priority value for each task. The control plane sorts all tasks according to priority and updates the JobID-to-Priority table in real time. When a long-lived connection terminates, the system notifies the switch to clear the record for that connection from the Conn-to-JobID table.

[0034] (2) Priority tracker After the task logger completes the mapping between connections and task priorities, the switch can determine the JobID and priority of each incoming data packet. Because distributed training tasks exhibit an "on-off" communication mode, with tasks only active during the transmission phase, the highest priority active task changes dynamically. The priority tracker infers the current highest priority active task by monitoring the priority of recently arrived data packets.

[0035] Specifically, for the n most recent packets passing through the switch (where n is the bandwidth-delay product BDP), their priority set is P={p1,p2,…,p...} n The priority tracker first excludes tasks whose priorities belong to the frozen task set F, and then takes the maximum value from the remaining priorities as the currently active highest priority P. active That is, P active =max{p i ∈P| Job(p i ) F}. This P active The value is then used for priority arbitration and ECN tagging decisions.

[0036] (3) ECN marker When a switch receives a data packet, the ECN tagger determines whether to perform ECN tagging based on the queue length and the current data packet's priority p. ECN tagging is enabled only when the following three conditions are met simultaneously: (i) The current queue length exceeds k min Threshold; (ii) The current task is not in a frozen state; (iii) The priority p of the current data packet is lower than P. active .

[0037] When all three conditions are met, the ECN marker marks the data packet with ECN to notify the sender to reduce the transmission rate.

[0038] Example 2: Based on Embodiment 1, but with a difference, in order to further verify the effectiveness of the present invention, the performance of the scheduler (SCCS) proposed by this invention was evaluated in a simulation environment, and a comparative experiment was conducted with a variety of existing technical solutions. The specific contents are as follows.

[0039] 1. Simulation Platform and Experimental Configuration The simulation platform in this embodiment is built on OMNeT++. The simulation experiment was conducted on a server running Ubuntu 20.04.6 LTS, which has a kernel version of 5.4.0-86-generic, and is equipped with an Intel® Xeon® Silver 4314 processor with a clock speed of 2.40GHz and 32GB of memory.

[0040] The simulator supports multi-tenant training mode and has a built-in job scheduler that can receive task requests and perform GPU allocation. The default configuration uses a front-end priority scheduling strategy on the worker nodes, with total communication transmitted in 256KB blocks. The network topology adopts the widely used Leaf-Spine architecture, with a default configuration containing 16 racks, each rack containing 8 hosts, each host equipped with 8 GPUs, for a total of 1024 GPUs in the cluster. It uses 100 network nodes, a maximum transmission unit (MTU) of 1500, a link propagation delay of 1µs, and a default maximum number of switch queues of 8. To verify performance at different scales, this embodiment also compares configurations equipped with 2048, 4096, and 8192 GPUs. DCTCP is used as the congestion control protocol by default.

[0041] 2. Comparison of Schemes To comprehensively evaluate the performance advantages of the SCCS of this invention, this embodiment selects the following five existing technical solutions as comparison benchmarks: ByteScheduler: A byte-level traffic scheduling scheme that is unaware of communication contention issues between jobs; Cassini: A centralized scheduler that uses a static timing offset mechanism to set start offsets for each task through offline analysis to achieve interleaved communication phases; Crux: A scheduler that performs static job-level priority sorting based on GPU utilization, limited by the number of hardware priority queues; Sincronia: A centralized scheduler that introduces the concept of collaborative flow scheduling, capable of detecting network bottlenecks and dynamically adjusting collective communication priorities; MLTCP: A distributed congestion control algorithm based on end-side feedback, which uses the percentage of bytes already transmitted as a congestion awareness indicator to dynamically adjust the transmission rate.

[0042] In addition, this embodiment sets Crux(∞) as a performance upper bound reference. This scheme assumes that the switch has an infinite number of priority queues, which can achieve ideal scheduling where each task occupies a queue exclusively, representing the theoretical optimal performance under this technical route.

[0043] 3. Experimental Results Please see Figure 5 , Figure 5 The results show the comparison between SCCS and the above-mentioned schemes in three core metrics: throughput, iteration time, and GPU utilization.

[0044] (1) Throughput performance ByteScheduler performs the worst among all compared solutions due to its inability to detect inter-job communication contention. SCCS achieves a 2.8x throughput improvement compared to ByteScheduler. Cassini relies solely on a static timing offset mechanism and cannot handle dynamic communication fluctuations in multi-tenant environments; SCCS achieves up to a 2.2x throughput improvement over Cassini. Crux, limited by the number of priority queues, experiences priority mapping conflicts when the number of tasks exceeds the number of queues; SCCS achieves up to a 2.0x throughput improvement compared to Crux. While Sincronia can dynamically adjust collective communication priorities, the micro-burst control overhead introduced by its centralized control plane limits further performance improvements; SCCS still achieves a 1.6x throughput improvement compared to Sincronia. MLTCP, lacking global state awareness, exhibits suboptimal behavior in its feedback-based scheduling decisions; SCCS achieves up to a 2.3x throughput improvement compared to MLTCP.

[0045] (2) Iteration time performance In terms of average iteration time, SCCS reduces it by 62.8% compared to ByteScheduler, 51.9% compared to Cassini, 43.8% compared to Crux, 35.1% compared to Sincronia, and 56.0% compared to MLTCP. These results demonstrate that SCCS effectively reduces iteration latency caused by bandwidth contention in multi-task concurrent scenarios through real-time priority awareness in the data plane and collaborative scheduling by two managers.

[0046] (3) GPU utilization performance In terms of GPU utilization, SCCS improves efficiency by 23.5% compared to ByteScheduler, 20.2% compared to Cassini, 16.9% compared to Crux, 12.8% compared to Sincronia, and 23.1% compared to MLTCP. This improvement in GPU utilization indicates that SCCS effectively reduces the blocking time of GPU computation during network communication, thus improving the efficiency of overlapping computation and communication.

[0047] 4. Experimental Conclusions Experimental results show that the SCCS proposed in this invention outperforms existing centralized scheduling schemes (ByteScheduler, Cassini, Crux, Sincronia) and distributed congestion control schemes (MLTCP) across all evaluation metrics. Specifically, compared to the second-best performing Sincronia, SCCS still achieves a 1.6x increase in throughput, a 35.1% reduction in iteration time, and a 12.8% increase in GPU utilization. Furthermore, experiments show that the performance of SCCS is close to the upper bound of the theoretically optimal scheme Crux(∞), fully validating that this invention, without relying on a large number of physical queue resources, can achieve efficient scheduling of concurrent communication traffic for multiple tasks through the collaborative mechanism of a priority manager, a freeze manager, and an ECN threshold manager, effectively alleviating network bandwidth contention problems in distributed training scenarios.

[0048] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A switch cooperative communication scheduler for distributed training, characterized in that, include: The priority manager is used to monitor and manage the priority of data streams, record tasks and their priority information, and determine whether packets need to be marked with an explicit congestion notification (ECN) tag based on their priority. The freeze manager is used to identify and freeze tasks that have the highest priority on the local switch but a low priority on the remote switch, in order to prevent the task from competing for bandwidth resources on the local switch. ECN Threshold Manager is used to dynamically adjust ECN thresholds to optimize bandwidth utilization; The freeze manager works in conjunction with the ECN threshold manager: The freeze manager avoids bandwidth waste caused by scheduling decision errors, and the ECN threshold manager uses the available bandwidth of low-priority tasks to compensate for bandwidth waste caused by the transmission rate of the highest-priority task sub-line.

2. The switch collaborative communication scheduler according to claim 1, characterized in that, The priority manager includes: The task logger is used to receive notification packets sent when a task establishes a long connection via a lightweight distributed notification protocol, extract connection identifiers, task identifiers, and priority metadata, and maintain connection-task mapping tables and task-priority mapping tables. A priority tracker is used to determine the currently active highest priority task by monitoring the priority of recently arriving data packets and combining this with a frozen task set. An ECN marker is used to mark data packets with ECN when the queue length exceeds a threshold, the current task is not frozen, and the current task priority is lower than the highest priority.

3. The switch collaborative communication scheduler according to claim 2, characterized in that, The priority tracker monitors the priority set of the n most recent packets passing through the switch, excludes the priority of frozen tasks, and determines the maximum value of the remaining priorities as the highest priority currently active, where n is the bandwidth-delay product (BDP).

4. The switch collaborative communication scheduler according to claim 1, characterized in that, The freeze manager configures a state machine for each task, and the state machine includes: The "cannot be frozen" status indicates that the task has been marked by the local ECN. The freezeable state indicates that the task is the highest priority task active locally and has not been marked by the local ECN. A frozen state indicates that the task has not been marked with a local ECN and the ECN mark comes from a remote switch; The state machine performs state transitions based on the local ECN tag state and the remote ECN tag state.

5. The switch collaborative communication scheduler according to claim 4, characterized in that, When a task is in a freezeable state and the task is detected to be marked with an ECN on a remote switch, the state machine will transition the task to a freeze state. When the remote ECN tag disappears or the local ECN tag becomes active, the state machine returns the task from the frozen state to the non-freezing state.

6. The switch collaborative communication scheduler according to claim 1, characterized in that, The ECN threshold manager uses k min and k max Two ECN thresholds, where k max The ECN threshold manager dynamically adjusts k as a fixed value based on link utilization and maximum queue length. min value.

7. The switch collaborative communication scheduler according to claim 6, characterized in that, The ECN threshold manager dynamically adjusts the kmin value in the following ways: When link utilization is lower than the target value, increase k min This is intended to provide more buffer space for low-priority data streams; When the link utilization exceeds the target value and the maximum queue length is lower than the set threshold, k will be... min Set the value to a very low value to prioritize the highest priority task. When the link utilization exceeds the target value and the maximum queue length is not lower than the set threshold, k is decreased step by step according to the BDP algorithm. min value.

8. A switch cooperative communication scheduling method for distributed training based on the scheduler described in any one of claims 1-7, characterized in that, include: Priority management steps: Monitor and manage the priority of data streams, record tasks and their priority information, and determine whether ECN tags need to be added to data packets based on their priority; Freeze management steps: Identify and freeze tasks that have the highest priority on the local switch but a low priority on the remote switch to prevent them from competing for bandwidth resources on the local switch; and ECN threshold management steps: Dynamically adjust ECN thresholds to optimize bandwidth utilization; The freeze management step and the ECN threshold management step are executed in concert: the freeze management step avoids bandwidth waste caused by scheduling decision errors, and the ECN threshold management step uses the available bandwidth of low-priority tasks to compensate for bandwidth waste caused by the transmission rate of the highest-priority task sub-line.

9. The switch cooperative communication scheduling method according to claim 8, characterized in that, The priority management steps include: The system receives notification packets sent when a task establishes a long connection via a lightweight distributed notification protocol, extracts the connection identifier, task identifier, and priority metadata, and maintains connection-task mapping tables and task-priority mapping tables. By monitoring the priority of recently arriving data packets and combining this with a frozen task set, the highest priority task currently active can be determined; and When the queue length exceeds the threshold, the current task is not frozen, and the current task priority is lower than the highest priority, the data packet is marked with ECN.

10. The switch cooperative communication scheduling method according to claim 8, characterized in that, The freeze management step configures a state machine for each task, which includes a non-freezeable state, a freezeable state, and a freeze state, and performs state transitions based on the local ECN tag state and the remote ECN tag state. When a task is in a freezeable state and it is detected that the task is marked with an ECN on a remote switch, the task is put into a freeze state. When the remote ECN tag disappears or the local ECN tag becomes active, the task will be returned from the frozen state to the non-frozen state.

Images