Apparatus and method for supporting path rescheduling in a parallel data communication system

Abstract

The application discloses a kind of equipment parallel data communication system and method supporting path rescheduling, system includes first switch, second switch and multiple devices to be communicated, first switch includes multiple first ports, second switch includes multiple second ports, each device is electrically connected with first port, second port respectively to form first, second path;First, second switch all include path monitoring module, the link congestion of the first, second path corresponding to the first, second port corresponding to the path monitoring module of first, second switch is determined and is transmitted to device;Device includes transmission control module, and transmission control module controls the data packet to be transmitted to pass one or more in first, second path according to the link congestion of first path and the link congestion of second path transmission.The application supports inter-device parallel communication, retransmission communication and can switch path according to link congestion in data transmission, can improve the communication efficiency between multiple devices.

Description

Technical Field

[0001] This invention relates to the fields of computer and communication technology, and in particular to a device parallel data communication system and method that supports path rescheduling. Background Technology

[0002] With the rapid development of artificial intelligence technology, the training of large-scale models (such as the GPT series and Transformer series) places increasingly higher demands on computing power and data transmission efficiency. To meet these computing power requirements, data centers typically employ multiple GPUs or ASICs for model training. A typical AI training topology is as follows: Figure 1 As shown, multiple ASICs or GPUs are interconnected through a PCIe switch and communicate with the CPU through the underlying PCIe switch.

[0003] During training, common communication modes include broadcast, all-reduce, and P2P transmission. Broadcast is used to synchronize the latest parameters from the master node (or a master parameter server) to all computing nodes; all-reduce is used for multiple nodes to collaboratively complete reduction operations (such as parameter summation or averaging); and P2P (peer-to-peer) transmission is used for devices to exchange data.

[0004] The aforementioned communication patterns typically account for 30% to 60% of the overall training time; therefore, communication efficiency directly determines model training efficiency. For example... Figure 1 The architecture shown depicts a current PCIe topology where all ASIC devices are interconnected via PCIe switches. However, current PCIe switches are mostly designed with static paths, meaning the bandwidth and resources of the connection channels are fixed. This approach presents the following significant problems.

[0005] Problem 1: Uneven communication traffic causes path congestion. In multi-GPU training, the communication traffic of each pair of devices is not the same. Some device groups frequently exchange data due to algorithm characteristics (such as model parallelism or hybrid parallelism strategies), resulting in high-traffic communication; while other devices communicate less, but the PCIe path bandwidth is fixed. This leads to congestion on some links, while the bandwidth of other links is not fully utilized.

[0006] Problem 2: Lack of communication awareness and path scheduling capabilities. Traditional PCIe switch architecture uses fixed-path transmission and cannot dynamically adjust according to actual communication needs. Especially in the later stages of training, when the parallel or pipelined parallel strategies of different levels of models change, the data transmission mode changes accordingly, but the path configuration remains fixed, further exacerbating the problems of low resource utilization and low throughput.

[0007] Question 3: Inability to support concurrent and efficient transmission. The current PCIe architecture is usually a strict single-path static route, which can easily lead to path contention when multiple pairs of devices initiate data transmission at the same time, making it impossible to fully parallelize communication and resulting in a decrease in training efficiency.

[0008] The above background information is provided only to assist in understanding the inventive concept and technical solution of this invention. It does not necessarily belong to the prior art of this application, nor does it necessarily provide technical teaching. In the absence of clear evidence that the above information was disclosed before the filing date of this application, the above background information should not be used to evaluate the novelty and inventiveness of this application. Summary of the Invention

[0009] The purpose of this invention is to provide a device parallel data communication system and method that supports path rescheduling, enabling parallel communication and retransmission between devices, as well as switching links based on link congestion during data transmission, thereby improving the communication efficiency between multiple devices.

[0010] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0011] A device parallel data communication system supporting path rescheduling includes a first switch, a second switch, and multiple devices to communicate with. The devices are PCIe devices such as GPUs and ASICs. The first switch includes multiple first ports, and the second switch includes multiple second ports. Each device is electrically connected to both the first and second ports. The transmission path between a device and the first port is configured as a first path, and the transmission path between a device and the second port is configured as a second path.

[0012] Both the first switch and the second switch include a path monitoring module. The path monitoring module of the first switch is configured to determine the link congestion status of the first path corresponding to each first port and transmit it to the device. The path monitoring module of the second switch is configured to determine the link congestion status of the second path corresponding to each second port and transmit it to the device.

[0013] The device includes a transmission control module configured to control the data packets to be transmitted to be transmitted through one or more of the first path and the second path based on the link congestion status of the first path and the second path.

[0014] Furthermore, following any one or a combination of the aforementioned technical solutions, the transmission control module controls the data packet to be transmitted to be transmitted through the path with less link congestion between the first path and the second path.

[0015] Furthermore, following any or a combination of the aforementioned technical solutions, the transmission control module is further configured to split the data packet to be transmitted into multiple sub-data packets, each sub-data packet being configured with a location tag, the location tag being configured to indicate the position of the sub-data packet within the data packet, and different sub-data packets having different location tags;

[0016] The transmission control module is also configured to control the sub-data packets to be transmitted through one of the first path and the second path, with different sub-data packets being transmitted through the same or different paths.

[0017] Furthermore, in accordance with any or a combination of the aforementioned technical solutions, the data packet is configured with a priority level. If the priority level of the data packet to be transmitted is higher than a preset level threshold, the transmission control module controls part of the sub-data packet to pass through the first path, and controls the other part of the sub-data packet to be transmitted through the second path.

[0018] or,

[0019] The data packets are configured with priority levels. If the priority level of the data packet to be transmitted is higher than a preset level threshold and the link congestion of the first path and the second path both meet the preset requirements, then the transmission control module controls some of the sub-data packets to pass through the first path, and controls the other part of the sub-data packets to be transmitted through the second path.

[0020] Furthermore, in accordance with any or a combination of the aforementioned technical solutions, the device further includes a data reassembly module, which is configured to reassemble multiple sub-data packets into the data packet based on the location tags corresponding to each of the sub-data packets.

[0021] Furthermore, following any or a combination of the aforementioned technical solutions, the data reassembly module is further configured to determine whether there are missing sub-data packets based on the location tags corresponding to each of the sub-data packets. If so, the data reassembly module sends a missing sub-data packet retransmission request to the transmission control module so that the transmission control module at least retransmits the missing sub-data packets.

[0022] Furthermore, following any one or a combination of the aforementioned technical solutions, the path monitoring module is configured to determine a link congestion score based on the link congestion status of the port, wherein the port includes the first port and the second port;

[0023] The transmission control module is configured to execute the following path transmission control strategy:

[0024] When the congestion scores corresponding to the first port and the second port are both lower than the first score threshold, the transmission control module controls the data packet to be transmitted to be transmitted through the first path and / or the second path.

[0025] And / or,

[0026] When the congestion score of one of the first port and the second port is higher than the second score threshold, and the congestion score of the other port is not higher than the second score threshold, the transmission control module controls the data packet to be transmitted to be transmitted through the path corresponding to the port whose congestion score is not higher than the second score threshold, where the second score threshold is greater than the first score threshold.

[0027] Furthermore, following any one or a combination of the aforementioned technical solutions, the transmission control module is also configured to execute the following path transmission control strategy:

[0028] When the congestion scores corresponding to the first port and the second port are both higher than the third score threshold, the transmission control module will split the data packet to be transmitted into multiple sub-data packets, and control some of the sub-data packets to be transmitted through the first path, while the other part of the sub-data packets will be transmitted through the second path.

[0029] Furthermore, following any one or a combination of the aforementioned technical solutions, the path monitoring module is configured to calculate the link congestion score based on the following formula:

[0030] Link congestion score = α × delay weight + β × FIFO occupancy weight + γ × error rate weight;

[0031] Wherein, the delay weight = (current delay - baseline delay) / baseline delay, the FIFO occupancy rate weight = current FIFO occupancy rate / 100%, the error rate weight = current CRC error rate / baseline error rate, and α, β, and γ are adjustment coefficients.

[0032] Furthermore, following any one or a combination of the aforementioned technical solutions, the transmission control module is also configured to execute the following path transmission control strategy:

[0033] If the link congestion of the current path does not meet the preset data transmission requirements, and the current path is one of the first path and the second path, then the transmission control module controls the data packet to be transmitted to switch to transmission through the other path.

[0034] According to another aspect of the present invention, a device parallel data communication system supporting path rescheduling is provided, including a first switch, a second switch and a plurality of devices to communicate, each of the devices including a third port and a fourth port, the third port being configured to be electrically connected to the first switch, the fourth port being configured to be electrically connected to the second switch, the transmission path between the third port and the first switch being configured as a first path, and the transmission path between the fourth port and the second switch being configured as a second path;

[0035] The device includes a path monitoring module and a transmission control module. The path monitoring module is configured to determine the link congestion status of the first path corresponding to the third port and transmit it to the transmission control module. The path monitoring module is configured to determine the link congestion status of the second path corresponding to the fourth port and transmit it to the transmission control module.

[0036] The transmission control module is configured to control the data packets to be transmitted to be transmitted through one or more of the first path and the second path based on the link congestion status of the first path and the second path.

[0037] Furthermore, following any one or a combination of the aforementioned technical solutions, the transmission control module controls the data packet to be transmitted to be transmitted through the path with less link congestion between the first path and the second path.

[0038] According to another aspect of the present invention, a device parallel data communication method supporting path rescheduling is provided, comprising the following steps:

[0039] For multiple devices to communicate, each device is pre-configured to be electrically connected to a first switch and a second switch respectively, and the communication link between the device and the first switch is determined as a first path, and the communication link between the device and the second switch is determined as a second path;

[0040] The link congestion status of the first path and the second path is obtained, and based on the link congestion status of the first path and the second path, the data to be transmitted is controlled to be transmitted through one or more of the first path and the second path.

[0041] Furthermore, following any one or a combination of the aforementioned technical solutions, the device parallel data communication method supporting path rescheduling further includes the following steps:

[0042] The data packets to be transmitted are controlled to be transmitted through the path with less link congestion between the first path and the second path.

[0043] The beneficial effects of the technical solution provided by this invention are as follows:

[0044] a. The device parallel data communication system supporting path rescheduling proposed in this invention can sense the communication mode between devices and the link congestion of each data transmission path through the path monitoring module. It can automatically pre-allocate path resources and switch to the appropriate path to transmit data according to the link congestion during the transmission process using the transmission control module, thereby realizing flexible resource scheduling and communication concurrency. This can improve the training efficiency of the overall system and significantly reduce the latency during data transmission. It is particularly suitable for application scenarios that require low latency, such as AI training, HPC and high-performance data exchange.

[0045] b. The device parallel data communication system and method supporting path rescheduling proposed in this invention can further improve the communication efficiency between multiple devices by splitting the data packet to be transmitted into multiple sub-data packets and flexibly adjusting the transmission path of each sub-data packet according to the link congestion of the first and second paths. This includes transmitting multiple sub-data packets through one or more optimal paths, transmitting corresponding sub-data packets through different paths, and switching to another path to transmit sub-data packets when one path is severely congested.

[0046] c. This invention utilizes a data reassembly module to reassemble each sub-data packet based on its location tag, which improves the reliability of data transmission, avoids data transmission errors, and can detect missing sub-data packets. It also initiates retransmission requests only for missing sub-data packets, thus balancing data transmission efficiency and reliability. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A schematic diagram of the existing PCIe device communication architecture;

[0049] Figure 2 A schematic diagram of a first type of switch supporting path rescheduling, provided as an exemplary embodiment of the present invention;

[0050] Figure 3 A schematic diagram of a second type of switch supporting path rescheduling, provided as an exemplary embodiment of the present invention;

[0051] Figure 4 A schematic diagram of a first device supporting path rescheduling, provided as an exemplary embodiment of the present invention;

[0052] Figure 5 A schematic diagram of a second device supporting path rescheduling, provided as an exemplary embodiment of the present invention;

[0053] Figure 6 A schematic diagram illustrating the principle of system switching data transmission paths, provided as an exemplary embodiment of the present invention;

[0054] Figure 7 A schematic diagram illustrating the principle of data splitting and reassembly provided for an exemplary embodiment of the present invention;

[0055] Figure 8 A flowchart of a device parallel data communication method supporting path rescheduling is provided as an exemplary embodiment of the present invention. Detailed Implementation

[0056] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0057] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0058] Based on the aforementioned problems in the existing technology, this invention proposes a device parallel data communication system and method that supports path rescheduling. By sensing the link congestion between devices, it automatically pre-allocates path resources and enables flexible resource scheduling and communication concurrency, thereby improving the overall system's data transmission efficiency and thus improving the model's training efficiency.

[0059] In one embodiment of the present invention, a device parallel data communication system supporting path rescheduling is provided, such as... Figures 2 to 4As shown, the device includes a first switch, a second switch, and multiple devices to communicate. The first switch includes multiple first ports, and the second switch includes multiple second ports. Each device is electrically connected to both the first and second ports. The transmission path between the device and the first port is configured as a first path, and the transmission path between the device and the second port is configured as a second path.

[0060] Both the first switch and the second switch include a path monitoring module. The path monitoring module of the first switch is configured to determine the link congestion status of the first path corresponding to each first port and transmit it to the device. The path monitoring module of the second switch is configured to determine the link congestion status of the second path corresponding to each second port and transmit it to the device.

[0061] The device includes a transmission control module configured to control the data packets to be transmitted to be transmitted through one or more of the first path and the second path based on the link congestion status of the first path and the second path.

[0062] Preferably, the transmission control module controls the data packet to be transmitted to be transmitted through the path with less link congestion between the first path and the second path. Alternatively, the transmission control module is further configured to split the data packet to be transmitted into multiple sub-data packets, each sub-data packet being configured with a location tag indicating its position within the data packet, and different sub-data packets having different location tags. The transmission control module is also configured to control the sub-data packets to be transmitted through one of the first path and the second path, with different sub-data packets being transmitted through the same or different paths.

[0063] In one embodiment of the present invention, such as Figure 2 As shown, the Path Monitor Unit (PMU) is electrically connected to each port of its corresponding switch. The PMU includes interconnected data acquisition submodules and data processing submodules.

[0064] The data acquisition submodule is configured to periodically collect the following important parameters in real time.

[0065] (1) Average transmission delay: Monitors the average time it takes for each data packet to travel through the link.

[0066] (2) FIFO Buffer Occupancy: The space occupied by the FIFO buffer inside the PCIe port. A high FIFO occupancy rate indicates serious data backlog. For example: The maximum FIFO buffer is 256KB, and the current buffer occupancy is 220KB, with a FIFO occupancy rate of 86%.

[0067] (3) CRC error rate and retransmission count: Monitor link stability. If the CRC error rate increases or a large number of retransmissions occur, it is considered that the link is abnormal or damaged.

[0068] The data acquisition submodule includes a latency counter, a FIFO status monitor, and a CRC error counter.

[0069] The latency counter works by inserting a timestamp into the header of PCIe data packets. When a packet enters the switch, it is recorded as T1; when it leaves the switch, it is recorded as T2. The transmission delay is calculated by subtracting T1 from T2. Specifically, a hardware timer with nanosecond-level precision can be used. Each PCIe port has an independent timer. Data processing employs a sliding window algorithm, calculating the average latency of the most recent 100 data packets every 10μs. The technical challenge is clock synchronization. This application proposes two solutions: one is to configure a PMU within the switch to collect data transmission information from each port; the other is to configure a PMU for each port, but all PMUs use a unified clock source to avoid clock skew between different ports.

[0070] The FIFO status monitor operates by monitoring the receive buffer (Rx FIFO) and transmit buffer (Tx FIFO) of each port within the PCIe switch. Monitoring is achieved by reading the write and read pointer positions in the FIFO control register to calculate the currently occupied space. This application employs hardware-level monitoring and updates the FIFO status every clock cycle to ensure real-time monitoring.

[0071] The CRC error counter works by setting error detection points at both the PCIe physical layer and the data link layer. Using a hardware counter, the count increments each time a CRC error is detected to count the total number of errors. For example, with a 1ms statistical period, the error rate is calculated as: number of erroneous packets / total number of packets. Anomaly detection: when the error rate exceeds 0.1%, it is considered a link anomaly.

[0072] like Figure 2 As shown, the data processing submodule includes a statistical analysis unit, a congestion score calculator, and a threshold comparator. The processing logic of the statistical analysis unit is as follows.

[0073] Data fusion: Collect raw data from delay counters, FIFO monitors, and error statisticians.

[0074] Filtering: Use the exponential moving average algorithm to eliminate sudden fluctuations. The formula is: New value = α × current value + (1-α) × historical average.

[0075] Trend analysis: Calculate the first and second derivatives of the data to determine the changing trend of performance indicators.

[0076] Anomaly detection: The 3σ criterion is used to detect outliers; data that exceeds the normal range will be marked.

[0077] The congestion score calculator preferably employs a multi-dimensional weighted algorithm. Specifically, the link congestion score can be calculated based on the following formula: Link congestion score = α × Delay weight + β × FIFO occupancy weight + γ × Error rate weight, where, delay weight = (current delay - baseline delay) / baseline delay, FIFO occupancy weight = current FIFO occupancy rate / 100%, error rate weight = current CRC error rate / baseline error rate, and α, β, and γ are adjustment coefficients. Preferably, α = 0.4, β = 0.4, and γ = 0.2.

[0078] More preferably, a non-linear mapping is used, such as employing the Sigmoid function, to map the linear score to the 0-1 range, avoiding the influence of extreme values. Dynamic threshold: The scoring threshold is dynamically adjusted based on historical data to adapt to different network load characteristics.

[0079] The threshold comparator is configured to compare one or more of the following: transmission delay, FIFO state (FIFO space occupancy), and CRC error, with their respective thresholds and output the comparison results.

[0080] Preferably, such as Figure 3As shown, the data processing submodule (PMU) further includes a communication interface submodule, which comprises a control signal transmitter and a status report generator. The status report generator is configured to determine link congestion based on data transmitted by the data processing submodule. This congestion can be expressed as a congestion score or a link status evaluation level. The control signal transmitter is configured to generate and send control signals / evaluation results from the PMU to the data transmission control module.

[0081] Preferably, the signal sent by the control signal transmitter to the data transmission control module includes three fields: path ID, congestion level, and suggested action. The suggested action may include: link idle, data packet allocation can be increased, link is in normal use, adding / reducing data packets is not recommended, link is congested, new data packet allocation needs to be reduced, or the path needs to be switched. For example, for a path, if its congestion score is <0.3, the link is idle and data packet allocation can be increased; 0.3 ≤ congestion score <0.7: normal use; congestion score ≥0.7: link is congested, new data packet allocation should be reduced, or the path should be switched.

[0082] See Figure 3 The GPU Transmission Control Unit (GTCU) is configured to perform data splitting, path selection, and transmission control on the data packets to be transmitted. Specifically, the GTCU module initiates a DMA request on the GPU memory side, reads the large data packet to be transmitted, and splits it into N sub-packets, each with a fixed size (e.g., 64KB). A Packet Header is then added to each sub-packet. Preferably, the Packet Header includes a source ID, a target ID, a sequence ID, and a path ID, for example, in the format [SourceID][TargetID][SeqID][PathID]. The GTCU writes different sub-packets into multiple Tx Queues, each Tx Queue using either a first path or a second path for transmission. Different Tx Queues can use different paths for parallel transmission; for example, Tx Queue 0 transmits through the port of the uplink switch (i.e., the first switch), while Tx Queue 1 transmits through the ports of the downlink switch and the second switch. This allows each transmission queue to operate independently, enabling concurrent packet transmission.

[0083] Specifically, the GTCU control signal receiving mechanism includes: priority processing: the GTCU maintains a priority queue, and emergency signals (such as path failures) are processed first; state synchronization: after receiving a signal, the GTCU updates its internal path state table to ensure the consistency of path selection.

[0084] The dynamic path adjustment process of the data transmission control module includes the following aspects: (1) Path evaluation: The GTCU re-evaluates all available paths based on the congestion score provided by the PMU; (2) Load redistribution: Calculate the load distribution of each path and find the optimal redistribution scheme; (3) Smooth switching: Instead of immediately switching all data streams, the migration is carried out gradually to avoid causing new congestion; (4) Effect verification: After switching, the performance of the new path is continuously monitored, and if the effect is not good, it is rolled back.

[0085] The specific implementation mechanism of multi-queue management includes two aspects: queue creation and binding process and queue scheduling algorithm principle. The queue creation and binding process includes: (1) initialization stage: GTCU creates a corresponding sending queue for each available path according to the physical topology of PCIe Switch; (2) dynamic binding: the binding relationship between queue and physical port can be dynamically adjusted, and a queue can be bound to different ports; (3) resource allocation: each queue is allocated a fixed memory space and DMA channel to avoid resource competition between queues.

[0086] The principles of queue scheduling algorithms include: (1) Priority scheduling: high-priority data (such as gradient synchronization) is sent first, and low-priority data (such as log transmission) is delayed; (2) Fairness guarantee: weighted fair queue algorithm is used to ensure that each queue has a minimum bandwidth guarantee; (3) Congestion control: when a queue is too crowded, new data packets are suspended from being sent to that queue.

[0087] The specific implementation mechanism for multi-queue management includes queue creation and binding processes. During the initialization phase: GTCU creates a corresponding sending queue for each available path based on the PCIe switch's physical topology; dynamic binding: the binding relationship between queues and physical ports can be dynamically adjusted, allowing a queue to be bound to different ports; resource allocation: each queue is allocated a fixed amount of memory space and DMA channels to avoid resource contention between queues. The queue scheduling algorithm principles include: priority scheduling: high-priority data (such as gradient synchronization) is sent first, while low-priority data (such as log transmission) is delayed; fairness guarantee: a weighted fair queue algorithm is used to ensure that each queue has a minimum bandwidth guarantee; congestion control: when a queue becomes too congested, the delivery of new data packets to that queue is suspended.

[0088] The transmission control module is configured to execute one or more of the following path transmission control strategies in combination.

[0089] The data packets are configured with priority levels. If the priority level of the data packet to be transmitted is higher than a preset level threshold, the transmission control module controls some of the sub-data packets to pass through the first path, and controls the other part of the sub-data packets to be transmitted through the second path. More preferably, the data packets are configured with priority levels. If the priority level of the data packet to be transmitted is higher than a preset level threshold and the link congestion of both the first path and the second path meets preset requirements, the transmission control module controls some of the sub-data packets to pass through the first path, and controls the other part of the sub-data packets to be transmitted through the second path.

[0090] When the congestion scores corresponding to the first port and the second port are both lower than the first score threshold, the transmission control module controls the data packet to be transmitted to be transmitted through the first path and / or the second path.

[0091] When the congestion score of one of the first port and the second port is higher than the second score threshold, and the congestion score of the other port is not higher than the second score threshold, the transmission control module controls the data packet to be transmitted to be transmitted through the path corresponding to the port whose congestion score is not higher than the second score threshold, where the second score threshold is greater than the first score threshold.

[0092] When the congestion scores corresponding to the first port and the second port are both higher than the third score threshold, the transmission control module will split the data packet to be transmitted into multiple sub-data packets, and control some of the sub-data packets to be transmitted through the first path, while the other part of the sub-data packets will be transmitted through the second path.

[0093] If the link congestion of the current path does not meet the preset data transmission requirements, and the current path is one of the first path and the second path, then the transmission control module controls the data packet to be transmitted to switch to transmission through the other path.

[0094] The Data Reassembly Unit (DRU) is configured to reassemble multiple sub-data packets into the data packet based on the location tags corresponding to each sub-data packet. The DRU is also configured to determine whether any sub-data packets are missing based on the location tags corresponding to each sub-data packet. If so, the DRU sends a missing sub-data packet retransmission request to the transmission control module so that the transmission control module at least retransmits the missing sub-data packets. Specifically, the DRU module reads data sub-packets from the Rx Queue; sorts them according to SeqID; waits for all packets to arrive; checks the CRC and integrity flag of each packet; merges the data blocks and transmits them to the target memory space, then releases the buffer.

[0095] The technical challenges of DRU data reassembly include preventing out-of-order data reassembly and recovering from data errors. The core mechanisms for handling out-of-order data include the following aspects.

[0096] Receive window management: (1) Window size: The receive window size is dynamically adjusted according to the link delay and bandwidth to balance memory usage and reassembly efficiency; (2) Sequence number management: A circular sequence number mechanism is used, with a sequence number space of 32 bits, which supports large data transmission; (3) Timeout mechanism: A receive timeout is set for each fragment, and fragments that have not arrived by the timeout trigger a retransmission request; (4) Key technologies of the reassembly algorithm: Missing fragment detection: By analyzing the received sequence number, missing fragments can be quickly identified; (5) Early reassembly: It is not necessary to wait for all fragments to arrive, and the consecutive fragments that have arrived can be reassembled first; (6) Memory management: Memory pool technology is used to avoid frequent memory allocation and release.

[0097] The following error recovery mechanisms are proposed for data error recovery: (1) Retransmission request generation, including selective retransmission: only retransmit the lost fragments, not the entire data block; fast retransmission: retransmission is triggered immediately when 3 duplicate ACKs are received, without waiting for timeout; retransmission limit: set the maximum number of retransmissions to avoid infinite retransmission; (2) Data integrity guarantee including end-to-end verification: in addition to fragment-level CRC verification, there is also MD5 verification of the entire data block; version control: each data transmission session has a version number to avoid mixing of old and new data.

[0098] Based on the above-described parallel data communication system supporting path rescheduling, a specific embodiment is described below. Implementation Background: In an 8-GPU server, all GPUs are interconnected via two PCIe switches. During a training task, GPU1 sends 256KB of data to GPU4. The goal is to utilize the link resources of the switches to achieve segmented concurrency, high-speed transmission, and flexible scheduling.

[0099] Step 1: Path monitoring initialization (completed by PMU).

[0100] The upper and lower PCIe switches activate their respective Path Monitoring Units (PMUs); the sampling period is configured (e.g., every 10μs); the PMUs begin collecting the following port status data: current average data transmission latency, FIFO buffer utilization, and CRC error count in the most recent period. The PMU module calculates the link congestion score and transmits it to the GTCU module.

[0101] Step 2: The GPU source initiates data transmission preparation (GTCU initialization).

[0102] Objective: To configure GTCU to complete data splitting, path planning, and multi-channel concurrency preparation.

[0103] Operation details: The GPU Transmission Control Unit (GTCU) inside GPU0 is triggered; the data block to be transmitted is read from memory: total length 256KB; it is split into 4 data sub-packets of 64KB each, and each sub-packet is attached with a header as shown in Table 1, where the uplink is equivalent to the first path above and the downlink is equivalent to the second path above.

[0104] Table 1. List of headers for each sub-data packet

[0105]

[0106] Based on the given path allocation: Packet1–4 are bound to different send queues on the GPU (Tx Queue0: uplink port, Tx Queue1: downlink port).

[0107] Step 3: Data scheduling and recovery mechanism in case of link failure.

[0108] Objective: To ensure that when a link (uplink / downlink) is damaged during communication, the system can complete data recovery and path switching without terminating transmission or restarting training.

[0109] The PMU monitors for anomalies on each path in real time. For example, if a physical disconnection occurs on downlink Ports 5–8, the CRC error rate is consistently >10%, or a FIFO deadlock occurs, the PMU marks the path as "unavailable".

[0110] PMU switches to an alternative path: Based on the uplink path status, it finds the alternative path for the current communication; modifies the path allocation for the current session; and notifies the source GTCU to modify the PathID field of the remaining or unsuccessful data packets.

[0111] GTCU retransmits failed sub-packets: GTCU reads incomplete packets or lost acknowledgment packets from the queue, updates the PathID in the Header, and retransmits the packets via a new path (upstream Switch).

[0112] The DRU re-receives, verifies, and reassembles the data, ensuring no data loss and uninterrupted training.

[0113] Route update log: GTCU updates the current route for this session to "alternate path"; when the original primary path fails, the system marks the currently used path as an alternate path so that it can correctly switch back during subsequent fault recovery. If the failed link recovers, path recovery can be attempted in the next session.

[0114] Step 4: Runtime scheduling mechanism to achieve continuous high concurrency.

[0115] Objective: During the long-running training of large models, the communication topology is constantly changing, requiring dynamic scheduling of path resources to continuously achieve optimal concurrent transmission.

[0116] Real-time Path Load Reassessment (PMU): The PMU refreshes the link status periodically and reassesses the path score for all active communication sessions.

[0117] High-concurrency strategy trigger: When some paths are detected to be excessively idle (<30% bandwidth usage), GTCU actively switches the allocation of subsequent new sub-packets to idle links, thereby achieving path traffic splitting and thus expanding concurrency.

[0118] GTCU multi-queue asynchronous operation: GTCU internally supports parallel scheduling of n Tx Queues, each bound to a different PCIe port; when each packet is sent, it is written to the corresponding Tx queue according to the latest PathID allocated by GTCU, achieving data concurrency.

[0119] DRU multi-channel reception: DRU supports receiving data packets from multiple Rx Queues simultaneously; multiple sub-packets of the same session can be received out of order; synchronous reassembly is achieved through SeqID, resulting in higher throughput.

[0120] In one embodiment of the present invention, a device parallel data communication system supporting path rescheduling is provided. The difference from the above embodiment is that the path monitoring module is not located in the switch, but in the PCIe device. The path monitoring module determines the congestion status of each link by detecting the data transmission status of each port of the PCIe device.

[0121] In this embodiment, the device parallel data communication system supporting path rescheduling includes a first switch, a second switch, and multiple devices to communicate. Each device includes a third port and a fourth port. The third port is configured to be electrically connected to the first switch, and the fourth port is configured to be electrically connected to the second switch. The transmission path between the third port and the first switch is configured as a first path, and the transmission path between the fourth port and the second switch is configured as a second path. The devices include a path monitoring module and a transmission control module. The path monitoring module is configured to determine the link congestion status of the first path corresponding to the third port and transmit this information to the transmission control module. The path monitoring module is configured to determine the link congestion status of the second path corresponding to the fourth port and transmit this information to the transmission control module. The transmission control module is configured to control the data packets to be transmitted to be transmitted through one or more of the first and second paths based on the link congestion status of the first and second paths. Preferably, the transmission control module controls the data packets to be transmitted to be transmitted through the path with less link congestion between the first and second paths.

[0122] It should be noted that, in addition to the path monitoring module and transmission control module, the device in this embodiment preferably also includes the data reconstruction module. The working methods and principles of the path monitoring module, transmission control module, and data reconstruction module are the same as in the above embodiments, and will not be repeated here.

[0123] In one embodiment of the present invention, a device supporting path rescheduling is provided. The device includes a path monitoring module, a transmission control module, and a data reassembly module. The device includes a plurality of third ports, each of which is configured with a corresponding equivalent fourth port. In this embodiment, the equivalent of the fourth port and the third port means that the device communicates with another device of the same type through its third port and the fourth port. The transmission path corresponding to the third port is configured as a first path, and the transmission path corresponding to the fourth port is configured as a second path.

[0124] The path monitoring module is configured to determine the link congestion status of the first path corresponding to the third port and transmit it to the transmission control module. The path monitoring module is also configured to determine the link congestion status of the second path corresponding to the fourth port and transmit it to the transmission control module. The transmission control module is configured to control the data packets to be transmitted to be transmitted through one or more of the first and second paths based on the link congestion status of the first and second paths.

[0125] The transmission control module is further configured to split the data packet to be transmitted into multiple sub-data packets. Each sub-data packet is equipped with a location tag, which indicates its position within the data packet. Different sub-data packets have different location tags. The transmission control module is also configured to control the sub-data packets to be transmitted via one of the first path and the second path, with different sub-data packets being transmitted via the same or different paths.

[0126] The device further includes a data reassembly module, which is configured to reassemble multiple sub-data packets into the data packet according to the location tags corresponding to each of the sub-data packets.

[0127] The data reconstruction module is further configured to determine whether there are missing sub-data packets based on the location tags corresponding to each sub-data packet. If there are, the data reconstruction module sends a missing sub-data packet retransmission request to the transmission control module so that the transmission control module at least retransmits the missing sub-data packets.

[0128] In one embodiment of the present invention, a device supporting path rescheduling is provided. In this embodiment, the device does not include the path monitoring module, but only includes the transmission control module and the data reassembly module. The configuration and working principle of the transmission control module and the data reassembly module are the same as those in the above embodiments, and will not be described again.

[0129] In one embodiment of the present invention, a device parallel data communication method supporting path rescheduling is provided, comprising the following steps:

[0130] For multiple devices to communicate, each device is pre-configured to be electrically connected to a first switch and a second switch respectively, and the communication link between the device and the first switch is determined as a first path, and the communication link between the device and the second switch is determined as a second path;

[0131] The link congestion status of the first path and the second path is obtained, and based on the link congestion status of the first path and the second path, the data to be transmitted is controlled to be transmitted through one or more of the first path and the second path.

[0132] In this embodiment, the device parallel data communication method supporting path rescheduling further includes the following step: controlling the data packets to be transmitted to be transmitted through the path with less link congestion between the first path and the second path.

[0133] It should be noted that the device parallel data communication method and device embodiment supporting path rescheduling provided by the present invention have the same inventive concept as the above-mentioned device parallel data communication system embodiment supporting path rescheduling. The entire contents of the device parallel data communication system embodiment supporting path rescheduling are incorporated into the device parallel data communication method and device embodiment supporting path rescheduling by means of introduction.

[0134] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0135] The above description is only a specific embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A device parallel data communication system supporting path rescheduling, characterized by The system includes a first switch, a second switch, and multiple devices to communicate with. The first switch includes multiple first ports, and the second switch includes multiple second ports. Each device is electrically connected to both the first and second ports. The transmission path between the device and the first port is configured as a first path, and the transmission path between the device and the second port is configured as a second path. Both the first switch and the second switch include a path monitoring module. The path monitoring module of the first switch is configured to determine a link congestion score based on the link congestion status of the first path corresponding to each first port and transmit it to the device. The path monitoring module of the second switch is configured to determine a link congestion score based on the link congestion status of the second path corresponding to each second port and transmit it to the device. The device includes a transmission control module configured to control the transmission of data packets to be transmitted through one or more of the first and second paths based on the link congestion status of the first path and the second path, including: When the congestion scores corresponding to the first port and the second port are both lower than the first score threshold, the transmission control module controls the data packet to be transmitted to be transmitted through the first path and the second path. When the congestion score of one of the first port and the second port is higher than the second score threshold and the congestion score of the other port is not higher than the second score threshold, the transmission control module controls the data packet to be transmitted to be transmitted through the path corresponding to the port whose congestion score is not higher than the second score threshold, where the second score threshold is greater than the first score threshold. When the congestion scores corresponding to the first port and the second port are both higher than the third score threshold, the transmission control module splits the data packet to be transmitted into multiple sub-data packets, and controls some of the sub-data packets to be transmitted through the first path, while the other part of the sub-data packets is transmitted through the second path, and the third score threshold is greater than the first score threshold. If the link congestion of the current path does not meet the preset data transmission requirements, and the current path is one of the first path and the second path, then the transmission control module controls the data packet to be transmitted to switch to transmission through the other path.

2. The device parallel data communication system supporting path rescheduling according to claim 1, wherein, The transmission control module is further configured to split the data packet to be transmitted into multiple sub-data packets, each sub-data packet being configured with a location tag, the location tag being configured to indicate the position of the sub-data packet within the data packet, and different sub-data packets having different location tags; The transmission control module is also configured to control the sub-data packets to be transmitted through one of the first path and the second path, and different sub-data packets to be transmitted through the same or different paths.

3. The device parallel data communication system supporting path rescheduling according to claim 2, wherein, The data packet is configured with a priority level. If the priority level of the data packet to be transmitted is higher than the preset level threshold, the transmission control module controls some of the sub-data packets to pass through the first path, and controls other parts of the sub-data packets to be transmitted through the second path. or, The data packets are configured with priority levels. If the priority level of the data packet to be transmitted is higher than a preset level threshold and the link congestion of the first path and the second path both meet the preset requirements, then the transmission control module controls some of the sub-data packets to pass through the first path, and controls the other part of the sub-data packets to be transmitted through the second path.

4. The device parallel data communication system supporting path rescheduling according to claim 2, wherein, The device further includes a data reassembly module, which is configured to reassemble multiple sub-data packets into the data packet according to the location tags corresponding to each of the sub-data packets.

5. The device parallel data communication system supporting path rescheduling according to claim 4, wherein, The data reconstruction module is further configured to determine whether there are missing sub-data packets based on the location tags corresponding to each sub-data packet. If there are, the data reconstruction module sends a missing sub-data packet retransmission request to the transmission control module so that the transmission control module at least retransmits the missing sub-data packets.

6. The parallel data communication system for devices supporting path rescheduling according to claim 1, characterized in that, The path monitoring module is configured to calculate the link congestion score based on the following formula: Link congestion score = α × delay weight + β × FIFO occupancy weight + γ × error rate weight; Wherein, the delay weight = (current delay - baseline delay) / baseline delay, the FIFO occupancy rate weight = current FIFO occupancy rate / 100%, the error rate weight = current CRC error rate / baseline error rate, and α, β, and γ are adjustment coefficients.

7. A device parallel data communication system supporting path rescheduling, characterized by The system includes a first switch, a second switch, and multiple devices to communicate with. Each device includes a third port and a fourth port. The third port is configured to be electrically connected to the first switch, and the fourth port is configured to be electrically connected to the second switch. The transmission path between the third port and the first switch is configured as a first path, and the transmission path between the fourth port and the second switch is configured as a second path. The device includes a path monitoring module and a transmission control module. The path monitoring module is configured to determine the link congestion status of the first path corresponding to the third port and transmit it to the transmission control module. The path monitoring module is configured to determine the link congestion status of the second path corresponding to the fourth port and transmit it to the transmission control module. The transmission control module is configured to control the data packets to be transmitted to be transmitted through one or more of the first path and the second path based on the link congestion status of the first path and the second path, including: When the congestion scores corresponding to the third port and the fourth port are both lower than the first score threshold, the transmission control module controls the data packet to be transmitted to be transmitted through the first path and the second path. When the congestion score of one of the third port and the fourth port is higher than the second score threshold and the congestion score of the other port is not higher than the second score threshold, the transmission control module controls the data packet to be transmitted to be transmitted through the path corresponding to the port whose congestion score is not higher than the second score threshold, where the second score threshold is greater than the first score threshold. When the congestion scores corresponding to the third port and the fourth port are both higher than the third score threshold, the transmission control module splits the data packet to be transmitted into multiple sub-data packets, and controls some of the sub-data packets to be transmitted through the first path, while the other part of the sub-data packets is transmitted through the second path, and the third score threshold is greater than the first score threshold. If the link congestion of the current path does not meet the preset data transmission requirements, and the current path is one of the first path and the second path, then the transmission control module controls the data packet to be transmitted to switch to transmission through the other path.

8. The device parallel data communication system supporting path rescheduling according to claim 7, wherein, The transmission control module controls the data packets to be transmitted to be transmitted through the path with less link congestion between the first path and the second path.

9. A method for parallel data communication of a device supporting path rescheduling, characterized by, The device parallel data communication system supporting path rescheduling as described in claim 1 includes the following steps: For multiple devices to communicate, each device is pre-configured to be electrically connected to a first switch and a second switch respectively, and the communication link between the device and the first switch is determined as a first path, and the communication link between the device and the second switch is determined as a second path; The link congestion status of the first path and the second path is obtained, and based on the link congestion status of the first path and the second path, the data to be transmitted is controlled to be transmitted through one or more of the first path and the second path.

10. The method of claim 9, wherein the device parallel data communication method supporting path rescheduling is characterized by, It also includes the following steps: The data packets to be transmitted are controlled to be transmitted through the path with less link congestion between the first path and the second path.

Images