A method, system and medium for fast loss identification of an RDMA multi-plane network

Abstract

This invention provides a method, system, and medium for fast packet loss identification in RDMA multi-plane networks, relating to the field of fast packet loss identification technology. The method includes: acquiring RDMA data packets containing packet sequence numbers (PSNs); determining the target network plane exit ID and target path ID; maintaining a multi-plane state bitmap and path state bitmap for each queue pair; performing pre-allocation forwarding according to the target exit ID and target path ID; if the bitmap shows an abnormal target plane or path, or if the buffer backlog exceeds a threshold, rescheduling the data packet to other planes and paths with normal status and no backlog, and adding a rescheduling packet marker to the data packet; if an RDMA data packet carrying the forward packet loss identification marker is received, determining whether a packet is lost; this invention enables balanced distribution of consecutive data packets in the same queue pair across multiple planes and paths, improving packet loss identification efficiency.

Description

Technical Field

[0001] This invention relates to the field of fast packet loss identification technology, and in particular to a fast packet loss identification method, system and medium for RDMA multiplane networks. Background Technology

[0002] In RDMA networks, data flows are typically forwarded using the ECMP mechanism on the switch side. Since the five-tuple information for the same session is fixed, its hash result usually remains unchanged, causing data packets from the same session to concentrate on a single path. When multiple data flows converge at the same forwarding exit, path congestion occurs. Limited by the lossless transmission requirements of RDMA, the backpressure mechanism leads to a decrease in the overall transmission bandwidth of the network interface card (NIC). Furthermore, when a physical link or optical module fails, the lack of flexible plane and path switching capabilities causes data flows to become stuck on the faulty path, resulting in the interruption of long-term training tasks. Simultaneously, in scenarios with concurrent transmission across multiple planes and paths, the order in which data packets arrive at the receiving end may be reversed compared to when they were sent. This out-of-order phenomenon makes traditional methods relying on packet sequence number order to determine packet loss prone to misjudgment, and relying on reordering buffers and timers to detect packet loss is inefficient, failing to meet the requirements of rapid packet loss location and response in RDMA networks. Summary of the Invention

[0003] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows:

[0004] According to a first aspect of this application, a method for fast packet loss identification in an RDMA multiplane network is provided. The method is applied to a network interface card (NIC) system, which includes a transport protocol engine, a load balancer, and Ethernet egress ports belonging to multiple independent physical network planes. The load balancer is positioned between the transport protocol engine and the multiple Ethernet egress ports. The method is executed by the load balancer and includes:

[0005] S100, Obtain an RDMA data packet containing a data packet sequence number (PSN) from the transmission protocol engine; the PSN increments sequentially with the data packet transmission order;

[0006] S200: Based on the total number of physical network planes and PSN, determine the target network plane exit ID, and based on the PSN mapping value in the target network plane and the total number of available data forwarding paths, determine the target path ID;

[0007] S300 maintains a multi-plane state bitmap and a path state bitmap for each queue pair to record the communication status of each network plane and path, and monitors the backlog of the egress queue buffer corresponding to each Ethernet egress port.

[0008] S400, if the multi-plane status bitmap and path status bitmap show that the target plane and path status are normal and the buffer backlog does not exceed the preset threshold, perform pre-allocation forwarding according to the target exit ID and target path ID, encapsulate the RDMA data packet according to the hash entropy value corresponding to the target path ID, and add a forward packet loss identification mark to the RDMA data packet.

[0009] S500, if the bitmap shows that the target plane or path is abnormal, or the buffer backlog exceeds the threshold, the data packet is rescheduled to another plane and path that is in normal condition and has no backlog, and a rescheduled packet mark is added to the data packet.

[0010] S600, at the receiving end, if an RDMA data packet carrying the forward packet loss identification mark is received, it is determined whether a packet has been lost based on the PSN of the RDMA data packet, the total number of network planes, and the total number of paths.

[0011] According to another aspect of this application, a network interface card (NIC) system for fast packet loss identification in an RDMA multiplane network is also provided, comprising:

[0012] The host bus interface is used for data interaction with the computing host.

[0013] The transport protocol engine is used to receive host data and encapsulate it into RDMA data packets containing packet sequence numbers (PSNs).

[0014] Multiple Ethernet egress ports are connected to multiple independent physical network planes;

[0015] A load balancer, positioned between the transport protocol engine and the multiple Ethernet egress ports, is configured to perform the following operations:

[0016] S100, Obtain an RDMA data packet containing a data packet sequence number (PSN) from the transmission protocol engine; the PSN increments sequentially with the data packet transmission order;

[0017] S200: Based on the total number of physical network planes and PSN, determine the target network plane exit ID, and based on the PSN mapping value in the target network plane and the total number of available data forwarding paths, determine the target path ID;

[0018] S300 maintains a multi-plane status bitmap and a path status bitmap for each queue pair to record the communication health status of each network plane and path, and monitors the backlog of the egress queue buffer corresponding to each Ethernet egress port.

[0019] S400, if the multi-plane status bitmap and path status bitmap show that the target plane and path status are normal and the buffer backlog does not exceed the preset threshold, perform pre-allocation forwarding according to the target exit ID and target path ID, encapsulate the RDMA data packet according to the hash entropy value corresponding to the target path ID, and add a forward packet loss identification mark to the RDMA data packet.

[0020] S500, if the bitmap shows an abnormal target plane or path, or the buffer backlog exceeds the threshold, the data packet is rescheduled to another normal plane and path without backlog and sent, and a rescheduled packet mark is added to the data packet; wherein, the receiving end determines whether a packet is lost based on the PSN of the received RDMA data packet carrying the forward packet loss identification mark, the total number of network planes, and the total number of paths.

[0021] According to another aspect of this application, a computer-readable storage medium is also provided, storing a computer program that, when executed by a processor within a network card or a host CPU, implements the fast packet loss identification method for RDMA multiplane networks as described in the first aspect.

[0022] The present invention has at least the following beneficial effects:

[0023] The fast packet loss identification method for RDMA multi-plane networks of this invention, by adding a load balancer inside the network card and performing a modulo operation on the total number of physical network planes and the total number of paths based on the packet sequence number (PSN), enables balanced distribution of consecutive packets in the same queue across multiple planes and paths, thereby effectively reducing single-egress congestion caused by switch hash polarization. Simultaneously, by maintaining multi-plane and path status bitmaps to record health status and monitoring egress queue buffer backlog in real time, it can quickly reschedule packets to other normal planes and paths when link failures or local congestion occur, preventing training tasks from being interrupted due to single-point failures. Furthermore, this application adds a forward packet loss identification mark to normally pre-allocated packets at the sending end. Based on this mark and the mathematical relationship between the packet PSN and the product of the number of planes and paths, the receiving end can quickly locate whether preceding packets are lost in out-of-order transmission scenarios, thereby improving packet loss identification efficiency and reducing identification latency. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1This is a schematic diagram of a multi-plane network architecture provided in an embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram of the physical topology of an RDMA multiplane network provided in an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of a multi-plane network logical interconnection structure provided in an embodiment of the present invention;

[0028] Figure 4 A schematic diagram of the Spine-Leaf network connection relationship provided in an embodiment of the present invention;

[0029] Figure 5 This is a block diagram of a network interface card (NIC) logical function module including a load balancer, provided in an embodiment of the present invention.

[0030] Figure 6 This is a schematic diagram of network plane balanced allocation based on PSN sequence number provided in an embodiment of the present invention;

[0031] Figure 7 This is a schematic diagram of a two-level load balancing and entropy encapsulation method involving multiple planes and in-plane multiple paths provided in an embodiment of the present invention.

[0032] Figure 8 This is a schematic diagram illustrating the rapid identification of preceding data packet loss provided in an embodiment of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] It should be noted that, based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Furthermore, this device and / or practice the method can be implemented using other structures and / or functionalities besides one or more of the aspects set forth herein.

[0035] The following describes a fast packet loss identification method for RDMA multiplane networks. The method is applied to a network interface card (NIC) system, which includes a transport protocol engine, a load balancer, and Ethernet egress ports belonging to multiple independent physical network planes. The load balancer is located between the transport protocol engine and the multiple Ethernet egress ports.

[0036] The method in this application operates within a multi-plane network architecture. For example... Figure 2 As shown, the network is physically divided into multiple independent network planes (e.g., Figure 2 (Network plane-1 and network plane-2 are shown in the diagram). These network planes are independently isolated in terms of physical links, switching equipment, and power supply, and serve as redundant backups for each other.

[0037] Furthermore, combined Figure 4 As can be seen, each independent network plane adopts a standard spine-leaf network topology. The upper layer consists of spine switches, and the lower layer consists of leaf switches, forming a non-blocking, fully interconnected network between the upper and lower layers. Each leaf switch is connected to various RDMA communication nodes via physical links.

[0038] like Figure 3 As shown, the multiple network chips / network devices (101a, 101b) on the left and the network devices (101c, 101d) on the right are connected and communicate via 2, 4, 8 or more independent computing network interconnection systems or network planes (102a-102h). In other words, the same pair of RDMA communication nodes logically possesses multiple completely physically isolated transmission paths, providing physical hardware support for the subsequent multi-plane load balancing and rapid fault bypass in this application.

[0039] Within an RDMA communication node (i.e., a network interface card system), such as Figure 5 As shown, the network card system mainly includes a host bus interface 201, a transmission protocol engine 203, a load balancer 204, and several Ethernet egress ports (such as 202a to 202h).

[0040] Specifically, the host bus interface 201 connects upwards to the computing host (such as a CPU or GPU), and supports bus types including but not limited to PCIe and UALink, for receiving communication data sent by the computing host. The transmission protocol engine 203 is connected to the host bus interface 201 and is used to encapsulate the received data into standard data packets according to the RDMA protocol. The Ethernet egress ports 202a to 202h are the physical connection ports between the network card and the external switch, and each Ethernet egress port is connected to... Figure 2 The diagram shows a single physical network plane (e.g., egress port 202a is connected to network plane-1, 202b is connected to network plane-2, etc.).

[0041] The load balancer 204 is positioned between the transmission protocol engine 203 and the plurality of Ethernet egress ports 202a to 202h. The RDMA data packets encapsulated by the transmission protocol engine 203 are first sent to the load balancer 204, and then the load balancer 204 dynamically determines which Ethernet egress port to send the data packets to based on the characteristics of the data packets and the real-time port status.

[0042] exist Figures 2 to 5 Based on the described isolated network plane and network interface card (NIC) internal equalizer architecture, Figure 1 This further demonstrates the multipath out-of-order transmission network configuration of this application in actual operation. Because the network interface card's internal load balancer 204 is configured between the protocol engine and the physical egress, it can actively change the sending characteristics of different data packets within the same session (i.e., the hash entropy value in subsequent steps, such as the source port number). Therefore, in switches (such as...) Figure 1 When the Spine and Leaf switches in the system receive data packets and perform ECMP hash routing, consecutive data packets of the same QP will be swapped at the physical layer. Figure 1 This enables multi-path out-of-order transmission driven by the network interface card (NIC) across multiple physical links.

[0043] Based on the aforementioned multi-plane physical isolation network and the hardware architecture of the network interface card's built-in load balancer 204, the routing conflict problem caused by switch hash polarization in traditional RDMA networks can be solved at the hardware level. The load balancer 204, located between the protocol layer and the physical egress, can flexibly intervene in the switch's routing behavior without modifying the main structure of the RDMA protocol messages, ensuring full utilization of the bandwidth of each switch egress.

[0044] Furthermore, the independent isolation of physical planes provides high network reliability. When a link, optical module, or switch in network plane-1 experiences a physical failure, the physical link on that plane is disconnected, but the network interface card (NIC) system can still maintain communication with the peer node through other Ethernet egress ports connected to network plane-2. This multi-plane redundancy architecture effectively prevents long-term AI model training tasks from being completely interrupted due to a single point of hardware failure.

[0045] Finally, since the load balancer 204 is designed between the transmission protocol engine 203 and multiple physical egress ports, this physical location allows packet scheduling to be completed without going through the host CPU or network card external buffer, and the plane selection and path selection of data packets can be completed with extremely low hardware processing latency, which fully guarantees the ultra-low latency characteristics of the RDMA network.

[0046] The method is executed by the load balancer and includes:

[0047] S100, obtain the RDMA data packet containing the data packet sequence number PSN from the transmission protocol engine; the PSN increments sequentially with the data packet transmission order.

[0048] like Figure 5 As shown, this step forms the data basis for the load balancer 204 to perform subsequent distribution actions. After receiving the service data to be sent from the computing host via the host bus interface 201, the transport protocol engine 203 within the network interface card system encapsulates it according to the RDMA protocol standard (e.g., RoCEv2). During the encapsulation process, the transport protocol engine 203 assigns a Packet Sequence Number (PSN) to each RDMA data packet.

[0049] In its implementation, the transport protocol engine 203 maintains a monotonically increasing sequence number counter for each queue pair (QP). Whenever a QP sends an RDMA data packet, the transport protocol engine 203 fills the current counter value into the corresponding PSN field in the packet header, and then immediately increments the counter by 1. This ensures that the PSN values ​​carried by different data packets sent by the same QP are strictly incremented in the order of transmission. For example, assuming the current counter value of a QP is 1000, the PSN of the first RDMA data packet sent will be 1000; when sending the second data packet, its PSN will be 1001, and so on.

[0050] After encapsulation, the transport protocol engine 203 sends the RDMA data packet carrying the PSN value to the load balancer 204 through its internal data interface. The load balancer 204 then obtains the data packet and parses the PSN value from it for subsequent plane and path distribution calculations in step S200. Figure 6 As shown, Figure 6 The diagram shows a continuous sequence of PSN values ​​for packets in a QP (increasing from 21 to 36), visually demonstrating that the PSN values ​​obtained in this step have a strictly lossless incremental characteristic.

[0051] This step generates a PSN value that increments sequentially with the transmission order when encapsulating data packets using the transport protocol engine. This value is then provided to the load balancer, providing a definite basis for the uniform cyclical distribution of subsequent data packets across multiple physical planes and paths. The strictly increasing nature of the PSN value ensures that the load balancer's "modulo" operation achieves even round-robin distribution, avoiding the uneven distribution of traditional hash distribution. At the same time, the incrementing and consecutive unique sequence number provides an indispensable data foundation for the subsequent receiver to implement "single-path forward packet loss detection," enabling the receiver to quickly locate preceding data packets through simple mathematical subtraction, reducing the computational overhead and judgment delay of packet loss detection.

[0052] S200 determines the target network plane exit ID based on the total number of physical network planes and PSN, and determines the target path ID based on the PSN mapping value in the target network plane and the total number of available data forwarding paths.

[0053] This step, based on the PSN value obtained in step S100, performs a two-layer scheduling process involving first-layer network plane selection and second-layer path selection within the plane. In the implementation scenario of this application, the total number of physical network planes (denoted as M) and the total number of available data forwarding paths within a single plane (denoted as N) are typically configured as integer powers of 2. Based on this configuration feature, to achieve extremely low latency scheduling at the hardware level, this step can be implemented through the following bitwise operations.

[0054] Furthermore, step S200 includes the following steps:

[0055] S210, by shifting the binary value of the PSN to the right by the corresponding bit, a modulo operation is performed on the total number of physical network planes to determine the exit ID of the target network plane.

[0056] The load balancer first performs a modulo operation on the PSN and the total number of network planes M (i.e., the mathematical expression: Egress ID = PSN % M). Since M is an integer power of 2 (e.g., M = 2, 4, 8, 16, etc.), this modulo operation is logically equivalent to extracting the low-order bits of the PSN binary data. In practice, the load balancer does not need to actually execute division or remainder instructions; it only needs to shift the binary value of the PSN to the right by log2(M) bits, and use the right overflow bit as the target egress ID. For example, if the total number of network planes is 4 (M = 4), the corresponding shift is 2 bits. Assuming the current data packet's PSN value is 21 (binary 00010101), shifting it to the right by 2 bits and taking the 2 overflow bits (i.e., 01) corresponds to a decimal value of 1, then the target network plane egress ID of this data packet is 1 (corresponding to the first network plane).

[0057] S220, when the total number of available paths in the physical network plane is also a power of 2, the modulo operation on the total number of paths is performed by extracting the corresponding low-order binary bits of the new value after right shift, so as to determine the target path ID.

[0058] After determining the target network plane via S210, the load balancer performs a second-level operation, namely calculating the path ID (mathematical expression: path ID = (PSN / M)%N). First, the mapping value of PSN within the network plane needs to be calculated (i.e., PSN / M; since M is a power of 2, this division is directly represented in hardware operations as right-shifting PSN by log2(M) bits). Then, based on the obtained mapping value, a modulo operation is performed on the total number of available paths N within a single plane. When N is also a power of 2 (e.g., N = 4, 8, 16, etc.), this modulo operation can also be achieved by extracting the low-order bits of the binary representation.

[0059] Combining the above example, the total number of network planes M is 4, and the total number of available paths N is 4. When the PSN value is 21, its binary representation is 00010101. After right-shifting by 2 bits in S210, the intermediate result 00000101 (corresponding to decimal 5) is obtained. In S220, the modulo of the number of paths N=4 needs to be taken, that is, the lowest 2 bits of the intermediate result 00000101 are truncated to obtain 01. The decimal value corresponding to this value is 1, that is, the second-layer target path ID is determined to be 1 (corresponding to path 1 in the network plane). Through the calculations in S210 and S220, the load balancer can accurately locate that the data packet should be sent out from path 1 of network plane 1.

[0060] In actual hardware implementation, the entire process described above does not require a divider or remainder circuit. Only a simple shift register and bit truncation logic are needed to complete the calculation. Therefore, the calculation delay for processing a single data packet can be controlled within a single clock cycle.

[0061] This step leverages the configuration characteristic that both the number of physical network planes and the number of paths are powers of 2, transforming complex mathematical modulo operations into high-speed binary shifting and bit-truncation operations at the hardware level, achieving low-latency computation without a divider. This hardware-friendly implementation ensures that the network card maintains minimal scheduling latency and a stable scheduling cycle when handling high-concurrency data streams with bandwidths of 800G or higher. Simultaneously, by employing modulo (round-robin) calculation strictly based on an increasing PSN sequence, it fundamentally eliminates the hash polarization defects of traditional 5-tuple hashing methods, ensuring absolutely uniform distribution of data packets from the same queue across various planes and paths, achieving theoretically optimal load balancing. Furthermore, this distribution pattern based on a fixed mathematical step size directly provides a strict periodic anchor point for the subsequent receiver to implement "single-path forward packet loss detection," allowing the receiver to pre-locate preceding packet losses using only simple mathematical subtraction.

[0062] S300 maintains a multi-plane state bitmap and a path state bitmap for each queue pair to record the communication status of each network plane and path, and monitors the backlog of the egress queue buffer corresponding to each Ethernet egress port.

[0063] In this embodiment, the load balancer maintains two tables (a multi-plane bitmap and a path bitmap) for each QP. These bitmaps are used to quickly record whether each network plane and data forwarding path is normal or faulty / abnormal. This is equivalent to real-time traffic conditions on a network map, providing a core reference for subsequent decisions on "which path to take".

[0064] The load balancer also monitors the amount of data queued in the buffer of the physical network interface card (NIC) egress point. Once the data exceeds a set threshold, it indicates that the egress point is congested.

[0065] Furthermore, step S300 includes the following steps:

[0066] S310: When a link interruption or packet loss occurs in any network plane or data forwarding path, the transport protocol layer identifies that the network plane or data forwarding path is in a communication abnormal state.

[0067] S311, update the marked position of the corresponding multi-plane state bitmap or path state bitmap to the fault state.

[0068] S312, after subsequent RDMA data packets have calculated the target network plane or target path, if the query of the multi-plane status bitmap and path status bitmap shows that the corresponding mark is a fault state, then the original pre-allocation logic is terminated and the rescheduling strategy is triggered.

[0069] In this embodiment, fault detection and marking are prerequisites for ensuring the triggering of rescheduling logic. First, the transport protocol layer (such as...) Figure 5In step 203), during the process of receiving and acknowledging data packets and retransmitting after timeout, the underlying link's communication quality signals are acquired in real time, including data packet loss, unacknowledged timeouts (NACK feedback), and physical link layer connection interruption signals. When any physical network plane or a data forwarding path frequently experiences the above-mentioned anomalies, the transport protocol layer marks this abnormal state and transmits it to the load balancer, i.e., identifying the abnormal state as described in S310. Secondly, after receiving the abnormal signal, the load balancer updates the corresponding marker to "fault" (e.g., set to 1) in its internally maintained multi-plane state bitmap or path state bitmap, i.e., S311. Finally, as... Figure 6 and Figure 7 As shown, after the subsequent RDMA data packets undergo S200 calculation, the load balancer first queries the status bitmap corresponding to the calculated target network plane egress ID and target path ID. If the corresponding marker is found to be in a fault state, the original pre-allocation logic is immediately terminated, and the rescheduling strategy (i.e., S500) is directly triggered, thereby ensuring that traffic will not continue to be sent to links that have already failed or suffered severe packet loss.

[0070] Furthermore, step S300 also includes the following steps:

[0071] S321 dynamically determines the communication load status of each network plane and path based on the real-time packet loss rate, transmission delay, and congestion feedback signals on each network plane and data forwarding path.

[0072] In this embodiment, to avoid misjudgments of health status due to oversensitivity or sluggishness in the system, a dynamic sensitivity threshold adjustment mechanism is introduced. The load balancer continuously collects real-time packet loss rate, end-to-end transmission delay, and PFC (Priority Flow Control) congestion feedback signals for each network plane and data forwarding path, and determines the current communication load status based on these parameters. Specifically, weight coefficients can be assigned to the three dimensions of packet loss rate, transmission delay, and congestion signal, and then the packet loss rate, transmission delay, and congestion signal are weighted and summed to obtain a communication load health score. The communication load status is determined based on the health score.

[0073] S322, When the overall network is detected to be in a preset low load state, increase the sensitivity threshold for judging abnormalities.

[0074] When the overall network is detected to be in a preset low-load state (i.e., there is still a large amount of bandwidth redundancy in the network), the load balancer will increase the sensitivity threshold for judging anomalies. This means that even if there is a slight packet loss rate or a small transmission delay in the network, the system will judge it as an "abnormal link" and reroute it, so as to prioritize ensuring low latency and extremely low packet loss rate for critical data flows.

[0075] S323, when the overall network is detected to be in a preset high load state, the sensitivity threshold for judging anomalies is reduced, allowing paths with slight congestion or high latency to continue to carry forward traffic in order to maintain the maximum effective throughput of the overall network.

[0076] When the system detects that the overall network is under a preset high load (i.e., the network lines are already very congested), it will proactively lower the sensitivity threshold for judging anomalies. At this time, as long as the link is not completely disconnected, even if it experiences slight congestion or high latency, the system will allow it to continue to handle a portion of the forwarding traffic, avoiding global network turbulence caused by frequent path switching, thereby maintaining the maximum effective throughput of the overall network.

[0077] Furthermore, step S300 also includes the following steps:

[0078] S310 retrieves the total number of currently concurrent queue pairs, the real-time data transmission rate of each queue pair, and the remaining bandwidth resources of the network interface card.

[0079] like Figure 5 The network interface card (NIC) hardware architecture shown describes a load balancer 204 that periodically reads the total number of currently active queue pairs (QPs) and the real-time data transmission rate of the corresponding packet senders for each QP (which can be calculated by dividing the number of data packets sent by each QP per unit time by the time window). Based on this, the load balancer 204 calculates the remaining bandwidth resources of the NIC by subtracting the real-time aggregated transmission rate of all QPs from the fixed total bandwidth limit of the NIC's physical link layer (e.g., 800Gbps).

[0080] S320 dynamically adjusts the preset threshold of each egress queue buffer based on the total number of concurrent queue pairs, real-time transmission rate, and remaining bandwidth resources.

[0081] After acquiring the remaining bandwidth resources, the load balancer 204 will dynamically adjust the various Ethernet egress ports (i.e., Figure 5The preset thresholds (202a~202h) correspond to the egress queue buffers. The dynamic adjustment strategy for these thresholds is as follows: When a large number of concurrent QPs, an extremely high sending rate, and limited remaining network card bandwidth are detected, it indicates that the network is under high pressure and prone to congestion. In this case, the load balancer 204 will proactively lower the preset thresholds for each egress queue buffer. The logic behind this adjustment is that under high pressure, even a small amount of data accumulating in the buffer may be a precursor to severe congestion. Lowering the threshold can prompt the system to initiate a rescheduling process earlier, preventing buffer overflow from triggering PFC flow control. Conversely, when the network card is under low load (sufficient remaining bandwidth), the load balancer will appropriately raise the preset thresholds to provide the egress queues with some buffering flexibility, avoiding frequent rescheduling due to small, instantaneous traffic spikes, thereby maintaining network stability.

[0082] S330, when the backlog of a certain egress queue exceeds the dynamically adjusted threshold, the allocation weight is calculated based on the remaining processing capacity of the other egress queues that have not exceeded the threshold, and the traffic exceeding the current sending capacity is distributed to the other egress queues that have not exceeded the threshold according to the allocation weight.

[0083] When the system detects that the backlog depth of the egress queue for a certain Ethernet egress port (e.g., 202a) exceeds the preset threshold dynamically adjusted in S320, the load balancer determines that the network plane egress is congested. At this time, the rescheduling policy in S500 is triggered. To ensure that rescheduling does not lead to congestion transfer (i.e., dropping all packets from the originally congested port A to port B, causing port B to also instantly exceed the threshold), the load balancer first obtains the remaining processing capacity (usually referring to the remaining free space in the buffer of each port and its current available transmission bandwidth redundancy) of all other healthy egress queues before distributing traffic. Based on this remaining processing capacity, the load balancer calculates the distribution weight for each healthy port (the port with larger remaining processing capacity has a larger distribution weight). Finally, the load balancer smoothly distributes the traffic exceeding the transmission capacity of the congested port to the remaining healthy egress queues according to the aforementioned calculated weight ratio.

[0084] S400, if the multi-plane status bitmap and path status bitmap show that the target plane and path status are normal and the buffer backlog does not exceed the preset threshold, perform pre-allocation forwarding according to the target exit ID and target path ID, encapsulate the RDMA data packet according to the hash entropy value corresponding to the target path ID, and add a forward packet loss identification mark to the RDMA data packet.

[0085] like Figure 7 and Figure 5After completing the S300 status monitoring, if it is determined that the target network plane and the target path are in normal condition and the corresponding egress queue buffer backlog does not exceed the threshold, the load balancer will perform standard pre-allocation forwarding.

[0086] Furthermore, step S400 includes the following steps:

[0087] S410 pre-establishes a mapping relationship between different target path IDs and different source port numbers locally.

[0088] During the actual network interface card (NIC) driver initialization or system configuration phase, the load balancer will store or dynamically generate a mapping table between path IDs and source port numbers in its internal lookup table (LUT) memory or local registers. The principle for constructing this mapping table is that different path IDs must correspond to unique source port numbers. For example, if there are 4 available forwarding paths in a single plane (path IDs are 0~3), the mapping table will pre-allocate 4 fixed source port numbers with large hash differences (e.g., path 0 corresponds to source port number 10000, path 1 corresponds to source port number 10001, path 2 corresponds to source port number 10002, and path 3 corresponds to source port number 10003).

[0089] S420, based on the determined target path ID, query the corresponding source port number in the mapping relationship.

[0090] After the load balancer calculates the second-level target path ID of the data packet to be sent in step S200, the system does not perform complex secondary dynamic generation calculations. Instead, it directly uses the target path ID as an index to quickly read the corresponding preset source port number from the mapping lookup table. For example, if the target path ID calculated in step S200 is 2, the source port number obtained by the load balancer after reading the mapping table will be 10002.

[0091] S430 encapsulates the source port number found into the header of the RDMA data packet to change the hash routing result triggered when the RDMA data packet passes through the intermediate switch, so that different data packets in the same queue can be evenly distributed on different physical links.

[0092] When the load balancer completes the encapsulation of the packet header (especially the UDP header in RoCEv2), it fills the source port field of the five-tuple with the destination source port number obtained from S420. Since the five-tuple (source IP, destination IP, source port, destination port, protocol number) is the core input parameter of the hash algorithm in traditional switch ECMP hash routing, this application, by continuously changing the source port number, enables packets with different PSNs in the same session (same QP) to trigger different hash calculations by the switch, causing them to be routed to different destinations. Figure 4Different uplink physical links. Since the PSN is strictly incrementally cyclically distributed, consecutive data packets of the same QP will sequentially use different source port numbers in the mapping table, thereby achieving absolute round-robin balanced distribution on multiple paths. At the same time, since this forwarding action occurs under the premise of "normal status" and "no congestion", the load balancer will add an extra "forward packet loss identification mark" to the data packet so that the receiving end in the subsequent step S600 can identify whether the packet was sent normally according to the preset mathematical period.

[0093] In this embodiment, by pre-establishing a mapping relationship between path IDs and source port numbers, the load balancer achieves fast and accurate replacement of packet hash entropy values ​​at the hardware level. This table-based lookup encapsulation method not only has extremely low processing latency and no complex computational burden, but also completely eliminates the hash polarization problem of the same session in traditional networks, enabling network bandwidth utilization to stably reach full line speed. Simultaneously, adding a "forward packet loss identification mark" to packets under normal conditions provides clear classification triggering conditions for rapid detection of out-of-order packet loss at the receiving end, improving the accuracy and timeliness of packet loss perception in long-cycle AI training services.

[0094] S500, if the bitmap shows an abnormal target plane or path, or the buffer backlog exceeds the threshold, the data packet is rescheduled to another plane and path that is in normal condition and has no backlog, and a rescheduled packet marker is added to the data packet.

[0095] When an abnormal situation occurs in the monitoring results of step S300 or the prediction of step S400, i.e., the multi-plane status bitmap or path status bitmap shows a target plane or path failure, or the backlog depth of the corresponding exit queue buffer exceeds the preset threshold, the load balancer will interrupt the original pre-allocation logic and enter the rescheduling process of step S500.

[0096] Furthermore, step S500 includes the following steps:

[0097] S510, take the remaining buffer space size of all currently normal Ethernet egress ports, and determine the forwarding weight of each healthy egress port based on the remaining space size; the remaining space size is positively correlated with the forwarding weight.

[0098] After a reschedule is triggered, the load balancer will first scan... Figure 5The system displays all Ethernet egress ports (i.e., 202a~202h). It needs to select ports that are currently in a healthy state (i.e., the multiplane status bitmap and path status bitmap are marked normally, and their corresponding egress queue buffer backlog does not exceed the current preset threshold). For each selected healthy egress port, the load balancer obtains the remaining space of its egress queue buffer (e.g., the number of remaining available cache bytes). Then, the system calculates the forwarding weight based on the remaining space of these healthy ports. The weight allocation follows a "positive correlation principle," meaning that the larger the remaining buffer space of a port, the less its current data backlog and the more abundant its processing capacity, thus receiving a higher forwarding weight. The specific weight calculation method can be expressed as: the forwarding weight of a certain healthy port... .

[0099] S520, based on the forwarding weight of each healthy egress port, sequentially distributes the RDMA data packets to be rescheduled to each target rescheduling egress queue to ensure load balancing of each port after rescheduling.

[0100] After calculating the forwarding weights of each healthy port in S510, the load balancer extracts the rescheduled data packets that were originally intended for the abnormal / congested ports. To prevent these sudden traffic surges from instantly congesting other healthy ports, the load balancer does not forward these packets arbitrarily. Instead, it strictly distributes the rescheduled traffic to the tail of the egress queue of each healthy port according to the forwarding weight ratio calculated in S510, using a weighted round-robin or weighted fair queue (WFQ) scheduling mechanism. For example, assuming that the remaining space of healthy port A is twice that of port B, then the weight calculated for port A is also twice that of port B. When distributing rescheduled traffic, the load balancer will allocate the data packets to ports A and B in a 2:1 ratio. At the same time, because the data packet has undergone rescheduling and deviated from the original strictly mathematical recursive path based on PSN, plane number, and path number, the load balancer will add a "rescheduled packet tag" to the data packet so that the receiving end can identify and skip packet loss detection for this packet in step S600.

[0101] In this embodiment, by introducing weight calculation based on the remaining buffer space size before rescheduling, the congestion transfer problem that may occur with traditional simple rescheduling is avoided. It ensures that when local congestion occurs, the migrated traffic can be received by the most idle and capable healthy port, thus maintaining load balancing across physical network planes after rescheduling. Simultaneously, these rescheduled packets are tagged with specific markers, providing the receiving end with clear criteria for judgment, letting it know that the sequence numbers of these packets no longer follow the original fixed mathematical step size rule. This directly shields forward packet loss detection for these packets, improving the accuracy and efficiency of packet loss identification in the entire RDMA multi-plane network, and strongly supporting the stable operation of large-scale intelligent computing center clusters under high load conditions.

[0102] S600, at the receiving end, if an RDMA data packet carrying the forward packet loss identification mark is received, it is determined whether a packet has been lost based on the PSN of the RDMA data packet, the total number of network planes, and the total number of paths.

[0103] like Figure 8 As shown, at the receiving end, after receiving the RDMA data packet, it is necessary to selectively perform packet loss detection according to the tag type carried by the data packet in order to avoid misjudgment in the context of out-of-order multipath transmission.

[0104] Furthermore, step S600 includes the following steps:

[0105] S610: Obtain the tag type carried in the RDMA data packet received by the receiving network card.

[0106] When parsing a received RDMA data packet, the receiving network card first reads the extended identifier bits in the packet header to obtain the tag type attached to the packet at the sending end. The tag type includes two mutually exclusive states: "forward packet loss identification tag" and "rescheduling packet tag".

[0107] S620, if the tag type is the rescheduling packet tag, then it is determined that the received RDMA data packet has changed its original plane and path at the sending end due to congestion or failure, and the current processing is skipped.

[0108] If the receiving end parses a "rescheduling packet marker" into the data packet, it means that the sending end, during the S500 step, had forcibly migrated the data packet to another backup network plane or path due to congestion or failure on the original path. Since the data packet's transmission path has deviated from the original mathematical recursive sequence based on PSN, if the receiving end still checks for preceding packet loss according to the original mathematical model, misjudgments will inevitably occur. Therefore, upon detecting this marker, the receiving end immediately exits the current packet loss detection process, neither performing further calculations nor reporting the packet loss event.

[0109] S630, if the tag type is the forward packet loss identification tag, then obtain the PSN value R of the currently received RDMA data packet. PSN .

[0110] If the marker parsed by the receiver is a "forward packet loss identification marker," it means that the data packet was sent strictly in accordance with the S400 pre-allocation strategy, and its PSN, total number of network planes, and total number of paths maintain a strict mathematical recursive relationship. The receiver then extracts the current data packet sequence number from the header of the data packet and records it as follows. .

[0111] S640, according to R PSN The PSN value V of the preceding RDMA data packet to be detected is determined by the total number of network planes M and the total number of data forwarding paths N within a single network plane. PSN =R PSN -M×N.

[0112] like Figure 8 The calculation logic is as follows: because the sending end transmits data according to the two-layer distribution mechanism in S200 (the step size is determined by the number of planes and paths), the PSN interval between adjacent data on the same path is fixed. The receiving end utilizes this characteristic to calculate the PSN value of the preceding data packet through simple mathematical subtraction. The specific formula is as follows: .in, The total number of network planes, The total number of available paths within a single network plane is M. Multiplying M and N gives the fixed PSN interval between consecutive data packets on the same path. For example, assuming the total number of network planes M is 4, the total number of paths in a single plane N is 4, and the PSN value of the currently received data packet is 37 ( Then, the receiving end calculates the PSN value of the preceding data packet to be detected using the formula: This means that the data packets located at PSN 21 on the current path have a definite sequential relationship with the current packet.

[0113] S650, if the receiving end queries V PSN If the corresponding RDMA data packet does not arrive within the current sliding time window, it is determined that the preceding RDMA data packet to be detected has been lost; otherwise, it is determined that the preceding RDMA data packet to be detected has not been lost.

[0114] The receiving network interface card (NIC) internally maintains a receive buffer record table based on packet PSN (or a receive status table based on a bitmap). Calculations are... Then, the receiving end immediately queries the record table to check the PSN. The system checks whether the data packet has been successfully received within the currently set sliding time window (e.g., 100 microseconds or a preset jitter tolerance period). If the data packet has not been received within the current time window, it is determined that the packet has not been received. The data packet has been lost on the current forwarding path. The receiving end can then trigger a retransmission mechanism. Conversely, if... If the corresponding data packet has arrived successfully, it is determined that no packet loss has occurred on that path.

[0115] In this embodiment, by introducing a conditional filtering mechanism based on tag type at the receiving end, the disruption of the original ordered sequence by cross-plane rescheduled packets is effectively avoided, fundamentally eliminating false packet loss misjudgments in rescheduling scenarios. Simultaneously, by utilizing the inherent mathematical step size rules of multi-plane distribution systems, the complex out-of-order reordering and packet loss identification are transformed into a single simple subtraction operation performed by the receiving end. This single-path forward detection mechanism significantly reduces the packet loss identification latency of RDMA networks in multi-path out-of-order transmission scenarios.

[0116] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0117] Embodiments of the present invention also provide a network interface card (NIC) system for fast packet loss identification in RDMA multiplane networks, characterized in that it includes:

[0118] The host bus interface is used for data interaction with the computing host.

[0119] The transport protocol engine is used to receive host data and encapsulate it into RDMA data packets containing packet sequence numbers (PSNs).

[0120] Multiple Ethernet egress ports are connected to multiple independent physical network planes.

[0121] A load balancer, positioned between the transport protocol engine and the multiple Ethernet egress ports, is configured to perform the following operations:

[0122] S100, obtain the RDMA data packet containing the data packet sequence number PSN from the transmission protocol engine; the PSN increments sequentially with the data packet transmission order.

[0123] S200 determines the target network plane exit ID based on the total number of physical network planes and PSN, and determines the target path ID based on the PSN mapping value in the target network plane and the total number of available data forwarding paths.

[0124] S300 maintains a multi-plane status bitmap and a path status bitmap for each queue pair to record the communication health status of each network plane and path, and monitors the backlog of the egress queue buffer corresponding to each Ethernet egress port.

[0125] S400, if the multi-plane status bitmap and path status bitmap show that the target plane and path status are normal and the buffer backlog does not exceed the preset threshold, perform pre-allocation forwarding according to the target exit ID and target path ID, encapsulate the RDMA data packet according to the hash entropy value corresponding to the target path ID, and add a forward packet loss identification mark to the RDMA data packet.

[0126] S500, if the bitmap shows an abnormal target plane or path, or the buffer backlog exceeds the threshold, the data packet is rescheduled to another normal plane and path without backlog and sent, and a rescheduled packet mark is added to the data packet; wherein, the receiving end determines whether a packet is lost based on the PSN of the received RDMA data packet carrying the forward packet loss identification mark, the total number of network planes, and the total number of paths.

[0127] Embodiments of the present invention also provide a non-transitory computer-readable storage medium that can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a method in the method embodiments, wherein the at least one instruction or the at least one program is loaded and executed by the processor to implement the method provided in the above embodiments.

[0128] While specific embodiments of the invention have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and are not intended to limit the scope of the invention. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the invention.

Claims

1. A fast packet loss identification method for RDMA multiplane networks, characterized in that, The method is applied to a network interface card (NIC) system, which includes a transmission protocol engine, a load balancer, and Ethernet egress ports belonging to multiple independent physical network planes. The load balancer is located between the transmission protocol engine and the multiple Ethernet egress ports. The method is executed by the load balancer and includes: S100, Obtain an RDMA data packet containing a data packet sequence number (PSN) from the transmission protocol engine; the PSN increments sequentially with the data packet transmission order; S200: Based on the total number of physical network planes and PSN, determine the target network plane exit ID, and based on the PSN mapping value in the target network plane and the total number of available data forwarding paths, determine the target path ID; S300 maintains a multi-plane state bitmap and a path state bitmap for each queue pair to record the communication status of each network plane and path, and monitors the backlog of the egress queue buffer corresponding to each Ethernet egress port. S400, if the multi-plane status bitmap and path status bitmap show that the target plane and path status are normal and the buffer backlog does not exceed the preset threshold, perform pre-allocation forwarding according to the target exit ID and target path ID, encapsulate the RDMA data packet according to the hash entropy value corresponding to the target path ID, and add a forward packet loss identification mark to the RDMA data packet. S500, if the bitmap shows that the target plane or path is abnormal, or the buffer backlog exceeds the threshold, the data packet is rescheduled to another plane and path that is in normal condition and has no backlog, and a rescheduled packet mark is added to the data packet. S600, at the receiving end, if an RDMA data packet carrying the forward packet loss identification mark is received, it is determined whether a packet has been lost based on the PSN of the RDMA data packet, the total number of network planes, and the total number of paths.

2. The fast packet loss identification method for RDMA multi-plane networks according to claim 1, characterized in that, Step S400 includes the following steps: S410, pre-establishes a mapping relationship between different target path IDs and different source port numbers locally; S420, based on the determined target path ID, query the mapping relationship to obtain the corresponding source port number; S430 encapsulates the source port number found into the header of the RDMA data packet to change the hash routing result triggered when the RDMA data packet passes through the intermediate switch, so that different data packets in the same queue can be evenly distributed on different physical links.

3. The fast packet loss identification method for RDMA multi-plane networks according to claim 1, characterized in that, Step S300 includes the following steps: S310, when any network plane or data forwarding path experiences link interruption or packet loss, the transport protocol layer identifies that the network plane or data forwarding path is in a communication abnormal state. S311, update the marked position of the corresponding multi-plane state bitmap or path state bitmap to the fault state; S312, after subsequent RDMA data packets have calculated the target network plane or target path, if the query of the multi-plane status bitmap and path status bitmap shows that the corresponding mark is a fault state, then the original pre-allocation logic is terminated and the rescheduling strategy is triggered.

4. The fast packet loss identification method for RDMA multi-plane networks according to claim 1, characterized in that, Step S600 includes the following steps: S610, obtain the tag type carried by the RDMA data packet received by the receiving network card; S620, if the tag type is the rescheduling packet tag, then it is determined that the received RDMA data packet has changed its original plane and path at the sending end due to congestion or failure, and the current processing is skipped. S630, if the tag type is the forward packet loss identification tag, then obtain the PSN value R of the currently received RDMA data packet. PSN ; S640, according to R PSN The PSN value V of the preceding RDMA data packet to be detected is determined by the total number of network planes M and the total number of data forwarding paths N within a single network plane. PSN =R PSN -M×N; S650, if the receiving end queries V PSN If the corresponding RDMA data packet does not arrive within the current sliding time window, it is determined that the preceding RDMA data packet to be detected has been lost; otherwise, it is determined that the preceding RDMA data packet to be detected has not been lost.

5. The fast packet loss identification method for RDMA multi-plane networks according to claim 1, characterized in that, Step S300 also includes the following steps: S321, dynamically determine the communication load status of each network plane and path based on the real-time packet loss rate, transmission delay and congestion feedback signal on each network plane and data forwarding path; S322, When the overall network is detected to be in a preset low load state, increase the sensitivity threshold for judging abnormalities; S323, when the overall network is detected to be in a preset high load state, the sensitivity threshold for judging anomalies is reduced, allowing paths with slight congestion or high latency to continue to carry forward traffic in order to maintain the maximum effective throughput of the overall network.

6. The fast packet loss identification method for RDMA multi-plane networks according to claim 1, characterized in that, Step S200 includes the following steps: S210, by shifting the binary value of the PSN to the right by the corresponding bit, a modulo operation is performed on the total number of physical network planes to determine the exit ID of the target network plane; S220, when the total number of available paths in the physical network plane is also a power of 2, the modulo operation on the total number of paths is performed by extracting the corresponding low-order binary bits of the new value after right shift, so as to determine the target path ID.

7. The fast packet loss identification method for RDMA multiplane networks according to claim 1, characterized in that, Step S500 includes the following steps: S510, take the remaining buffer space size of all currently normal Ethernet egress ports, and determine the forwarding weight of each healthy egress port based on the remaining space size; the remaining space size is positively correlated with the forwarding weight. S520, based on the forwarding weight of each healthy egress port, sequentially distributes the RDMA data packets to be rescheduled to each target rescheduling egress queue to ensure load balancing of each port after rescheduling.

8. The method for fast packet loss identification in RDMA multi-plane networks according to claim 1, characterized in that, Step S300 also includes the following steps: S310: Obtain the total number of currently concurrent queue pairs, the real-time data transmission rate of each queue pair, and the remaining bandwidth resources of the network card. S320 dynamically adjusts the preset threshold of each egress queue buffer based on the total number of concurrent queue pairs, real-time transmission rate, and remaining bandwidth resources. S330, when the backlog of a certain egress queue exceeds the dynamically adjusted threshold, the allocation weight is calculated based on the remaining processing capacity of the other egress queues that have not exceeded the threshold, and the traffic exceeding the current sending capacity is distributed to the other egress queues that have not exceeded the threshold according to the allocation weight.

9. A network interface card (NIC) system for fast packet loss identification in an RDMA multiplane network, characterized in that, include: The host bus interface is used for data interaction with the computing host. The transport protocol engine is used to receive host data and encapsulate it into RDMA data packets containing packet sequence numbers (PSNs). Multiple Ethernet egress ports are connected to multiple independent physical network planes; A load balancer, positioned between the transport protocol engine and the multiple Ethernet egress ports, is configured to perform the following operations: S100, Obtain an RDMA data packet containing a data packet sequence number (PSN) from the transmission protocol engine; the PSN increments sequentially with the data packet transmission order; S200: Based on the total number of physical network planes and PSN, determine the target network plane exit ID, and based on the PSN mapping value in the target network plane and the total number of available data forwarding paths, determine the target path ID; S300 maintains a multi-plane status bitmap and a path status bitmap for each queue pair to record the communication health status of each network plane and path, and monitors the backlog of the egress queue buffer corresponding to each Ethernet egress port. S400, if the multi-plane status bitmap and path status bitmap show that the target plane and path status are normal and the buffer backlog does not exceed the preset threshold, perform pre-allocation forwarding according to the target exit ID and target path ID, encapsulate the RDMA data packet according to the hash entropy value corresponding to the target path ID, and add a forward packet loss identification mark to the RDMA data packet. S500, if the bitmap shows an abnormal target plane or path, or the buffer backlog exceeds the threshold, the data packet is rescheduled to another normal plane and path without backlog and sent, and a rescheduled packet mark is added to the data packet; wherein, the receiving end determines whether a packet is lost based on the PSN of the received RDMA data packet carrying the forward packet loss identification mark, the total number of network planes, and the total number of paths.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor in the network card or the host CPU, it implements the fast packet loss identification method for RDMA multiplane networks as described in any one of claims 1 to 8.

Images