Method and system of receiving data packets in a remote direct memory access network

Abstract

A system (100, 200, 300, 400, 500, 600) of sending and receiving a plurality of data packets (108A-C, 208A-C, 308A-C, 408A- C, 508A-D, 608A-D) between a requesting node (104, 204, 304, 404, 504, 604) and a responding node (106, 206, 306, 406, 506, 606) in a Remote Direct Memory Access, RDMA, network (102, 202, 302, 402, 502, 602) is provided. The responding node receives the plurality of data packets from the requesting node. The responding node determines whether the plurality of data packets contain a path reroute marking indicator. The responding node sends an adaptive routing notification indicator in a header of a data packet back to the requesting node if the plurality of data packets contains the path reroute marking indicator.

Description

[0001] METHOD AND SYSTEM OF R EC E IVING DATA PACKETS IN A REMOTE DIRECT MEMORY ACCESS NETWORK

[0002] TECHNICAL FIELD

[0003] The disclosure relates generally to data packet transmission and, more particularly, to a method and a system for receiving one or more data packets in a Remote Direct Memory Access, RDMA, network. The method involves communication between a requesting node and a responding node. The disclosure also details the functionalities of a responding node in a Remote Direct Memory Access, RDMA, network.

[0004] BACKGROUND

[0005] A Remote Direct Memory Access, RDMA, network is a computer network architecture that enables direct memory access from the memory of one computer to another computer without utilizing processors. The RDMA network facilitates efficient data transfer between computers by reducing reliance on a central processing unit, CPU, and minimizing latency. This enhances overall performance within the RDMA network.

[0006] Traditionally, the RDMA network has been structured as a lossless fabric to ensure the transmission of all packets without loss or corruption. Both InfiniBand and Ethernet, utilizing RDMA network over Converged Ethernet, RoCE, technologies implement flow control mechanisms such as credit-based flow control, and Priority-based Flow Control, PFC, protocols tailored to InfiniBand and Ethernet, respectively. The lossless fabric is particularly advantageous for traditional High- Performance Computing, HPC, applications.

[0007] The RDMA network employs a non-complex Automatic Repeat Request, ARQ, protocol, such as Go-Back-N, for efficient handling of retransmission when a packet is lost. This choice of ARQ protocol is driven by the lossless nature of the RDMA network. Additionally, the traditional RDMA network does not support Out of Order, OOO, packet delivery. In the event of packet loss requiring retransmission during transmission, the traditional RDMA network implements the Go-back-N ARQ protocol to manage packet retransmissions.

[0008] In recent years, managing network congestion has emerged as a critical issue within the network system, referring to interconnected devices equipped with the RDMA network. The network system may be a data center or a cluster. Various methods, such as the adaptive routing method and selective repeat ARQ protocol, have been developed to address and mitigate network congestion. This is particularly relevant to the RDMA network, where efficient traffic and congestion handling are vital for maintaining high performance and reliability.

[0009] The adaptive routing method is employed for load-balancing traffic or is used to adjust the path of data traffic within the network system based on current network conditions such as network congestion and failures. Therefore, the adaptive routing method redirects data away from areas with network congestion or failures within the network system. The adaptive routing method is a process wherein a router can reroute data within the network system via an alternate route for a specified destination based on the current network conditions. The adaptive routing method dynamically determines optimal routes based on network topology and changes in traffic loads. By detecting congestion status on network links, the adaptive routing method preferentially selects either the shortest path or a non-shortest path for packet forwarding. This enhances throughput, resilience and reduces latency within the network system. The adaptive routing method is deployed and utilized in large supercomputing data centers with topologies such as Fat-Tree or Folded Clos network topology, and DragonFly network topology. The Fat-Tree network topology is a standard network topology in large supercomputing data centers. The Fat-Tree network topology includes interconnected groups of fully bipartite graphs across different levels. In the Fat-Tree network topology, the adaptive routing method efficiently distributes the traffic load among multiple possible routing paths.

[0010] Conversely, the DragonFly network topology is employed as a direct topology, with each node directly linked to a compute node. The DragonFly network topology is divided into multiple groups, where within each group, all network nodes are interconnected by full-mesh connections. Between different groups, there are connections or network links facilitating communication. These network links enable direct communication between network nodes in different groups, allowing for efficient data transmission and overall network connectivity. However, the limited number of shortest paths between network nodes in the DragonFly network topology may lead to network congestion issues. To mitigate network congestion, the DragonFly network topology heavily relies on alternative non-shortest paths extensively.

[0011] The adaptive routing method is deployed and utilized in the DragonFly network topology to optimize network performance. The adaptive routing method preferentially (i) selects the shortest path for packet forwarding when the network links are normal or (ii) selects the non-shortest path that is not congested to forward the packets when the shortest path is congested. This means that when network congestion occurs on the shortest path, the non-shortest path is selected for packet transmission to distribute the traffic loads. Implementing the adaptive routing method in the DragonFly network topology optimizes the utilization of network links. This enhances bandwidth utilization, increases throughput, reduces latency, and lowers costs for large supercomputing data centers. The adaptive routing method leads packets to traverse different network routes with varying propagation delays or congestion points. This variation can cause some packets to arrive at the destination faster than others, leading to OOO delivery. This increased latency can impact the overall performance of the RDMA network, as the receiver may require computational resources and time to reorder received packets.

[0012] The selective repeat ARQ protocol manages a sequence of packets in reliable network connections or communications. The selective repeat ARQ protocol avoids redundant retransmission of lost packets within the network system, thereby reducing network congestion. The selective repeat ARQ protocol is a control flow protocol, or a type of error-control protocol. The selective repeat ARQ protocol is used in data communication and high-speed networks where efficient bandwidth utilization and quick error recovery are needed.. The Selective Repeat ARQ protocol, while more complex than protocols like Go-Back- N, offers greater bandwidth efficiency.. This means that only damaged or lost packets are retransmitted, rather than retransmitting the sequence of packets.

[0013] In the selective repeat ARQ protocol, a window of packets sent by a sender and a window of packets received by a receiver remains the same. The sender sends frames, or the packets specified by a window size without waiting for individual acknowledgements, ACK, from the receiver. The window size is set by the sender, the receiver, or the selective repeat ARQ protocol. The sender's window represents the packets that have been sent but not yet acknowledged, while the receiver's window represents the packets that can be accepted in any order. Upon receiving each packet, the receiver acknowledges each received packet individually and correctly. If the receiver receives some packets out of order (OOO), the receiver still accepts the packets and sends a Negative Acknowledgment, NACK, to the sender, indicating unreceived or missing packets. Upon receiving the NACK from the receiver, the sender retransmits only the damaged or lost packets, avoiding the need to retransmit the entire window of packets. In other words, the receiver may selectively reject a single frame, which may be retransmitted alone. This selective retransmission feature distinguishes the selective repeat ARQ protocol from alternative protocols like the Go-Back- N ARQ protocol. Unlike the Go-Back-N protocol which retransmits the entire window upon receiving the NACK, the selective repeat ARQ protocol retransmits only the selective missing packets upon receiving the NACK. The receiver accepts the frames that are received out of order (OOO) and acknowledges the reception of these frames by buffering them. If a timeout occurs without receiving acknowledgments for certain frames, the sender resends those specific frames that were not acknowledged by the receiver, rather than resending the entire set of frames. However, managing selective retransmissions can be challenging. In scenarios where the receiver acknowledges frames after their timeout because the packet was received out of order (OOO), the sender might redundantly resend the same packet to the receiver. This redundancy can lead to inefficiencies in network utilization and exacerbate network congestion issues.

[0014] Therefore, there arises a need to address the aforementioned technical problem / drawbacks in the interaction between a sender and a receiver for receiving data packets in a Remote Direct Memory Access, RDMA, network.

[0015] SUMMARY

[0016] It is an object of the disclosure to provide a method and a system of receiving one or more data packets in a Remote Direct Memory Access, RDMA, network being sent between a requesting node and a responding node. The disclosure also relates to a responding node in a Remote Direct Memory Access, RDMA, network while avoiding one or more disadvantages of prior art approaches.

[0017] This object is achieved by the features of the independent claims. Further, implementation forms are apparent from the dependent claims, the description, and the figures.

[0018] According to a first aspect, a method of receiving one or more data packets in a Remote Direct Memory Access, RDMA, network being sent between a requesting node and a responding node is provided. The method includes receiving the one or more data packets from the requesting node. The method includes determining whether at least one of the received one or more data packets contains a path reroute marking indicator in a header of the at least one of the received one or more data packets. The path reroute marking indicator indicates that the at least one data packet has been rerouted onto a second path through the RDMA network that is different from a first path through the RDMA network over which others of the received one or more data packets have traversed. The method includes sending an adaptive routing notification indicator in the header of a data packet back to the requesting node upon determining that at least one of the received one or more data packets contains the path reroute marking indicator in the header. The adaptive routing notification indicator notifies the requesting node, upon receipt, that the at least one of the received one or more data packets has been received out of order, OOO, due to the rerouting.

[0019] The method, as described above, integrates multipath adaptive routing with a selective repeat mechanism to ensure efficient operation even in a lossy network environment. The method utilizes a novel control protocol for Remote Direct Memory Access, RDMA, over Converged Ethernet, ROCE, in the lossy network environment. The method enables identification of path reroutes through a dedicated bit or a specific bit marking in a Base Transport Header, BTH, field. The method utilizes the dedicated bit to generate a new Adaptive Routing Notification, ARN, message or an acknowledge, ACK, message to notify the requesting node about an Out of Order, OOO, path reroute packet event. The method efficiently corrects path markings for the one or more data packets by implementing error agnostic or routing table-based methods in modem switches. The method allows the adaptive routing method and the selective repeat mechanism to enhance the performance of the RDMA network. The method identifies the origin of the OOO packet arrivals at the requesting node, enabling the retransmission of the data packets individually in case of packet loss. The method utilizes a sniffer to detect the one or more data packets passing through the RDMA network and analyze them to monitor RDMA network activity, detect anomalies, or troubleshoot issues within the RDMA network. The method collects and analyzes switch logs to identify security threats, performance issues, and configuration errors within the RDMA network. The switch logs may include records of events, activities, and errors that occur within the RDMA network. The method prevents the unnecessary re-transmission of the data packets that are received out of order (OOO). This method performs path rerouting of the one or more data packets at lower congestion granularity, thus enabling the switch to bypass local congestion points in case of adaptive routing. Preferably, the first path is changed to the second path by adaptive routing. Optionally, when the determining step determines that none of the received one or more data packets contains the path reroute marking indicator but detecting that a packet has been received OOO, a negative acknowledgement, NACK, packet is sent to the requesting node, requesting that a missing packet be resent.

[0020] Preferably, the received one or more data packets are part of a write request. Preferably, the path reroute marking indicator is represented by at least one bit in a Base Transport Header, BTH. The adaptive routing notification indicator may be represented by at least one bit in an acknowledgment Extended Transport Header, AETH, syndrome field.

[0021] According to a second aspect, a responding node in a Remote Direct Memory Access, RDMA, network configured to receive one or more data packets from a requesting node is provided. The responding node is configured to determine whether at least one of the received one or more data packets contains a path reroute marking indicator in a header of the at least one of the received one or more data packets. The path reroute marking indicator indicates that the at least one data packet has been rerouted onto a second path through the RDMA network that is different from a first path through the RDMA network over which others of the received one or more data packets have traversed. The responding node is configured to send an adaptive routing notification indicator in the header of a data packet back to the requesting node upon determining that at least one of the received one or more data packets contains the path reroute marking indicator in the header. The adaptive routing notification indicator notifies the requesting node, upon receipt, that the at least one of the received one or more data packets has been received out of order, OOO, due to the rerouting.

[0022] Preferably, the responding node is configured to change the first path to the second path by adaptive routing. Optionally, when the determining step determines that none of the received one or more data packets contains the path reroute marking indicator but detecting that a packet has been received OOO, a negative acknowledgement, NACK, packet is sent to the requesting node, requesting that a missing packet be resent.

[0023] Preferably, the received one or more data packets are part of a write request. Preferably, the path reroutes marking indicator is represented by at least one bit in a Base Transport Header, BTH. The adaptive routing notification indicator may be represented by at least one bit in an acknowledgment Extended Transport Header, AETH, syndrome field.

[0024] According to a third aspect, a system includes means adapted for carrying out all the steps of the method is provided.

[0025] According to a fourth aspect, a computer program includes instructions for carrying out all the steps of the method when the computer program is executed on a computer system is provided.

[0026] These and other aspects of the disclosure will be apparent from the implementations) described below.

[0027] BRIEF DESCRIPTION OF DRAWINGS

[0028] Implementations of the disclosure will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0029] FIG. 1 illustrates a block diagram of a system that depicts an interaction between a requesting node and a responding node for sending one or more data packets in a Remote Direct Memory Access, RDMA, network during an RDMA write operation in accordance with an implementation of the disclosure;

[0030] FIG. 2 illustrates a block diagram of a system that depicts an interaction between a requesting node and a responding node for sending one or more data packets in a Remote Direct Memory Access, RDMA, network when at least one of the one or more data packets is lost during an RDMA write operation in accordance with an implementation of the disclosure;

[0031] FIG. 3 illustrates a block diagram of a system that depicts an interaction between a requesting node and a responding node for receiving one or more data packets in a Remote Direct Memory Access, RDMA, network when at least one of the one or more data packets includes a path reroute marking indicator during an RDMA read operation in accordance with an implementation of the disclosure;

[0032] FIG. 4 shows a block diagram of a system that illustrates an interaction between a requesting node and a responding node for receiving one or more data packets in a Remote Direct Memory Access, RDMA, network when at least one of the one or more data packets is lost during an RDMA read operation in accordance with an implementation of the disclosure;

[0033] FIG. 5 illustrates a block diagram of a system that depicts an interaction between a requesting node and a responding node for receiving one or more data packets in a Remote Direct Memory Access, RDMA, network when a false negative error occurs in accordance with an implementation of the disclosure;

[0034] FIG. 6 illustrates a block diagram of a system that depicts an interaction between a requesting node and a responding node for receiving one or more data packets in a Remote Direct Memory Access, RDMA, network when a false positive error occurs in accordance with an implementation of the disclosure;

[0035] FIG. 7 is a flow diagram that illustrates a method of receiving one or more data packets in a Remote Direct Memory Access, RDMA, network being sent between a requesting node and a responding node in accordance with an implementation of the disclosure;

[0036] FIG. 8 is a flow diagram that illustrates a method of Remote Direct Memory Access, RDMA, write operation of a responding node in accordance with an implementation of the disclosure;

[0037] FIG. 9 is a flow diagram that illustrates a method of Remote Direct Memory Access, RDMA, read operation of a requesting node in accordance with an implementation of the disclosure; and

[0038] FIG. 10 is an illustration of a computer system (e.g., a requesting node, and a responding node) in which the various architectures and functionalities of the various previous implementations may be implemented.

[0039] DETAILED DESCRIPTION OF THE DRAWINGS

[0040] This disclosure describes implementations that provide a method and a system of receiving one or more data packets in a Remote Direct Memory Access, RDMA, network being transmitted between a requesting node and a responding node.

[0041] To make solutions of the disclosure more comprehensible for a person skilled in the art, the following implementations of the disclosure are described with reference to the accompanying drawings.

[0042] Terms such as “a first”, “a second”, “a third”, and “a fourth” (if any) in the summary, claims, and foregoing accompanying drawings of the disclosure are used to distinguish between similar objects and are not necessarily used to describe a specific sequence or order. It should be understood that the terms so used are interchangeable under appropriate circumstances, so that the implementations of the disclosure described herein are, for example, capable of being implemented in sequences other than the sequences illustrated or described herein. Furthermore, the terms “include” and “have” and any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, a method, a system, a product, or a device that includes a series of steps or units, is not necessarily limited to expressly listed steps or units but may include other steps or units that are not expressly listed or that are inherent to such process, method, product, or device.

[0043] Definitions:

[0044] Path Marking: Path marking refers to the process by which a switch marks a packet header to indicate a rerouted path. It is important to note that only a first data packet following the rerouted path is marked in this manner.

[0045] Adaptive Routing Notification, ARN: Adaptive Routing Notification is a special negative acknowledgment, NACK, that notifies that an out-of-order (OOO) packet reaches the destination due to a path reroute.

[0046] FIG. 1 illustrates a block diagram of a system 100 that depicts an interaction between a requesting node 104 and a responding node 106 for sending one or more data packets 108A-C in a Remote Direct Memory Access, RDMA, network 102 during an RDMA write operation in accordance with an implementation of the disclosure. The system 100 includes the RDMA network 102 including the requesting node 104, and the responding node 106. The requesting node 104 transmits the one or more data packets 108A-C to the responding node 106. The one or more data packets 108A-C may be a part of a write request. The write request includes an RDMA write operation. The RDMA write operation instructs the RDMA network 102 to transfer data in the one or more data packets 108A-C directly from a memory of the requesting node 104 to a memory of the responding node 106. The one or more data packets 108A-C include at least one of a first data packet 108A, a second data packet 108B, and a Nth data packet 108C. The responding node 106 receives the one or more data packets 108A-C from the requesting node 104. The responding node 106 receives the one or more data packets (e.g. 108A, 108B) through one or more first paths (e.g.,110A, HOB) from the requesting node 104.

[0047] Upon the arrival of each data packet 108A at the responding node 106, the responding node 106 determines whether at least one of the one or more data packets 108A-C contains a path reroute marking indicator in a header of the at least one of the one or more data packets 108A-C. The path reroute marking indicator indicates that the at least one data packet is rerouted onto a second path 112 through the RDMA network 102. The second path 112 is different from the first path 110A-B where other data packets of the one or more data packets 108A-C are traversed. For example, the Nth data packet 108C from the one or more data packets 108A-C is rerouted to the second path 112 through the RDMA network 102 from the first path 110A-B. This rerouting occurs while transmitting the one or more data packets 108A-C from the requesting node 104 to the responding node 106. The RDMA network 102 marks the Nth data packet 108C with the path reroute marking indicator by a relevant switch. The responding node 106 identifies that the Nth data packet 108C is marked or rerouted and it reaches the responding node 106 in a series of Out of Order, OOO. The path reroute marking indicator may be represented by at least one bit in a Base Transport Header, BTH.

[0048] The responding node 106 sends an adaptive routing notification indicator 114 in the header of the one or more data packets (e.g. 108A-C) back to the requesting node 104 if the path reroute marking indicator is set on the header of at least one of the one or more data packets 108A-C and each data packet arrives in the responding node 106 from the requesting node 104 in series of OOO. The adaptive routing notification indicator 114 may include an Adaptive Routing Notification, ARN, and Negative acknowledgment, NACK. Upon receiving the adaptive routing notification indicator 114 by the requesting node 104, the adaptive routing notification indicator 114 notifies the requesting node 104 that the at least one of the one or more data packets 108A-Cis received in the series of OOO due to the rerouting. The requesting node 104 then interprets that the one or more data packets 108A-C have been transmitted and stored to the responding node 106. Therefore, the requesting node 104 does not need to resend the Nth data packet 108C to the responding node 106, as the Nth data packet 108A is received in the series of OOO. The adaptive routing notification indicator 114 may be represented by at least one bit in an acknowledgment Extended Transport Header, AETH, syndrome field.

[0049] FIG. 2 illustrates a block diagram of a system 200 that depicts an interaction between a requesting node 204 and a responding node 206 for sending one or more data packets 208A-C in a Remote Direct Memory Access, RDMA, network 202 when at least one of the one or more data packets 208A-C is lost during an RDMA write operation in accordance with an implementation of the disclosure. The system 200 includes the RDMA network 202, the requesting node 204, and the responding node 206. The requesting node 204 transmits the one or more data packets 208A-C to the responding node 206 through one or more first paths (e.g., 210A, 210B, 210C). The one or more data packets 208A-C include a first data packet 208A, a second data packet 208B, and a third data packet 208C. The responding node 206 receives the one or more data packets 208A-C from the requesting node 204.

[0050] Upon the arrival of each data packet 208A at the responding node 204„ the responding node 206 checks the reception status of the one or more data packets 208 A-C by determining whether the one or more data packets 208 A-C are received or not. For example, if the responding node 206 identifies a loss of the second data packet 208B among the one or more data packets 208A- C during transmission, the third packet 208C may not be marked by a relevant switch. Subsequently, upon the arrival of the third data packet 208C at the responding node 206, it is identified as part of a series of Out of Order, OOO, and remains unmarked. In response to this, the responding node 206 is configured to send a negative acknowledgement, NACK, 212 to the requesting node 204 when it determines that none of the received one or more data packets 208A-C contains a path reroute marking indicator. Simultaneously, the requesting node 204 detects that the first data packet 208A and the third data packet 208C have been received in the series of OOO. The NACK includes a request to transmit the second data packet 208B and is transmitted to responding node 206 as the second data packet 208B (i.e., a missed packet) was lost during the previous transmission. Subsequently, the requesting node 204 receives the NACK and resends the second data packet 208B to the responding node 206.

[0051] FIG. 3 illustrates a block diagram of a system 300 that depicts an interaction between a requesting node 304 and a responding node 306 for receiving one or more data packets 308A-C in a Remote Direct Memory Access, RDMA, network 302 when at least one of the one or more data packets 308A-C includes a path reroute marking indicator during an RDMA read operation in accordance with an implementation of the disclosure. The system 300 includes the RDMA network 302, the requesting node 304, and the responding node 306. The requesting node 304 sends a read request to the responding node 306. The read request includes an RDMA read operation. The RDMA read operation may instruct the RDMA network 302 to retrieve the one or more data packets 308 A-C directly from a memory of the responding node 306 to a memory of the requesting node 304. The one or more data packets 308A-C include at least one of a first data packet 308A, a second data packet 308B, or a third data packet 308C. The responding node 306 receives the read request from the requesting node 304. The responding node 306 transmits the one or more data packets 308A-C to the requesting node 304 through one or more first paths (e.g., 310A, 310B) in the RDMA network 302.

[0052] Upon the arrival of each data packet 308A at the requesting node 304 the requesting node 304 determines whether at least one of the one or more data packets 308A-C contains a path reroute marking indicator in the header of the at least one of the one or more data packets 308A-C. For example, the third data packet 308C in the one or more data packets 308A-C may be rerouted to a second path 312 from the first path 310A-B within the RDMA network 302, while the requesting node 304 receives other data packets of the one or more data packets 308A-C through the one or more first paths (e.g., 310A, 310B). The RDMA network 302 marks the third data packet 308C with a path reroute marking indicator through a relevant switch. Upon reaching the requesting node 304, the third data packet 308C is marked as rerouted when it arrives in a series of Out of Order, OOO. If the third data packet 308C is marked as rerouted by the path reroute marking indicator, the requesting node 304 does not send a negative acknowledgment, NACK, to the responding node 306. Instead, the requesting node 304 sends a read request to the responding node 306. This is because the requesting node 304 interprets that it has to wait for a next data packet upon identifying that the third data packet 308C is marked by the path reroute marking indicator. The third data packet 308C is marked by the path reroute marking indicator to indicate that the third data packet 308C reached the requesting node 304 in the series of OOO.

[0053] FIG. 4 illustrates a block diagram of a system 400 that depicts an interaction between a requesting node 404 and a responding node 406 for receiving one or more data packets 408A-C in a Remote Direct Memory Access, RDMA, network 402 when at least one of the one or more data packets 408A-C is lost during an RDMA read operation in accordance with an implementation of the disclosure. The system 400 includes the RDMA network 402 including the requesting node 404, and the responding node 406.

[0054] The requesting node 404 sends a read request to the responding node 406. The one or more data packets 408A-C include at least one of a first data packet 408A, a second data packet 408B, or a third data packet 408C. The responding node 406 receives the read request from the requesting node 404. The responding node 406 transmits the one or more data packets 408 A-C to the requesting node 404 through the RDMA network 402 in one or more first paths (e.g., 410A, 410B, 410C).

[0055] Upon receiving each data packet 408A from the responding node 406, the requesting node 404 checks the reception status of the one or more data packets 408 A-C by determining whether the one or more data packets 408 A-C are received or not. For example, if the requesting node 404 identifies a loss of the second data packet 408B among the one or more data packets 408A- C during transmission, the third data packet 408C may not be marked by a relevant switch. Subsequently, upon the arrival of the third data packet 408C at the requesting node 404, it is identified as part of a series of Out of Order, OOO, and remains unmarked. In response to this, the requesting node 404 sends a read request to responding node 406 to transmit the second data packet 408B again, as the second data packet 408B was lost during the previous transmission. The responding node 406 receives the read request from the requesting node 404 and resends the second data packet 408B to the requesting node 404.

[0056] FIG. 5 illustrates a block diagram of a system 500 that depicts an interaction between a requesting node 504 and a responding node 506 for receiving one or more data packets 508A-D in a Remote Direct Memory Access, RDMA, network 502 when a false negative error occurs in accordance with an implementation of the disclosure. The system 500 includes the RDMA network 502, the requesting node 504, and the responding node 506. The requesting node 504 transmits the one or more data packets 508A-D to the responding node 506. The one or more data packets 508A-D may be a part of a write request. The write request includes an RDMA write operation. The one or more data packets 508A-D include at least one of a first data packet 508A, a second data packet 508B, a third data packet 508C, or a fourth data packet 508D. The responding node 506 receives the one or more data packets 508A-D from the requesting node 504 through one or more first paths (e.g., 510A, 510B, 510C).

[0057] Upon receiving each data packet 508A from the requesting node 504, the responding node 506 determines whether at least one of the one or more data packets 508A-D contains the path reroute marking indicator in the header of the at least one of the one or more data packets 508A-D. If the responding node 506 determines that at least one of the one or more data packets 508A-D does not contain the path reroute marking indicator, the responding node 506 identifies them as arriving in a series of Out of Order, OOO, assuming that the at least one of the one or more received data packets 508A-D is lost during transmission due to being received out of order, OOO.

[0058] In this case, if the at least one of the one or more data packets 508A-D reroutes onto a second path 512 within the RDMA network 502 that is different from the first path 510A-C over which others of the one or more data packets 508 A-D are traversed, the RDMA network 502 fails to mark the rerouted data packet with the path reroute marking indicator. This results in a false negative error as the at least one of the one or more received data packets 508A-D is not correctly marked with the path reroute marking indicator. In other words, the responding node 506 incorrectly marks the path of the at least one of the one or more data packets 508A-D. The responding node 506 sends a negative acknowledgment, NACK, 514 to the requesting node 504 upon the data packet 508D arriving through the second path 512 in the responding node 506. The responding node 506 sends the NACK 514 to the requesting node 504 to request retransmission of a missed data packet, if the responding node 506 determines that the path reroute marking indicator is not set on the header of the one or more received data packets and the one or more data packets 508A-D are received in the series of OOO. Meanwhile, the missed data packet reaches the responding node 506 in the series of OOO. The requesting node 504 retransmits the missed data packet to the responding node 506. The responding node 506 may discard the retransmitted missed data packet (i.e., duplicate data packet), if the missed data packet (i.e., original data packet), is received in the series of OOO.

[0059] For example, if the fourth data packet 508D in the one or more received data packets 508 A-D is rerouted to the second path 512 from the first path 510A-C, while other data packets in the one or more data packets 508A-D are received through the first path 510A-C, the responding node 506 identifies that the fourth data packet 508D reaches the responding node 506 in the series of OOO without being marked by the path reroute marking indicator, assuming that the third data packet 508C is lost during transmission. The responding node 506 sends the NACK 514 to the requesting node 504 to request retransmission of the third data packet 508C. The requesting node 504 receives the NACK 514 and re-transmits the third data packet 508C to the responding node 506. Meanwhile, if the responding node 506 receives the third data packet 508C (i.e., the original third data packet that was initially transmitted by the requesting node 504 and that was assumed as the missed packet by the responding node 506), in the series of OOO, the responding node 506 discards the re- transmitted third data packet (i.e., a duplicate data packet) when the third data packet 508C (i.e., the original third data packet) reaches the responding node 506.

[0060] FIG. 6 illustrates a block diagram of a system 600 that depicts an interaction between a requesting node 604 and a responding node 606 for receiving one or more data packets 608A-D in a Remote Direct Memory Access, RDMA, network 602 when a false positive error occurs in accordance with an implementation of the disclosure. The system 600 includes the RDMA network 602 including the requesting node 604, and the responding node 606. The requesting node 604 transmits the one or more data packets 608A-D to the responding node 606. The requesting node 604 starts a Retransmission Time-Out, RTO, timer to wait for an acknowledgment of the one or more data packets 608 A-D from the responding node 606. The one or more data packets 608A-D may be a part of a write request. The write request includes an RDMA write operation. The one or more data packets 608A-D include at least one of a first data packet 608A, a second data packet 608B, a third data packet 608C, and a fourth data packet 608D. The responding node 606 receives the one or more data packets 608A-D from the requesting node 604.

[0061] Upon the arrival of each data packet at the responding node 606, the responding node 606 determines whether at least one of the one or more data packets 608A-D contains a path reroute marking indicator in a header of the at least one of the one or more data packets 608A-D. If the responding node 606 determines that at least one of the one or more data packets 608A-D contains the path reroute marking indicator, the responding node 606 identifies that the at least one of the one or more data packets 608A-D reaches a series of Out of Order, OOO.

[0062] In this case, none of the one or more data packets 608A-D are rerouted onto a second path from one or more first paths (e.g., 610A, 610B, 610C, 610D) during transmission. However, the RDMA network 602 incorrectly marks at least one of the one or more data packets 608A-D as the path reroute marking indicator. This results in a false positive error since the at least one of the one or more data packets 608A-D is marked as the path reroute marking indicator.

[0063] For example, if the fourth data packet 608D in the one or more data packets 608 A-D is not rerouted to the second path from the first path 610A-D, but the third data packet 608C is lost during transmission, the RDMA network 602 incorrectly marks the fourth data packet 608D with the path reroute marking indicator by a relevant switch. The responding node 606 identifies that the fourth data packet 608D reaches the responding node 606 in the series of OOO, as the fourth data packet 608D is marked by the path reroute marking indicator. The responding node 606 then sends an Adaptive Routing Notification, ARN, message 612 to the requesting node 604 upon identifying that at least one of the one or more data packets 608A-D is received in the series of OOO. The responding node 606 assumes that the third data packet 608C has reached the responding node 606 in the series of OOO upon receiving the ARN message 612. Meanwhile, the requesting node 604 assumes that the third data packet 608C was lost and retransmits the third data packet 608C to the responding node 606 when the RTO timer of the third data packet 608C expires before receiving the acknowledgment for the third data packet 608C. Consequently, the responding node 606 receives the missed data packet, i.e., the third data packet 608C.

[0064] The system 600 includes an RDMA network over Converged Ethernet, ROCE, protocol that uses a transport layer of the RDMA network 602 for connection management through a Base Transport Eleader, BTEI. The system 600 selects a specific bit in the BTH header for marking the path of the one or more data packets 608A-D. The responding node 606 accurately interprets the path taken by the one or more data packets 608A-D if the specific bit is selected within a certain range of reserved bits i.e. 6 reserved bits, to prevent cyclic redundancy check, CRC, error detection from detecting changes made by the RDMA network 602. If the selected bits for path marking fall outside a certain range of the reserved bits, changes made by switches in the RDMA network 602 might trigger the CRC error detection leading to an error.

[0065] Table 1: This table depicts the RDMA network 602 that marks the path of the one or more data packets 608A-D within a bit (i.e., the specific bit) on the BTH header.

[0066] The selected specific bit is marked by a square box in the table 1.

[0067] Table 2: This table depicts the specific bit in a syndrome field i.e., any code within a range of OlOxxxxx can be used to generate the ARN message. The ARN message utilizes an acknowledge extension header.

[0068] AETH syndrome field FIG. 7 is a flow diagram that illustrates a method of receiving one or more data packets in a Remote Direct Memory Access, RDMA, network being sent between a requesting node and a responding node in accordance with an implementation of the disclosure. At a step 702, the method includes receiving the one or more data packets in the responding node from the requesting node. At a step 704, the method includes determining whether at least one of the one or more received data packets contains a path reroute marking indicator in a header of the at least one of the one or more received data packets in the responding node. The path reroute marking indicator indicates that the at least one data packet has been rerouted onto a second path through the RDMA network that is different from a first path through the RDMA network over which others of the one or more received data packets have traversed. At a step 706, the method includes sending an adaptive routing notification indicator in the header of a data packet back to the requesting node by the responding node, upon determining that at least one of the one or more received data packets contains the path reroute marking indicator in the header. The adaptive routing notification indicator notifies the requesting node, upon receipt, that the at least one of the one or more received data packets has been received out of order, OOO, due to the rerouting.

[0069] FIG. 8 is a flow diagram that illustrates a method of Remote Direct Memory Access, RDMA, write operation of a responding node in accordance with an implementation of the disclosure. At a step 802, the method includes receiving one or more data packets from a requesting node. At a step 804, the method includes checking whether a path reroute marking indicator, PRMI, in a BTH header of the one or more received data packets if the responding node receives the one or more data packets in a sequence of Out of Order, OOO. At a step 806, the method includes sending an adaptive routing notification, ARN, message to the requesting node if the PRMI is set on the BTH header of the one or more received packets. At a step 808, the method includes sending a Negative Acknowledgement, NACK, message to the requesting node if the PRMI is not set on the BTH header of the one or more received packets.

[0070] FIG. 9 is a flow diagram that illustrates a method of Remote Direct Memory Access, RDMA, read operation of a requesting node in accordance with an implementation of the disclosure. At a step 902, the method includes receiving one or more data packets from a responding node. At a step 904, the method includes checking whether a path reroute marking indicator, PRMI, in a BTH header of the one or more received data packets if the requesting node receives the one or more data packets in a sequence of Out of Order, OOO. At a step 906, the method includes sending an adaptive routing notification, ARN, message to the responding node if the PRMI is set on the BTH header of the one or more received packets. At a step 908, the method includes sending a read request with missing packets to the responding node if the PRMI is not set on the BTH header of the one or more received packets.

[0071] FIG. 10 is an illustration of a computer system (e.g., a requesting node, and a responding node) 1000 in which the various architectures and functionalities of the various previous implementations may be implemented. As shown, the computer system 1000 includes at least one processor 1003 that is connected to a bus 1002, wherein the computer system 1000 may be implemented using any suitable protocol, such as Peripheral Component Interconnect, PCI, Express, Accelerated Graphics Port, AGP, Hyper Transport, or any other bus or point-to-point communication protocol. The computer system 1000 also includes a memory 1006.

[0072] Control logic (software) and data are stored in the memory 1006 which may take a form of random-access memory, RAM. In the disclosure, a single semiconductor platform may refer to a sole unitary semiconductor-based integrated circuit or chip. It should be noted that the term single semiconductor platform may also refer to multi-chip modules with increased connectivity which simulate on-chip modules with increased connectivity which simulate on-chip operation, and make substantial improvements over utilizing a conventional central processing unit, CPU, and bus implementation. Of course, the various modules may also be situated separately or in various combinations of semiconductor platforms per the desires of the user. The computer system 1000 may also include a secondary storage 1010. The secondary storage 1010 includes, for example, a hard disk drive and a removable storage drive, representing a floppy disk drive, a magnetic tape drive, a compact disk drive, a digital versatile disk, a DVD drive, a recording device, a universal serial bus, a USB flash memory. The removable storage drives at least one of reads from and writes to a removable storage unit in a well-known manner.

[0073] Computer programs, or computer control logic algorithms, may be stored in at least one of the memory 1006 and the secondary storage 1010. Such computer programs, when executed, enable the computer system 1000 to perform various functions as described in the foregoing. The memory 1006, the secondary storage 1010, and any other storage are possible examples of computer-readable media.

[0074] In an implementation, the architectures and functionalities depicted in the various previous figures may be implemented in the context of the processor 1004, a graphics processor coupled to a communication interface 1012, an integrated circuit (not shown) that is capable of at least a portion of the capabilities of both the processor 1004 and a graphics processor, a chipset (namely, a group of integrated circuits designed to work and sold as a unit for performing related functions, and so forth).

[0075] Furthermore, the architectures and functionalities depicted in the various previous-described figures may be implemented in a context of a general computer system, a circuit board system, a game console system dedicated to entertainment purposes, an application- specific system. For example, the computer system 1000 may take the form of a desktop computer, a laptop computer, a server, a workstation, a game console, or an embedded system.

[0076] Furthermore, the computer system 1000 may take the form of various other devices including, but not limited to a personal digital assistant, PDA, device, a mobile phone device, a smart phone, a television, and so forth. Additionally, although not shown, the computer system 1000 may be coupled to a network (for example, a telecommunications network, a local area network, LAN, a wireless network, a wide area network, WAN, such as the Internet, a peer-to-peer network, a cable network, or the like) for communication purposes through an I / O interface 1008.

[0077] It should be understood that the arrangement of components illustrated in the figures described is exemplary and that other arrangement may be possible. It should also be understood that the various system components (and means), defined by the claims, described below, and illustrated in the various block diagrams, represent components in some systems configured according to the subject matter disclosed herein. For example, one or more of these system components (and means) may be realized, in whole or in part, by at least some of the components illustrated in the arrangements illustrated in the described figures.

[0078] In addition, while at least one of these components is implemented at least partially as an electronic hardware component, and therefore constitutes a machine, the other components may be implemented in software that when included in an execution environment constitutes a machine, hardware, or a combination of software and hardware.

[0079] Although the disclosure and its advantages have been described in detail, it should be understood that various changes, substitutions, and alterations can be made herein without departing from the spirit and scope of the disclosure as defined by the appended claims.

Claims

CLAIMS1. Amethod of receiving a plurality of data packets (108A-C, 208A-C, 308A-C, 408A-C, 508A-D, 608A-D) in a Remote Direct Memory Access, RDMA, network being sent between a requesting node (104, 204, 304, 404, 504, 604) and a responding node (106, 206, 306, 406, 506, 606), comprising steps of: receiving the plurality of data packets (108A-C, 208A-C, 308A-C, 408A-C, 508A-D, 608A-D) from the requesting node (104, 204, 304, 404, 504, 604); determining whether at least one of the received plurality of data packets ( 108A-C, 208A-C, 308A-C, 408A-C, 508A- D, 608A-D) contains a path reroute marking indicator in a header of the at least one of the received plurality of data packets (108A-C, 208A-C, 308A-C, 408A-C, 508A-D, 608A-D), wherein the path reroute marking indicator indicates that the at least one data packet has been rerouted onto a second path through the RDMA network that is different from a first path through the RDMA network over which others of the received plurality of data packets (108A-C, 208A-C, 308A-C, 408A-C, 508A-D, 608A-D) have traversed; and upon determining that at least one of the received plurality of data packets (108A-C, 208A-C, 308A-C, 408A-C, 508A-D, 608A-D) contains the path reroute marking indicator in the header, sending an adaptive routing notification indicator in the header of a data packet back to the requesting node (104, 204, 304, 404, 504, 604), wherein the adaptive routing notification indicator notifies the requesting node (104, 204, 304, 404, 504, 604), upon receipt, that the at least one of the received plurality of data packets (108A-C, 208A-C, 308A-C, 408A-C, 508A-D, 608A-D) has been received out of order, OOO, due to the rerouting.

2. The method of claim 1 wherein the first path is changed to the second path by adaptive routing.

3. The method of claim 1 wherein when the determining step determines that none of the received plurality of data packets (108A-C, 208A-C, 308A-C, 408A-C, 508A-D, 608A-D) contains the path reroute marking indicator but detecting that a packet has been received OOO, sending a negative acknowledgement, NACK, packet to the requesting node (104, 204, 304, 404, 504, 604), requesting that a missing packet be resent.

4. The method of claim 1 wherein the received plurality of data packets (108A-C, 208A-C, 308A-C, 408A-C, 508A-D, 608A-D) are part of a write request.

5. The method of claim 1 wherein the path reroute marking indicator is represented by at least one bit in a Base Transport Header, BTH.

6. The method of claim 1 wherein the adaptive routing notification indicator is represented by at least one bit in an acknowledgment Extended Transport Header, AETH, syndrome field.

7. A responding node (106, 206, 306, 406, 506, 606) in a Remote Direct Memory Access, RDMA, network configured to: receive a plurality of data packets (108A-C, 208A-C, 308A-C, 408A-C, 508A-D, 608A-D) from a requesting node (104, 204, 304, 404, 504, 604);determine whether at least one of the received plurality of data packets (108A-C, 208A-C, 308A-C, 408A-C, 508A- D, 608A-D) contains a path reroute marking indicator in a header of the at least one of the received plurality of data packets (108A-C, 208A-C, 308A-C, 408A-C, 508A-D, 608A-D), wherein the path reroute marking indicator indicates that the at least one data packet has been rerouted onto a second path through the RDMA network that is different from a first path through the RDMA network over which others of the received plurality of data packets (108A-C, 208A-C, 308A-C, 408A-C, 508A-D, 608A-D) have traversed; and upon determining that at least one of the received plurality of data packets (108A-C, 208A-C, 308A-C, 408A-C, 508A-D, 608A-D) contains the path reroute marking indicator in the header, sending an adaptive routing notification indicator in the header of a data packet back to the requesting node (104, 204, 304, 404, 504, 604), wherein the adaptive routing notification indicator notifies the requesting node (104, 204, 304, 404, 504, 604), upon receipt, that the at least one of the received plurality of data packets (108A-C, 208A-C, 308A-C, 408A-C, 508A-D, 608A-D) has been received out of order, OOO, due to the rerouting.

8. The responding node (106, 206, 306, 406, 506, 606) of claim 7 is configured to change the first path to the second path by adaptive routing.

9. The responding node (106, 206, 306, 406, 506, 606) of claim 7 wherein when after determining that none of the received plurality of data packets (108A-C, 208A-C, 308A-C, 408A-C, 508A-D, 608A-D) contains the path reroute marking indicator, but detecting that a packet has been received OOO, the responding node (106, 206, 306, 406, 506, 606) is configured to send a negative acknowledgment, NACK, packet to the requesting node (104, 204, 304, 404, 504, 604), requesting that a missing packet be resent.

10. The responding node (106, 206, 306, 406, 506, 606) of claim 7, wherein the received plurality of data packets (108A- C, 208A-C, 308A-C, 408A-C, 508A-D, 608A-D) is part of a write request.

11. The responding node (106, 206, 306, 406, 506, 606) of claim 7, wherein the path reroute marking indicator is represented by at least one bit in a Base Transport Header, BTH.

12. The responding node (106, 206, 306, 406, 506, 606) of claim 7, wherein the adaptive routing notification indicator is represented by at least one bit in an acknowledgment Extended Transport Header, AETH, syndrome field.

13. A system (100, 200, 300, 400, 500, 600) comprising means adapted for carrying out all the steps of the method according to any preceding method claim.

14. A computer program comprising instructions for carrying out all the steps of the method according to any preceding method claim, when said computer program is executed on a computer system.

Images