Apparatus and method for remote direct memory access

Abstract

A sending device and a receiving device are proposed. The sending device is configured to: maintain a sending queue, wherein the sending queue comprises one or more WQEs, each WQE comprising destination information of the WQE; assign an XID to each of the one or more WQEs, and convert each WQE into a corresponding data packet, wherein each corresponding data packet comprises the XID of the corresponding WQE, and is associated with a PSN; send each data packet to the receiving device of the destination information of the WQE; receive, from the receiving device, a notification message indicating, for each sent data packet, whether the sent data packet is received; and determine, based on the notification message, information carried in the WQEs, and information held by the sending device, whether to generate a completion for each WQE.

Description

[0001] This application is a divisional application, the original application number is 202180101453.1, the original application date is August 12, 2021, and the entire contents of the original application are incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to high-performance computing technology, in particular to remote direct memory access (RDMA) technology. The present application relates to the transmission of RDMA transactions over a packet-based network. To this end, the present application provides a device, a method and a packet format for RDMA. BACKGROUND

[0003] Applications often require a computer to be able to access memory residing on a remote computer. RDMA enables a computer to perform such memory access operations without involving the operating system running on the computer. RDMA is widely used in modern data centers and computer clusters because it provides low-latency remote operations and high network bandwidth. RDMA offloads memory operations from the central processing unit (CPU) to the RDMA network interface card (NIC) (RDMA network interface card, RNIC), which can directly access the memory.

[0004] There are two common RDMA technologies: one is defined in the InfiniBand specification, and the other is defined in the Internet Engineering Task Force (IETF). The IETF protocol is also known as the Internet wide area RDMA protocol (iWARP). In particular, there are two variants of InfiniBand RDMA, which enable it to run on IP / Ethernet networks, such as RoCE and RoCEv2.

[0005] Both of these RDMA technologies define operations as software layer "verbs" that are then translated into work requests posted to the RNIC's queues. This work request is called a work queue element (WQE). RDMA also defines various transports on which the RDMA verbs execute. Transports are either reliable or unreliable, where the software is responsible for retransmission of lost packets, and are either connection-oriented or datagram-oriented. The various transports are instructed when a pair of network nodes establishes an RDMA communication channel specified by a queue pair (QP). All current RDMA transports provide a software programming model in which operations send a done signal in the order (in time) in which the operations were posted by the software layer.

[0006] Today, the RDMA technologies do not support a reliable transport service that can manage short messages multiplexed from multiple sources or to multiple destinations. In the context of the present application, "reliable transport" means that a message is guaranteed to be exchanged only once, and "short message" indicates that the maximum data length of the operation is the maximum transfer unit (MTU) of the network. Therefore, there is a need for a new reliable RDMA transport scheme for handling messages of a single packet length. SUMMARY

[0007] In view of the above, the device and method of the present application aim at providing a device and method for RDMA. The goal is to propose a new reliable RDMA transport for short messages, and in particular, to enable out-of-order processing of short messages. One object is to support the execution of each message without dependency on previous or subsequent operations. Another object is to provide a scheme for multiplexing RDMA operations from different upper-layer processes / protocols (ULPs) to multiple destinations.

[0008] The above and other objects are achieved by the subject matter claimed in the independent claims. Other implementations are evident from the dependent claims, the description, and the attached drawings.

[0009] A first aspect of the present application provides a sending device for RDMA. The sending device is configured to: maintain a sending queue, wherein the sending queue comprises one or more WQEs, wherein each WQE comprises destination information of the WQE, wherein the destination information comprises at least one of: an address of a receiving device, an identification of a receiving device, and an identification of a target application of a receiving device; assign a transaction identifier (XID) to each of the one or more WQEs, and convert each WQE into a corresponding data packet, wherein each corresponding data packet comprises the XID of the corresponding WQE, and is associated with a packet sequence number (PSN); send each data packet to the receiving device of the destination information of the WQE; receive a notification message for each sent data packet from the receiving device, wherein the notification message indicates whether the sent data packet is received at the receiving device; determine whether to generate a completion for each WQE based on the notification message, information carried in the WQE, and information held by the sending device.

[0010] For example, a WQE is an RDMA operation or transaction issued or posted by a source ULP (e.g. an application or software) into a QP. A WQE is sent to or received from a peer. A QP represents a communication endpoint, consisting of a send queue (SQ) and a receive queue (RQ) or shared receive queue (SRQ). According to the present application, a scheme is provided to support independent processing of each WQE or each message, i.e. each WQE or each message is executed independently of previous or subsequent operations. Each WQE comprises its own destination information (e.g. target ID, destination IP address, etc.), thereby enabling the sending device of the first aspect to process multiple consecutive WQEs, each of which can be destined to a different target. Each WQE is associated with a unique XID, which can be used by the receiving end to detect duplicate operations and to guarantee that a message is exchanged only once.

[0011] The sending device can decide whether to send a signal of WQE completion based on certain conditions. Traditionally, a RDMA QP always generates a completion on certain operation codes (RQ processing, etc.). There is no control to operate on conditional completion. According to the present application, the completion can be configurable.

[0012] In an implementation form of the first aspect, each WQE carries a message that fits into a single network maximum transmission unit only.

[0013] In particular, the present application proposes a new reliable RDMA transport scheme for handling messages up to the length of a single data packet, i.e. short messages.

[0014] In an implementation form of the first aspect, when the WQE corresponding to the respective data packet comprises a write operation request or a send operation request, and the notification message indicates that the respective data packet is received at the receiving device, the sending device is further configured to determine that the WQE is successfully completed.

[0015] RDMA operations include read, write, send, receive, atomic operation, etc. For a write operation or a send operation, for example, the sending device considers the write operation or the send operation successfully completed once the receiving device acknowledges the reception of the corresponding data packet.

[0016] In an implementation form of the first aspect, when the WQE corresponding to the respective data packet comprises a read operation request or an atomic operation request, the sending device is further configured to: receive, as an atomic operation response or a read operation response to the atomic operation request or the read operation request, respectively, a response data packet indicating that the respective data packet has been processed by the receiving device, wherein the response data packet comprises an XID of a response WQE and an XID of the WQE comprising the read operation request or the atomic operation request; determine that the WQE is successfully completed.

[0017] For a read operation or an atomic operation, for example, the sending device further expects a read operation response or an atomic operation response from the receiving device. That is, when the receiving device acknowledges the reception of the corresponding data packet, the read / atomic WQE is not completed, but only when the sending device receives the appropriate response data packet. In this case, the receiving device operates accordingly to provide the response data packet carrying the required data. For example, the read operation response comprises the data that the sending device intends to get from the receiving device. According to the present application, the response data packet carries the XID of the response WQE converted to the response data packet, and the XID of the WQE corresponding to the operation request. It is to be noted that in this case, the receiving device can also be considered as a "sending device" sending the data packet carrying the WQE.

[0018] In an implementation form of the first aspect, a maximum acceptable number of reception timeouts associated with the receiving device is obtained; when the number of reception timeouts exceeds the maximum acceptable number of reception timeouts, it is determined that the WQE corresponding to the respective data packet is completed but with an error.

[0019] Optionally, the service transmission according to the application comprises a threshold for signaling network related error conditions. For example, a receive timeout can also be named a retransmission timeout. For example, when a data packet (carrying the same WQE) is retransmitted too many times, e.g. exceeding a maximum acceptable retransmission number, the sending device stops retransmitting and determines the WQE completion with error.

[0020] In an implementation form of the first aspect, the notification message of each transmitted data packet comprises a receiver not ready (RNR) indication, wherein the RNR indication indicates that the receiving device is not able to process the transmitted data packet, wherein the sending device is further configured to determine an XID comprised in the transmitted data packet, and postpone transmitting another data packet comprising the same XID until expiry of a timeout value.

[0021] The application proposes an RNR signal for delaying the transmission of other messages. In particular, the sending device receives an RNR indication when a received data packet carries a message that the receiving device is temporarily not able to process. The RNR signal only affects the specific message on the sending device, avoiding any Head-of-Line (HoL) blocking.

[0022] In an implementation form of the first aspect, the WQE further comprises a parameter indicating a maximum acceptable number of RNR indications, wherein the sending device is further configured to receive another notification message comprising an RNR indication for a retransmitted data packet, and determine the WQE corresponding to the retransmitted data packet as completed with error when the number of received RNR indications exceeds the maximum acceptable number of RNR indications.

[0023] Thus, the sending device is implicitly informed about the WQE completion error.

[0024] In an implementation form of the first aspect, the notification message comprises a plurality of RNR indications, each RNR indication indicating that the receiving device is not able to process a specific transmitted data packet.

[0025] It is possible that there can be more than one message that the receiving device is temporarily not able to process, e.g. if the receiving device has no resources available.

[0026] In an implementation form of the first aspect, the RNR indication further indicates the timeout value.

[0027] It is possible that only a limited number of RNR timeout values are defined, to keep the overall completion latency low and to reduce the number of timers required.

[0028] In an implementation form of the first aspect, each RNR indication further indicates that the receiving device is not able to process a specific transmitted data packet within a specific timeout value.

[0029] It is possible that different timeout values can be set for different messages.

[0030] In an implementation form of the first aspect, the sending device is further configured to manage WQEs corresponding to sent data packets for which a RNR indication is received based on the indicated timeout value indicated by the RNR indication.

[0031] Optionally, the sending device can group delayed WQEs in RNR "state" according to the received RNR timeout value. According to the application, when a WQE is in RNR state, it indicates that a RNR indication for this WQE is received. WQEs in RNR "state" can be managed together by the sending device.

[0032] In an implementation form of the first aspect, the number of supported timeout values is configurable and agreed on all communicating devices.

[0033] In an implementation form of the first aspect, the one or more WQEs of the sending queue comprise a first WQE and a second WQE, wherein the first WQE and the second WQE comprise different destination information.

[0034] According to the application, a solution is proposed that WQEs towards different destination nodes (e.g. different receiving nodes) can be managed.

[0035] In an implementation form of the first aspect, the one or more WQEs of the sending queue comprise a set of WQEs comprising the same destination information, wherein the sending device is further configured to group the data packets corresponding to the set of WQEs into a data packet stream and to send the data packet stream to the receiving device of the set of WQEs.

[0036] For example, each WQE comprises its own destination information. For WQEs sharing the same destination node, data packets corresponding to such WQEs can be grouped into a data packet stream and managed together by the sending device.

[0037] In an implementation form of the first aspect, each WQE is further associated with a priority sent from the sending device.

[0038] The sending device can configure a priority for a WQE to be sent. WQEs with high priority can be processed before WQEs with low priority.

[0039] In an implementation form of the first aspect, the sending device is further configured to obtain the one or more WQEs from at least one ULP.

[0040] It is possible that the messages or WQEs processed by the sending device can be posted or issued by different source ULPs.

[0041] In one implementation form of the first aspect, the set of WQEs comprising the same destination information are obtained from different ULPs.

[0042] For example, when WQEs obtained from different source ULPs share the same destination, the data packets corresponding to such WQEs can be grouped into a data packet flow and sent together. In this way, WQEs from different sources are multiplexed.

[0043] In one implementation form of the first aspect, the XIDs of the WQEs obtained from different ULPs are allocated from the same pool of XIDs, and the PSNs of the data packets associated with the WQEs obtained from different ULPs are allocated from the same pool of PSNs.

[0044] Optionally, the sending device can maintain a contiguous number space for XIDs and PSNs respectively.

[0045] In one implementation form of the first aspect, the priority of each WQE is determined based on the ULP from which the WQE is obtained.

[0046] For example, the priority of WQEs or tasks posted from certain ULPs can be higher than the priority of other WQEs.

[0047] In one implementation form of the first aspect, each data packet is further associated with a flow block ID identifying a flow, wherein the flow comprises a plurality of data packets, and the data packets corresponding to the WQEs further comprise the flow block ID.

[0048] According to the present application, equal cost multiple path (ECMP) flow blocks can also be applied. ECMP flow blocks support (relatively easily) by dividing a data packet flow between an initiator node and a target node into sub-flows. It is assumed that data packets of different sub-flows use different routes to traverse the network.

[0049] In one implementation form of the first aspect, the sending device is further configured to send each data packet to the receiving device through a flow block associated with the receiving device.

[0050] For each sent data packet associated with a particular flow block ID, it is routed through a flow identifiable by the flow block ID.

[0051] The second aspect of the present application provides a receiving device for RDMA, wherein the receiving device is configured to: receive one or more data packets from a sending device for RDMA, wherein each data packet of the one or more data packets is associated with a PSN and comprises an XID of a corresponding WQE, wherein each WQE comprises destination information of the WQE, wherein the destination information comprises at least one of: an address of the receiving device, an identification of the receiving device, and an identification of a target application of the receiving device; and send a notification message to the sending device, wherein the notification message indicates that the one or more data packets are received at the receiving device.

[0052] According to the present application, a receiving device is provided, which operates correspondingly to the sending device of the first aspect. The receiving device proposed in the present application processes each received data packet independently. That is, each received data packet can be executed safely upon reception, regardless of the status of any previous data packet.

[0053] In an implementation form of the second aspect, when the WQE corresponding to a respective data packet comprises a write operation request or a send operation request, the receiving device is further configured to: put data carried on the write operation request into a destination memory location in the receiving device; or put data carried on the send operation request from a receive queue of the receiving device into a buffer.

[0054] For example, for a write operation or a send operation, the receiving device correspondingly places data carried in the data packet.

[0055] In an implementation form of the second aspect, when the WQE corresponding to a respective data packet comprises a read operation request or an atomic operation request, the receiving device is further configured to: send a response data packet to the sending device, which indicates that the respective data packet has been processed by the receiving device.

[0056] For example, for a read operation or an atomic operation, the receiving device correspondingly operates to provide a response data packet carrying the required data. For example, the read operation response comprises data that the sending device intends to obtain from the receiving device.

[0057] In an implementation form of the second aspect, the receiving device is further configured to: create a response WQE according to the read operation request or the atomic operation request; assign an XID to the response WQE, and convert the response WQE into the response data packet, wherein the response data packet comprises the XID of the response WQE and the XID of the WQE comprising the read operation request or the atomic operation request.

[0058] Accordingly, the receiving device generates a response WQE for the read operation or atomic operation request. It is to be noted that in this case, the receiving device can be regarded as the “sending device” of the first aspect, which allocates an XID for the response WQE, converts the WQE into a data packet, and further sends the data packet. The response data packet differs from a “normal” sent data packet in that the response data packet also carries the XID of the WQE corresponding to the operation request (e.g. the sent data packet). That is, two XIDs are carried in the response data packet.

[0059] In an implementation form of the second aspect, the receiving device is further configured to generate an RNR indication for the received data packet when the receiving device is unable to process the respective data packet, wherein the notification message for the respective data packet further comprises the RNR indication indicating to the sending device that the receiving device is unable to process the respective data packet.

[0060] As mentioned before, the present application proposes an RNR signal for delaying the transmission of other messages. In particular, the receiving device is temporarily unable to process a received data packet carrying a message, e.g. if the receiving device has no resources available, and sends an RNR indication. The RNR signal only affects the specific message at the sending device, avoiding any HoL blocking.

[0061] In an implementation form of the second aspect, the RNR indication further indicates a preset timeout value, wherein the RNR indication indicates that the receiving device is unable to process the respective data packet within the preset timeout value.

[0062] In an implementation form of the second aspect, the notification message comprises a plurality of RNR indications, each RNR indication indicating that the receiving device is unable to process a specific sent data packet.

[0063] In an implementation form of the second aspect, the one or more data packets comprise a first data packet and a second data packet, wherein the PSN of the first data packet is smaller than the PSN of the second data packet, wherein the receiving device is further configured to send the notification message for the first data packet and the notification message for the second data packet out of order; and / or to process the first data packet and the second data packet out of order; to send the response message for the first data packet and the response message for the second data packet out of order to the sending device.

[0064] Embodiments of the present application propose a new scheme which enables the receiving device to perform the received data packets out of order. Thus, also the acknowledgments (ACKs) of the received data packets, e.g. the notification messages, can be sent out of order. Likewise, also the responses can be sent out of order.

[0065] In an implementation form of the second aspect, the receiving device is further configured to maintain a database of XIDs of WQEs received at the receiving device.

[0066] The receiving device can record the XID carried in the received data packet.

[0067] In an implementation form of the second aspect, the receiving device is further configured to determine whether each received data packet is a duplicate data packet by checking whether the XID of the WQE included in the received data packet is already stored in the database, and to ignore any duplicate data packet.

[0068] To avoid executing the same WQE more than once, the receiving device checks the XIDs stored in the database. When the XID carried in a newly received data packet is already stored in the database, the receiving device ignores the data packet instead of executing it. In this way, it is guaranteed that a message will be executed only once.

[0069] In an implementation form of the second aspect, the database further comprises a status of each WQE indicating whether the WQE has been processed.

[0070] A third aspect of the application provides a method for RDMA. The method comprises: maintaining a sending queue, wherein the sending queue comprises one or more WQEs, wherein each WQE comprises destination information of the WQE, wherein the destination information comprises at least one of: an address of a receiving device, an identification of a receiving device, and an identification of a target application of the receiving device; assigning an XID to each of the one or more WQEs, and converting each WQE into a data packet, wherein each data packet comprises the XID of the corresponding WQE, and is associated with a PSN; sending each data packet to the receiving device of the destination information of the WQE; receiving a notification message of each sent data packet from the receiving device, wherein the notification message indicates whether the sent data packet is received at the receiving device; determining whether to generate a completion for each WQE based on the notification message, information carried in the WQE, and information held by the sending device.

[0071] The method of the third aspect and its implementation forms provide the same advantages and effects as described above for the sending device of the first aspect and its respective implementation forms.

[0072] A fourth aspect of the present application provides a method for RDMA. The method comprises: receiving one or more data packets from a sending device for RDMA, wherein each data packet of the one or more data packets is associated with a PSN and comprises an XID of a corresponding WQE, wherein each WQE comprises destination information of the WQE, wherein the destination information comprises at least one of: an address of the receiving device, an identification of the receiving device, and an identification of a target application of the receiving device; sending a notification message to the sending device, wherein the notification message indicates that the one or more data packets are received and / or processed at the receiving device.

[0073] The method of the fourth aspect and implementation forms thereof provide the same advantages and effects as described above for the receiving device of the second aspect and respective implementation forms thereof.

[0074] A fifth aspect of the present application provides a computer program comprising program code for implementing the method of any one of the third aspect and implementation forms thereof or any one of the fourth aspect and implementation forms thereof when implemented on a processor.

[0075] It has to be noted that all devices, elements, units and means described in the present application can be implemented in software or hardware elements or any kind of combination thereof. All steps, functions, procedures and / or actions described in the present application can be performed in software or hardware elements or any kind of combination thereof. All entities described in the present application performing the steps, functions, procedures and / or actions described in the present application are intended to be implemented in software or hardware elements or any kind of combination thereof. BRIEF DESCRIPTION OF DRAWINGS

[0076] The above described aspects and implementation forms thereof will be explained in the following description of specific embodiments in relation to the enclosed drawings in which:

[0077] Figure 1 A sending device is shown;

[0078] Figure 2 A schematic diagram is shown;

[0079] Figure 3 A signaling flow diagram is shown;

[0080] Figure 4 A signaling flow diagram is shown;

[0081] Figure 5 A flow diagram of a WQE state machine is shown;

[0082] Figure 6A flow diagram illustrating a WQE state machine is shown;

[0083] Figure 7 A schematic diagram is shown;

[0084] Figure 8 A receiving device is shown;

[0085] Figure 9 A method is shown.

[0086] Figure 10 Another method is shown. DETAILED DESCRIPTION

[0087] Exemplary embodiments of methods, devices, and program products for short message processing in RDMA systems are described in connection with the appended drawings. While the description provides detailed examples of possible implementations, it is noted that the details are intended to be illustrative and not limiting of the scope of the present application.

[0088] Furthermore, one embodiment / example can refer to other multiple embodiments / examples. For example, any description given in one embodiment / example, including but not limited to terminology, elements, procedures, explanations, and / or technical advantages, apply to other multiple embodiments / examples.

[0089] Figure 1A transmitting device 100 suitable for RDMA is shown. The transmitting device 100 can include processing circuitry (not shown) to perform, implement, or initiate various operations of the transmitting device 100 described herein. The processing circuitry can include hardware and software. The hardware can include analog circuitry or digital circuitry, or both analog and digital circuitry. The digital circuitry can include components such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), or multi-purpose processors, etc. The transmitting device 100 can also include memory circuitry storing one or more instructions executable by the processor or processing circuitry, particularly under control of software. For example, the memory circuitry can include a non-transitory storage medium storing executable software code that, when executed by the processor or processing circuitry, causes the transmitting device 100 to perform various operations. In one embodiment, the processing circuitry includes one or more processors and a non-transitory memory coupled with the one or more processors. The non-transitory memory can carry executable program code that, when executed by the one or more processors, causes the transmitting device 100 to perform, implement, or initiate the operations or methods described herein.

[0090] In particular, the sending device 100 is configured to maintain a sending queue 101 comprising one or more WQEs 102, 102'. In particular, each WQE 102, 102' comprises destination information of the WQE 102, 102'. The destination information comprises at least one of: an address of the receiving device 200, an identification of the receiving device (200, 200') and an identification of a target application of the receiving device 200, 200'. The sending device 100 is further configured to assign an XID to each of the one or more WQEs 102, 102' and to convert each WQE 102, 102' into a respective data packet 103, 103'. Each respective data packet 103, 103' comprises the XID of the corresponding WQE 102, 102' and is associated with a PSN. The sending device 100 is then configured to send each data packet 103, 103' to the receiving device (200, 200') of the destination information of the WQE 102, 102'. The sending device 100 is further configured to receive a notification message 201 from the receiving device 200 for each sent data packet 103, wherein the notification message 201 indicates whether the sent data packet 103 was received at the receiving device 200. Furthermore, the sending device 100 is configured to determine whether to generate a completion for each WQE 102 based on the notification message 201, information carried in the WQE 102 and information held by the sending device 100.

[0091] Embodiments of the present application introduce a new approach, wherein each WQE (each RDMA operation) or each message can be processed independently, e.g. executed without relying on previous or subsequent operations. Each WQE comprises all information required to execute the operation, in particular an indication of the destination node (e.g. the receiving device 200) and the process (ID, address, etc.).

[0092] For example, an RDMA transaction involves an initiator node and a destination target node (or target node). The initiator node initiates or sends the RDMA operation request and the target node receives the RDMA operation request and responds accordingly. Figure 1 The sending device 100 shown can be considered as the initiator node, Figure 1 The receiving device 200 shown can be considered as the target node.

[0093] According to embodiments of the present application, each WQE is associated with a unique XID, which can be used at the receiving end to detect duplicate operations. Each sent data packet is associated with a unique PSN. It is noted that each data packet needs to be acknowledged as received. The initiator node uses these ACK signals to detect missing data packets and retransmits, e.g. using a selective repeat retransmission scheme.

[0094] Traditionally, RDMA QPs always generate completions on certain opcodes (RQ processing, etc.). There is no control to manipulate conditional completions. According to embodiments of the present application, completions can be configurable. That is, the sending device 100 can decide whether to send a signal of completion of a WQE based on certain conditions (e.g., information held by the sending device 100).

[0095] For example, the sending device 100 can obtain information from the ULP that sets the sending device 100 to obtain a completion signal after every X completed WQEs, such that the ULP checks the completion queue events (CQEs) of the completion queue (CQ) at a lower frequency than for every individual WQE. In another example, the sending device 100 can be instructed to generate a completion when there are no more WQEs to be processed by a certain node-to-node context (NNC). Optionally, the sending device 100 can also send a completion signal for every high-priority WQE and only send a completion signal for a group of low-priority WQEs. There are many implementations in this regard. These examples are not intended to limit the present application.

[0096] The new service method as described in the present application can be referred to as short-lived datagram (SLD). As mentioned previously, the transport type can be reliable or unreliable, and connection-oriented or datagram-oriented. The SLD as described in the present application is applicable to most transports, such as reliable connection (RC), reliable datagram (RD), and unreliable datagram (UD), as shown in Figure 2 That is, the present application discloses a new service between traditional RD, RC, and UD transports.

[0097] It is noted that the new RDMA transport scheme proposed by embodiments of the present application is for handling short messages, e.g., messages up to the length of a single data packet. According to embodiments of the present application, each WQE 102, 102' carries a message that fits in a single network maximum transmission unit.

[0098] Generally, RDMA operations include read, write, send, receive, atomic operations, etc.

[0099] Figure 3 A signaling flow diagram between an initiator node (e.g., the sending device 100) and a target node (e.g., the receiving device 200) is shown. The sending device 100 can be as described in Figure 1The sending device shown, the receiving device 200 can be as Figure 1 In particular, Figure 3 The exchange of data packets for a write operation and a send operation is shown.

[0100] In this embodiment, a transmit (Tx) WQE from a source send queue (SSQ) (e.g., transmit queue 102) is assigned an XID W. Figure 1 The WQE 102 shown is converted into a write request carrying XID W, for example, Figure 1 After sending device 100 sends data packet 103, an ACK is received at sending device 100. Accordingly, sending device 100 can determine that the WQE with XID W is successfully completed. After receiving the write request, receiving device 200 can place the data carried in the write request into the destination memory location in receiving device 200.

[0101] Hereinafter, another WQE with XID S is converted into a send request carrying XID S. Accordingly, receiving device 200 may place the data carried in the send request from the RQ of receiving device 200 into a buffer. After receiving an ACK for the send request from receiving device 100, sending device 100 may determine that the WQE with XID S has successfully completed. It should be noted that a WQE may send a completion signal immediately after receiving an ACK for the WQE, without waiting for the previous WQE to complete.

[0102] According to an embodiment of the present invention, when the WQE 102 corresponding to the corresponding data packet 103 includes a write operation request or a send operation request, and the notification message 201 indicates that the corresponding data packet 103 is received at the receiving device 200, the sending device 100 is further configured to determine that the WQE 102 is successfully completed.

[0103] Figure 4 FIG. 1 shows another signaling flow chart between an initiator node (eg, a sending device 100) and a target node (eg, a receiving device 200) provided by an embodiment of the present invention. The sending device 100 may be Figure 1 The sending device shown, the receiving device 200 can be as Figure 1 In particular, Figure 4 The exchange of data packets for a read operation or an atomic operation is shown.

[0104] In this embodiment, XID R is assigned to Tx WQE. WQE (e.g. Figure 1 The WQE 102 shown is converted into a read request carrying an XID R, for example, asFigure 1 The ACK is received at the sending device 100 after the sending device 100 sends the data packet 103.

[0105] For example, for a read operation, the sending device 100 also expects a read operation response from the receiving device 200. That is, the read WQE is not completed when the receiving device 200 acknowledges the reception of the corresponding data packet, but only when the sending device 100 receives the appropriate response data packet. In this case, the receiving device 200 internally creates a response WQE for the WQE 102.

[0106] For example, the receiving device 200 intends to send the response WQE back to the sending device. In this case, it can be understood that the roles of the “sending device” and the “receiving device” are switched. That is, the receiving device 200 here can also be considered as a “sending device” sending a data packet carrying a WQE, which is the response WQE. According to an embodiment of the present application, the receiving device 200 generates a response WQE for the read operation request, and allocates another XID, for example, as shown in the XID Q, for the response WQE. The receiving device 200 also converts the response WQE into a response data packet 202, and sends the response data packet 202 to the sending device 100. Figure 4

[0107] It is worth mentioning that the response data packet 202 is different from the “normally” sent data packet 103 from the sending device 100 in that the response data packet 202 also carries the original XID (for example, XID R) of the WQE corresponding to the operation request (for example, the sent data packet 103). That is, two XIDs (for example, XID R and XID Q) are carried in the response data packet 202.

[0108] In response to receiving the response data packet 202, the sending device 100 should send an ACK to inform the receiving device 200 that the response data packet 202 has been received. For example, in this case, the sending device 100 behaves here similar to a “receiving device”.

[0109] According to an embodiment of the present application, when the Tx WQE carries an atomic operation request, the data packet exchange between the sending device 100 and the receiving device 200 is similar to the case of the read operation. Here, it is not repeated.

[0110] ​According to embodiments of the present application, when the WQE 102 corresponding to the respective data packet 103 comprises a read operation request or an atomic operation request, the sending device 100 is further configured to receive, as an atomic operation response or a read operation response to the atomic operation request or the read operation request, respectively, a response data packet 202 indicating that the respective data packet 103 has been processed by the receiving device 200. In particular, the response data packet 202 comprises the XID of the response WQE 102 and the XID of the WQE 102 comprising the read operation request or the atomic operation request. Further, the sending device 100 is configured to determine that the WQE 102 is successfully completed.

[0111] Figure 5 An illustration of a state machine controlling the processing of each WQE at the initiator node (e.g. sending device 100) is shown. Each WQE is processed independently of all other WQEs. In particular, even for WQEs issued by the same source ULP, they are still processed independently.

[0112] As can be seen from the diagram, once a WQE (e.g. WQE Tx) is sent, an ACK is expected.

[0113] For the case where an ACK is received, a database (e.g. bitmap) storing the status of the sent data packets can be updated accordingly. If a data packet loss is detected, a fast retransmission mechanism can be applied to retransmit the lost data packet. For the retransmitted data packet, an ACK is again expected.

[0114] In case no ACK is received, the sending device can wait for a retransmission timeout to expire before retransmitting the data packet. For example, a maximum acceptable receive timeout number “RTOMAXN” can be configured for the sending device. If the value of “RTOMAXN” is greater than 0, the value of “RTOMAXN” is decremented (e.g. by 1) each time a retransmission timeout expires without receiving an ACK. Once the value reaches 0, the sending device 100 determines that the WQE is completed but with error.

[0115] According to embodiments of the present application, the sending device 100 can be configured to obtain a maximum acceptable receive timeout number associated with the receiving device 200. When the receive timeout number exceeds the maximum acceptable receive timeout number, the sending device 100 can be configured to determine that the WQE 102 corresponding to the respective data packet 103 is completed but with error.

[0116] The present application also proposes a RNR signal for delaying the transmission of other messages. In particular, the sending device receives a RNR indication when a received data packet carries a message that the receiving device cannot temporarily process (e.g. if the receiving device has no available resources). The performance limiting phenomenon that occurs when a line of data packets is blocked by the first data packet is called HoL blocking. The RNR signal only affects the specific message at the sending device, avoiding any HoL blocking.

[0117] It is possible that a RNR indication (or a notification message comprising a RNR indication) indicating that the receiving device 200 cannot temporarily process the sent data packet is received instead of an ACK. Then, a counter of the maximum number of acceptable RNR indications "RNRMAXN" is checked. If this counter reaches 0, the sending device 100 determines that the WQE is completed but with an error. If the counter has not reached 0, the counter is decreased (e.g. by 1) for each received RNR indication and, once the counter reaches 0, the sending device 100 determines that the WQE is completed but with an error.

[0118] According to an embodiment of the present application, Figure 1 The illustrated notification message 201 can comprise a RNR indication, wherein the RNR indication indicates that the receiving device 200 cannot process the sent data packet. The sending device 100 is configured to determine the XID comprised in the sent data packet 103. The sending device 100 is further configured to postpone the sending of another data packet containing the same XID until the expiry of a timeout value.

[0119] After the expiry of the timeout, the sending device 100 retransmits the data packet comprising the same XID. The receiving device 200 can still be unable to process the sent data packet. In this case, another RNR indication for retransmitting the data packet can be sent from the receiving device 200. According to an embodiment of the present application, the WQE 102 can further comprise a parameter indicating the maximum number of acceptable RNR indications, i.e. as Figure 5 illustrated "RNRMAXN". It is possible that the sending device 100 can receive another notification message comprising a RNR indication for retransmitting the data packet. The sending device 100 can be further configured to determine that the WQE 102 corresponding to the retransmitted data packet is completed but with an error when the number of received RNR indications exceeds "RNRMAXN".

[0120] According to an embodiment of the present application, the notification message 201 can comprise a plurality of RNR indications, each RNR indication indicating that the receiving device 200 cannot process a specific sent data packet. For example, the timeout value can be indicated by the RNR indication.

[0121] Optionally, each RNR indication further indicates that the receiving device 200 cannot process the specific sent data packet within a specific timeout value.

[0122] According to embodiments of the application, the sending device 100 can further be configured to manage the WQE 102, 102' corresponding to the sent data packet 103, 103' for which an RNR indication is received, based on the indicated timeout value of the RNR indication.

[0123] Optionally, the sending device 100 can group the delayed WQE in RNR "state" according to the received RNR timeout value. According to the application, when a WQE is in RNR state, this indicates that an RNR indication for this WQE is received. The WQE in RNR "state" can be managed together by the sending device 100 (e.g. the initiator node) instead of each NNC. Typically, only a limited number of RNR timeout values are defined to keep the overall completion latency low and to reduce the number of timers needed. The initiator node groups the delayed WQE in RNR "state" according to the received RNR timeout value, which can be set by the target node (e.g. the receiving device 200).

[0124] According to embodiments of the application, the number of supported timeout values is configurable and agreed on all communicating devices.

[0125] Optionally, the receiving device 200 can be configured to set the timeout value based on the current conditions of the receiving device 200. For example, the receiving device 200 can adjust the timeout value based on its current load, the number of connected NNCs, the number of remaining RQ elements (RQ internal).

[0126] Figure 6 A diagram of the read / atomic WQE state machine provided by embodiments of the application is shown. Figure 6 is based on Figure 5 and Figure 5 A more general case of all RDMA operations is shown.

[0127] Similarly, as in Figure 5 , once a WQE (e.g. WQE Tx) is sent, an ACK is expected. In case no ACK is received, or in case the sending device 100 receives an RNR indication instead of an ACK indicating that the receiving device 200 is temporarily unable to process the sent data packet, the operation of the sending device 100 is similar to the one described in Figure 5 . This is not repeated here.

[0128] For the case where an ACK is received, the database (e.g. bitmap) storing the status of the sent data packet can be updated accordingly. In case a data packet loss is detected, a fast retransmission mechanism can be applied to retransmit the lost data packet. For the retransmitted data packet, again an ACK is expected. This part is also similar to the one described in Figure 5 .

[0129] As previously mentioned, the read and / or atomic "verb" operation requires the target node to send data back to the initiator node. That is, the read / atomic WQE is not completed when the ACK of the request packet is received by the target node, but only when the initiator node receives the appropriate response packet.

[0130] That is, upon reception of a read and / or atomic request packet, the target node receiving device 200 stores an internal response WQE, for managing the transmission of the response packet, and possible retransmissions. This internal attribute of the WQE means that it is "hidden" from the upper layer software, the ULP does not publish the WQE, and the completion event / signal is not reported to any upper layer software at the target node.

[0131] In case the receiving device 200 does not have resources available to store this internal response WQE, a RNR signal packet is generated and sent back to the sending device 100.

[0132] According to embodiments of the application, the target node (e.g. the receiving device 200) handles the internal response WQE as if it was published by the ULP. The same state machine rules and conditions apply to this case as well, e.g. retransmit the response packet until an ACK is received or until an error condition is identified (but this error is not reported to any ULP).

[0133] According to embodiments of the application, one or more WQEs 102, 102' of the sending queue 101 comprise a first WQE and a second WQE, wherein the first WQE and the second WQE comprise different destination information.

[0134] As previously mentioned, each WQE comprises its own destination information (e.g. target ID, destination IP address, etc.) to enable the source ULP to publish multiple consecutive WQEs, each WQE can be directed to a different target ULP and network node using the same QP class data structure.

[0135] According to embodiments of the application, one or more WQEs 102, 102' of the sending queue 101 comprise a WQE set comprising the same destination information, wherein the sending device 100 is further configured to: group the packets corresponding to the WQE set into a packet stream; and send the packet stream to a receiving device 200 of the WQE set.

[0136] Various WQEs can be grouped into a packet stream directed to their destination target node by their destination. The management of this packet stream is done by using a NNC data structure. A single NNC manages the WQEs between an initiator-target node pair.

[0137] According to embodiments of the present application, each WQE 102, 102' is also associated with a priority sent from the sending device 100. Optionally, the priority of each WQE 102, 102' can be determined based on the ULP from which the WQE (102, 102') is fetched. For example, if the SLD QP created by the ULP is associated with a higher priority, then the WQE fetched from the SQ will be associated with a higher priority. Thus, such WQE will be processed before WQE from other ULP SQs, which can be created with lower priority.

[0138] According to embodiments of the present application, the sending device 100 can also be configured to fetch one or more WQE 102, 102' from at least one ULP.

[0139] According to embodiments of the present application, the set of WQE comprising the same destination information are fetched from different ULPs. That is, various WQE from different source ULPs can be grouped into a packet stream by their destination.

[0140] According to embodiments of the present application, the XIDs of the WQE fetched from different ULPs are allocated from the same pool of XIDs, and the PSNs of the packets associated with the WQE fetched from different ULPs are allocated from the same pool of PSNs.

[0141] A unique XID is allocated to each WQE from a continuous numerical space for all WQEs sharing the same destination target node. A unique PSN is allocated to each Tx packet from a continuous numerical space for all packets sent to the same destination target node.

[0142] According to embodiments of the present application, the node-to-node context (NNC) can be divided into two operational parts: NNC-Tx used by the initiator function and NNC-Rx used by the target function. The NNC-Tx maintains different continuous numerical spaces for XIDs and PSNs. The NNC-Rx processes the received ACK and RNR indications to detect missing packets and triggers the NNC-Tx to retransmit these packets. It is also noted that the expiration of timers also results in the detection of missing packets, which triggers retransmission.

[0143] Figure 7An exemplary schematic diagram of the SLD WQE distributor provided by embodiments of the application is shown. The NNC-Rx can store a "received XID" state and identify duplicate WQEs, e.g., using a PSN database and an XID database. These duplicates are only acknowledged but not executed again. This is to guarantee that a WQE is executed only once. The NNC-Rx builds ACKs based on PSNs to indicate received and missing data packets. The NNC-Rx builds RNR indications to indicate multiple RNR signals for multiple data packets.

[0144] Furthermore, the NNC can include thresholds for signaling network related error conditions, e.g., period of receive / retransmit timers, maximum number of timer expirations (no reception of any ACK / RNR data packet).

[0145] The NNC includes congestion control (CC) related information and parameters that control the transmission of data packets managed by the NNC.

[0146] It is worth mentioning that the "Rx XID" state database of the NNC-Rx can be a shared resource pool from which any NNC-Rx can allocate the required resources as needed. Alternatively, a single RTO timer can be used for all NNC-Tx instances. The memory used to store the WQEs can be part of the host's memory, a dedicated memory connected to the RNIC, an internal memory, or any combination thereof. To avoid that a single ULP consumes too many RNIC resources (for WQEs) and thus starve other ULPs from their valid SLD services, a fairness mechanism can be added to enforce a maximum number of outstanding WQEs per ULP.

[0147] According to embodiments of the application, each data packet 103, 103' can further be associated with a stream block ID identifying a stream block. In particular, a stream block comprises a plurality of data packets, and the data packet 103, 103' corresponding to the WQE 102, 102' further comprises the stream block ID.

[0148] According to embodiments of the application, the sending device 100 can be further configured to send each data packet 103 by a stream to the receiving device 200, wherein the stream block is associated with the receiving device 200.

[0149] It is worth mentioning that ECMP flowlets are supported by the new approach presented in this invention and the transmission is designed for flowlets. ECMP flowlets are supported (relatively easily) by splitting the packet flow between the initiator node and the target node into sub-flows. It is assumed that packets of different sub-flows use different routes to traverse the network. To support ECMP flowlets, NNC-Tx maintains a separate (continuous) PSN space for each sub-flow (e.g. flowlet). Each WQE is assigned a flowlet ID which is also included in the Tx packet. Each ACK report also includes the flowlet ID and when an ACK is received, the initiator node only triggers retransmission of packets of that flowlet. There is decision flexibility which enables the initiator node to dynamically select different flowlets for retransmission. The decision can be based on any combination of criteria such as "better" flowlets with less load, faster speed, less congestion, etc. Embodiments of the invention also support each NNC to have different number of flowlets, enabling more efficient use of the initiator node, e.g. resources at the sending device 100.

[0150] Figure 8 An embodiment of the invention provides a receiving device 200 adapted for RDMA is shown. The receiving device 200 can comprise processing circuitry (not shown) for performing, implementing or initiating various operations of the receiving device 200 described herein. The processing circuitry can comprise hardware and software. The hardware can comprise analog circuitry or digital circuitry, or both analog circuitry and digital circuitry. The digital circuitry can comprise components such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), or multi-purpose processors, etc. The receiving device 200 can also comprise memory circuitry storing one or more instructions executable by the processor or processing circuitry, in particular under control of software. For example, the memory circuitry can comprise a non-transitory storage medium storing executable software code which, when executed by the processor or processing circuitry, causes the receiving device 200 to perform various operations. In one embodiment, the processing circuitry comprises one or more processors and a non-transitory memory connected to the one or more processors. The non-transitory memory can carry executable program code which, when executed by the one or more processors, causes the receiving device 200 to perform, implement or initiate the operations or methods described herein.

[0151] In particular, the receiving device 200 is configured to receive one or more data packets 103 from a sending device 100 for RDMA. It is possible that the sending device 100 here can be the sending device 100 described above with respect to Fig. 1. Figure 1The sending device 100 is shown. In particular, each of the one or more data packets 103 is associated with a PSN and comprises an XID of the corresponding WQE 102. Each WQE 102 comprises destination information of the WQE 102, wherein the destination information comprises at least one of: an address of the receiving device 200, an identification of the receiving device 200 and an identification of a target application of the receiving device 200. Further, the receiving device 200 is configured to send a notification message 201 to the sending device 100, wherein the notification message 201 indicates that the one or more data packets have been received at the receiving device 200.

[0152] Embodiments of the present application further provide a receiving device 200, which operates correspondingly with respect to the sending device 100 previously described in the present application. The receiving device 200 presented in the present application processes each received data packet 103 independently. That is, each received data packet can be executed safely upon reception, regardless of the status of any previous data packet. According to the present application, WQEs can be executed out-of-order and a signal of out-of-order completion is sent immediately after the ACK of the WQE is received. The order refers to the chronological order in which the WQEs are issued to the RNIC, e.g. by the ULP.

[0153] It is worth mentioning that the ACK supports the notification of the aggregation of multiple data packets and supports "Selective ACK" (SACK) to report received and missing data packets. The number of such reported data packets, e.g. the size of the ACK data structure, is an implementation parameter.

[0154] According to embodiments of the present application, when the WQE 102 corresponding to the respective data packet 103 comprises a write operation request, the receiving device 200 is further configured to put the data carried on the write operation request into a destination memory location in the receiving device 200. Similarly, when the WQE 102 corresponding to the respective data packet 103 comprises a send operation request, the receiving device 200 can be further configured to put the data carried on the send operation request from a receive queue of the receiving device 200 into a buffer.

[0155] According to embodiments of the present application, when the WQE 102 corresponding to the respective data packet 103 comprises a read operation request or an atomic operation request, the receiving device 200 is further configured to send a response data packet 202 to the sending device 100 indicating that the respective data packet 103 has been processed by the receiving device 200.

[0156] For example, for a read operation or an atomic operation, the receiving device 200 operates correspondingly to provide a response data packet carrying the required data. For example, the read operation response comprises the data that the sending device 100 intended to get from the receiving device 200.

[0157] According to embodiments of the application, the receiving device 200 can further be configured to create a response WQE in response to the read operation request or the atomic operation request. The receiving device 200 is further configured to assign an XID to the response WQE and to convert the response WQE into a response data packet 202. In particular, the response data packet 202 comprises the XID of the response WQE and the XID of the WQE 102 comprising the read operation request or the atomic operation request.

[0158] It is to be understood that in this case the receiving device 200 can be regarded as a “sending device” which assigns an XID to a (response) WQE, converts the WQE into a (response) data packet and further sends the data packet. The response data packet differs from a “normal” sent data packet in that the response data packet further carries the XID of the WQE corresponding to the operation request (e.g. the sent data packet). I.e. two XIDs are carried in the response data packet.

[0159] According to embodiments of the application, the receiving device 200 can further be configured to generate an RNR indication for a received data packet 103 when the receiving device 200 is unable to process the respective data packet 103. In particular, the notification message 201 of the respective data packet 103 further comprises the RNR indication indicating to the sending device 100 that the receiving device 200 is unable to process the respective data packet 103.

[0160] Received messages that the receiving device 200 as the target node is temporarily unable to process are delayed by the RNR signal. For example, the RNR signal only affects specific messages at the initiator, thus avoiding any HoL blocking. WQEs in the RNR “state” are managed by the initiator node together, not by each NNC. Only a small, limited number of RNR TO values are defined to keep the overall completion latency low and to reduce the number of timers needed. The initiator node groups the delayed WQEs in the RNR “state” according to the received RNR TO values set by the target node.

[0161] According to embodiments of the application, the RNR indication further indicates a preset timeout value, wherein the RNR indication indicates that the receiving device 200 is unable to process the respective data packet 103 within the preset timeout value.

[0162] According to embodiments of the application, the notification message 201 comprises a plurality of RNR indications, each RNR indication indicating that the receiving device 200 is unable to process a specific sent data packet.

[0163] It is possible that one or more of the data packets 103 comprises a first data packet and a second data packet, wherein the PSN of the first data packet is smaller than the PSN of the second data packet. According to embodiments of the present application, the receiving device 200 is further configured to send a notification message for the first data packet and a notification message for the second data packet out of order. The receiving device 200 is further configured to process the first data packet and the second data packet out of order; send a response message for the first data packet and a response message for the second data packet to the sending device 100 out of order.

[0164] According to embodiments of the present application, the receiving device 200 is further configured to maintain a database of XIDs of WQEs 102 received at the receiving device 200.

[0165] According to embodiments of the present application, the receiving device 200 is further configured to determine whether each received data packet 103 is a duplicate data packet by checking whether the XID of the WQE 102 comprised in the received data packet 103 is already stored in the database; ignore any duplicate data packet. These duplicates are only acknowledged, but not executed again. This is to guarantee that a WQE is executed only once.

[0166] According to embodiments of the present application, the database further comprises a status of each WQE 102 indicating whether the WQE 102 has already been processed.

[0167] In the present application, SLD is designed for lossy networks, not the traditional IB's RDMA. This network property means that data packet loss events are more likely to happen. SLD adopts a retransmission scheme, which can have two complementary mechanisms: PSN-based and timer-based. PSN-based retransmission can be based on the unique strictly monotonically increasing PSN assigned to each sent data packet and quickly identify the lost data packet when a "gap" is detected in the PSN reported back by ACK. Timer-based retransmission is similar to the traditional way of triggering retransmission events: if no ACK is received within a configurable time period, all outstanding data packets are retransmitted. Outstanding data packets are defined as data packets that have been sent but for which no ACK has been received.

[0168] In summary, embodiments of the present application propose a new RDMA service and transport (SLD) between RC, RD and UC services. SLD covers all RDMA operations and has a more efficient retransmission scheme. Embodiments of the present application are able to multiplex operations / WQEs from different ULPs and also to different targets. ECMP flowlet is also supported.

[0169] Figure 9 A method 900 for RDMA provided by embodiments of the present application is shown. In certain embodiments of the present application, the method 900 is performed by a device 100 as Figure 1 or Figure 8The sending device 100 shown performs. The method 900 comprises a step 901 of maintaining a sending queue 101, wherein the sending queue 101 comprises one or more WQEs 102, 102'. Each WQE 102, 102' comprises destination information of the WQE 102, 102', wherein the destination information comprises at least one of: an address of the receiving device 200, an identification of the receiving device 200 and an identification of a target application of the receiving device 200. The method 900 further comprises a step 902 of assigning an XID to each of the one or more WQEs 102, 102' and converting each WQE 102, 102' into a data packet 103, 103'. In particular, each data packet 103, 103' comprises the XID of the corresponding WQE 102, 102' and is associated with a PSN. The method 900 further comprises a step 903 of sending each data packet 103, 103' to the receiving device (200, 200') of the destination information of the WQE 102, 102'. The method 900 further comprises a step 904 of receiving a notification message 201 from the receiving device 200 for each sent data packet 103, wherein the notification message 201 indicates whether the sent data packet 103 is received at the receiving device 200. Further, the method 900 comprises a step 905 of determining whether to generate a completion for each WQE 102 based on the notification message 201, information carried in the WQE 102 and information held by the sending device 100.

[0170] Figure 10 A method 1000 for RDMA is shown to be provided by embodiments of the present application. In particular embodiments of the present application, the method 1000 is performed by a receiving device 200 as shown in Figure 1 or Figure 8 The method 1000 comprises a step 1001 of receiving one or more data packets 103 from a sending device 100 for RDMA. In particular, each data packet 103 of the one or more data packets 103 is associated with a PSN and comprises an XID of a corresponding WQE 102. Each WQE 102 comprises destination information of the WQE 102, wherein the destination information comprises at least one of: an address of the receiving device 200, an identification of the receiving device 200 and an identification of a target application of the receiving device 200. The method 1000 further comprises a step 1002 of sending a notification message 201 to the sending device 100, wherein the notification message 201 indicates that the one or more data packets are received and / or processed at the receiving device 200.

[0171] The application has been described in connection with various embodiments and implementations as examples. However, those skilled in the art will appreciate and understand that other variations are possible in light of the teachings and studies of the drawings, the present application, and the appended claims. In the claims and specification, the word "comprising" does not exclude other elements or steps, and the word "a" or "an" does not exclude a plurality. A single element or other unit can fulfill the functions of several entities or items described in the claims. Listing of means for accomplishing a certain task in the claims does not exclude that other means can be used for accomplishing the same task, and listing of a certain task in the claims does not exclude that other tasks can be performed in addition to the listed ones.

[0172] Furthermore, any method provided by embodiments of the application can be implemented in a computer program, having code means which, when run by processing means, causes the processing means to execute the steps of the method. The computer program is included in a computer readable medium of a computer program product. The computer readable medium can comprise essentially any memory, such as a ROM (Read-Only Memory), a PROM (Programmable Read-Only Memory), an EPROM (Erasable PROM), an EEPROM (Electrically Erasable PROM), a hard disk, or a floppy disk.

[0173] Furthermore, the skilled person realizes that embodiments of the transmitting device 100 or the receiving device 200 comprise the necessary communication capabilities in the form of functions, modules, units, elements etc. for performing the schemes. Examples of other such modules, units, elements and functions are: processors, memories, buffers, control logic, encoders, decoders, rate matchers, de-rate matchers, mapping units, multipliers, decision units, selection units, switches, interleavers, de-interleavers, modulators, demodulators, inputs, outputs, antennas, amplifiers, receiving units, transmitting units, DSPs, TCM (Trellis-Coded Modulation) encoders, TCM decoders, power supply units, power feeders, communication interfaces, communication protocols, etc. which are suitably arranged together to perform the schemes.

[0174] In particular, the one or more processors of the transmitting device 100 or the receiving device 200 can comprise one or more instances of a Central Processing Unit (CPU), a processing unit, a processing circuit, a processor, an Application Specific Integrated Circuit (ASIC), a microprocessor, or other processing logic that can interpret and execute instructions. The expression “processor” can thus represent a processing circuitry comprising a plurality of processing circuits, such as, e.g., any, some or all of the ones enumerated above. The processing circuitry can further perform data processing functions for inputting, outputting, and processing of data comprising data buffering and device control functions, such as call processing, user interface control, or the like.

Claims

1. A method for remote direct memory access (RDMA), the method comprising: The method is performed by a device adapted for RDMA, comprising: maintaining a send queue comprising one or more work queue elements (WQE), wherein a first WQE comprises destination information, wherein the first WQE is one WQE in the send queue, the destination information comprises at least one of: an address of a receiving device, an identification of a receiving device, and an identification of a target application of a receiving device; there are WQEs in the send queue with different destination information; allocating a transaction identification (XID) for the first WQE; converting the first WQE into a first data packet, wherein the first data packet comprises the XID of the first WQE, the first data packet is associated with a packet sequence number (PSN); the first data packet is further associated with a flowlet ID identifying a flowlet; the flowlet ID is used to group multiple data packets into the same data flow for transmission through equal multi-path; sending the first data packet to a receiving device according to the destination address information; receiving a notification message of the first data packet from the receiving device, wherein the notification message indicates whether the first data packet is received at the receiving device; determining whether to generate a completion for the first WQE based on the notification message, information carried in the first WQE, and information held by the device adapted for RDMA.

2. The method of claim 1, wherein, The first WQE carries information limited to a message that fits into a single network maximum transmission unit.

3. The method according to claim 1 or 2, characterized in that, The determining whether to generate a completion for the first WQE based on the notification message, information carried in the first WQE, and information held by the device adapted for RDMA, comprises: when the first WQE comprises a write operation request or a send operation request, and the notification message indicates that the first data packet is received at the receiving device, determining that the first WQE is successfully completed.

4. The method according to claim 1 or 2, characterized in that, The determining whether to generate a completion for the first WQE based on the notification message, information carried in the first WQE, and information held by the device adapted for RDMA, comprises: receiving a response data packet indicating that the first data packet has been processed by the receiving device as an atomic operation response or a read operation response to an atomic operation request or a read operation request, wherein the response data packet comprises an XID of a response first WQE and an XID of the first WQE comprising the read operation request or the atomic operation request; determining that the first WQE is successfully completed.

5. The method according to claim 1 or 2, characterized in that, The method further comprises: obtaining a maximum acceptable number of receive timeouts associated with the receiving device; when a number of receive timeouts exceeds the maximum acceptable number of receive timeouts, determining that the WQE of the first data packet is completed but has an error.

6. The method of claim 1 or 2, wherein, The method further comprises: when the notification message of the first data packet comprises a receiver not ready (RNR) indication, wherein the RNR indication indicates that the receiving device is unable to process the sent data packet, determining an XID included in the first data packet; postponing sending another data packet comprising the same XID before a timeout value expires.

7. The method of claim 6, wherein, The method further comprises: when the first WQE further comprises a parameter indicating a maximum acceptable number of RNR indications, receiving another notification message comprising RNR indications of retransmitted data packets; when the number of received RNR indications exceeds the maximum acceptable number of RNR indications, determining that the first WQE corresponding to the retransmitted data packets is completed but has errors.

8. The method of claim 6, wherein, The notification message comprises a plurality of RNR indications, each RNR indication indicating that a particular transmitted data packet cannot be processed by the receiving device.

9. The method of claim 8, wherein, The RNR indication comprises the timeout value.

10. The method of claim 9, wherein, The method further comprises: based on the timeout value of the RNR indication, managing the first WQE corresponding to the first data packet of the transmitted data packet for which the RNR indication is received.

11. The method of claim 10, wherein, The number of supported timeout values is configurable and agreed upon by all communication devices.

12. The method of any one of claims 1-2, 7-11, wherein, The sending queue comprises one or more WQE sets, a first WQE set being one WQE set in the sending queue, the first WQE set comprising one or more WQEs comprising the same destination information, the method further comprising: grouping the data packets of the first WQE set into a data packet stream; sending the data packet stream to the receiving device associated with the destination information of the WQE set.

13. The method of claim 12, wherein, The WQE set comprising the same destination information is obtained from different upper layer processes (ULPs).

14. The method of claim 13, wherein, The method further comprises: XIDs of WQEs obtained from different ULPs are allocated from the same XID pool, and PSNs of data packets associated with the WQEs obtained from different ULPs are allocated from the same PSN pool.

15. The method of claim 14, wherein, The priority of the first WQE is determined based on the ULP from which the first WQE is obtained.

16. The method of any one of claims 1-2, 7-11, wherein, The method further comprises: sending the first data packet to the receiving device through the stream block, wherein the stream block is associated with the receiving device.

17. The method of claim 3, wherein, The method further comprises: receiving the first data packet from other RDMA-capable devices, wherein the first data packet is associated with a packet sequence number (PSN) and comprises a transaction identifier (XID) of a corresponding work queue element (WQE); sending a notification message to the other RDMA-capable devices, wherein the notification message indicates that the one or more data packets are received at the receiving device.

18. The method of claim 17, wherein, The method further comprises: putting data carried on the write operation request into a destination memory location in the RDMA-capable device; or putting data carried on the send operation request from a receive queue of the RDMA-capable device into a buffer.

19. An apparatus adapted for RDMA, the apparatus comprising: The device comprises a processor and a memory, the memory being configured to store computer-executable instructions, and when the device is running, the processor executes the computer-executable instructions in the memory to perform the operation steps of the method of any one of claims 1-18 using hardware resources in the device.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium comprises instructions which, when executed on a computer, cause the computer to perform the operation steps of the method of any one of claims 1-18.

Images