Apparatus and method for remote direct memory access

Abstract

The present application relates to an apparatus and method for RDMA. Specifically, the present application proposes a sending apparatus for RDMA. The sending apparatus is configured to: send a first group of packets to a receiving apparatus, wherein the first group of packets comprises one or more first RDMA operations; after sending the first group of packets, send a second group of packets to the receiving apparatus, wherein the second group of packets comprises one or more second RDMA operations, the one or more second RDMA operations being associated with a fence indication; and provide a notification to the receiving apparatus, wherein the notification indicates that the one or more second RDMA operations are associated with the fence indication. Furthermore, the present application proposes a receiving apparatus for RDMA.

Description

TECHNICAL FIELD

[0001] The present invention relates to high performance computing technology, and in particular to Remote Direct Memory Access (RDMA) technology. The present invention can transfer RDMA transactions over a packet-based network. To this end, the present invention provides an apparatus, method and data packet format for RDMA. BACKGROUND

[0002] RDMA is the primary transport method for remote memory operations. The viability of RDMA is largely dependent on its low latency characteristics.

[0003] RDMA operations need to be executed in order, so it provides a mechanism to implement the required order. Typically, the RDMA receiver needs to know the correct order of operations. In some cases, the RDMA receiver can be able to process RDMA operations out of order.

[0004] It is noted that the network can deliver packets out of order, for example, due to different network paths used by different packets. Thus, the packets of an RDMA data can arrive out of order at the RDMA receiver, for example, due to network delivery, or packets that are lost and retransmitted.

[0005] There are two common RDMA technologies today: one is defined by the InfiniBand Transport Alliance (IBT) in the InfiniBand (IB) specification, and the other is defined by the Internet Engineering Task Force (IETF). The IETF protocol is also known as the internet wide area RDMA protocol (iWARP). RDMA only provides a simple mechanism for ordering operations. Both RDMA technologies define two different strategies for ordering operations. Specifically, the IB specification defines a "strict order" of execution, while iWARP defines a "loose order" of execution with some restrictions.

[0006] One of the mechanisms for ordering the execution of RDMA operations is a fence. A fence can only be used at the RDMA sender side to force the delivery of operations in order. A fence causes the RDMA sender to delay sending an operation until the RDMA receiver sends a completion response for the previous dependent operation. However, this delay degrades the low latency characteristics of RDMA, and thus the viability of RDMA. SUMMARY

[0007] In view of the above challenges, embodiments of the present invention aim at providing an apparatus and a method for RDMA. In particular, it is an object to reduce the high latency of RDMA operations including a fence indication. It is an object to allow a sender to send RDMA operations with a fence indication without having to wait for a completion signal of a previous operation.

[0008] The objects are achieved by the embodiments of the present invention described in the attached independent claims. Advantageous implementations of the embodiments of the present invention are further defined in the dependent claims.

[0009] A first aspect of the present invention provides a sending device for RDMA. The sending device is configured to: send, to a receiving device, a first set of packets, wherein the first set of packets comprises one or more first RDMA operations; send, to the receiving device, a second set of packets after sending the first set of packets, wherein the second set of packets comprises one or more second RDMA operations, the one or more second RDMA operations being associated with a fence indication; and provide, to the receiving device, a notification, wherein the notification indicates that the one or more second RDMA operations are associated with the fence indication.

[0010] Embodiments of the present invention provide a technical solution for the sending device to communicate the presence of a fence indication associated with an operation to a receiving device. The technical solution helps the receiving device to handle the fence, instead of the sending device, thereby reducing the latency of RDMA operations including a fence indication.

[0011] In an implementation form of the first aspect, the second set of packets is sent without any indication of a completion status of the one or more first RDMA operations being received at the receiving device.

[0012] In contrast to conventional technical solutions, the present invention helps the sending device to send a fence operation, i.e. the one or more second RDMA operations associated with a fence indication, immediately after sending the first RDMA operation. That is, the sending device does not have to wait until a completion signal of the first RDMA operation is received.

[0013] In an implementation form of the first aspect, one or more packets of the second set of packets comprise one or more headers, the fence indication being carried in the one or more headers.

[0014] For example, the fence indication is carried in the one or more headers, in particular by overwriting one or more reserved or unused fields in the one or more headers.

[0015] In an implementation form of the first aspect, the fence indication is carried in a packet having an RDMA operation code.

[0016] Currently, there is no operation code associated with the fence indication. An operation with the fence indication can be referred to as a fence operation. The embodiment proposes to extend the protocol to have specific new operation codes designed for fence operations. For example, if the RDMA "write-only" has an operation code: 0xA (1010b), a new operation code for a fence operation: fence RDMA "write-only" can be set to 0x1A (11010b).

[0017] In an implementation form of the first aspect, one or more packets of the second group of packets comprise one or more payloads, and the fence indication is carried in the one or more payloads.

[0018] Alternatively, the fence indication can be carried in a payload of a packet packet, instead of in a header.

[0019] In an implementation form of the first aspect, the fence indication is used to instruct the receiving device to complete the one or more first RDMA operations before starting to process the second RDMA operation.

[0020] It is noted that a fence is one of the RDMA mechanisms used to enforce the order of execution of RDMA operations. Upon receiving a notification of an operation associated with a fence indication, the receiving device will complete the previous operations before starting to process the fence operation.

[0021] In an implementation form of the first aspect, the sending device is further configured to send a third group of packets to the receiving device, wherein the third group of packets comprises one or more third RDMA operations; and the third group of packets further comprises a fence indication associated with the one or more second RDMA operations when the sending device has not received an indication that the one or more first RDMA operations are completed at the receiving device side.

[0022] Optionally, the fence indication can also be added to the next operation of the fence operation, i.e. the third RDMA operation, to prevent ambiguity in case of packet loss.

[0023] In an implementation form of the first aspect, the fence indication associated with the one or more second RDMA operations is further used to instruct the receiving device to complete the one or more first RDMA operations before starting to process the third RDMA operation.

[0024] In an implementation form of the first aspect, the wrap indication is a read wrap indication or a local wrap indication, the read wrap indication indicating the receiving device to complete read operations in the one or more first RDMA operations before starting processing the second RDMA operation, and the local wrap indication indicating the receiving device to complete all operations in the one or more first RDMA operations before starting processing the second RDMA operation.

[0025] Generally, there can be two types of wrap indications: read wrap and local wrap.

[0026] A second aspect of the present application provides a receiving device for RDMA. The receiving device is configured to: maintain a list of received RDMA operations; receive a first group of packets from a sending device, wherein the first group of packets comprises one or more first RDMA operations; receive a second group of packets from the sending device, wherein the second group of packets comprises one or more second RDMA operations associated with a wrap indication; receive a notification from the sending device, wherein the notification indicates that the one or more second RDMA operations are associated with the wrap indication; and add the one or more first RDMA operations, the one or more second RDMA operations, and the associated wrap indication to the list.

[0027] Embodiments of the present application provide a technical solution for the sending device to transmit the existence of the wrap indication associated with the operation to the receiving device. The technical solution helps the receiving device to process the wrap, instead of the sending device, thereby reducing the delay of the RDMA operation comprising the wrap indication.

[0028] In an implementation form of the second aspect, the second group of packets is received after the first group of packets.

[0029] That is, the sending device can send the wrap operation, i.e., the one or more second RDMA operations associated with the wrap indication, immediately after sending the first RDMA operation.

[0030] In an implementation form of the second aspect, the receiving device is further configured to process the one or more second RDMA operations after the one or more first RDMA operations are completed at the receiving device side.

[0031] Even though the sending device does not have to wait to send the one or more second RDMA operations until receiving the completion signal of the first RDMA operation, the receiving device preferably does not process the one or more second RDMA operations before the processing of the one or more first RDMA operations is completed. In this way, the execution order of the RDMA operations can be maintained without high latency.

[0032] In an implementation form of the second aspect, one or more of the second group of packets comprises one or more headers, and the net indication is carried in the one or more headers.

[0033] In particular, the net indication can be carried in the headers by overwriting one or more reserved or unused fields in the one or more headers.

[0034] In an implementation form of the second aspect, the net indication is carried in a packet having an RDMA operation code.

[0035] In an implementation form of the second aspect, one or more of the second group of packets comprises one or more payloads, and the net indication is carried in the one or more payloads.

[0036] In an implementation form of the second aspect, the net indication is for instructing the receiving device to complete the one or more first RDMA operations before starting to process the second RDMA operation.

[0037] In an implementation form of the second aspect, the receiving device is further configured to receive, from the sending device, a third group of packets, wherein the third group of packets comprises one or more third RDMA operations, and the third group of packets further comprises a net indication associated with the one or more second RDMA operations when the sending device does not receive the indication that the one or more first RDMA operations are completed at the receiving device side.

[0038] In an implementation form of the second aspect, the net indication associated with the one or more second RDMA operations is further for instructing the receiving device to complete the one or more first RDMA operations before starting to process the one or more third RDMA operations.

[0039] In an implementation form of the second aspect, the fence indication is a read fence indication or a local fence indication, the read fence indication indicating to the receiving device to complete read operations in the one or more first RDMA operations before starting processing the second RDMA operation, and the local fence indication indicating to the receiving device to complete all operations in the one or more first RDMA operations before starting processing the second RDMA operation.

[0040] A third aspect of the present application provides a method for RDMA, wherein the method comprises: sending a first group of packets to a receiving device, wherein the first group of packets comprises one or more first RDMA operations; sending a second group of packets to the receiving device after sending the first group of packets, wherein the second group of packets comprises one or more second RDMA operations, the one or more second RDMA operations being associated with a fence indication; providing a notification to the receiving device, wherein the notification indicates that the one or more second RDMA operations are associated with the fence indication.

[0041] The method of the third aspect and implementation forms thereof provide the same advantages and effects as described above for the sending device of the first aspect and respective implementation forms thereof.

[0042] A fourth aspect of the present application provides a method for RDMA, wherein the method comprises: maintaining a list of received operations; receiving a first group of packets from a sending device, wherein the first group of packets comprises one or more first RDMA operations; receiving a second group of packets from the sending device, wherein the second group of packets comprises one or more second RDMA operations, the one or more second RDMA operations being associated with a fence indication; receiving a notification from the sending device, wherein the notification indicates that the one or more second RDMA operations are associated with the fence indication; adding the one or more first RDMA operations, the one or more second RDMA operations and the associated fence indication to the list.

[0043] The method of the fourth aspect and implementation forms thereof provide the same advantages and effects as described above for the device of the second aspect and respective implementation forms thereof.

[0044] A fifth aspect of the present application provides a computer program comprising program code for performing the method provided by the third aspect or the fourth aspect or any implementation form thereof when the program code is implemented on a processor.

[0045] The sixth aspect of the present application provides a computer readable storage medium, comprising computer program code instructions executable by a computer, when the computer program code instructions run on the computer, performing the method according to the third aspect or the fourth aspect or any implementation manner thereof.

[0046] It should be noted that all the devices, elements, units and modules described in the present application can be implemented in software or hardware elements or any combination thereof. The steps performed by various entities described in the present application and the functions described by various entities are intended to indicate that the respective entities perform the respective steps and functions. Even if the specific functions or steps to be performed by external entities are not reflected in the description of the specific detailed elements of the entity performing the specific steps or functions in the following description of specific embodiments, it should be clear to those skilled in the art that these methods and functions can be implemented in corresponding software or hardware elements, or in any combination of such elements. BRIEF DESCRIPTION OF DRAWINGS

[0047] The above aspects and their implementation manners will be described in detail in the following description of specific embodiments in combination with the accompanying drawings, in which:

[0048] Figure 1 An example of packets exchanged between a sender and a receiver is shown;

[0049] Figure 2 An example of packets exchanged between a sender and a receiver is shown;

[0050] Figure 3 A sending device provided by an embodiment of the present application is shown;

[0051] Figure 4 A receiving device provided by an embodiment of the present application is shown;

[0052] Figure 5 An example of packets exchanged between a sender and a receiver provided by an embodiment of the present application is shown;

[0053] Figure 6 An example of packets exchanged between a sender and a receiver provided by an embodiment of the present application is shown;

[0054] Figure 7 A method provided by an embodiment of the present application is shown;

[0055] Figure 8 A method provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0056] Illustrative embodiments of methods, devices, and program products for efficient packet transmission in a communication system are described with reference to the drawings. While the description provides detailed examples, it should be noted that the details are intended to be examples and in no way limit the scope of application.

[0057] Furthermore, embodiments / examples can reference other embodiments / examples. For example, any description including, but not limited to, terminology, elements, procedures, explanations, and / or technical advantages mentioned in one embodiment / example apply to other embodiments / examples.

[0058] Two types of fence indications are defined: read fence, which sends a block of operations before all previous read operations have completed; local fence, which sends a block of operations before all previous operations have completed. Initially, local fence was defined as an invalid operation in the IB specification, and later extended to cover all operations.

[0059] Figure 1 An example of packets exchanged between a sender and a receiver is shown. Specifically, Figure 1 An example of packets exchanged with a write operation including a read fence with IB "strictly ordered" execution is shown. Note that "strictly ordered" means that the operations carried in all packets must be executed in order.

[0060] Note that Work Queue Entry (WQE) and Queue Pair (QP) are terms in the RDMA specification. WQE is a RDMA operation or transaction that is pushed into a QP, either to be transmitted to a peer or has been received from a peer. QP represents a communication endpoint, consisting of a send queue and a receive queue.

[0061] It can be seen in Figure 1 that there are three WQEs issued for a particular QP, namely, two read operations (read 1 and read 2) and one write operation (write 1). For example, read 1 requests three responses and read 2 requests two responses. Write 1, which includes a read fence indication, is a multi-packet operation with two packets. The two read operations will be sent immediately once the sender can process the WQEs, regardless of whether the previous read operations have completed. In this example, read 1 and read 2 are processed in the order dictated by "strictly ordered" execution. The sender will send the write operation only after receiving two completion signals indicating that the read operations have completed. Note, however, that the time delay is caused by the read fence indication. Specifically, the delay occurs between sending read 2 and starting write 1.

[0062] Note that while in Figure 1In this example shown, read 2 starts after read 1 completes; it is also possible to send read 2 before the last response to read 1 is accepted. The read fence indication on a write operation indicates that the write operation will not be sent until all previous read operations complete.

[0063] However, if there is no read fence indication on write 1, read 2 and write 1 requests will not wait for all responses to read 1 to arrive before being issued. That is, both read 2 and write 1 will be output to the network in order, but the write operation (write 1) can arrive at the receiver before the previous read operation completes. The response to read 2 will arrive after read 1. That is, the receiver will not process read 2 until read 1 completes. However, write 1 can be processed before either of the two read operations complete. The ordering rules for strict ordering scenarios are defined in the IB specification.

[0064] Figure 2 Another example of packets exchanged between a sender and a receiver is shown. Specifically, Figure 2 Packets exchanged with a write operation that includes a local fence with iWARP "loose ordering" execution are shown. Note that "loose ordering" means that packets can be placed out of order.

[0065] As can be seen from Figure 2 Three WQEs are issued for a particular QP, i.e., three write operations (write 1, write 2, write 3). Specifically, write 1 is a multi-packet operation with three packets; write 2, which includes a local fence indication, is a multi-packet operation with three packets; and write 3 is a multi-packet operation with two packets. Write 2 and write 3 will not be sent until the sender receives a completion signal for write 1. Since this example shows "loose ordering" execution, write 3 will be sent even before write 2 is signaled complete. Note that the time delay between the last packet of write 1 being sent and the start of write 2 is caused by the local fence indication.

[0066] Note that even in the strict ordering case without a fence, the packets carrying the operations will be sent to the wire. The ordering rules occur at the receiving end. In the read-before-write scenario, the write operation can be processed before the read operation completes. The fence indication prevents packets from being sent in both the strict and loose ordering execution scenarios. Typically, multiple WQEs are sent in an RDMA at a time.

[0067] As can be seen, as an example of the conventional approach described in Figure 1 and Figure 2 Both the read fence indication and the local fence indication result in high latency before the sender is allowed to send the next operation, as an example of the conventional approach described in

[0068] Figure 3A sending device 300 for RDMA is shown. The sending device 300 can include processing circuitry (not shown) to perform, conduct or initiate the various operations of the sending device 300 described herein. The processing circuitry can include hardware and software. The hardware can include analog circuitry or digital circuitry, or both analog and digital circuitry. The digital circuitry can include components such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), or multi-purpose processors, etc. The sending device 300 can also include storage circuitry to store one or more instructions that can be executed by the processor or the processing circuitry, particularly under the control of software. For example, the storage circuitry can include a non-transitory storage medium storing executable software code that, when executed by the processor or the processing circuitry, causes the various operations of the sending device 300 to be performed. In one embodiment, the processing circuitry includes one or more processors and a non-transitory memory connected to the one or more processors. The non-transitory memory can carry executable program code that, when executed by the one or more processors, causes the sending device 300 to perform, conduct or initiate the operations or methods described herein.

[0069] In particular, the sending device 300 is configured to send a first set of packets 301 to a receiving device 310. The first set of packets 301 comprises one or more first RDMA operations. The sending device 300 is further configured to send a second set of packets 302 to the receiving device 310 after sending the first set of packets 301. The second set of packets 302 comprises one or more second RDMA operations. In particular, the one or more second RDMA operations are associated with a fence indication. Furthermore, the sending device 300 is configured to provide a notification 303 to the receiving device 310, wherein the notification 303 indicates that the one or more second RDMA operations are associated with the fence indication.

[0070] To reduce the high latency of RDMA operations including fence indications, embodiments of the present application propose a method of sending and receiving RDMA operations that facilitates the communication of fence indications to the receiver, i.e. the receiving device 310. In particular, the fence is signaled by the sender, i.e. the sending device 300, to the receiver. By delegating the processing of the fence to the receiver, instead of processing the fence by the sender, the latency can be reduced.

[0071] It is noted that, according to embodiments of the present application, the second set of packets 302 can be sent without the receiving device 310 receiving any indication of completion status of one or more of the first RDMA operations. This is in contrast to conventional approaches in which, if a previous operation includes a fence indication (as in the examples shown in Figure 1 and Figure 2 the next operation is only sent after the sender receives a completion signal for the previous operation.

[0072] Figure 4 A receiving device 310 for RDMA provided by embodiments of the present application is shown. The receiving device 310 can include processing circuitry (not shown) for performing, conducting or initiating various operations of the sending device 300 described herein. The processing circuitry can include hardware and software. The hardware can include analog circuitry or digital circuitry, or both analog and digital circuitry. The digital circuitry can include components such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), or multi-purpose processors, etc. The receiving device 310 can also include storage circuitry storing one or more instructions executable by the processor or processing circuitry, particularly under control of software. For example, the storage circuitry can include a non-transitory storage medium storing executable software code that, when executed by the processor or processing circuitry, causes various operations of the receiving device 310 to be performed. In one embodiment, the processing circuitry includes one or more processors and a non-transitory memory connected to the one or more processors. The non-transitory memory can carry executable program code that, when executed by the one or more processors, causes the receiving device 310 to perform, conduct or initiate the operations or methods described herein.

[0073] In particular, the receiving device 310 is configured to maintain a list 311 of received RDMA operations. The receiving device 310 is also configured to receive a first set of packets 301 from the sending device 300. In particular implementations, the sending device 300 can be a network interface card (NIC) or a host bus adapter (HBA). Figure 3The sending device 300 is shown. The first set of packets 301 comprises one or more first RDMA operations. The receiving device 310 is further configured to receive a second set of packets 302 from the sending device 300. The second set of packets 302 comprises one or more second RDMA operations. In particular, the one or more second RDMA operations are associated with a fence indication. Furthermore, the receiving device 310 is configured to receive a notification 303 from the sending device 300, wherein the notification 303 indicates that the one or more second RDMA operations are associated with the fence indication. The receiving device 310 is then further configured to add the one or more first RDMA operations, the one or more second RDMA operations and the associated fence indication to the list 311.

[0074] It is noted that the second set of packets 302 is received after the first set of packets 301.

[0075] Embodiments of the present application propose a receiving device 310 which maintains a list 311 of received RDMA operations. It is possible that the list 311 can also comprise metadata of the RDMA operations, e.g. a fence indication associated with a RDMA operation. In particular, when the receiving device 310 receives a RDMA operation, it keeps the received operation and the fence indication associated with the operation, if a fence indication exists, in the list 311. Optionally, the list 311 of received RDMA operations can be stored in a storage space of the receiving device 310. The list 311 can be stored in a buffer or a queue.

[0076] Figure 5 An example of packets exchanged between a sender and a delegated receiver is shown. In particular, the sender can be Figure 3 the sending device 300 shown, the receiver can be Figure 4 the receiving device 310 shown. Figure 5 An example of packets exchanged with a write operation comprising a read fence with IB "strictly ordered" execution is shown. As mentioned before, "strictly ordered" means that the operations are executed in order. The ordering rules for the strictly ordered scenario are defined in the IB specification.

[0077] The example can be implemented in a Figure 5As seen in the diagram, for a specific QP, there are two WQE publications: one read operation and one write operation. Specifically, the read operation requests three responses. The write operation is a multi-packet operation with two packets and includes a read fencing indication. According to this embodiment, the write operation is sent by the sending device (i.e., sending device 300) without any delay after the read request and includes a fencing indication. The receiver (i.e., receiving device 310) stores these write requests, but will not process them due to the fencing indication. It should be understood that the fencing indication notifies the receiver that all previous read operations need to be completed before a write operation can be performed. Accordingly, the receiver processes the read operation and sends three responses. Once the receiver has completed executing the read request, it begins executing the stored write request.

[0078] Figure 6 This illustration shows another example of packets exchanged between a transmitter and a delegated receiver according to an embodiment of the present invention. Specifically, the transmitter may be... Figure 3 The transmitting device 300 shown can be a receiver. Figure 4 The receiving device 310 shown. Figure 6 This illustrates a packet exchanged with a local fence that includes write operations executed with iWARP "loose ordering". As previously mentioned, "loose ordering" means that operations may be executed out of order. The ordering rules for loose ordering scenarios are defined in the iWARP specification.

[0079] from Figure 6 As can be seen, three WQEs are issued for a specific QP, namely three write operations (Write 1, Write 2, and Write 3). Specifically, Write 1 is a multi-packet operation with three packets; Write 2, which includes a local fencing indication, is a multi-packet operation with three packets; and Write 3 is a multi-packet operation with two packets. According to this embodiment, all write operations are sent once the transmitter (i.e., transmitting device 300) processes its WQE. Specifically, even though Write 2 includes a local fencing indication, Write 2 is sent before Write 1 completes. Write 3 is sent before Write 2 completes because this example demonstrates a “loose order” of execution. Write 3 can be executed before or after Write 2 completes because the “loose order” of iWARP execution allows for such an implementation, and because Write 3 itself is not fencing.

[0080] Embodiments of the present invention define a new procedure for performing RDMA operations. Specifically, even in the presence of a fence indication, it is recommended that the RDMA operation be transmitted from the transmitter to the receiver via the network. Specifically, the transmitter may be... Figure 3 The transmitting device 300 shown can be a receiver. Figure 4The receiving device 310 is shown. The sending device 300 transmits the fence indication (if present) to the receiving device 310. Optionally, the sending device 300 can continue to transmit the fence indication for the subsequent operations until the completion indication for the first fence operation is received. This is to ensure that the receiving device 310 obtains the fence indication even in the case of receiving out-of-order operations or packets. The receiving device 310 can maintain a queue of the received operations and when there is a fence indication for a particular operation and for all the following operations, it will not process this operation and the following operations until the receiving device 310 completes all the previous operations on which they depend.

[0081] It is noted that the present application reduces the latency of the RDMA operations including the fence indication by transmitting the fence indication to the receiver, in particular, the delegate receiver. The delegate receiver, i.e. Figure 3 or Figure 4 The receiving device 310 is shown, becomes the entity to perform the fence, instead of the sender, i.e. the sending device 300. Therefore, the sending device 300 does not wait for the completion of the previous dependent operation to send the fence operation. In this way, the sending device 300 does not delay the operation because of the fence indication, instead, the receiving device 310 delays the execution of the operation according to the transmitted fence indication (if present).

[0082] Optionally, the fence indication can be the metadata of the operation, which can be incorporated into all the packets of the operation. In particular, the fence indication can also be added to the next operation to prevent ambiguity in the case of packet loss.

[0083] Further, the metadata of the operation can be carried on the unused fields in the header or sent as a new packet with a different operation code. Now, there is no operation code defined for the fence operation. The embodiment of the present application proposes to extend the protocol to have a specific new operation code designed for the fence operation. For example, when the RDMA write-only operation has the operation code 0xA (1010b), the new operation code for the fence RDMA write-only operation can be created as 0x1A (11010b).

[0084] The receiving device 310 can maintain a queue of the operations, the metadata of the operation can be stored until all the previous dependent operations are processed in the case of the presence of the fence indication.

[0085] It is noted that according to the embodiment of the present application, the latency of the operation with the fence indication can reduce at least one round trip time (RTT) of the channel between the sending device 300 and the receiving device 310. The RTT is the sum of the time taken for the completion signal of the previous operation to reach the sending device 300 from the receiving device 310 and the time taken for the fence operation to reach the receiving device 310 from the sending device 300.

[0086] Figure 7 A method 700 for RDMA provided in an embodiment of the present invention is illustrated. In a particular embodiment of the invention, method 700 is performed by… Figure 3 The transmitting device 300 shown performs the method 700. Method 700 includes the step 701 of sending a first group of packets to the receiving device 310. Specifically, the first group of packets 301 includes one or more first RDMA operations. In a specific implementation, the receiving device 310 may be... Figure 4 The receiving device is shown in the diagram. Method 700 further includes a step 702 of sending a second set of packets 302 to the receiving device 310. Specifically, the second set of packets 302 includes one or more second RDMA operations associated with a fencing indication. Step 702 is specifically performed after step 701, i.e., the step of sending the first set of packets 301. Furthermore, method 700 includes a step 703 of providing a notification 303 to the receiving device 310, wherein the notification 303 indicates that one or more second RDMA operations are associated with a fencing indication.

[0087] Figure 8 A method 800 for RDMA provided in an embodiment of the present invention is illustrated. In a particular embodiment of the invention, method 800 is performed by… Figure 4 The receiving device 310 shown performs the following method 800: Step 801 includes maintaining a received operation list 311; Step 802 includes receiving a first group of packets 301 from the transmitting device 300, wherein the first group of packets 301 includes one or more first RDMA operations. In a specific implementation, the transmitting device 300 may be... Figure 3 The transmitting device is shown. Method 800 further includes step 803 of receiving a second set of packets 302 from transmitting device 300. The second set of packets 302 includes one or more second RDMA operations, wherein the one or more second RDMA operations are associated with a fence indication. Furthermore, method 800 includes step 804 of receiving a notification 303 from transmitting device 300, wherein the notification 303 indicates that one or more second RDMA operations are associated with a fence indication; and step 805 of adding one or more first RDMA operations, one or more second RDMA operations, and the associated fence indication to list 311.

[0088] The present application has been described in connection with various embodiments and implementations as examples. However, those skilled in the art will appreciate and understand that other variations are possible in light of the teachings and disclosure provided herein and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps not listed in the claims. In the claims, the word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. An element specified in one claim as preceding another can be understood as preceded also by "a" or "an". Reference to an item in the singular does not exclude the plural. The mere fact that measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0089] Further, any of the methods of embodiments of the application can be implemented in a computer program, having code means which when run by processing means cause the processing means to execute the steps of the methods. The computer program is included in a computer readable medium of a computer program product. The computer readable medium can comprise essentially any memory, for example a ROM, a programmable read-only memory (PROM), an erasable PROM (EPROM), a flash memory, an electrically erasable PROM (EEPROM), or a hard disk.

[0090] Furthermore, it is realized by the skilled person that embodiments of the transmitting device 300 and the receiving device 310 comprise the necessary communication capabilities in the form of functions, modules, units, elements etc. for performing the technical solutions. Examples of other such modules, units, elements and functions are: processors, memory, buffers, control logic, encoders, decoders, rate matchers, de-rate matchers, mapping units, multipliers, decision units, selection units, switches, interleavers, de-interleavers, modulators, demodulators, inputs, outputs, antennas, amplifiers, receiving units, transmitting units, DSPs, trellis-coded modulation (TCM), TCM encoder, TCM decoder, power supply units, power feeders, communication interfaces, communication protocols, etc. which are suitably arranged together to perform the technical solutions.

[0091] In particular, the processors of the transmitting device 300 and the receiving device 310 can comprise one or more instances from a Central Processing Unit (CPU), a processing unit, a processing circuit, a processor, an Application Specific Integrated Circuit (ASIC), a microprocessor, or other processing logic that can interpret and execute instructions. The expression “processor” can thus represent a processing circuitry comprising a plurality of processing circuits, such as, e.g., any, some or all of the ones enumerated above. The processing circuitry can further perform data processing functions for inputting, outputting, and processing of data comprising data buffering and device control functions, such as call processing control, user interface control, or the like.

Claims

1. A sending device (300) for Remote Direct Memory Access (RDMA), characterized in that, The sending device (300) is configured to: send a first group of packets (301) to a receiving device (310), wherein the first group of packets (301) comprises one or more first RDMA operations; after sending the first group of packets (301), and without receiving any indication from the receiving device (310) that the one or more first RDMA operations are completed, send a second group of packets (302) to the receiving device (310), wherein the second group of packets (302) comprises one or more second RDMA operations, the one or more second RDMA operations being associated with a fence indication; provide a notification (303) to the receiving device (310), wherein the notification (303) indicates that the one or more second RDMA operations are associated with the fence indication; send a third group of packets to the receiving device (310), wherein the third group of packets comprises one or more third RDMA operations; when the sending device (300) does not receive an indication that the one or more first RDMA operations are completed at the receiving device (310) side, the third group of packets further comprises a fence indication associated with the one or more second RDMA operations.

2. The transmitting device (300) according to claim 1, c h a r a c t e r i z e d b y The one or more packets in the second group of packets (302) comprise one or more headers, and the fence indication is carried in the one or more headers.

3. The transmitting device (300) according to claim 1, characterized in that, The fence indication is carried in a packet with an RDMA operation code.

4. The transmitting device (300) according to claim 1, characterized in that, The one or more packets in the second group of packets (302) comprise one or more payloads, and the fence indication is carried in the one or more payloads.

5. The transmitting device (300) according to any one of claims 1 to 4, characterized in that, The fence indication is used to instruct the receiving device (310) to complete the one or more first RDMA operations before starting to process the second RDMA operations.

6. The transmitting device (300) according to claim 1, characterized in that The fence indication associated with the one or more second RDMA operations is further used to instruct the receiving device (310) to complete the one or more first RDMA operations before starting to process the third RDMA operations.

7. The transmitting device (300) according to any one of claims 1 to 6, wherein, The fence indication is a read fence indication or a local fence indication, the read fence indication is used to instruct the receiving device (310) to complete read operations in the one or more first RDMA operations before starting to process the second RDMA operations, and the local fence indication is used to instruct the receiving device (310) to complete all operations in the one or more first RDMA operations before starting to process the second RDMA operations.

8. A receiving device (310) for Remote Direct Memory Access (RDMA), characterized in that, The receiving device (310) is configured to: maintain a list (311) of received RDMA operations; receive a first group of packets (301) from a sending device (300), wherein the first group of packets (301) comprises one or more first RDMA operations; after the first group of packets (301), receive a second group of packets (302) from the sending device (300), wherein the second group of packets (302) comprises one or more second RDMA operations, the one or more second RDMA operations being associated with a fence indication; provide a notification (303) to the sending device (300), wherein the notification (303) indicates that the one or more second RDMA operations are associated with the fence indication; receiving a notification (303) from the sending device (300), wherein the notification (303) indicates that the one or more second RDMA operations are associated with the fence indication; adding the one or more first RDMA operations, the one or more second RDMA operations and the associated fence indication into the list (311); receiving a third set of packets from the sending device (300), wherein the third set of packets comprises one or more third RDMA operations; when the sending device (300) does not receive an indication that the one or more first RDMA operations are completed at the receiving device (310) side, the third set of packets further comprises a fence indication associated with the one or more second RDMA operations.

9. The receiving device (310) according to claim 8, characterized by Further used for: processing the one or more second RDMA operations after the one or more first RDMA operations are completed at the receiving device (310) side.

10. The receiving device (310) according to claim 8 or 9, characterized by, one or more packets in the second set of packets (302) comprise one or more headers, and the fence indication is carried in the one or more headers.

11. The receiving device (310) according to claim 8 or 9, characterized by, the fence indication is carried in a packet with an RDMA opcode.

12. The receiving device (310) according to claim 8 or 9, characterized by, one or more packets in the second set of packets (302) comprise one or more payloads, and the fence indication is carried in the one or more payloads.

13. The receiving device (310) according to any of claims 8 to 12, characterized by, the fence indication is used to instruct the receiving device (310) to complete the one or more first RDMA operations before starting to process the second RDMA operation.

14. The receiving device (310) according to claim 8, characterized in that, the fence indication associated with the one or more second RDMA operations is further used to instruct the receiving device (310) to complete the one or more first RDMA operations before starting to process the one or more third RDMA operations.

15. The receiving device (310) according to any of claims 8 to 14, characterized by, the fence indication is a read fence indication used to instruct the receiving device (310) to complete read operations in the one or more first RDMA operations before starting to process the second RDMA operation, or a local fence indication used to instruct the receiving device (310) to complete all operations in the one or more first RDMA operations before starting to process the second RDMA operation.

16. A method (700) for Remote Direct Memory Access (RDMA), characterized by, The method is applied to a sending device (300), and the method comprises: sending (701) a first set of packets (301) to a receiving device (310), wherein the first set of packets (301) comprises one or more first RDMA operations; after sending the first set of packets (301), and without any indication of completion status of the one or more first RDMA operations being received by the receiving device (310), sending (702) a second set of packets (302) to the receiving device (310), wherein the second set of packets (302) comprises one or more second RDMA operations, and the one or more second RDMA operations are associated with a fence indication; providing (703) a notification (303) to the receiving device (310), wherein the notification (303) indicates that the one or more second RDMA operations are associated with the fence indication; sending a third set of packets to the receiving device (310), wherein the third set of packets comprises one or more third RDMA operations; when the sending device (300) does not receive an indication that the one or more first RDMA operations are completed on the receiving device (310) side, the third set of packets further comprises a fence indication associated with the one or more second RDMA operations.

17. A method (800) for Remote Direct Memory Access (RDMA), characterized by, The method is applied to a receiving device (310), the method comprising: maintaining (801) a list (311) of received operations; receiving (802) a first set of packets (301) from a sending device (300), wherein the first set of packets (301) comprises one or more first RDMA operations; receiving (803) a second set of packets (302) from the sending device (300) after the first set of packets (301), wherein the second set of packets (302) comprises one or more second RDMA operations, the one or more second RDMA operations being associated with a fence indication; receiving (804) a notification (303) from the sending device (300), wherein the notification (303) indicates that the one or more second RDMA operations are associated with the fence indication; adding (805) the one or more first RDMA operations, the one or more second RDMA operations, and the associated fence indication to the list (311); receiving a third set of packets from the sending device (300), wherein the third set of packets comprises one or more third RDMA operations; when the sending device (300) does not receive an indication that the one or more first RDMA operations are completed on the receiving device (310) side, the third set of packets further comprises a fence indication associated with the one or more second RDMA operations.

18. A computer program product comprising a program code, characterized in that, The program code, when implemented on a processor, is configured to perform the method according to claim 16 or 17.

19. A computer-readable storage medium, characterized in that, A computer program product comprising computer program code instructions which, when the computer program code instructions are run on a computer, perform the method according to claim 16 or 17.

Images