Remote direct memory access with striping over an unreliable datagram transport

Abstract

In a multinode data processing system in which nodes exchange information over a network or through a switch, a structure and mechanism are provided which enables data packets to be sent and received in any order. Normally, if in-order transmission and receipt are required, then transmission over a single path is essential to insure proper reassembly. However, the present mechanism avoids this necessity and permits Remote Direct Memory Access (RDMA) operations to be carried out simultaneously over multiple paths. This provides a data striping mode of operation in which data transfers can be carried out much faster since packets of single or multiple RDMA messages can be portioned and transferred over several paths simultaneously, thus providing the ability to utilize the full system bandwidth that is available.

Description

[0001] This application claims priority based upon Provisional patent application having Provisional Ser. No. 60 / 605,659 filed on Aug. 30, 2004.BACKGROUND OF THE INVENTION [0002] The present invention is generally directed to the transfer of information residing in one computer system or on one data processing node to another data processing node. The present invention is more particularly directed to data transfers in which data is transferred by the network adapter directly into the target user buffer in the address space of the receiving system or node from the address space of the sending system or node. This is referred to as remote direct memory access (RDMA). Even more particularly, the present invention is directed to systems and methods for carrying out RDMA without the automatic assumption that data which has been sent is data which has also been received. This assumption is referred to as reliable transport but which should really be thought of as the “reliable transport ...

AI Technical Summary

Benefits of technology

[0014] In accordance with a preferred embodiment of the present invention a mechanism is provided for the transfer of data from the memory space of one data processing node to the memory space of one or more other data processing nodes. In particular, the present invention provides a data transfer structure and mechanism in which the data is transferred in at least one and typically in many packets which are not constrained to arrive at the destination node in any given order. The presence of the potentially out-of-order aspect provides the ability to structure a number of other transfer modalities and to provide a number of ancillary advantages all of which are described below under their respective headings.

Problems solved by technology

There is no mechanism in existing RDMA transport mechanisms to accept out-of-order packet delivery.

The state of the art RDMA paradigm also includes the limitation that data packets sent over the communications fabric are received in the same order as they were sent since they assume the underlying network transport to be reliable for RDMA to function correctly.

This tends to bog down adapter operation and produces its own form of bandwidth limiting effect in addition to the bandwidth limiting effect caused by the fact that the source and destination nodes are constrained by having all of the packets pass in order through a single communications path.

In addition, adapter design itself is unnecessarily complicated since this paradigm requires the buffering of unacknowledged in-flight packets.

This provides advantages but it also entails some risk and limitations.

Just as importantly, the “send and forget” RDMA model (RDMA with the underlying reliable network transport assumption) unnecessarily limits bandwidth and precludes the use of many other features and functions such as efficient striping multiple packets across a plurality of paths.

None of these functions can be carried out as efficiently within the existing state of the art RDMA “send and forget” paradigm as they are herein.

However, the use of DMA technology has been limited by the need for tight hardware controls and coordination of memory operations.

The tight coupling between memory operations and CPU operations poses some challenges, however, when the data processing system comprises a plurality of processing nodes that communicate with one another over a network.

As pointed out above, prior art RDMA models (such as Infiniband referred to above) do not tolerate receipt of packets in other than their order of transmission.

If a packet was received in a different order than it was transmitted, serious data integrity problems could result.

Thus, in order to prevent packets from being received out of transmission order, transactions in existing RDMA technologies have to be assigned small time-out values, so that a time-out is forced to occur unless the expected action (namely, the receipt of an acknowledgment of the packet from the receiver to the sender) occurs within an undesirably short period of time.

All of these restrictions impact the effective bandwidth that is apparent to a node for the transmission of RDMA messages across the network.

Images