Apparatus and method for remote direct memory access

Abstract

The invention provides a transmitting apparatus and a receiving apparatus. The sending device is used for maintaining a sending queue, the sending queue comprises one or more WQEs, and each WQE comprises destination information of the WQE; an XID is distributed to each of the one or more WQEs, each WQE is converted into a corresponding data packet, and each corresponding data packet comprises the XID of the corresponding WQE and is associated with a PSN; transmitting each data packet to the receiving device of the destination information of the WQE; receiving, from the receiving device, a notification message indicating, for each transmitted packet, whether the transmitted packet is received; and based on the notification message, the information carried in the WQE and the information held by the sending device, determining whether generation of each WQE is completed.

Description

[0001] This application is a divisional application. The application number of the original application is 202180101453.1, and the original application date is August 12, 2021. The entire contents of the original application are incorporated into this application by reference. Technical Field

[0002] The present invention relates to high performance computing technology, and more particularly to remote direct memory access (RDMA) technology. The present invention relates to transmitting RDMA transactions over a packet-based network. To this end, the present invention provides a device, method, and packet format for RDMA. Background Art

[0003] Applications often require computers to be able to access memory that resides on a remote computer. RDMA enables computers to perform such memory access operations without involving the operating system running on the computer. RDMA is widely used in modern data centers and computer clusters because it provides low-latency remote operations and high network bandwidth. RDMA offloads memory operations from the central processing unit (CPU) to an RDMA network interface card (NIC) (RNIC), which can directly access the memory.

[0004] There are two common RDMA technologies: one is defined in the InfiniBand specification and the other is defined in the Internet Engineering Task Force (IETF). The IETF protocol is also known as the Internet wide area RDMA protocol (iWARP). Specifically, InfiniBand RDMA has two variants that allow it to run on IP / Ethernet networks, such as RoCE and RoCEv2.

[0005] Both RDMA technologies define operations as software layer "verbs" which are then translated into work requests that are posted to the RNIC queue. Such work requests are called work queue elements (WQEs). RDMA also defines various transports over which the RDMA verbs are performed. Transports are either reliable or unreliable, where the software is responsible for retransmission of lost packets, and are either connection-oriented or datagram-oriented. Various transports are indicated when a pair of network nodes establish an RDMA communication channel specified by a queue pair (QP). All current RDMA transports provide a software programming model in which operations signal completion in the order (in time) in which they were posted by the software layer.

[0006] Today, RDMA technology does not support reliable transport services that can manage short messages multiplexed from multiple sources or to multiple destinations. In the context of this application, "reliable transport" means guaranteeing that a message is exchanged only once, and "short message" means that the maximum data length of the operation is the network's maximum transfer unit (MTU). Therefore, a new reliable RDMA transport scheme is needed to handle messages of a single packet length. Summary of the invention

[0007] In view of the above, the apparatus and method of the present invention are intended to provide an apparatus and method for RDMA. The goal is to propose a new reliable RDMA transmission for short messages, and in particular, to achieve out-of-order processing of short messages. One purpose is to support the execution of each message without relying on previous or subsequent operations. Another purpose is to provide a scheme for multiplexing RDMA operations from different source upper-layer processes or protocols (ULP) to multiple destinations.

[0008] The above and other objects are achieved by the subject matter claimed in the independent claims. Other implementations are obvious from the dependent claims, the description and the drawings.

[0009] The first aspect of the present invention provides a sending device for RDMA. The sending device is used to: maintain a sending queue, wherein the sending queue includes one or more WQEs, wherein each WQE includes the destination information of the WQE, wherein the destination information includes at least one of the following items: an address of a receiving device, an identifier of the receiving device, and an identifier of a target application of the receiving device; assign a transaction identifier (XID) to each of the one or more WQEs, and convert each WQE into a corresponding data packet, wherein each corresponding data packet includes the XID of the corresponding WQE and is associated with a packet sequence number (PSN); send each data packet to the receiving device of the destination information of the WQE; receive a notification message of each sent data packet from the receiving device, wherein the notification message indicates whether the sent data packet is received at the receiving device; determine whether the generation of each WQE is completed based on the notification message, the information carried in the WQE, and the information held by the sending device.

[0010] For example, a WQE is an RDMA operation or transaction that is published or issued by a source ULP (e.g., an application or software) and pushed into a QP. A WQE is sent to a peer or received from a peer. A QP represents a communication endpoint and consists of a send queue (SQ) and a receive queue (RQ) or a shared receive queue (SRQ). According to the present invention, a scheme is provided that supports independent processing of each WQE or each message, that is, each WQE or each message is executed independently of previous or subsequent operations. Each WQE includes its own destination information (e.g., target ID, destination IP address, etc.), thereby enabling the sending device of the first aspect to process multiple consecutive WQEs, each of which can be sent to a different target. Each WQE is associated with a unique XID, which can be used by the receiving end to detect duplicate operations and ensure that messages are exchanged only once.

[0011] The sending device may decide whether to signal WQE completion based on certain conditions. Traditionally, RDMA QPs always generate completions on certain opcodes (RQ processing, etc.). No control is performed to operate conditional completions. According to the present invention, completions may be configurable.

[0012] In an implementation of the first aspect, each WQE carries only messages that fit within a single network maximum transmission unit.

[0013] In particular, the present invention proposes a new reliable RDMA transmission scheme for processing messages up to the length of a single data packet, ie, short messages.

[0014] In an implementation of the first aspect, when the WQE corresponding to the corresponding data packet includes a write operation request or a send operation request, and the notification message indicates that the corresponding data packet is received at the receiving device, the sending device is further used to determine that the WQE is successfully completed.

[0015] RDMA operations include reading, writing, sending, receiving, atomic operations, etc. For a write operation or a send operation, for example, once the receiving device confirms receipt of the corresponding data packet, the sending device considers that the write operation or the send operation is successfully completed.

[0016] In an implementation of the first aspect, when the WQE corresponding to the corresponding data packet includes a read operation request or an atomic operation request, the sending device is further used to: receive a response data packet indicating that the corresponding data packet has been processed by the receiving device as an atomic operation response or a read operation response to the atomic operation request or the read operation request, wherein the response data packet includes the XID of the response WQE and the XID of the WQE including the read operation request or the atomic operation request; and determine that the WQE is successfully completed.

[0017] For example, for a read operation or an atomic operation, the sending device also expects a read operation response or an atomic operation response from the receiving device. That is, when the receiving device confirms receipt of the corresponding data packet, the read / atomic WQE is not completed, but is only completed when the sending device receives a suitable response data packet. In this case, the receiving device operates accordingly to provide a response data packet carrying the required data. For example, the read operation response includes the data that the sending device intends to obtain from the receiving device. According to the present invention, the response data packet carries the XID of the response WQE converted into the response data packet, and the XID of the WQE corresponding to the operation request. It should be noted that, in this case, the receiving device can also be considered as a "sending device" that sends a data packet carrying a WQE.

[0018] In an implementation of the first aspect, a maximum acceptable number of receive timeouts associated with the receiving device is obtained; when the number of receive timeouts exceeds the maximum acceptable number of receive timeouts, it is determined that the WQE corresponding to the corresponding data packet is completed but with errors.

[0019] Optionally, the service transmission according to the present invention includes a threshold for signaling a network-related error condition. For example, the receive timeout may also be named a retransmission timeout. For example, when a data packet (carrying the same WQE) is retransmitted too many times, for example, exceeding the maximum acceptable number of retransmissions, the sending device stops retransmissions and determines that the WQE is completed with errors.

[0020] In an implementation of the first aspect, the notification message of each sent data packet includes a receiver not ready (RNR) indication, wherein the RNR indication indicates that the receiving device cannot process the sent data packet, wherein the sending device is also used to: determine the XID included in the sent data packet; and postpone sending another data packet including the same XID before a timeout value expires.

[0021] The present invention proposes an RNR signal for delaying the transmission of other messages. In particular, when a received data packet carries a message that the receiving device cannot process temporarily, the sending device receives an RNR indication. The RNR signal only affects the specific message on the sending device, avoiding any "head-of-line" (HoL) blocking.

[0022] In an implementation of the first aspect, the WQE also includes a parameter indicating a maximum acceptable number of RNR indications, wherein the sending device is further used to: receive another notification message including an RNR indication of a retransmitted data packet; when the number of received RNR indications exceeds the maximum acceptable number of RNR indications, determine that the WQE corresponding to the retransmitted data packet is completed but with errors.

[0023] Therefore, the sending device is implicitly informed of the WQE completion error.

[0024] In an implementation manner of the first aspect, the notification message includes multiple RNR indications, each RNR indication indicating that the receiving device cannot process a specific sent data packet.

[0025] It is possible that there may be more than one message that the receiving device is temporarily unable to process, for example if the receiving device has no available resources.

[0026] In an implementation manner of the first aspect, the RNR indication also indicates the timeout value.

[0027] It is possible that only a limited number of RNR timeout values ​​are defined to keep the overall completion latency low and to reduce the number of timers required.

[0028] In an implementation of the first aspect, each RNR indication further indicates that the receiving device is unable to process a specific transmitted data packet within a specific timeout value.

[0029] Possibly, for different messages different timeout values ​​may be set.

[0030] In an implementation manner of the first aspect, the sending device is further configured to manage the WQE of the data packet sent corresponding to the received RNR indication based on the timeout value indicated by the RNR indication.

[0031] Optionally, the sending device may group delayed WQEs in the RNR "state" according to the received RNR timeout value. According to the present invention, when a WQE is in the RNR state, this indicates that an RNR indication for the WQE is received. WQEs in the RNR "state" may be managed together by the sending device.

[0032] In an implementation of the first aspect, the number of supported timeout values ​​is configurable and agreed upon on all communication devices.

[0033] In an implementation manner of the first aspect, the one or more WQEs of the sending queue include a first WQE and a second WQE, wherein the first WQE and the second WQE include different destination information.

[0034] According to the present invention, a solution is proposed to manage WQEs towards different destination nodes (eg, different receiving nodes).

[0035] In an implementation of the first aspect, one or more WQEs of the sending queue include a WQE set, and the WQE set includes the same destination information, wherein the sending device is further used to: group the data packets corresponding to the WQE set into a data packet stream; and send the data packet stream to the receiving device of the WQE set.

[0036] For example, each WQE includes its own destination information. For WQEs that share the same destination node, data packets corresponding to such WQEs can be grouped into data packet flows and managed together by the sending device.

[0037] In an implementation manner of the first aspect, each WQE is further associated with a priority sent from the sending device.

[0038] The sending device can configure the priority of the WQE to be sent. A WQE with a higher priority can be processed before a WQE with a lower priority.

[0039] In an implementation manner of the first aspect, the sending device is further configured to obtain the one or more WQEs from at least one ULP.

[0040] It is possible that a message or WQE processed by a sending device may be issued or originated by a different source ULP.

[0041] In an implementation manner of the first aspect, the WQE set including the same destination information is obtained from different ULPs.

[0042] For example, when WQEs obtained from different source ULPs share the same destination, packets corresponding to such WQEs can be grouped into a packet stream and sent together. In this way, WQEs from different sources are multiplexed.

[0043] In an implementation of the first aspect, XIDs of WQEs obtained from different ULPs are allocated from the same XID pool, and PSNs of data packets associated with the WQEs obtained from different ULPs are allocated from the same PSN pool.

[0044] Optionally, the sending device may maintain consecutive number spaces for the XID and PSN respectively.

[0045] In an implementation manner of the first aspect, the priority of each WQE is determined based on the ULP from which the WQE is obtained.

[0046] For example, WQEs or tasks issued from certain ULPs may have a higher priority than other WQEs.

[0047] In an implementation manner of the first aspect, each data packet is further associated with a flow block ID identifying a flow, wherein the flow includes a plurality of data packets, and the data packet corresponding to the WQE further includes the flow block ID.

[0048] According to the present invention, an equal cost multiple path (ECMP) flow block can also be applied. The ECMP flow block is supported (relatively easily) by dividing the packet flow between the initiator node and the target node into sub-flows. It is assumed that the packets of different sub-flows use different routes to traverse the network.

[0049] In an implementation manner of the first aspect, the sending device is further configured to send each data packet to the receiving device through the flow block, wherein the flow block is associated with the receiving device.

[0050] For each transmitted packet associated with a particular flow block ID, it is routed through the flow identifiable by that flow block ID.

[0051] A second aspect of the present invention provides a receiving device for RDMA, wherein the receiving device is used to: receive one or more data packets from a sending device for RDMA, wherein each of the one or more data packets is associated with a PSN and includes an XID of a corresponding WQE, wherein each WQE includes destination information of the WQE, wherein the destination information includes at least one of the following: an address of the receiving device, an identifier of the receiving device, and an identifier of a target application of the receiving device; and send a notification message to the sending device, wherein the notification message indicates that the one or more data packets are received at the receiving device.

[0052] According to the present invention, there is provided a receiving device, which operates accordingly to the sending device of the first aspect. The receiving device proposed in the present invention processes each received data packet independently. That is, each received data packet can be safely executed when received, regardless of the state of any previous data packet.

[0053] In an implementation of the second aspect, when the WQE corresponding to the corresponding data packet includes a write operation request or a send operation request, the receiving device is further used to: place the data carried on the write operation request into a destination memory location in the receiving device; or place the data carried on the send operation request into a buffer from the receiving queue of the receiving device.

[0054] For example, for a write operation or a send operation, the receiving device places the data carried in the data packet accordingly.

[0055] In an implementation of the second aspect, when the WQE corresponding to the corresponding data packet includes a read operation request or an atomic operation request, the receiving device is further used to: send a response data packet to the sending device indicating that the corresponding data packet has been processed by the receiving device.

[0056] For example, for a read operation or an atomic operation, the receiving device operates accordingly to provide a response data packet carrying the required data. For example, a read operation response includes the data that the sending device intends to obtain from the receiving device.

[0057] In an implementation of the second aspect, the receiving device is also used to: create a response WQE based on the read operation request or the atomic operation request; assign an XID to the response WQE, and convert the response WQE into the response data packet, wherein the response data packet includes the XID of the response WQE and the XID of the WQE including the read operation request or the atomic operation request.

[0058] Accordingly, the receiving device generates a response WQE for the read operation or atomic operation request. It should be noted that in this case, the receiving device can be regarded as the "sending device" described in the first aspect, which assigns an XID to the response WQE, converts the WQE into a data packet, and further sends the data packet. The difference between the response data packet and the "normal" sent data packet is that the response data packet also carries the XID of the WQE corresponding to the operation request (such as the sent data packet). That is, the response data packet carries two XIDs.

[0059] In an implementation of the second aspect, the receiving device is also used to: generate an RNR indication of the received data packet when the receiving device cannot process the corresponding data packet, wherein the notification message of the corresponding data packet also includes the RNR indication indicating to the sending device that the receiving device cannot process the corresponding data packet.

[0060] As mentioned above, the present invention proposes an RNR signal for delaying the transmission of other messages. In particular, the receiving device is temporarily unable to process a received data packet carrying a message, for example, if the receiving device has no available resources, then the RNR indication is sent. The RNR signal only affects the specific message at the sending device, avoiding any HoL blocking.

[0061] In an implementation manner of the second aspect, the RNR indication further indicates a preset timeout value, wherein the RNR indication indicates that the receiving device cannot process the corresponding data packet within the preset timeout value.

[0062] In an implementation manner of the second aspect, the notification message includes multiple RNR indications, each RNR indication indicating that the receiving device cannot process a specific sent data packet.

[0063] In an implementation of the second aspect, the one or more data packets include a first data packet and a second data packet, wherein the PSN of the first data packet is less than the PSN of the second data packet, and wherein the receiving device is further used to: send the notification message of the first data packet and the notification message of the second data packet in random order; and / or perform random processing on the first data packet and the second data packet; and send the response message of the first data packet and the response message of the second data packet to the sending device in random order.

[0064] The embodiment of the present invention proposes a new solution, which enables the receiving device to execute the received data packets out of order. Therefore, the acknowledgment (ACK) of the received data packet, such as a notification message, can also be sent out of order. Similarly, the response can also be sent out of order.

[0065] In an implementation manner of the second aspect, the receiving device is further configured to maintain a database storing XIDs of WQEs received at the receiving device.

[0066] The receiving device may record the XID carried in the received data packet.

[0067] In an implementation of the second aspect, the receiving device is further used to: determine whether each received data packet is a duplicate data packet by checking whether the XID of the WQE included in the received data packet has been stored in the database; and ignore any duplicate data packets.

[0068] To avoid executing the same WQE more than once, the receiving device checks the XID stored in the database. When the XID carried in the newly received packet is already stored in the database, the receiving device ignores the packet instead of executing it. In this way, it can be guaranteed that the message will be executed only once.

[0069] In an implementation of the second aspect, the database further includes a status of each WQE indicating whether the WQE has been processed.

[0070] The third aspect of the present invention provides a method for RDMA. The method includes: maintaining a sending queue, wherein the sending queue includes one or more WQEs, wherein each WQE includes the destination information of the WQE, wherein the destination information includes at least one of the following: an address of a receiving device, an identifier of the receiving device, and an identifier of a target application of the receiving device; assigning an XID to each of the one or more WQEs, and converting each WQE into a data packet, wherein each data packet includes the XID of the corresponding WQE and is associated with a PSN; sending each data packet to the receiving device of the destination information of the WQE; receiving a notification message of each sent data packet from the receiving device, wherein the notification message indicates whether the sent data packet is received at the receiving device; determining whether the generation of each WQE is completed based on the notification message, the information carried in the WQE, and the information held by the sending device.

[0071] The method of the third aspect and its implementation method provides the same advantages and effects as described above for the sending device of the first aspect and its corresponding implementation method.

[0072] A fourth aspect of the present invention provides a method for RDMA. The method comprises: receiving one or more data packets from a sending device for RDMA, wherein each of the one or more data packets is associated with a PSN and includes an XID of a corresponding WQE, wherein each WQE includes destination information of the WQE, wherein the destination information includes at least one of the following: an address of the receiving device, an identifier of the receiving device, and an identifier of a target application of the receiving device; and sending a notification message to the sending device, wherein the notification message indicates that the one or more data packets are received and / or processed at the receiving device.

[0073] The method of the fourth aspect and its implementation method provides the same advantages and effects as described above for the receiving device of the second aspect and its corresponding implementation method.

[0074] A fifth aspect of the present invention provides a computer program comprising program code for executing the method of any one of the third aspect and its implementations or any one of the fourth aspect and its implementations when implemented on a processor.

[0075] It should be noted that all devices, elements, units and modules described in this application can be implemented in software or hardware elements or any combination thereof. All steps performed by various entities described in this application and the functions to be performed by various entities described are intended to refer to the corresponding sending devices for performing the corresponding steps and functions. Although in the description of the following specific embodiments, the specific functions or steps performed by the external entities are not reflected in the description of the specific detailed elements of the sending devices that perform the specific steps or functions, it should be clear to the technician that these methods and functions can be implemented by corresponding hardware or software elements or any combination thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In conjunction with the accompanying drawings, the following description of specific embodiments will illustrate the above aspects and their implementation methods, wherein:

[0077] Figure 1 A transmitting device is shown;

[0078] Figure 2 A schematic diagram is shown;

[0079] Figure 3 shows a signaling flow chart;

[0080] Figure 4 shows a signaling flow chart;

[0081] Figure 5 shows a flow chart of the WQE state machine;

[0082] Figure 6shows a flow chart of the WQE state machine;

[0083] Figure 7 A schematic diagram is shown;

[0084] Figure 8 A receiving device is shown;

[0085] Fig. 9 One method is shown.

[0086] Fig.10 Another approach is shown. DETAILED DESCRIPTION

[0087] Exemplary embodiments of methods, devices, and program products for short message processing in an RDMA system are described in conjunction with the accompanying drawings. Although the description provides detailed examples of possible implementations, it should be noted that these details are intended to be exemplary and will not limit the scope of the present application.

[0088] In addition, one embodiment / example may refer to other multiple embodiments / examples. For example, any description mentioned in one embodiment / example, including but not limited to terms, elements, processes, explanations and / or technical advantages, is applicable to other multiple embodiments / examples.

[0089] Figure 1A sending device 100 suitable for RDMA provided by an embodiment of the present invention is shown. The sending device 100 may include a processing circuit (not shown) for executing, implementing or starting various operations of the sending device 100 described herein. The processing circuit may include hardware and software. The hardware may include an analog circuit or a digital circuit, or both an analog circuit and a digital circuit. The digital circuit may include components such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP) or a multi-purpose processor. The sending device 100 may also include a memory circuit that stores one or more instructions that can be executed by a processor or a processing circuit, especially under the control of software. For example, the memory circuit may include a non-transitory storage medium that stores executable software code, and when the executable software code is executed by a processor or a processing circuit, the sending device 100 performs various operations. In one embodiment, the processing circuit includes one or more processors and a non-transitory memory connected to the one or more processors. The non-transitory memory may carry executable program code that, when executed by one or more processors, causes the sending device 100 to perform, implement, or initiate the operations or methods described herein.

[0090] In particular, the sending device 100 is used to maintain a sending queue 101, wherein the sending queue 101 includes one or more WQEs 102, 102'. In particular, each WQE 102, 102' includes destination information of the WQE 102, 102'. The destination information includes at least one of the following: an address of the receiving device 200, an identifier of the receiving device (200, 200'), and an identifier of a target application of the receiving device 200, 200'. The sending device 100 is also used to assign an XID to each of the one or more WQEs 102, 102', and convert each WQE 102, 102' into a corresponding data packet 103, 103'. Each corresponding data packet 103, 103' includes the XID of the corresponding WQE 102, 102' and is associated with a PSN. Then, the sending device 100 is used to send each data packet 103, 103' to the receiving device (200, 200') of the destination information of the WQE 102, 102'. The sending device 100 is also used to receive a notification message 201 for each sent data packet 103 from the receiving device 200, wherein the notification message 201 indicates whether the sent data packet 103 is received at the receiving device 200. In addition, the sending device 100 is used to determine whether the generation is completed for each WQE 102 based on the notification message 201, the information carried in the WQE 102, and the information held by the sending device 100.

[0091] Embodiments of the present invention introduce a new approach where each WQE (each RDMA operation) or each message can be processed independently, e.g., executed without dependency on previous or subsequent operations. Each WQE includes all the information required to allow the operation to be performed, specifically, an indication of the destination node (e.g., receiving device 200) and the process (ID, address, etc.).

[0092] For example, an RDMA transaction involves an initiator node and a destination target node (or target node). The initiator node initiates or sends an RDMA operation request, and the target node receives the RDMA operation request and responds accordingly. Figure 1 The sending device 100 shown can be regarded as an initiator node. Figure 1 The receiving device 200 shown may be considered as a target node.

[0093] According to an embodiment of the present invention, each WQE is associated with a unique XID, which can be used to detect duplicate operations at the receiving end. Each transmitted data packet is associated with a unique PSN. It should be noted that each data packet needs to be confirmed as received. The initiator node uses these ACK signals to detect lost data packets and retransmits them, for example, by applying a selective repeat retransmission scheme.

[0094] Traditionally, RDMA QPs always generate completions on certain opcodes (RQ processing, etc.). No control is performed to operate conditional completions. According to an embodiment of the present invention, completions may be configurable. That is, the sending device 100 may decide whether to signal the completion of a WQE based on certain circumstances (e.g., information held by the sending device 100).

[0095] For example, the transmitting device 100 may obtain information from a ULP that sets the transmitting device 100 to obtain a completion signal after every X completed WQEs, so that the ULP checks the completion queue event (CQE) of the completion queue (CQ) at a lower frequency rather than for each individual WQE. In another example, the transmitting device 100 may be instructed to generate a completion when there are no more WQEs to be processed by a node-to-node context (NNC). Optionally, the transmitting device 100 may also send a completion signal for each high priority WQE and only send a completion signal for a group of low priority WQEs. There are many implementations in this regard. These examples are not intended to limit the present invention.

[0096] The new service method described in the present invention may be referred to as short-lived datagram (SLD). As mentioned above, the transmission type may be reliable or unreliable, and connection-oriented or datagram-oriented. The SLD described in the present invention is applicable to most transmissions, such as reliable connection (RC), reliable datagram (RD) and unreliable datagram (UD), such as Figure 2 That is, the present invention discloses a new service between traditional RD, RC and UD transmissions.

[0097] It should be noted that the new RDMA transmission scheme proposed in the embodiment of the present invention is used to process short messages, for example, messages as long as a single data packet. According to the embodiment of the present invention, each WQE 102, 102' carries messages that are limited to a single network maximum transmission unit.

[0098] Typically, RDMA operations include read, write, send, receive, atomic operations, etc.

[0099] Figure 3 FIG. 1 shows a signaling flow chart between an initiator node (eg, a sending device 100) and a target node (eg, a receiving device 200) provided by an embodiment of the present invention. The sending device 100 may be as follows: Figure 1The sending device shown in FIG. 200 may be a receiving device 200 as shown in FIG. Figure 1 In particular, Figure 3 The exchange of data packets for a write operation and a send operation is shown.

[0100] In this embodiment, a transmit (Tx) WQE from a source send queue (SSQ) (e.g., the transmit queue 102) is assigned an XID W. Figure 1 The WQE 102 shown is converted into a write request carrying XID W, for example, Figure 1 After the sending device 100 sends the data packet 103, an ACK is received at the sending device 100. Accordingly, the sending device 100 can determine that the WQE with XID W is successfully completed. After receiving the write request, the receiving device 200 can put the data carried in the write request into the destination memory location in the receiving device 200.

[0101] Hereinafter, another WQE with XID S is converted into a send request carrying XID S. Accordingly, the receiving device 200 may put the data carried on the send request into the buffer from the RQ of the receiving device 200. After receiving the ACK of the send request from the receiving device 100, the sending device 100 may determine that the WQE with XID S is successfully completed. It should be noted that the WQE may send a completion signal immediately after receiving the ACK of the WQE without waiting for the previous WQE to be completed.

[0102] According to an embodiment of the present invention, when the WQE 102 corresponding to the corresponding data packet 103 includes a write operation request or a send operation request, and the notification message 201 indicates that the corresponding data packet 103 is received at the receiving device 200, the sending device 100 is further configured to determine that the WQE 102 is successfully completed.

[0103] Figure 4 FIG. 2 shows another signaling flow chart between an initiator node (eg, a sending device 100) and a target node (eg, a receiving device 200) provided by an embodiment of the present invention. The sending device 100 may be as follows: Figure 1 The sending device shown in FIG. 200 may be a receiving device 200 as shown in FIG. Figure 1 In particular, Figure 4 The exchange of data packets for a read operation or an atomic operation is shown.

[0104] In this embodiment, XID R is assigned to the Tx WQE. Figure 1 The WQE 102 shown is converted into a read request carrying, for example, an XID R, such as Figure 1 The data packet 103 is shown. After the sending device 100 sends the data packet 103, an ACK is received at the sending device 100.

[0105] For example, for a read operation, the sending device 100 also expects a read operation response from the receiving device 200. That is, the read WQE is not completed when the receiving device 200 acknowledges receipt of the corresponding data packet, but is completed only when the sending device 100 receives a suitable response data packet. In this case, the receiving device 200 internally creates a response WQE for the WQE 102.

[0106] For example, the receiving device 200 intends to send a response WQE back to the sending device. In this case, it can be understood that the roles of the "sending device" and the "receiving device" are switched. That is, the receiving device 200 here can also be considered as a "sending device" that sends a data packet carrying a WQE, and the WQE is a response WQE. According to an embodiment of the present invention, the receiving device 200 generates a response WQE for the read operation request and assigns another XID to the response WQE, for example, Figure 4 The receiving device 200 also converts the response WQE into a response data packet 202 and sends the response data packet 202 to the sending device 100 .

[0107] It is worth mentioning that the difference between the response packet 202 and the "normal" sent packet 103 from the sending device 100 is that the response packet 202 also carries the original XID (e.g., XID R) of the WQE corresponding to the operation request (e.g., the sent packet 103). That is, the response packet 202 carries two XIDs (e.g., XID R and XID Q).

[0108] In response to receiving the response data packet 202, the sending device 100 should send an ACK to inform the receiving device 200 that the response data packet 202 has been received. For example, in this case, the sending device 100 behaves here like a "receiving device".

[0109] According to the embodiment of the present invention, when the Tx WQE carries an atomic operation request, the data packet exchange between the transmitting device 100 and the receiving device 200 is similar to the case of a read operation, which will not be described in detail here.

[0110] According to an embodiment of the present invention, when the WQE 102 corresponding to the corresponding data packet 103 includes a read operation request or an atomic operation request, the sending device 100 is further configured to receive, as an atomic operation response or a read operation response to the atomic operation request or the read operation request, a response data packet 202 indicating that the corresponding data packet 103 has been processed by the receiving device 200. In particular, the response data packet 202 includes the XID of the responding WQE 102 and the XID of the WQE 102 including the read operation request or the atomic operation request. In addition, the sending device 100 is configured to determine that the WQE 102 is successfully completed.

[0111] Figure 5 A diagram of a state machine controlling the processing of each WQE at an initiator node (eg, a transmitting device 100) is shown. Each WQE is processed independently of all other WQEs. In particular, even for WQEs issued by the same source ULP, they are still processed independently.

[0112] As can be seen from the figure, once a WQE (eg, WQE Tx) is sent, an ACK is expected.

[0113] In the case of receiving an ACK, a database (e.g., a bitmap) storing the status of the transmitted data packets can be updated accordingly. If data packet loss is detected, a fast retransmit mechanism can be applied to retransmit the lost data packet. For the retransmitted data packet, an ACK is also expected.

[0114] In the case where no ACK is received, the sending device may wait for the retransmission timeout to expire before retransmitting the data packet. For example, the maximum acceptable number of receive timeouts "RTOMAXN" may be configured for the sending device. If the value of "RTOMAXN" is greater than 0, the value of "RTOMAXN" is reduced (e.g., minus 1) each time the retransmission timeout expires without receiving an ACK. Once the value reaches 0, the sending device 100 determines that the WQE is completed with errors.

[0115] According to an embodiment of the present invention, the sending device 100 may be configured to obtain a maximum acceptable receive timeout number associated with the receiving device 200. When the receive timeout number exceeds the maximum acceptable receive timeout number, the sending device 100 may be configured to determine that the WQE 102 corresponding to the corresponding data packet 103 is completed with errors.

[0116] The present invention also proposes an RNR signal for delaying the transmission of other messages. In particular, a sending device receives an RNR indication when a received data packet carries a message that the receiving device is temporarily unable to process (e.g., if the receiving device has no available resources). For example, the performance limitation phenomenon that occurs when a row of data packets is blocked by the first data packet is called HoL blocking. The RNR signal only affects the specific message at the sending device, avoiding any HoL blocking.

[0117] It is possible that an RNR indication (or a notification message including an RNR indication) indicating that the receiving device 200 is temporarily unable to process the transmitted data packet may be received instead of an ACK. Then, a counter of the maximum acceptable number of RNR indications "RNRMAXN" is checked. If the counter reaches 0, the transmitting device 100 determines that the WQE is completed with errors. If the counter has not reached 0, the counter is decreased (e.g., minus 1) for each received RNR indication, and once the counter reaches 0, the transmitting device 100 determines that the WQE is completed with errors.

[0118] According to an embodiment of the present invention, Figure 1 The notification message 201 shown may include an RNR indication, wherein the RNR indication indicates that the receiving device 200 cannot process the sent data packet. The sending device 100 is used to determine the XID included in the sent data packet 103. The sending device 100 is also used to postpone sending another data packet containing the same XID before the timeout value expires.

[0119] After the timeout expires, the sending device 100 retransmits the data packet including the same XID. The receiving device 200 may still be unable to process the sent data packet. In this case, another RNR indication for the retransmitted data packet may be sent from the receiving device 200. According to an embodiment of the present invention, the WQE 102 may also include a parameter indicating the maximum acceptable number of RNR indications, i.e., Figure 5 The "RNRMAXN" shown. It is possible that the sending device 100 can receive another notification message including the RNR indication of the retransmitted data packet. The sending device 100 can also be used to: when the number of received RNR indications exceeds "RNRMAXN", determine that the WQE 102 corresponding to the retransmitted data packet is completed but with errors.

[0120] According to an embodiment of the present invention, the notification message 201 may include a plurality of RNR indications, each RNR indication indicating that the receiving device 200 is unable to process a particular transmitted data packet. For example, a timeout value may be indicated by the RNR indication.

[0121] Optionally, each RNR indication also indicates that the receiving device 200 was unable to process a particular transmitted data packet within a particular timeout value.

[0122] According to an embodiment of the present invention, the sending device 100 may also be configured to manage the WQE 102, 102' of the sent data packet 103, 103' corresponding to the received RNR indication based on the timeout value indicated by the RNR indication.

[0123] Optionally, the sending device 100 can group delayed WQEs in the RNR "state" according to the received RNR timeout value. According to the present invention, when a WQE is in the RNR state, this indicates that an RNR indication of the WQE is received. The WQEs in the RNR "state" can be managed together by the sending device 100 (e.g., the initiator node) rather than by each NNC. Typically, only a few limited number of RNR timeout values ​​are defined to maintain an overall low completion delay and reduce the number of required timers. The initiator node groups the delayed WQEs in the RNR "state" according to the received RNR timeout value, which can be set by the target node (e.g., the receiving device 200).

[0124] According to an embodiment of the present invention, the number of supported timeout values ​​is configurable and agreed upon across all communicating devices.

[0125] Optionally, the receiving device 200 may be configured to set the timeout value based on the current condition of the receiving device 200. For example, the receiving device 200 may adjust the timeout value based on its current load, the number of connected NNCs, the number of remaining RQ elements (inside RQ).

[0126] Figure 6 A diagram of a read / atomic WQE state machine provided by an embodiment of the present invention is shown. Figure 6 is based on Figure 5 ,and Figure 5 A more general case of all RDMA operations is shown.

[0127] Similarly, with Figure 5 Similarly, once a WQE (e.g., WQE Tx) is sent, an ACK is expected. In the event that an ACK is not received, or in the event that the transmitting device 100 receives an RNR indication indicating that the receiving device 200 is temporarily unable to process the transmitted data packet instead of an ACK, the operation of the transmitting device 100 is the same as Figure 5 The description is similar to that in . It will not be repeated here.

[0128] In the case of receiving an ACK, a database (e.g., a bitmap) storing the status of the sent data packets can be updated accordingly. If data packet loss is detected, a fast retransmit mechanism can be applied to retransmit the lost data packet. For the retransmitted data packet, an ACK is also expected. This part is also similar to Figure 5 The content described in .

[0129] As mentioned previously, read and / or atomic "verb" operations require the target node to send data back to the initiator node. That is, the read / atomic WQE is not completed when an ACK is received for the request packet of the read / atomic WQE, but only when the initiator node receives the appropriate response packet.

[0130] That is, upon receiving a read and / or atomic request packet, the target node receiving device 200 stores an internal response WQE for managing the transmission of the response packet, and possible retransmission. This internal attribute of the WQE means that it is "hidden" from the upper layer software, the ULP does not issue the WQE, and the completion event / signal is not reported to any upper layer software at the target node.

[0131] In the case where the receiving device 200 has no available resources to store the internal response WQE, an RNR signal data packet is generated and sent back to the transmitting device 100 .

[0132] According to an embodiment of the present invention, the target node (e.g., receiving device 200) processes the internal response WQE as if it were issued by the ULP. The same state machine rules and conditions apply in this case, for example, retransmitting the response packet until an ACK is received or until an error condition is identified (but this error will not be reported to any ULP).

[0133] According to an embodiment of the present invention, one or more WQEs 102, 102' of the send queue 101 include a first WQE and a second WQE, wherein the first WQE and the second WQE include different destination information.

[0134] As mentioned above, each WQE includes its own destination information (eg, target ID, destination IP address, etc.) to enable a source ULP to issue multiple consecutive WQEs, each of which can be sent to different target ULPs and network nodes using the same QP class data structure.

[0135] According to an embodiment of the present invention, one or more WQEs 102, 102' of the sending queue 101 include a WQE set, which includes the same destination information, wherein the sending device 100 is further used to: group data packets corresponding to the WQE set into a data packet flow; and send the data packet flow to the receiving device 200 of the WQE set.

[0136] Various WQEs can be grouped by their destination into a packet flow that is sent to its destination target node. The management of this packet flow is done by using NNC data structures. A single NNC manages the WQEs between an initiator-target node pair.

[0137] According to an embodiment of the present invention, each WQE 102, 102' is also associated with a priority sent from the sending device 100. Optionally, the priority of each WQE 102, 102' can be determined based on the ULP from which the WQE (102, 102') is obtained. For example, if the SLD QP created by the ULP is associated with a higher priority, the WQE extracted from the SQ will be associated with a higher priority. Therefore, such WQEs will be processed before processing WQEs from other ULP SQs, which may have been created with a lower priority.

[0138] According to an embodiment of the present invention, the transmitting device 100 may also be configured to obtain one or more WQEs 102, 102' from at least one ULP.

[0139] According to an embodiment of the present invention, a set of WQEs including the same destination information is obtained from different ULPs. That is, various WQEs from different source ULPs can be grouped into a data packet flow according to their destinations.

[0140] According to an embodiment of the present invention, XIDs of WQEs obtained from different ULPs are allocated from the same XID pool, and PSNs of data packets associated with WQEs obtained from different ULPs are allocated from the same PSN pool.

[0141] Each WQE is assigned a unique XID from a continuous number space used for all WQEs that share the same destination target node. Each Tx packet is assigned a unique PSN from a continuous number space used for all packets that are sent to the same destination target node.

[0142] According to an embodiment of the present invention, the node-to-node context (NNC) can be divided into two operation parts: NNC-Tx used by the initiator function and NNC-Rx used by the target function. NNC-Tx maintains different continuous digital spaces for XID and PSN. NNC-Rx processes the received ACK and RNR indications to detect lost data packets and triggers NNC-Tx to retransmit these data packets. It should also be noted that the expiration of the timer will also result in the detection of lost data packets, which will trigger retransmission.

[0143] Figure 7An exemplary schematic diagram of an SLD WQE distributor provided by an embodiment of the present invention is shown. NNC-Rx can store the "received XID" state and identify repeated WQEs, for example, using a PSN database and an XID database. These repetitions are only acknowledged but not executed again. This is to ensure that the WQE is only executed once. NNC-Rx constructs an ACK based on the PSN to indicate received and lost data packets. NNC-Rx constructs an RNR indication to indicate multiple RNR signals for multiple data packets.

[0144] Furthermore, the NNC may include thresholds for signaling network related error conditions, such as the period of receive / retransmit timers, the maximum number of timer expiration (not receiving any ACK / RNR packets).

[0145] The NNC includes congestion control (CC) related information and parameters, which control the transmission of data packets managed by the NNC.

[0146] It is worth mentioning that the "Rx XID" state database of the NNC-Rx can be a shared resource pool from which any NNC-Rx can allocate the required resources as needed. Alternatively, a single RTO timer can be used for all NNC-Tx instances. The memory used to store WQEs can be part of the host's memory, a dedicated memory connected to the RNIC, an internal memory, or any combination thereof. In order to avoid a single ULP consuming too many RNIC resources (for WQEs), thereby making other ULPs "starved of resources" and denying them valid SLD services, a fairness mechanism can be added to enforce a maximum number of outstanding WQEs per ULP.

[0147] According to an embodiment of the present invention, each data packet 103, 103' may also be associated with a stream block ID identifying the stream block. In particular, a stream block includes multiple data packets, and the data packets 103, 103' corresponding to the WQE 102, 102' also include a stream block ID.

[0148] According to an embodiment of the present invention, the sending device 100 may also be configured to send each data packet 103 to the receiving device 200 via a flow, wherein the flow block is associated with the receiving device 200 .

[0149] It is worth mentioning that ECMP flow blocks are supported by the new method proposed in the present invention, and the transmission is designed for flow blocks. ECMP flow blocks are supported (relatively easily) by dividing the data packet flow between the initiator node and the target node into sub-flows. It is assumed that the data packets of different sub-flows use different routes to traverse the network. In order to support ECMP flow blocks, NNC-Tx maintains a separate (continuous) PSN space for each sub-flow (e.g., flow block). Each WQE is assigned a flow block ID, which is also included in the Tx data packet. Each ACK report also includes the flow block ID, and when the ACK is received, the initiator node only triggers the retransmission of the data packets of the flow block. There is decision flexibility, so that the initiator node can dynamically select different flow blocks for retransmission. The decision can be based on any combination of criteria, such as "better" flow blocks with less load, faster speed, less congestion, etc. Embodiments of the present invention also support each NNC having a different number of flow blocks, thereby supporting more efficient use of initiator nodes, such as resources at the sending device 100.

[0150] Figure 8 A receiving device 200 suitable for RDMA provided by an embodiment of the present invention is shown. The receiving device 200 may include a processing circuit (not shown) for executing, implementing or starting various operations of the receiving device 200 described herein. The processing circuit may include hardware and software. The hardware may include an analog circuit or a digital circuit, or both an analog circuit and a digital circuit. The digital circuit may include components such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP) or a multi-purpose processor. The receiving device 200 may also include a memory circuit that stores one or more instructions that can be executed by a processor or a processing circuit, especially under the control of software. For example, the memory circuit may include a non-transitory storage medium that stores executable software code, and when the executable software code is executed by a processor or a processing circuit, the receiving device 200 performs various operations. In one embodiment, the processing circuit includes one or more processors and a non-transitory memory connected to the one or more processors. The non-transitory memory may carry executable program code that, when executed by one or more processors, causes the receiving device 200 to perform, implement, or initiate the operations or methods described herein.

[0151] In particular, the receiving device 200 is used to receive one or more data packets 103 from the sending device 100 for RDMA. It is possible that the sending device 100 here can be Figure 1The sending device 100 is shown. In particular, each of the one or more data packets 103 is associated with a PSN and includes an XID of a corresponding WQE 102. Each WQE 102 includes destination information of the WQE 102, wherein the destination information includes at least one of the following items: an address of a receiving device 200, an identifier of the receiving device 200, and an identifier of a target application of the receiving device 200. In addition, the receiving device 200 is used to send a notification message 201 to the sending device 100, wherein the notification message 201 indicates that one or more data packets are received at the receiving device 200.

[0152] The embodiment of the present invention also provides a receiving device 200, which operates accordingly with respect to the sending device 100 previously described in the present invention. The receiving device 200 proposed in the present invention processes each received data packet 103 independently. That is, each received data packet can be safely executed when received, regardless of the state of any previous data packet. According to the present invention, WQE can be executed out of order and send an out-of-order completion signal immediately after receiving the ACK of the WQE. The order refers to the time order in which the WQE (e.g., by the ULP) is issued to the RNIC.

[0153] It is worth mentioning that ACK supports notification of multiple packets in aggregate, and supports "selective ACK" (SACK) to report received and lost packets. The number of packets reported, such as the size of the ACK data structure, is an implementation parameter.

[0154] According to an embodiment of the present invention, when the WQE 102 corresponding to the corresponding data packet 103 includes a write operation request, the receiving device 200 is further configured to place the data carried in the write operation request into a destination memory location in the receiving device 200. Similarly, when the WQE 102 corresponding to the corresponding data packet 103 includes a send operation request, the receiving device 200 is further configured to place the data carried in the send operation request from the receiving queue of the receiving device 200 into a buffer.

[0155] According to an embodiment of the present invention, when the WQE 102 corresponding to the corresponding data packet 103 includes a read operation request or an atomic operation request, the receiving device 200 is further configured to send a response data packet 202 to the sending device 100 indicating that the corresponding data packet 103 has been processed by the receiving device 200 .

[0156] For example, for a read operation or an atomic operation, the receiving device 200 operates accordingly to provide a response data packet carrying the required data. For example, a read operation response includes the data that the sending device 100 intends to obtain from the receiving device 200.

[0157] According to an embodiment of the present invention, the receiving device 200 may also be used to create a response WQE according to the read operation request or the atomic operation request. The receiving device 200 is also used to assign an XID to the response WQE and convert the response WQE into a response data packet 202. In particular, the response data packet 202 includes the XID of the response WQE and the XID of the WQE 102 including the read operation request or the atomic operation request.

[0158] It should be understood that in this case, the receiving device 200 can be regarded as a "sending device" that assigns an XID to a (response) WQE, converts the WQE into a (response) data packet, and further sends the data packet. The difference between the response data packet and the "normal" sent data packet is that the response data packet also carries the XID of the WQE corresponding to the operation request (e.g., the sent data packet). That is, the response data packet carries two XIDs.

[0159] According to an embodiment of the present invention, the receiving device 200 may also be used to: generate an RNR indication for the received data packet 103 when the receiving device 200 cannot process the corresponding data packet 103. In particular, the notification message 201 of the corresponding data packet 103 also includes an RNR indication indicating to the sending device 100 that the receiving device 200 cannot process the corresponding data packet 103.

[0160] The received message that the receiving device 200 as the target node is temporarily unable to process is delayed by the RNR signal. For example, the RNR signal only affects specific messages at the initiator, thereby avoiding any HoL blocking. The WQEs in the RNR "state" are managed collectively by the initiator node, rather than by each NNC. Only a few limited number of RNR TO values ​​are defined to keep the overall completion delay low and reduce the number of required timers. The initiator node groups the delayed WQEs in the RNR "state" according to the received RNR TO value set by the target node.

[0161] According to an embodiment of the present invention, the RNR indication further indicates a preset timeout value, wherein the RNR indication indicates that the receiving device 200 is unable to process the corresponding data packet 103 within the preset timeout value.

[0162] According to an embodiment of the present invention, the notification message 201 includes a plurality of RNR indications, each RNR indication indicating that the receiving device 200 is unable to process a particular transmitted data packet.

[0163] It is possible that the one or more data packets 103 include a first data packet and a second data packet, wherein the PSN of the first data packet is less than the PSN of the second data packet. According to an embodiment of the present invention, the receiving device 200 may also be used to send a notification message of the first data packet and a notification message of the second data packet out of order. The receiving device 200 is also used to process the first data packet and the second data packet out of order; and send a response message of the first data packet and a response message of the second data packet to the sending device 100 out of order.

[0164] According to an embodiment of the present invention, the receiving device 200 may also be configured to maintain a database storing the XIDs of the WQEs 102 received at the receiving device 200 .

[0165] According to an embodiment of the present invention, the receiving device 200 may also be used to: determine whether each received data packet 103 is a duplicate data packet by checking whether the XID of the WQE 102 included in the received data packet 103 has been stored in the database; and ignore any duplicate data packets. These duplicates are only confirmed but not executed again. This is to ensure that the WQE is executed only once.

[0166] According to an embodiment of the present invention, the database also includes a status for each WQE 102 indicating whether the WQE 102 has been processed.

[0167] In the present invention, SLD is designed for lossy networks, rather than traditional IB's RDMA. This network property means that packet loss events are more likely to occur. SLD adopts a retransmission scheme, which may have two complementary mechanisms: PSN-based and timer-based. PSN-based retransmission can be based on a unique strictly monotonically increasing PSN assigned to each sent data packet, and quickly identify lost data packets when a "gap" is detected in the PSN reported back by the ACK. Timer-based retransmission is similar to the traditional way of triggering a retransmission event: if an ACK is not received within a configurable time period, all outstanding data packets are retransmitted. An outstanding data packet is defined as a data packet that has been sent but for which an ACK has not yet been received.

[0168] In summary, embodiments of the present invention propose a new RDMA service and transport (SLD) between RC, RD and UC services. SLD covers all RDMA operations and has a more efficient retransmission scheme. Embodiments of the present invention can multiplex operations / WQEs from different ULPs and can also be multiplexed to different targets. ECMP flow blocks are also supported.

[0169] Fig. 9 900 for RDMA provided by an embodiment of the present invention. In a specific embodiment of the present invention, the method 900 is as follows: Figure 1 or Figure 8The sending device 100 shown is executed. The method 900 includes step 901: maintaining a sending queue 101, wherein the sending queue 101 includes one or more WQEs 102, 102'. Each WQE 102, 102' includes destination information of the WQE 102, 102', wherein the destination information includes at least one of the following items: an address of the receiving device 200, an identifier of the receiving device 200, and an identifier of a target application of the receiving device 200. The method 900 also includes step 902: allocating an XID to each of the one or more WQEs 102, 102', and converting each WQE 102, 102' into a data packet 103, 103'. In particular, each data packet 103, 103' includes the XID of the corresponding WQE 102, 102' and is associated with a PSN. The method 900 further includes step 903: sending each data packet 103, 103' to the receiving device (200, 200') of the destination information of the WQE 102, 102'. The method 900 further includes step 904: receiving a notification message 201 for each transmitted data packet 103 from the receiving device 200, wherein the notification message 201 indicates whether the transmitted data packet 103 is received at the receiving device 200. In addition, the method 900 includes step 905: determining whether the generation is completed for each WQE 102 based on the notification message 201, the information carried in the WQE 102, and the information held by the transmitting device 100.

[0170] Fig.10 1000 for RDMA provided by an embodiment of the present invention is shown. In a specific embodiment of the present invention, the method 1000 is as follows: Figure 1 or Figure 8 The method 1000 is performed by the receiving device 200 shown. The method 1000 includes step 1001: receiving one or more data packets 103 from the sending device 100 for RDMA. In particular, each data packet 103 of the one or more data packets 103 is associated with a PSN and includes an XID of a corresponding WQE 102. Each WQE 102 includes destination information of the WQE 102, wherein the destination information includes at least one of the following items: an address of the receiving device 200, an identifier of the receiving device 200, and an identifier of a target application of the receiving device 200. The method 1000 also includes step 1002: sending a notification message 201 to the sending device 100, wherein the notification message 201 indicates that one or more data packets are received and / or processed at the receiving device 200.

[0171] The invention has been described in conjunction with various embodiments and implementations as examples. However, those skilled in the art will be able to understand and implement other variations in practicing the claimed invention, based on a study of the drawings, the present invention, and the appended claims. In the claims and the specification, the word "comprising" does not exclude other elements or steps, and "a" or "an" does not exclude a plurality. A single element or other unit may fulfil the functions of several entities or items described in the claims. The enumeration of certain measures in mutually different dependent claims does not indicate that a combination of these measures cannot be used in an advantageous implementation.

[0172] In addition, any method provided by an embodiment of the present invention can be implemented in a computer program having a code module, which, when executed by a processing module, causes the processing module to perform the method steps. The computer program is included in a computer-readable medium of a computer program product. The computer-readable medium can basically include any memory, such as a read-only memory (ROM), a programmable read-only memory (PROM), an erasable PROM (EPROM), a flash memory, an electrically erasable EPROM (EEPROM), or a hard disk drive.

[0173] In addition, the skilled person recognizes that embodiments of the transmitting device 100 or the receiving device 200 include necessary communication capabilities in the form of functions, modules, units, elements, etc. for performing the scheme. Examples of other such modules, units, elements and functions are: processors, memories, buffers, control logic, encoders, decoders, rate matchers, down-rate matchers, mapping units, multipliers, decision units, selection units, switches, interleavers, deinterleavers, modulators, demodulators, inputs, outputs, antennas, amplifiers, receiving units, transmitting units, DSPs, trellis-coded modulation (TCM) encoders, TCM decoders, power supply units, power supply feeds, communication interfaces, communication protocols, etc., which are appropriately arranged together to perform the scheme.

[0174] Specifically, for example, one or more processors of the transmitting device 100 or the receiving device 200 may include a central processing unit (CPU), a processing unit, a processing circuit, a processor, an application specific integrated circuit (ASIC), a microprocessor, or one or more instances of other processing logic that can interpret and execute instructions. The expression "processor" can therefore refer to a processing circuit including multiple processing circuits, such as any, some or all of the items listed above. The processing circuit system can also perform data processing functions for inputting, outputting and processing data, including data buffering and device control functions, such as call processing control, user interface control, etc.

Claims

1. A method for remote direct memory access (RDMA), characterized in that: The method is performed by a device suitable for RDMA, and includes: maintaining a sending queue, the sending queue comprising one or more working queue elements (WQEs), wherein a first WQE comprises destination information, wherein the first WQE is a WQE in the sending queue, and the destination information comprises at least one of the following: an address of a receiving device, an identifier of the receiving device, and an identifier of a target application of the receiving device; Allocating a transaction identifier (XID) to the first WQE; Converting the first WQE into a first data packet, wherein the first data packet includes the XID of the first WQE, and the first data packet is associated with a packet sequence number (PSN); The first data packet is sent to a receiving device according to the destination address information.

2. The method according to claim 1, characterized in that The method further comprises: receiving a notification message of the first transmitted data packet from the receiving device, wherein the notification message indicates whether the first data packet is received at the receiving device; Based on the notification message, the information carried in the first WQE, and the information held by the RDMA-compatible device, it is determined whether the generation of the first WQE is complete.

3. The method according to claim 1 or 2, characterized in that: The first WQE carries only messages that fit within a single network maximum transmission unit.

4. The method according to any one of claims 1 to 3, characterized in that: The determining whether the generation of the first WQE is completed based on the notification message, the information carried in the first WQE, and the information held by the device suitable for RDMA includes: When the first WQE includes a write operation request or a send operation request, and the notification message indicates that the first data packet is received at the receiving device, it is determined that the first WQE is successfully completed.

5. The method according to any one of claims 1 to 3, characterized in that: The determining whether the generation of the first WQE is completed based on the notification message, the information carried in the first WQE, and the information held by the device suitable for RDMA includes: receiving, as an atomic operation response or a read operation response to the atomic operation request or the read operation request, a response data packet indicating that the first data packet has been processed by the receiving device, wherein the response data packet includes an XID of a response first WQE and an XID of the first WQE including the read operation request or the atomic operation request; It is determined that the first WQE completed successfully.

6. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: obtaining a maximum acceptable number of receive timeouts associated with the receiving device; When the number of receive timeouts exceeds the maximum acceptable number of receive timeouts, it is determined that the WQE of the first data packet is completed with errors.

7. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: When the notification message of the first data packet includes a receiver not ready (RNR) indication, wherein the RNR indication indicates that the receiving device cannot process the sent data packet, determining the XID included in the first data packet; Sending another packet including the same XID is deferred until the timeout value expires.

8. The method according to claim 7, characterized in that The method further comprises: When the first WQE further includes a parameter indicating a maximum acceptable number of RNR indications, receiving another notification message including an RNR indication of a retransmitted data packet; When the number of received RNR indications exceeds the maximum acceptable number of RNR indications, it is determined that the first WQE corresponding to the retransmitted data packet is completed but with errors.

9. The method according to any one of claims 1 to 8, characterized in that: The notification message includes a plurality of RNR indications, each RNR indicating that the receiving device is unable to process a specific transmitted data packet.

10. The method according to claim 8 or 9, characterized in that: The RNR indication also indicates the timeout value.

11. The method according to claim 10, characterized in that The method further comprises: The first WQE corresponding to a first data packet sent for which an RNR indication is received is managed based on the timeout value indicated by the RNR indication.

12. The method according to claim 11, characterized in that The number of supported timeout values ​​is configurable and agreed upon across all communicating devices.

13. The method according to any one of claims 1 to 12, characterized in that: The sending queue includes one or more WQE sets, the first WQE set is a WQE set in the sending queue, the first WQE set includes one or more WQEs with the same destination information, and the method further includes: grouping the packets of the first WQE set into a packet flow; The data packet stream is sent to the receiving device associated with the destination information of the WQE set.

14. The method according to any one of claims 1 to 13, characterized in that The WQE set including the same destination information is obtained from different upper layer processes (ULPs).

15. The method according to claim 14, characterized in that The method further comprises: The XIDs of WQEs obtained from different ULPs are allocated from the same XID pool, and the PSNs of data packets associated with the WQEs obtained from different ULPs are allocated from the same PSN pool.

16. The method according to claim 15, characterized in that The priority of the first WQE is determined based on the ULP from which the first WQE is obtained.

17. The method according to any one of claims 1 to 16, characterized in that The first data packet is also associated with a stream block ID identifying a stream block, wherein the stream block includes a plurality of data packets.

18. The method according to claim 17, characterized in that The method further comprises: The first data packet is sent to the receiving device via the flow block, wherein the flow block is associated with the receiving device.

19. The method according to any one of claims 1 to 18, characterized in that The method further comprises: Receiving the first data packet from another device suitable for RDMA, wherein the first data packet is associated with a data packet sequence number (PSN) and includes a transaction identifier (XID) of a corresponding work queue element (WQE); A notification message is sent to the other device suitable for RDMA, wherein the notification message indicates that the one or more data packets are received at the receiving device.

20. The method according to claim 19, characterized in that The method further comprises: placing the data carried in the write operation request into a destination memory location in the RDMA-compatible device; or The data carried in the sending operation request is placed into a buffer from a receiving queue of the device suitable for RDMA.

21. A device suitable for RDMA, characterized in that: The device includes a processor and a memory, wherein the memory is used to store computer-executable instructions. When the device is running, the processor executes the computer-executable instructions in the memory to utilize the hardware resources in the device to perform the operating steps of any one of the methods described in claims 1 to 20.

22. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes instructions, which, when executed on a computer, enable the computer to execute the operating steps of any one of the methods described in claims 1 to 20.

Images