Multipath RDMA transmission

Abstract

According to the implementation of the present disclosure, a multi-path RDMA transmission scheme is proposed. In the scheme, at least one packet is generated based on an RDMA message to be sent from a first device to a second device. There is an RDMA connection via multiple paths between the first device and the second device. The first packet in at least one packet includes at least a plurality of fields, and the plurality of fields contain information for transmitting the first packet on a first path in the plurality of paths. The multi-path RDMA transmission scheme according to the present disclosure can effectively utilize abundant network paths while maintaining a low occupancy of the memory in the network interface card.

Description

[0001] Related Applications

[0002] This application is a divisional application of the invention patent application with application number 201711352625.4 and invention name “Multipath RDMA Transmission”. Background Art

[0003] Modern data center applications require high throughput and low latency network transmission to meet the growing needs of customers. Compared with traditional transport layer protocols such as the Transmission Control Protocol (TCP), Remote Direct Memory Access (RDMA) implements the entire transmission logic in the Network Interface Card (NIC) and allows direct access to remote memory without the involvement of the Central Processing Unit (CPU) or the operating system. Therefore, RDMA can achieve extremely high throughput and extremely low latency network transmission with almost no CPU involvement.

[0004] However, existing RDMA usually only supports single-path transmission. That is, there is only an RDMA connection between the sending device and the receiving device via a single network path. When a single network path fails, the RDMA connection between the sending device and the receiving device may not work properly. In addition, single-path transmission cannot fully utilize the rich network paths in the data center network. Summary of the invention

[0005] According to the implementation of the present disclosure, a multi-path RDMA transmission scheme is proposed. In the scheme, at least one packet is generated based on an RDMA message to be sent from a first device to a second device. There is an RDMA connection via multiple paths between the first device and the second device. The first packet in at least one packet includes at least a plurality of fields, and the plurality of fields contain information for transmitting the first packet on a first path in the plurality of paths. The multi-path RDMA transmission scheme according to the present disclosure can effectively utilize abundant network paths while maintaining a low occupancy of the memory in the network interface card.

[0006] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 A block diagram showing an example environment in which implementations of the present disclosure can be implemented;

[0008] Figure 2 A schematic diagram showing an example packet for MP-RDMA according to an implementation of the present disclosure;

[0009] Figure 3 A schematic diagram illustrating an example acknowledgement (ACK) for MP-RDMA according to an implementation of the present disclosure;

[0010] Figure 4 A schematic diagram showing an example data structure for tracking received packets according to an implementation of the present disclosure;

[0011] Figure 5 A schematic diagram illustrating an example negative acknowledgement (NACK) for MP-RDMA according to an implementation of the present disclosure;

[0012] Figure 6 A schematic diagram showing a plurality of control information for MP-RDMA according to an implementation of the present disclosure;

[0013] Figure 7 A flowchart showing a process for MP-RDMA according to some implementations of the present disclosure is shown;

[0014] Figure 8 A flowchart illustrating a process for MP-RDMA according to some implementations of the present disclosure; and

[0015] Fig. 9 A block diagram of a computing device capable of implementing various implementations of the present disclosure is shown.

[0016] In these drawings, the same or similar reference symbols are used to designate the same or similar elements. DETAILED DESCRIPTION

[0017] The present disclosure will now be discussed with reference to several example implementations. It should be understood that these implementations are discussed only to enable those skilled in the art to better understand and thus implement the present disclosure, and do not imply any limitation on the scope of the present subject matter.

[0018] As used herein, the term "including" and variations thereof are to be interpreted as open-ended terms meaning "including but not limited to". The term "based on" is to be interpreted as "based at least in part on". The terms "an implementation" and "an implementation" are to be interpreted as "at least one implementation". The term "another implementation" is to be interpreted as "at least one other implementation". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0019] RDMA in a nutshell

[0020] RDMA implements direct memory access to remote systems through NIC hardware. Therefore, RDMA can achieve extremely high throughput and extremely low latency network transmission with almost no local or remote CPU involvement. RDMA can be implemented on two transmission networks: InfiniBand network and Ethernet. On Ethernet, according to different protocol stacks integrated with Ethernet, the RDMA protocol can be divided into TCP-based RDMA protocol (iWARP) and RDMA protocol on converged Ethernet (RoCE, including RoCEv1 and RoCEv2). In this article, the RoCEv2 protocol will be described as an example. However, it should be understood that this is for illustrative purposes only and is not intended to imply any limitation on the scope of the present disclosure. The implementation of the present disclosure is also applicable to the iWARP protocol or the RDMA protocol based on the infinite bandwidth network.

[0021] In RDMA, an RDMA connection can be identified by a pair of work queues, also called a queue pair (QP). A QP consists of a send queue and a receive queue, both of which are maintained on the NICs of the sending device and the receiving device, respectively. When an application initiates an RDMA operation (also called an "RDMA primitive") to send or obtain data, the application assigns a work queue element (WQE) to the send queue or the receive queue. In addition, in order to notify the application of the completion of the operation, there is usually a completion queue (CQ) associated with each QP. When the WQE is completed, the corresponding completion queue element (CQE) will be delivered to the CQ.

[0022] There are usually four RDMA operations (also called RDMA primitives) in RDMA: send, receive, write, and read. Among them, send and receive are two-sided operations, which means that a send operation on one side always requires a receive operation on the other side. Read and write are one-sided operations, which means that applications can directly read or write remote memory without involving the remote CPU.

[0023] RDMA transmission is message-based, that is, RDMA operations are translated into RDMA messages for transmission. RDMA messages can be further divided into multiple segments of equal size to be encapsulated into User Datagram Protocol (UDP) packets. In RoCEv2, all RDMA data packets can use the same UDP destination port (e.g., 4791), while the UDP source port is arbitrary. The UDP source port is different for different RDMA connections, which allows the load balancing routing protocol to distinguish. In the UDP payload, the RDMA header is inserted into each data packet. The RDMA header contains a packet sequence number (PSN), which provides a continuous sequence number of the RDMA data packets in the RDMA connection. At the receiving device, the RDMA message is recovered according to the PSN. In addition, the receiving device can generate a positive acknowledgment (ACK) or a negative acknowledgment (NACK) to notify the sending device of received or lost data packets.

[0024] RDMA transmission requires a lossless network, which is provided by priority-based flow control (PFC). Specifically, PFC can use pre-configured priorities to implement hop-by-hop flow control on communication services. In PFC, when a downstream switch detects that the amount of data in its input queue exceeds a threshold, it will return a pause frame to the upstream switch to slow down data forwarding. PFC can effectively prevent switch packet loss, but this behavior of flow control by returning pause frames may spread congestion and slow down the entire network.

[0025] As described above, existing RDMA usually only supports single-path transmission. Specifically, all RDMA messages on an RDMA connection will use the same UDP source port and the same UDP destination port. Such single-path RDMA transmission has the following main disadvantages: First, single-path transmission is not robust to path failures, and some small failures in the path may significantly affect the performance of upper-layer applications. Second, a single path cannot achieve effective utilization of the entire network bandwidth.

[0026] The inventors have recognized that the above problems can be solved by implementing multi-path RDMA (MP-RDMA) transmission. In addition, the inventors have further recognized that MP-RDMA transmission may have many challenges.

[0027] RDMA transmission is implemented in the NIC, and the on-chip memory of the NIC is small and expensive. Installing a large-capacity memory on the NIC may not only occupy a large die space but also increase its power consumption, leading to other problems such as heat dissipation. Therefore, the on-chip memory of the NIC usually acts as a cache for the host memory to store information of the RDMA connection. When a cache miss occurs, the NIC needs to read the missing data from the host memory (for example, via the Peripheral Component Interconnect Express (PCIe) bus). Frequent cache misses will cause the NIC throughput to decrease. Therefore, in order to avoid performance degradation due to frequent cache misses, the memory occupancy for each path should be small enough to support more paths in the on-chip memory. However, it will be difficult to reduce the memory occupancy in multipath RDMA transmission.

[0028] First, multipath transmission needs to track the transmission and congestion status on each path in order to dynamically distribute the network load to each path according to the congestion situation. However, the number of these states will increase linearly with the number of paths used.

[0029] Secondly, multipath may cause data packets to arrive at the receiving device out of order. Therefore, the receiving device needs to record additional information to track whether the data packet has arrived. If the conditions of different paths are significantly different, the size of this additional information may be very large.

[0030] In addition, the NIC at the receiving end may not have enough memory to buffer out-of-order data packets, and thus has to store part of the data packets in the host memory. Therefore, data updates to the host memory may be out of order. For example, an application may use a first write operation to update the remote memory, and then issue a second write operation to set the remote flag to notify the remote application to read the data updated by the first write operation. However, if the second write operation updates the remote host memory before the first write operation, the remote application may prematurely read part of the data that has not been updated by the first write operation.

[0031] The above discusses some problems existing in the current RDMA transmission. According to the implementation of the present disclosure, a multipath RDMA transmission solution is provided to solve one or more of the above problems and other potential problems. The basic principles and several example implementations of the present disclosure are explained below with reference to the accompanying drawings.

[0032] Example Environment

[0033] Figure 1A block diagram of an example environment 100 in which implementations of the present disclosure can be implemented is shown. It should be understood that the structure and functionality of the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure. The implementations of the present disclosure can also be applied to environments with different structures and / or functions.

[0034] like Figure 1 As shown, environment 100 may include host 110 and host 120. Host 110 may include CPU 111, memory 112, and NIC 113. Host 110 may be any physical computer, virtual machine, server, etc. User application ( Figure 1 The host 110 may be connected to the network 130 via the NIC 113. Similarly, the host 120 may include a CPU 121, a memory 122, and a NIC 123. The user application ( Figure 1 The host 120 may be connected to the network 130 via the NIC 123.

[0035] In environment 100, host 110 and host 120 may be interconnected via network 130. Network 130 may be, for example, an Ethernet or a wireless broadband network. Figure 1 As shown, there may be multiple network paths 140-1, 140-2, ... 140-N (collectively or individually referred to as network paths 140, where N>1) between NIC 113 and NIC 123. Figure 1 Not shown, but it should be understood that in some implementations, additional network devices, such as switches, routers, etc., may or may not be included on each of the network paths 140.

[0036] In some implementations, the NIC 113 and the NIC 123 may establish an RDMA connection via multiple network paths 140 to enable multipath RDMA operations between the host 110 and the host 120. For example, when an application executed by the CPU 111 in the host 110 initiates a request for an RDMA write operation, the RDMA request may be sent to the NIC 113. The NIC 113 may read the corresponding data from the memory 112 based on the RDMA request and generate an RDMA message to be sent to the NIC 123. For example, the RDMA message may include the data to be sent and the address in the target memory 122 to be written. The RDMA message may be sent to the NIC 123 via multiple network paths 140. In response to receiving the RDMA message sent by the NIC 113, the NIC 123 may directly write the corresponding data contained therein into the memory 122. For another example, when an application executed by the CPU 111 in the host 110 initiates a request for an RDMA read operation, the RDMA request may be sent to the NIC 113. NIC 113 can read corresponding data from remote memory 122 based on the RDMA request. For example, the read data together with the address in the target memory 112 to be written can be included in the RDMA message. The RDMA message can be sent from NIC 123 to NIC 113 via multiple network paths 140. In response to receiving the RDMA message sent by NIC 123, NIC 113 can write the corresponding data contained therein directly into memory 112. RDMA send / receive operations are similar to read / write operations. That is, the NIC acting as a sending device transmits the RDMA message to the NIC acting as a receiving device via multiple network paths 140, and the NIC acting as a receiving device writes the data in the received RDMA message directly into the target memory, thereby realizing zero-copy data transmission between host 110 and host 120 without the participation of CP420U 111 and / or 121.

[0037] The example implementation of the present disclosure will be described in further detail below. For ease of description, the NIC 113 will be used as an example of a sending device, and the NIC 123 will be used as an example of a receiving device. In this document, the NIC 113 acting as a sending device is used interchangeably with the "first device", and the NIC 123 acting as a receiving device is used interchangeably with the "second device".

[0038] Grouping structure

[0039] In some implementations, the NIC 113 may generate at least one packet based on the RDMA message to be sent from the NIC 113 to the NIC 123. For example, the NIC 113 may divide the RDMA message into at least one segment of equal size to be encapsulated in at least one UDP / IP / Ethernet packet. The NIC 113 may then send the generated at least one packet from the NIC 113 to the NIC 123 via the RDMA protocol in the plurality of paths 140.

[0040] Figure 2 FIG. 2 is a schematic diagram of an example packet 200 for MP-RDMA transmission according to an implementation of the present disclosure. Figure 2 In the embodiment, the structure of the packet 200 sent from the NIC 113 to the NIC 123 will be described in conjunction with the RoCEv2 protocol. However, it should be understood that this is for illustrative purposes only and is not intended to imply any limitation on the scope of the present disclosure. The implementation of the present disclosure may also be applied to the iWARP protocol or the RDMA protocol based on the infinite bandwidth network. For illustrative purposes, in Figure 2 Only the header of packet 200 is shown.

[0041] like Figure 2 As shown, in some implementations, NIC 113 can generate packet 200 by reusing several fields in UDP and RoCEv2. The reused fields may include, for example, Ethernet header 210, IP header 220, UDP header 230, and RoCEv2 data header 240 inserted into the UDP payload.

[0042] UDP header 230 may include field 231, which originally indicates a UDP source port. In some implementations, NIC 113 may utilize field 231 (also referred to herein as a "first field") in packet 200 to identify a network path for transmitting the packet. For example, the network path indicated by field 231 may be a virtual network path (VP), which may be mapped to a corresponding physical path (such as, for example, Figure 1One of the paths 140 shown in FIG. 14 ). Typically, packets with the same VP will be mapped to the same physical path for transmission. In some implementations, when generating packet 200, NIC 113 may determine the VP for transmitting packet 200 and write the identifier (ID) of the VP into field 231. For example, in an initial situation where no ACK has been received or when a new path is being detected, NIC 113 may randomly select a VP for transmitting packet 200. Additionally or alternatively, in the event that an ACK is received, NIC 113 may determine the VP for transmitting packet 200 based on the VP to which the ACK is received. The selection of VP will be further described in detail in the “Congestion Control and Multipath Distribution” section below.

[0043] The RoCEv2 data header 240 may include a PSN (also referred to herein as a “second field”) of the packet 200 in the RDMA connection between the NIC 113 and the NIC 123 . In addition, the RoCEv2 data header 240 may include fields 241 and 242 .

[0044] In some implementations, the NIC 113 may use the field 241 to indicate whether the packet 200 is a retransmission packet. For example, when the field 241 is '1', it may indicate that the packet 200 is a retransmission packet; when the field 241 is '0', it may indicate that the packet 200 is not a retransmission packet. Regarding the retransmission of packets, further details will be described in the "Packet Loss Recovery" section below.

[0045] Additionally or alternatively, in some implementations, the NIC 113 may also utilize field 242 (also referred to herein as the "ninth field") to indicate whether packet 200 is a synchronization packet. The "synchronization packet" described herein refers to a packet that is not allowed to be delivered to the target memory out of order. For example, when field 242 is '1', it may indicate that packet 200 is a synchronization packet. That is, packet 200 can be delivered to the target memory only after the packet before packet 200 (that is, a packet whose sequence number is less than the sequence number of packet 200) is stored in the target memory (e.g., memory 122). When field 242 is '0', it may indicate that packet 200 is not a synchronization packet. Regarding the processing of synchronization packets, further detailed description will be given below in the "Synchronization Packet Processing" section.

[0046] In some implementations, such as Figure 2As shown, the packet 200 may further include an extension field 250 (also referred to herein as a "third field") and an extension field 260 (also referred to herein as a "fourth field"). For example, the extension field 250 may indicate the sequence number of the RDMA message to which the packet 200 belongs. The extension field 260 may indicate the sequence number of the packet 200 within the RDMA message to which it belongs. In particular, the extension fields 250 and 260 may be used for RDMA send / receive operations.

[0047] As described above, in the case where packet 200 is not a synchronous packet, when NIC 123 receives packet 200, NIC 123 can store the data in packet 200 directly in the corresponding address of memory 122. For RDMA read operations or write operations, RoCEv2 data header 240, for example, already includes the target address in memory 122 to be written, so NIC 123 can parse RoCEv2 data header 240 to obtain the target address and store the data in packet 200 directly at the target address. However, for RDMA send / receive operations, the receiving device NIC 123 needs additional information to determine the target address where the data is placed. For example, the target address is contained in a specific receive work queue element (WQE), and NIC 123 can determine the specific WQE based on field 250 in packet 200. NIC 123 can further determine the address offset of packet 200 within the RDMA message based on field 260.

[0048] On the receiving side, when the NIC 123 receives the packet 200 transmitted from the NIC 113, the NIC 123 may generate an ACK for the packet 200. Then, the NIC 123 may return the ACK for the packet 200 to the NIC 113 on the path where the packet 200 was received.

[0049] Figure 3 FIG. 3 is a schematic diagram of an example ACK 300 for MP-RDMA transmission according to an implementation of the present disclosure. Figure 3 Only the header of ACK 300 is shown.

[0050] like Figure 3 As shown, in some implementations, NIC 123 can generate ACK 300 by reusing several fields in UDP and RoCEv2. The reused fields may include, for example: Ethernet header 310, IP header 320, UDP header 330, and RoCEv2 ACK header 340 inserted into the UDP payload.

[0051] UDP header 330 may include field 331 (also referred to herein as the "fifth field"), which originally indicates the UDP source port. In some implementations, NIC 123 may copy field 231 in packet 200 to field 331 to identify the network path for transmitting ACK 300.

[0052] The RoCEv2 ACK header 340 may include a field 341 (also referred to herein as the "sixth field"), which is originally used by the receiver to indicate to the sender information such as which packets are lost, which packets are retransmitted, and which packets have been received in advance. In some implementations, the NIC 123 may use field 341 to indicate the PSN of the received packet 200. In addition, the RoCEv2 ACK header 340 may also include fields 342, 343, and 344.

[0053] In some implementations, field 342 (also referred to herein as the "seventh field") may be used to indicate an explicit congestion notification (represented herein as "ECN"). For example, an ECN may be generated by a router in the transmission path of packet 200 to indicate a congestion condition in the transmission path detected by the router. The ECN may be provided to NIC 113 for subsequent congestion control. The application of the ECN will be further described in detail in the "Congestion Control and Multipath Distribution" section below.

[0054] In some implementations, the NIC 123 may copy field 241 in packet 200 to field 343 to indicate that ACK 300 is an ACK for a retransmitted packet. Additionally or alternatively, in some implementations, the NIC 123 may copy field 242 in packet 200 to field 344 to indicate that ACK 300 is an ACK for a synchronization packet.

[0055] In some implementations, such as Figure 3 As shown, ACK 300 may also include an extension field 350 (also referred to herein as an "eighth field"). For example, NIC 123 may use extension field 350 to indicate to NIC 113 the maximum sequence number of the packet received at NIC 123. The maximum sequence number indicated by extension field 350 may be used by NIC 113 for subsequent congestion control. The application of the maximum sequence number will be further described in detail in the "Congestion Control and Multipath Distribution" section below.

[0056] Combination of the above Figure 2 and Figure 3The structure of example packets and example ACKs according to implementations of the present disclosure are depicted. It should be understood that this is for illustrative purposes only and does not imply any limitation on the scope of the present disclosure. Implementations of the present disclosure may also be embodied using packets and / or ACKs having different structures.

[0057] Congestion control and multi-path distribution

[0058] As described previously, MP-RDMA needs to track the transmission and congestion status on each of the multiple paths so that packets can be dynamically distributed to the paths according to the congestion situation. In order to minimize the occupation of the on-chip memory in the NIC, congestion control can be performed at the NIC 113 acting as a sending device based on congestion control information for multiple paths. In some implementations, the congestion control information can be represented by a unique congestion control window for multiple paths, which, for example, indicates the total number of packets that can be transmitted on multiple paths (hereinafter, the congestion control window is represented as "cwnd").

[0059] In the initial case when the NIC 113 has not received any ACK, cwnd may have an initial value greater than zero, such as 16. Assuming that the sequence numbers of packets to be sent start from 0, the NIC 113 may randomly distribute packets No. 0 to 15 (hereinafter represented as packets #0 to #15) in 16 VPs to send to the NIC 123.

[0060] When NIC 113 receives an ACK from NIC 123, NIC 113 may update cwnd based on the received ACK. For example, the ACK received by NIC 113 may have the following format: Figure 3 The structure shown.

[0061] In some implementations, the NIC 113 may update cwnd based on field 342 (ie, ECN) in the ACK. For example, the NIC 113 may update cwnd according to the following formula (1):

[0062]

[0063] In some implementations, the NIC 113 may further determine an available window (denoted herein as “awnd”) based on the updated cwnd, which may indicate the number of packets currently allowed to be sent to the NIC 123. When the determined available window is greater than 0, the NIC 113 may send the next packet on the path where the ACK is received.

[0064] In some implementations, in the absence of out-of-order reception (i.e., packets arrive at NIC 123 in the order in which they are sent, and ACKs for packets also arrive at NIC 113 in the order in which they are sent), NIC 113 may determine awnd according to the following formula (2):

[0065] awnd=cwnd-(snd_next-snd_una) formula (2)

[0066] Where snd_next indicates the sequence number of the next packet to be sent by NIC 113, and snd_una indicates the minimum sequence number of packets for which NIC 113 has not received ACK. That is, snd_next-snd_una can indicate the number of packets that NIC 113 has sent but has not received ACK.

[0067] For example, assuming that the initial cwnd is 16, NIC 113 has sent packets #0 to #15 to NIC 123, so snd_next=16 and snd_una=0. At this time, NIC 113 receives an ACK for packet #0 from NIC 123 (represented as "ACK#0" in this document), and the ECN in ACK#0 is 0, so cwnd is updated to 16+1 / 16 and snd_una is updated to 1. Therefore, awnd=16+1 / 16-(16-1)=1+1 / 16. Since awnd is greater than 0, NIC 113 can send packet #16 to NIC 123 on the path where ACK#0 is received, and snd_next is updated to 17.

[0068] Alternatively, in some implementations, taking into account the possible out-of-order reception (i.e., packets may not arrive at NIC 123 in the order in which they are sent, and ACKs for packets may not arrive at NIC 113 in the order in which they are sent), NIC 113 may determine awnd according to the following formula (3):

[0069] awnd=cwnd+inflate-(snd_next-snd_una) Formula (3)

[0070] Where snd_next represents the sequence number of the next packet to be sent by NIC 113, and snd_una indicates the minimum sequence number of the packet for which NIC 113 has not received an ACK. That is, snd_next-snd_una can represent the number of packets that NIC 113 has sent but has not received an ACK. inflate represents an expansion factor for the available window awnd, which estimates the number of out-of-order data packets that NIC 113 has received (e.g., ACKs for these out-of-order data packets have been received by NIC 113), and adds the number of these packets to the available window awnd so that as many packets as possible can be sent. inflate can be calculated based on the following: increment inflate for each received ACK; then subtract (ack_aack-snd_una) from the incremented inflate before updating snd_una based on the ACK, where ack_aack represents the value of field 350 in the ACK (e.g., in practice it can be equal to the maximum sequence number of the packet received at NIC 123 plus 1).

[0071] For example, assuming that the initial cwnd is 16, NIC 113 has sent packets #0 to #15 to NIC 123. At this time, snd_next=16, snd_una=0 and inflate=0. Then, NIC 113 receives ACK#1 for packet #1 from NIC 123, the ECN in ACK#1 is 0 and ack_aack is 0 (because ACK#0 before ACK#1 has not been received), so cwnd is updated to 16+1 / 16. inflate is first incremented by 1, and then (ack_aack-snd_una) is subtracted. Therefore, the updated inflate is 1, which indicates that NIC 123 has received 1 out-of-order data packet, namely, packet #1. According to formula (3), awnd=16+1 / 16+1-(16-0)=1+1 / 16. Since awnd is greater than 0, NIC 113 can send packet #16 to NIC 123 on the path where ACK #1 is received, and snd_next is updated to 17. Assume that NIC 113 further receives ACK #2 for packet #2 from NIC 123, whose ECN is 0. According to the above rule, inflate will be updated to 2, and awnd = 16 + 1 / 8 + 2 - (17 - 0) = 1 + 1 / 8. Since awnd is greater than 0, NIC 113 can send packet #17 to NIC 123 on the path where ACK #2 is received, and snd_next is updated to 18.

[0072] It can be seen that the implementation of the present disclosure can properly balance the packet traffic between multiple paths. At the same time, since only one congestion control window is used to perform congestion control on multiple paths, the implementation of the present disclosure can minimize the occupation of the on-chip memory of the NIC.

[0073] Burst Control

[0074] In some implementations, NIC 113 may send more than one packet for a returned ACK. For example, when the determined available window awnd is greater than 1, NIC 113 may send more than one packet. In some cases, if more than one packet is sent on a single path where ACK is received, congestion may occur or worsen. In some implementations, in order to solve the above problem, a threshold may be pre-set for burst packets. For example, the predetermined threshold indicates the maximum number of packets allowed to be sent on a single path at one time. When NIC 113 determines that the number of packets currently allowed to be sent to NIC 123 exceeds the predetermined threshold, NIC 113 only sends the predetermined threshold number of packets on the single path where ACK is received. The sending of the remaining packets will be triggered by the subsequently received ACK.

[0075] In some cases, if no subsequent ACK arrives, the remaining packets may never be sent. In some implementations, a timer may be set for these remaining packets. For example, the length of the timer may be set to half of the pre-estimated round trip delay (represented as "RTT" in this article) between NIC 113 and NIC 123. If the timer times out and a subsequent ACK has not been received, the remaining packets may be sent separately on a random path.

[0076] Congestion control window shrinkage

[0077] In some implementations, when there is no data to be transmitted, NIC 113 can gradually shrink the congestion control window cwnd. For example, when NIC 113 receives an ACK that triggers the transmission of the next packet but there is no data to be transmitted, NIC 113 can decrement cwnd by 1 to correspondingly reduce the transmission rate on multiple paths to save network resources. In addition, in some implementations, when the time when the sender lacks data to be transmitted exceeds a predetermined threshold (e.g., 3 times the RTT), the above-mentioned congestion control window and related states can be reset, that is, returned to the initial situation. In this way, the congestion control window can more accurately reflect the current congestion situation on multiple paths.

[0078] New path detection

[0079] As previously described, in the initial case, the NIC 113 may randomly select a path for transmitting packets. In the case of receiving an ACK, the NIC 113 may select a path for transmitting subsequent packets based on the path on which the ACK is received. Additionally or alternatively, in some implementations, the NIC 113 may also actively detect new paths based on a predetermined probability. For example, the NIC 113 may periodically send packets on a new random path based on a predetermined packet interval, rather than always determining the path for transmitting packets based on an ACK. Since congestion control is not required for each path, the implementation of the present disclosure allows many paths to be used simultaneously without incurring additional memory usage overhead. In addition, through periodic new path detection and the elimination of slow paths described previously, according to the implementation of the present disclosure, MP-RDMA can always use better paths for RDMA transmission, thereby further improving throughput and reducing transmission delays.

[0080] Out-of-order processing

[0081] As described above, due to the parallelism of multipath transmission, data packets may arrive at the receiving device out of order. When data packets arrive at the receiving end out of order, the NIC at the receiving end may not have enough memory to buffer the out-of-order data packets. The implementation of the present disclosure can control the disorder degree of MP-RDMA within an acceptable range, so as to minimize the occupation of the on-chip memory of the NIC.

[0082] Out-of-order processing at the receiving end

[0083] In some implementations, a bitmap data structure may be employed at the NIC 123 acting as a receiving device to track arriving packets. Figure 4 FIG. 4 is a schematic diagram of an example bitmap data structure 400 for tracking received packets according to an implementation of the present disclosure. Figure 4 As shown, the bitmap 400 can be organized into a circular array. For example, the bitmap 400 can have L slots, each of which contains two bits, for example, to record the state of a packet. The header 410 of the bitmap 400 corresponds to the packet with a sequence number of rcv_next, where rcv_next represents the sequence number of the next packet that the receiving device expects to receive. That is, the bitmap 400 can record the states of the packets with sequence numbers from rcv_next to rcv_next+L-1.

[0084] In some implementations, the RDMA message is sent based on the type of RDMA message (e.g., the type is contained in the Figure 2 In the RoCEv2 data header 240 shown in FIG. 1 ), each packet can have one of the following four states: 1) "Not received" state (in Figure 42) "Received" state (indicated as "E" in Figure 4 3) "end of message" state (in Figure 4 4) "end of message requiring completion notification" state (in Figure 4 The initial state of each slot in the bitmap 400 may be set to a "not received" state.

[0085] In response to receiving a packet from NIC 113, NIC 123 may determine the packet based on the packet's PSN (e.g., contained in the Figure 2 14 (shown in the RoCEv2 data header 240) to determine whether there is a slot corresponding to the packet in the bitmap 400. For example, if the PSN of the packet is less than rcv_next+L, the NIC 113 may determine that there is a slot corresponding to the packet in the bitmap 400; otherwise, the NIC 113 may determine that there is no slot corresponding to the packet in the bitmap 400. If there is a slot corresponding to the packet in the bitmap 400, the NIC 113 may update the state of the packet at the slot corresponding to the packet.

[0086] In some implementations, the NIC 113 may include a Figure 2 The field 250 shown in the figure determines whether the received packet is the last packet of the RDMA message. If the packet is not the last packet of the RDMA message, the status of the packet can be updated to the "received" state. If the packet is the last packet of the RDMA message, the NIC 113 can further check the message type field in the RoCEv2 data header 240 to determine whether the RDMA message requires completion notification. For example, if the RDMA message indicates that the RDMA operation is a read operation or a send operation, the NIC 113 can determine that the packet requires completion notification, and thereby update the status of the packet to the "end of message requiring completion notification" state; otherwise, the NIC 113 can update the status of the packet to the "end of message" state.

[0087] In some implementations, the NIC 113 may continuously scan the bitmap 400 to determine whether an RDMA message has been completely received. For example, a completely received RDMA message may be indicated by a plurality of consecutive slots marked as a "received" state together with a slot marked as a "message end requiring completion notification" state or a "message end" state, such as Figure 4When the NIC 113 determines that the RDMA message has been completely received, the NIC 113 may set the state of the slot corresponding to the RDMA message to the "not received" state, and update rcv_next to move the pointer 410 indicating the head of the loop array 400 to the slot 440 pointing to the RDMA message. Figure 4 As shown, the pointer indicating the head of the circular array 400 may be moved to the slot after the message 430, and the status of the slots corresponding to the messages 420 and 430 may be set to "E" so that these slots may be used to record the status of packets arriving later. Additionally, if a message requires completion notification (e.g., message 420), the NIC 113 may pop the WQE from the receive work queue and generate a corresponding CQE in the CQ.

[0088] Out-of-order processing at the sender

[0089] The above discusses the out-of-order processing performed at the receiving end. As described above, in order to minimize the occupancy of the receiving end NIC on-chip memory, the size of the bitmap (i.e., L) used to track the status of the received packets may be limited. That is, if the PSN of the packet received by NIC 123 exceeds rcv_next+L, the status of the packet cannot be recorded in the bitmap. If NIC 123 drops the packet as a result, the overall performance will be impaired. The implementation of the present disclosure can control the degree of out-of-order of MP-RDMA within an acceptable range through the path selection of the sending end, thereby solving the above-mentioned problem.

[0090] In some implementations, when NIC 113, acting as a sending device, receives an ACK from NIC 123, NIC 113 may determine the maximum packet sequence number of the packet for which the ACK has been received (represented herein as "snd_ooh"). NIC 113 may determine the threshold sequence number of the packet allowed to be sent on the path for which the ACK is received based on snd_ooh and a predetermined out-of-order tolerance for MP-RDMA (represented herein as "Δ"). For example, the threshold sequence number snd_ool may be determined according to the following formula (4):

[0091] snd_ool=snd_ooh-Δ Formula (4)

[0092] Where Δ≤L. When NIC 113 determines the sequence number of the packet for which the ACK is intended (e.g., by Figure 3Field 341 shown indicates that when the ACK is less than the threshold sequence number snd_ool, the NIC 113 may avoid sending the next packet on the path where the ACK is received by updating the congestion control window cwnd (eg, decrementing cwnd).

[0093] It can be seen from the above description that the implementation of the present disclosure uses snd_ooh to mark the out-of-order packets that arrive at the receiving device earlier through the fast path. In order to control the disorder degree of MP-RDMA, it is necessary to remove the slow path that causes the disorder degree to exceed the predetermined threshold value Δ. When the PSN of the packet targeted by the ACK is less than snd_ool, it identifies such a slow path. By avoiding sending subsequent packets on such a slow path, the implementation of the present disclosure can control the disorder degree of MP-RDMA within an acceptable range.

[0094] It should be noted that the above description does not consider the case of retransmitted packets. Since the sequence number of the retransmitted packet is likely to be less than snd_ool, the slow path cannot be identified for the retransmitted packet only by the PSN of the packet to which the ACK is directed. Therefore, in some implementations, for the ACK with a retransmit flag (e.g., Figure 3 The above operation may not be performed (indicated by the field 342 shown). That is, the path receiving the ACK for the retransmitted packet is always regarded as a normal path, and the ACK for the retransmitted packet can trigger the transmission of the next packet on the path. The processing of the retransmitted packet will be further described in detail below.

[0095] Synchronous Packet Processing

[0096] As described above, since the NIC 123 may not have enough memory to buffer the data packets that arrive out of order, it may have to store at least part of the data packets in the host memory 122. Therefore, the update of data in the memory 122 may be out of order. For ordinary operations, such out-of-order updates may not be a problem. However, for some operations that rely on a specific transmission order (referred to as "synchronous operations" herein), such out-of-order updates may cause the application to be unable to obtain the expected correct data. The implementation of the present disclosure can solve the above problem by delaying the synchronization operation for a specific time interval.

[0097] As previously combined Figure 2 As described, the NIC 113 may use field 242 in the packet to indicate whether the packet is a synchronous packet, i.e., a packet that is not allowed to be delivered to the target memory out of order. In some implementations, the NIC 113 may determine a time interval Δt for sending such a synchronous packet. For example, the NIC 113 may determine Δt according to the following formula (5):

[0098]

[0099] Wherein, Δ represents the predetermined out-of-order tolerance for MP-RDMA, RTT represents the pre-estimated round-trip delay between NIC 113 and NIC 123, α represents the predetermined adjustable coefficient, and R s Indicates the send rate on the RDMA connection.

[0100] In some implementations, at the transmitting end, the NIC 113 must wait at least a time interval Δt after sending the packet before the synchronization packet before sending the synchronization packet. In this way, the packet before the synchronization packet has a greater probability of arriving at the receiving end earlier than the synchronization packet.

[0101] In some implementations, on the receiving end, when the NIC 123 receives a packet with a synchronization marker (e.g., Figure 2 When receiving a packet (indicated by the field 242 shown in the figure), the NIC 123 will first determine whether packets with sequence numbers less than the sequence number of the synchronization packet have been delivered to the memory 122. Only when all packets with sequence numbers less than the sequence number of the synchronization packet have been delivered to the memory 122, the synchronization packet can be delivered to the memory 122. In this way, the implementation of the present disclosure can ensure that operations that rely on a specific transmission order are performed in order.

[0102] Packet Loss Recovery

[0103] For traditional single-path RDMA, packet loss can be detected by the discontinuity of the PSN of the received packets. However, in MP-RDMA, out-of-order arrival of packets is common and has nothing to do with packet loss in most cases. In response to the above problems, the implementation of the present disclosure can identify and recover real packet loss.

[0104] In some implementations, when the PSN of a packet received by NIC 123 acting as a receiving device exceeds rcv_next+L, NIC 123 may generate a negative acknowledgement (NACK) and send it to NIC 113. The structure of NACK is similar to Figure 3 The structure of the ACK shown is similar. For example, Figure 5 FIG. 5 is a schematic diagram of an example NACK 500 for MP-RDMA transmission according to an implementation of the present disclosure. Figure 5 Only the header of NACK 500 is shown.

[0105] like Figure 5 As shown, NACK 500 may include: Ethernet header 510, IP header 520, UDP header 530, and RoCEv2 NACK header 540 inserted into the UDP payload. Figure 3 The difference between the RoCEv2 ACK header 340 shown in FIG. 1 is that the RoCEv2 ACK header 340 includes a tag indicating a positive acknowledgment, while the RoCEv2 NACK header 540 includes a corresponding tag indicating a negative acknowledgment. In addition, the NACK 500 also includes fields 541, 542, 543, 544 and 550, which correspond to the following respectively: Figure 3 341, 342, 343, 344, and 350 in ACK 300 shown. In some implementations, fields 542 and 550 may be populated similarly to fields 342 and 350, respectively, while fields 543 and 544 may optionally not be populated. Unlike field 341, field 541 may be populated with the sequence number of the packet to be retransmitted. For example, NIC 123 may select a sequence number from the example shown in FIG. Figure 4 The head pointer of the bitmap 400 shown starts to sequentially scan the bitmap 400 and fills the sequence number of the first scanned packet with the status "E" into the field 541.

[0106] In some implementations, in response to receiving a NACK from the NIC 123, the NIC 113 may enter a packet loss recovery mode. In response to entering the packet loss recovery mode, the NIC 113 may determine the values ​​of two variables, recovery and snd_retx, where snd_retx indicates the starting point of the sequence numbers of the packets to be retransmitted, and recovery indicates the ending point of the sequence numbers of the packets to be retransmitted. For example, the NIC 113 may determine the value of recovery based on snd_next indicating the sequence number of the next packet currently to be sent, and determine the value of snd_retx based on the value in field 541 in the NACK 500. Figure 6 A schematic diagram showing a plurality of control information according to an implementation of the present disclosure is shown. Figure 6 The relationship between the information recovery and snd_retx used for the packet loss recovery mode and the several information used for path selection described above is schematically shown.

[0107] In some implementations, when entering in response to entering packet loss recovery mode, in response to receiving an ACK from NIC 123, NIC 113 may retransmit the packet indicated by snd_retx on the path where the ACK was received, rather than a new packet. For example, the retransmitted packet may carry a retransmission flag (i.e., Figure 2 The field 241 shown is set to '1'). The NIC 113 may update snd_retx and snd_una accordingly. When the value of snd_una exceeds the value of recovery, that is, when all packets to be retransmitted are sent, the packet loss recovery mode ends.

[0108] When NIC 123 receives a packet with a retransmission tag, it may include the retransmission tag in the ACK for the packet and send the ACK with the retransmission tag to NIC 113. In some implementations, for the ACK with the retransmission tag, NIC 113 may not perform the path selection shown in formula (3) or (4). That is, NIC 113 may always treat the path where the ACK for the retransmission packet is received as a normal path, and the ACK for the retransmission packet can always trigger the transmission of the next packet (e.g., a retransmission packet or a non-retransmission packet) on the path.

[0109] As described above, the sending of NACK is triggered in response to NIC 123 receiving a packet with a sequence number exceeding rcv_next+L. In some cases, if packet loss occurs but the sender does not have enough subsequent data to send, there may be a possibility that the packet loss recovery mode cannot be triggered. In some implementations, to avoid this situation, when the congestion control window allows the next packet to be sent and NIC 113 does not have enough data to be sent, NIC 113 can actively retransmit the packet for which the ACK is not received to trigger the packet loss recovery mode as early as possible.

[0110] In summary, the MP-RDMA scheme implemented according to the present disclosure can effectively utilize abundant network paths while maintaining low memory usage. By triggering packet transmission based on ACK on multiple paths and implementing out-of-order path selection, the scheme can select the best network path and distribute the packets appropriately on these paths for transmission. No matter how many network paths are used, the scheme only needs to occupy a small amount of additional memory space in the NIC. In addition, the MP-RDMA scheme implemented according to the present disclosure can well cope with PFC in the network. Using the congestion control scheme in the implementation of the present disclosure, PFC will rarely be triggered. When PFC is triggered, if PFC suspends all network paths, MP-RDMA will also stop transmission because there is no returned ACK. When PFC is restored, MP-RDMA will resume execution. If PFC suspends part of the path, these suspended paths will generate longer delays than other paths, and the path selection scheme in the implementation of the present disclosure can effectively eliminate these slow paths. Therefore, the MP-RDMA transmission scheme according to the present disclosure can improve the robustness against path failures and achieve efficient utilization of the entire network bandwidth.

[0111] Example Process

[0112] Figure 7Flowchart of process 700 for MP-RDMA according to some implementations of the present disclosure is shown. Process 700 may be implemented by a NIC acting as a sending device among NIC 113 and NIC 123. NIC 113 will be used as an example of a sending device, and NIC 123 will be used as an example of a receiving device. For example, process 700 may be implemented at NIC 113.

[0113] like Figure 7 As shown, at 710, based on the RDMA message to be sent from NIC 113 to NIC 123, at least one packet including at least a first packet is generated. NIC 113 and NIC 123 have an RDMA connection via multiple paths. The first packet includes at least multiple fields, and the multiple fields contain information for transmitting the first packet on a first path among the multiple paths. That is, the information in these multiple fields will be used to support or allow the transmission of the first packet on the first path. At 720, at least one packet is sent to NIC 123 on multiple paths via the RDMA protocol, and the first packet is sent on the first path.

[0114] In some implementations, the multiple fields include: a first field for indicating an identifier of the first path; a second field for indicating a packet sequence number of the first packet; a third field for indicating a message sequence number of the RDMA message; and a fourth field for indicating a sequence number of the first packet in the RDMA message.

[0115] In some implementations, NIC 113 sends at least one packet by: obtaining congestion control information indicating congestion conditions of multiple paths; determining whether a first packet is allowed to be sent based on the congestion control information; and in response to determining that the first packet is allowed to be sent, sending the first packet to NIC 123 via the RDMA protocol on the first path.

[0116] In some implementations, the at least one packet also includes a second packet subsequent to the first packet, and the NIC 113 sends the at least one packet by: updating congestion control information in response to receiving a first positive acknowledgement for the first packet from the NIC 123 on the first path; and controlling the sending of the second packet based on the updated congestion control information.

[0117] In some implementations, the first affirmative response includes at least: a fifth field for indicating an identifier of the first path; a sixth field for indicating a packet sequence number of a first packet received by NIC 123; a seventh field for indicating a congestion notification generated by a packet forwarding device on the first path, the congestion notification indicating a congestion condition of the first path detected by the packet forwarding device; and an eighth field for indicating a maximum sequential sequence number of packets received at NIC 123.

[0118] In some implementations, the NIC 113 updates the congestion control information by, in response to receiving the first positive acknowledgement, updating the congestion control information based on at least the congestion notification indicated by the seventh field and the maximum sequential sequence number indicated by the eighth field.

[0119] In some implementations, the NIC 113 updates the congestion control information by: determining, in response to receiving the first positive acknowledgement, a maximum packet sequence number of packets for which a positive acknowledgement has been received; determining an out-of-order tolerance level for transmission of at least one packet; determining, based on the maximum packet sequence number and the out-of-order tolerance level, a threshold sequence number of packets allowed to be sent on the first path; and updating the congestion control information based on at least the packet sequence number indicated by the sixth field and the threshold sequence number.

[0120] In some implementations, the first packet further includes a ninth field for indicating a transmission order on which the first packet depends. The NIC 113 sends the at least one packet by: determining a time interval for sending the first packet based at least on the congestion control information and the out-of-order tolerance; and in response to a third packet in the at least one packet immediately preceding the first packet being sent, waiting at least for the time interval before sending the first packet.

[0121] In some implementations, process 700 also includes: in response to receiving a negative acknowledgement from NIC 123, determining from at least one packet a fourth packet to which the negative acknowledgement is directed; processing the fourth packet by adding a retransmission marker in the fourth packet; and retransmitting the processed fourth packet to NIC 123 via the RDMA protocol on a second path among the multiple paths, on which a positive acknowledgement was previously received.

[0122] Figure 8 800 for MP-RDMA according to some implementations of the present disclosure. Process 800 may be implemented by a NIC acting as a receiving device among NIC 113 and NIC 123. NIC 113 will be used as an example of a sending device, and NIC 123 will be used as an example of a receiving device. For example, process 800 may be implemented at NIC 123.

[0123] like Figure 8 As shown, at 810, at least one packet including at least a first packet is received from NIC 113 via the RDMA protocol over multiple paths at NIC 123. NIC 113 and NIC 123 have an RDMA connection over multiple paths. The at least one packet includes multiple fields, and the multiple fields contain information for transmitting the first packet over a first path in the multiple paths.

[0124] In some implementations, at least one packet is generated based on an RDMA message, and the plurality of fields include: a first field for indicating an identifier of a first path; a second field for indicating a first packet sequence number of the first packet; a third field for indicating a message sequence number of the RDMA message; and a fourth field for indicating a sequence number of the first packet in the RDMA message.

[0125] Alternatively or additionally, at 820, in response to receiving the first packet on the first path, a first positive acknowledgement for the first packet is generated at the NIC 123. The first positive acknowledgement includes at least: a fifth field for indicating an identifier of the first path; a sixth field for indicating a first packet sequence number of the first packet received by the NIC 123; a seventh field for indicating a congestion notification generated by a packet forwarding device on the first path, the congestion notification indicating a congestion condition of the first path detected by the packet forwarding device; and an eighth field for indicating a maximum sequential sequence number of the packet received at the NIC 123. At 830, the first positive acknowledgement is sent to the NIC 113 on the first path.

[0126] In some implementations, process 800 also includes: obtaining a bitmap for recording the status of received packets; in response to receiving a first packet on a first path, determining whether there is a position corresponding to the first packet in the bitmap; and in response to determining that there is a position corresponding to the first packet in the bitmap, updating the status associated with the first packet at the position.

[0127] In some implementations, process 800 further includes: in response to determining that there is no position corresponding to the first packet in the bitmap, determining a packet to be retransmitted based on the bitmap; generating a negative acknowledgement indicating the packet to be retransmitted; and sending the negative acknowledgement to NIC 113 on the first path.

[0128] In some implementations, process 800 further includes: in response to the bitmap indicating that at least one packet associated with the RDMA message has been received, clearing a corresponding position in the bitmap for the at least one packet for recording status of packets associated with other RDMA messages.

[0129] In some implementations, process 800 also includes: in response to receiving a second packet including a retransmission marker on a second path of the multiple paths via the RDMA protocol, generating a second positive acknowledgement by adding the retransmission marker to a second positive acknowledgement for the second packet; and sending the second positive acknowledgement to NIC 113 on the second path.

[0130] In some implementations, process 800 also includes, in response to receiving the first packet, determining an address for storing the first packet in memory 122 based on the first packet; and storing the first packet at the address.

[0131] In some implementations, the first packet also includes a ninth field for indicating a transmission order on which the first packet depends, and the NIC 123 stores the first packet at the address by performing the following operations: in response to a third packet having a second packet sequence number having been stored in the memory 122, the first packet is stored at the address, and the second packet sequence number is less than the first packet sequence number.

[0132] Example Device

[0133] Fig. 9 900 is a block diagram of a computing device that can implement multiple implementations of the present disclosure. For example, the computing device 900 can be used to implement Figure 1 The hosts 110 and 120 are shown. It should be understood that Fig. 9 The computing device 900 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the implementation described in the present disclosure. Fig. 9 As shown, computing device 900 includes a computing device in the form of a general computing device 900. Components of computing device 900 may include, but are not limited to, one or more processors or processing units 910, memory 920, storage device 930, one or more communication units 940, one or more input devices 950, and one or more output devices 960.

[0134] In some implementations, the computing device 900 may be implemented as various user terminals or service terminals with computing capabilities. The service terminal may be a server, a large computing device, etc. provided by various service providers. The user terminal may be any type of mobile terminal, fixed terminal, or portable terminal, including a mobile phone, a multimedia computer, a multimedia tablet, an Internet node, a communicator, a desktop computer, a laptop computer, a tablet computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), a digital camera / camcorder, a positioning device, a gaming device, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof.

[0135] Processing unit 910 may be a real or virtual processor and may be capable of performing various processes according to a program stored in memory 920. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to increase the parallel processing capabilities of computing device 900. Processing unit 910 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0136] The computing device 900 typically includes a plurality of computer storage media. Such media may be any available media accessible to the computing device 900, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 920 may be a volatile memory (e.g., registers, cache, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof.

[0137] The storage device 930 may be a removable or non-removable medium and may include machine-readable media that can be used to store information and / or data and can be accessed within the computing device 900. The computing device 900 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not described herein, Fig. 9 As shown in , a disk drive for reading or writing from a removable, nonvolatile disk and an optical drive for reading or writing from a removable, nonvolatile optical disk can be provided. In these cases, each drive can be connected to a bus (not shown) by one or more data media interfaces.

[0138] The communication unit 940 enables communication with another computing device via a communication medium. The communication unit 940 may include a NIC 941 that enables MP-RDMA according to the implementation of the present disclosure. The NIC 941 enables the computing device 900 to perform MP-RDMA transmission with another computing device via a communication medium. For example, when the computing device 900 acts as a Figure 1 When the host 110 is shown, the NIC 941 can act as Figure 1 NIC 113 shown; and when computing device 900 acts as Figure 1 When the host 120 is shown, the NIC 841 can act as Figure 1 NIC 123 is shown.

[0139] Additionally, the functions of the components of the computing device 900 may be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the computing device 900 may operate in a networked environment using logical connections to one or more other servers, personal computers (PCs), or another general network node.

[0140] Input device 150 may be one or more various input devices, such as a mouse, keyboard, trackball, voice input device, image input device, etc. Output device 160 may be one or more output devices, such as a display, speaker, printer, etc. Computing device 900 may also communicate with one or more external devices (not shown) as needed, such as storage devices, display devices, etc. Such communication may be performed via an input / output (I / O) interface (not shown).

[0141] Example Implementation

[0142] Some example implementations of the present disclosure are listed below.

[0143] In a first aspect, the present disclosure provides a device. The device includes: a processing unit; a memory, which is coupled to the processing unit and stores instructions for execution by the processing unit, and the instructions, when executed by the processing unit, cause the device to perform an action, the action including: based on a remote direct memory access (RDMA) message to be sent from the device to another device, generating at least one packet including at least a first packet, the device and the other device having an RDMA connection via multiple paths. The first packet includes at least a plurality of fields, and the plurality of fields contain information for transmitting the first packet on a first path among the plurality of paths. The action also includes: sending at least one packet to another device on multiple paths via the RDMA protocol, the first packet being sent on the first path.

[0144] In some implementations, the multiple fields include: a first field for indicating an identifier of the first path; a second field for a packet sequence number of the first packet; a third field for indicating a message sequence number of the RDMA message; and a fourth field for indicating a sequence number of the first packet in the RDMA message.

[0145] In some implementations, sending at least one packet includes: obtaining congestion control information indicating congestion conditions of multiple paths; determining whether a first packet is allowed to be sent based on the congestion control information; and in response to determining that the first packet is allowed to be sent, sending the first packet to another device via the RDMA protocol on the first path.

[0146] In some implementations, the at least one packet also includes a second packet subsequent to the first packet, and sending the at least one packet includes: updating congestion control information in response to receiving a first positive acknowledgement for the first packet from another device on the first path; and controlling the sending of the second packet based on the updated congestion control information.

[0147] In some implementations, the first affirmative response includes at least: a fifth field for indicating an identifier of the first path; a sixth field for indicating a packet sequence number of the first packet; a seventh field for indicating a congestion notification generated by a packet forwarding device on the first path, the congestion notification indicating a congestion condition of the first path detected by the packet forwarding device; and an eighth field for indicating a maximum sequential sequence number of a packet received at another device.

[0148] In some implementations, updating the congestion control information includes, in response to receiving the first positive acknowledgement, updating the congestion control information based on at least the congestion notification indicated by the seventh field and the maximum sequential sequence number indicated by the eighth field.

[0149] In some implementations, updating the congestion control information includes: in response to receiving the first affirmative acknowledgement, determining a maximum packet sequence number of packets for which an affirmative acknowledgement has been received; determining an out-of-order tolerance level for transmission of at least one packet; determining a threshold sequence number of packets allowed to be sent on the first path based on the maximum packet sequence number and the out-of-order tolerance level; and updating the congestion control information based on at least the packet sequence number indicated by the sixth field and the threshold sequence number.

[0150] In some implementations, the first packet further includes a ninth field for indicating a transmission order on which the first packet depends. Sending at least one packet includes: determining a time interval for sending the first packet based at least on congestion control information and out-of-order tolerance; and in response to a third packet in the at least one packet immediately preceding the first packet being sent, waiting at least for the time interval before sending the first packet.

[0151] In some implementations, the actions also include: in response to receiving a negative acknowledgement from another device, determining from at least one packet a fourth packet to which the negative acknowledgement is directed; processing the fourth packet by adding a retransmission marker in the fourth packet; and retransmitting the processed fourth packet to another device via the RDMA protocol on a second path among the multiple paths, on which a positive acknowledgement was previously received.

[0152] In a second aspect, the present disclosure provides a device. The device includes: a processing unit; a memory, the memory is coupled to the processing unit and stores instructions for execution by the processing unit, the instructions, when executed by the processing unit, cause the device to perform an action, the action comprising: receiving at least one packet including at least a first packet from another device on multiple paths via a remote direct memory access (RDMA) protocol. The device and the other device have an RDMA connection via multiple paths. The first packet includes at least a plurality of fields, the plurality of fields containing information for transmitting the first packet on a first path on the multiple paths.

[0153] In some implementations, at least one packet is generated based on an RDMA message, and the plurality of fields include: a first field for indicating an identifier of a first path; a second field for indicating a first packet sequence number of the first packet; a third field for indicating a message sequence number of the RDMA message; and a fourth field for indicating a sequence number of the first packet in the RDMA message.

[0154] In some implementations, the actions further include: in response to receiving the first packet on the first path, generating a first positive acknowledgment for the first packet. The first positive acknowledgment includes at least: a fifth field for indicating an identifier of the first path; a sixth field for indicating a first packet sequence number of the first packet received by the device; a seventh field for indicating a congestion notification generated by a packet forwarding device on the first path, the congestion notification indicating a congestion condition of the first path detected by the packet forwarding device; and an eighth field for indicating a maximum sequential sequence number of packets received at the device.

[0155] In some implementations, the actions also include: obtaining a bitmap for recording the status of received packets; in response to receiving a first packet on a first path, determining whether there is a position corresponding to the first packet in the bitmap; and in response to determining that there is a position corresponding to the first packet in the bitmap, updating the status associated with the first packet at the position.

[0156] In some implementations, the actions further include: in response to determining that there is no position in the bitmap corresponding to the first packet, determining a packet to be retransmitted based on the bitmap; generating a negative acknowledgement indicating the packet to be retransmitted; and sending the negative acknowledgement to another device on the first path.

[0157] In some implementations, the actions further include: in response to the bitmap indicating that at least one packet associated with the RDMA message has been received, clearing a corresponding position in the bitmap for the at least one packet for recording status of packets associated with other RDMA messages.

[0158] In some implementations, the actions also include: in response to receiving a second packet including a retransmission marker on a second path of the multiple paths via the RDMA protocol, generating a second affirmative acknowledgement by adding the retransmission marker to a second affirmative acknowledgement for the second packet; and sending the second affirmative acknowledgement to another device on the second path.

[0159] In some implementations, the actions further include: in response to receiving the first packet, determining an address for storing the first packet in the target memory based on the first packet; and storing the first packet at the address.

[0160] In some implementations, the first packet also includes a ninth field for indicating a transmission order on which the first packet depends, and storing the first packet at the address includes: in response to a third packet having a second packet sequence number having been stored in the target memory, storing the first packet at the address, the second packet sequence number being less than the first packet sequence number.

[0161] In a third aspect, the present disclosure provides a method. The method includes: generating at least one packet including at least a first packet based on a remote direct memory access (RDMA) message to be sent from a first device to a second device, and the first device and the second device have an RDMA connection via multiple paths. The first packet includes at least a plurality of fields, and the plurality of fields contain information for transmitting the first packet on a first path among the plurality of paths. The method also includes: sending at least one packet to the second device on the plurality of paths via the RDMA protocol, the first packet being sent on the first path.

[0162] In some implementations, the multiple fields include: a first field for indicating an identifier of the first path; a second field for a packet sequence number of the first packet; a third field for indicating a message sequence number of the RDMA message; and a fourth field for indicating a sequence number of the first packet in the RDMA message.

[0163] In some implementations, sending at least one packet includes: obtaining congestion control information indicating congestion conditions of multiple paths; determining whether a first packet is allowed to be sent based on the congestion control information; and in response to determining that the first packet is allowed to be sent, sending the first packet to a second device via an RDMA protocol on a first path.

[0164] In some implementations, at least one packet also includes a second packet subsequent to the first packet, and sending the at least one packet includes: updating congestion control information in response to receiving a first positive acknowledgement for the first packet from the second device on the first path; and controlling the sending of the second packet based on the updated congestion control information.

[0165] In some implementations, the first affirmative response includes at least: a fifth field for indicating an identifier of the first path; a sixth field for indicating a packet sequence number of the first packet; a seventh field for indicating a congestion notification generated by a packet forwarding device on the first path, the congestion notification indicating a congestion condition of the first path detected by the packet forwarding device; and an eighth field for indicating a maximum sequential sequence number of packets received at the second device.

[0166] In some implementations, updating the congestion control information includes, in response to receiving the first positive acknowledgement, updating the congestion control information based on at least the congestion notification indicated by the seventh field and the maximum sequential sequence number indicated by the eighth field.

[0167] In some implementations, updating the congestion control information includes: in response to receiving the first affirmative acknowledgement, determining a maximum packet sequence number of packets for which an affirmative acknowledgement has been received; determining an out-of-order tolerance level for transmission of at least one packet; determining a threshold sequence number of packets allowed to be sent on the first path based on the maximum packet sequence number and the out-of-order tolerance level; and updating the congestion control information based on at least the packet sequence number indicated by the sixth field and the threshold sequence number.

[0168] In some implementations, the first packet further includes a ninth field for indicating a transmission order on which the first packet depends. Sending at least one packet includes: determining a time interval for sending the first packet based at least on congestion control information and out-of-order tolerance; and in response to a third packet in the at least one packet immediately preceding the first packet being sent, waiting at least for the time interval before sending the first packet.

[0169] In some implementations, the method also includes: in response to receiving a negative acknowledgement from the second device, determining from at least one packet a fourth packet to which the negative acknowledgement is directed; processing the fourth packet by adding a retransmission marker in the fourth packet; and retransmitting the processed fourth packet to the second device via the RDMA protocol on a second path among the multiple paths, on which a positive acknowledgement was previously received.

[0170] In a fourth aspect, the present disclosure provides a method. The method includes: receiving at least one packet including at least a first packet from a first device on multiple paths via a remote direct memory access (RDMA) protocol at a second device. The first device and the second device have an RDMA connection via multiple paths. The at least one packet includes a first packet received on a first path in the multiple paths. The first packet includes at least a plurality of fields, the plurality of fields containing information for transmitting the first packet on the first path on the multiple paths.

[0171] In some implementations, at least one packet is generated based on an RDMA message, and the plurality of fields include: a first field for indicating an identifier of a first path; a second field for indicating a first packet sequence number of the first packet; a third field for indicating a message sequence number of the RDMA message; and a fourth field for indicating a sequence number of the first packet in the RDMA message.

[0172] In some implementations, the method further includes: in response to receiving the first packet on the first path, generating a first positive acknowledgement for the first packet. The first positive acknowledgement includes at least: a fifth field for indicating an identifier of the first path; a sixth field for indicating a first packet sequence number of the first packet; a seventh field for indicating a congestion notification generated by a packet forwarding device on the first path, the congestion notification indicating a congestion condition of the first path detected by the packet forwarding device; and an eighth field for indicating a maximum sequential sequence number of the packet received at the second device.

[0173] In some implementations, the method further includes: obtaining a bitmap for recording the status of received packets; in response to receiving a first packet on a first path, determining whether there is a position corresponding to the first packet in the bitmap; and in response to determining that there is a position corresponding to the first packet in the bitmap, updating the status associated with the first packet at the position.

[0174] In some implementations, the method further includes: in response to determining that there is no position corresponding to the first packet in the bitmap, determining a packet to be retransmitted based on the bitmap; generating a negative acknowledgement indicating the packet to be retransmitted; and sending the negative acknowledgement to the first device on the first path.

[0175] In some implementations, the method further includes: in response to the bitmap indicating that at least one packet associated with the RDMA message has been received, clearing a corresponding position in the bitmap for at least one packet for recording status of packets associated with other RDMA messages.

[0176] In some implementations, the method further includes: in response to receiving a second packet including a retransmission marker on a second path of the multiple paths via the RDMA protocol, generating a second affirmative acknowledgement by adding the retransmission marker to a second affirmative acknowledgement for the second packet; and sending the second affirmative acknowledgement to the first device on the second path.

[0177] In some implementations, the method further includes: in response to receiving the first packet, determining an address for storing the first packet in the target memory based on the first packet; and storing the first packet at the address.

[0178] In some implementations, the first packet also includes a ninth field for indicating a transmission order on which the first packet depends, and storing the first packet at the address includes: in response to a third packet having a second packet sequence number having been stored in the target memory, storing the first packet at the address, the second packet sequence number being less than the first packet sequence number.

[0179] In a fifth aspect, the present disclosure provides a computer program product, which is tangibly stored in a non-transitory computer storage medium and includes computer executable instructions, which when executed by a device cause the device to perform the method in the third aspect or the fourth aspect.

[0180] In a sixth aspect, the present disclosure provides a computer-readable medium having computer-executable instructions stored thereon, which, when executed by a device, causes the device to perform the method in the third aspect or the fourth aspect.

[0181] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip systems (SOCs), load programmable logic devices (CPLDs), and the like.

[0182] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0183] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0184] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain the desired result. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of a separate implementation can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination.

[0185] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A device for data transmission, comprising: at least one processor; as well as The memory includes instructions that, when executed by the at least one processor, cause the at least one processor to perform the following operations: generating a plurality of packets for a remote direct memory access (RDMA) message to be sent from the device to another device, wherein a header of each packet in the plurality of packets comprises a packet sequence number, and wherein an RDMA connection via a plurality of paths is provided between the device and the another device; determining a congestion condition of a first path among the plurality of paths; Determining a time interval for sending a first packet based at least on the congestion control information of the plurality of paths and out-of-order tolerance of the plurality of packets; as well as In response to a third packet immediately preceding the first packet in transmission of the RDMA message, waiting at least the time interval before sending the first packet of the multiple packets to the other device via the first path based on the congestion condition, wherein the first packet includes a transmission order on which the first packet depends.

2. The apparatus of claim 1, wherein the first grouping comprises: a plurality of fields including information for transmitting the first packet on the first path among the plurality of paths, wherein the plurality of fields include a congestion notification field and a maximum sequence number field, the congestion notification field being generated by a packet forwarding device on the first path, the congestion notification field indicating the congestion condition of the first path detected by the packet forwarding device, and the maximum sequence number field indicating a maximum sequence number in packets received at the other device.

3. The device of claim 2, wherein the memory further comprises instructions that cause the at least one processor to: In response to receiving a first positive acknowledgement for the first packet from the other device on the first path, the congestion control information indicating the plurality of paths is updated based on at least the congestion condition indicated by the congestion notification field and the maximum sequence number indicated by the maximum sequence number field.

4. The apparatus of claim 1 , wherein the instructions for sending the first packet further comprise instructions for causing the at least one processor to: obtaining the congestion control information indicating the plurality of paths; determining whether the first packet is allowed to be sent based on the congestion control information; and In response to determining that the first packet is allowed to be sent, the first path is identified as a transmission path for the first packet based on corresponding congestion conditions of the multiple paths.

5. The device of claim 2, wherein the memory further comprises instructions that cause the at least one processor to: in response to receiving a first positive acknowledgement for the first packet from the other device on the first path, updating the congestion control information; and Transmission of a second packet among the plurality of packets is controlled based on the updated congestion control information, wherein the second packet is sent over a second path among the plurality of paths based on the updated congestion control information.

6. The apparatus of claim 5, wherein the instructions for updating the congestion control information comprise instructions that cause the at least one processor to: In response to receiving the first positive acknowledgement, determining a maximum packet sequence number of packets for which a positive acknowledgement has been received; determining an out-of-order tolerance level for transmission of the packets; Determining a threshold sequence number of the first packet based on the maximum packet sequence number and the out-of-order tolerance level; as well as The congestion control information is updated based on at least the packet sequence number and the threshold sequence number.

7. A method for data transmission, comprising: Generate a plurality of packets for a remote direct memory access (RDMA) message to be sent from a transmission device to another device, wherein a header of each of the plurality of packets includes a packet sequence number, and wherein an RDMA connection via a plurality of paths exists between the transmission device and the another device; determining a congestion condition of a first path among the plurality of paths; Determining a time interval for sending a first packet based at least on the congestion control information of the plurality of paths and out-of-order tolerance of the plurality of packets; as well as In response to a third packet immediately preceding the first packet in transmission of the RDMA message, waiting at least the time interval before sending the first packet of the multiple packets to the other device via the first path based on the congestion condition, wherein the first packet includes a transmission order on which the first packet depends.

8. The method of claim 7, wherein the first grouping comprises: a plurality of fields including information for transmitting the first packet on the first path among the plurality of paths, wherein the plurality of fields include a congestion notification field and a maximum sequence number field, the congestion notification field being generated by a packet forwarding device on the first path, the congestion notification field indicating the congestion condition of the first path detected by the packet forwarding device, and the maximum sequence number field indicating a maximum sequence number in packets received at the other device.

9. The method according to claim 8, further comprising: In response to receiving a first positive acknowledgement for the first packet from the other device on the first path, the congestion control information indicating the plurality of paths is updated based on at least the congestion condition indicated by the congestion notification field and the maximum sequence number indicated by the maximum sequence number field.

10. The method of claim 7, wherein sending the first packet further comprises: Acquire the congestion control information indicating the plurality of paths; determining whether the first packet is allowed to be sent based on the congestion control information; as well as In response to determining that the first packet is allowed to be sent, the first path is identified as a transmission path for the first packet based on corresponding congestion conditions of the multiple paths.

11. The method according to claim 8, further comprising: in response to receiving a first positive acknowledgement for the first packet from the other device on the first path, updating the congestion control information; as well as Transmission of a second packet among the plurality of packets is controlled based on the updated congestion control information, wherein the second packet is sent over a second path among the plurality of paths based on the updated congestion control information.

12. The method of claim 11, wherein updating the congestion control information comprises: In response to receiving the first positive acknowledgement, determining a maximum packet sequence number of packets for which a positive acknowledgement has been received; determining an out-of-order tolerance level for transmission of the packets; Determining a threshold sequence number of the first packet based on the maximum packet sequence number and the out-of-order tolerance level; as well as The congestion control information is updated based on at least the packet sequence number and the threshold sequence number.

13. At least one non-transitory machine-readable medium comprising instructions for data transmission, which instructions, when executed by at least one processor, cause the at least one processor to perform the following operations: generating a plurality of packets for a remote direct memory access (RDMA) message to be sent from a device to another device, wherein a header of each packet in the plurality of packets comprises a packet sequence number, and wherein an RDMA connection via a plurality of paths is provided between the device and the another device; determining a congestion condition of a first path among the plurality of paths; Determining a time interval for sending a first packet based at least on the congestion control information of the plurality of paths and out-of-order tolerance of the plurality of packets; as well as In response to a third packet immediately preceding the first packet in transmission of the RDMA message, waiting for at least the time interval before sending a first packet of the multiple packets to the other device via the first path based on the congestion condition, wherein the first packet includes a transmission order on which the first packet depends.

14. The at least one non-transitory machine-readable medium of claim 13, wherein the first packet comprises: a plurality of fields including information for transmitting the first packet on the first path among the plurality of paths, wherein the plurality of fields include a congestion notification field and a maximum sequence number field, the congestion notification field being generated by a packet forwarding device on the first path, the congestion notification field indicating the congestion condition of the first path detected by the packet forwarding device, and the maximum sequence number field indicating a maximum sequence number in packets received at the other device.

15. The at least one non-transitory machine-readable medium of claim 14, further comprising instructions that cause the at least one processor to: In response to receiving a first positive acknowledgement for the first packet from the other device on the first path, the congestion control information indicating the plurality of paths is updated based on at least the congestion condition indicated by the congestion notification field and the maximum sequence number indicated by the maximum sequence number field.

16. The at least one non-transitory machine-readable medium of claim 13, wherein the instructions for sending the first packet further comprise instructions for causing the at least one processor to: Acquire the congestion control information indicating the plurality of paths; determining whether the first packet is allowed to be sent based on the congestion control information; and In response to determining that the first packet is allowed to be sent, the first path is identified as a transmission path for the first packet based on corresponding congestion conditions of the multiple paths.

17. The at least one non-transitory machine-readable medium of claim 14, further comprising instructions that cause the at least one processor to: in response to receiving a first positive acknowledgement for the first packet from the other device on the first path, updating the congestion control information; and Transmission of a second packet among the plurality of packets is controlled based on the updated congestion control information, wherein the second packet is sent over a second path among the plurality of paths based on the updated congestion control information.

18. The at least one non-transitory machine-readable medium of claim 17, wherein the instructions for updating the congestion control information comprise instructions that cause the at least one processor to: In response to receiving the first positive acknowledgement, determining a maximum packet sequence number of packets for which a positive acknowledgement has been received; determining an out-of-order tolerance level for transmission of the packets; Determining a threshold sequence number of the first packet based on the maximum packet sequence number and the out-of-order tolerance level; as well as The congestion control information is updated based on at least the packet sequence number and the threshold sequence number.

Images