RDMA out-of-order direct writing method and system

Abstract

The invention provides an RDMA out-of-order direct writing method and system, and the system comprises a data transmitting end and a data receiving end, and does not need to carry a loaded memory writing address in each out-of-order packet. For the RDMA write message, when the data receiving end does not cache the RDMA message memory initial address, the data sending end embeds the load address in the data message, and after the data receiving end calculates and caches the RDMA message initial memory address, the subsequent data message does not need to embed the load address; for the RDMA read message, the data receiving end caches the memory initial address of the RDMA message; and the data receiving end can obtain the memory address of the load through calculation or from the message, and directly write the load into the memory. According to the invention, the RDMA protocol supports out-of-order packet receiving, the load of the out-of-order arriving data message can be directly written into the memory, and packet sorting is not needed; and the memory write address of the load is embedded in the data message only if necessary, so that the load memory utilization rate is high.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of communication, and particularly relates to an RDMA out-of-order direct writing method and system, and especially relates to a multi-path RDMA high-speed application scenario for a wisdom calculation center or a wide area network. BACKGROUND

[0002] Remote direct memory access (RDMA) is increasingly widely applied to data center networks. RDMA greatly improves the throughput of network data transmission, reduces the delay, and releases the performance burden of a CPU by implementing the main process of data transmission from the CPU to a network interface card (NIC).

[0003] Existing RDMA transmission protocols, such as RoCEv2, support single-path transmission. However, using multi-path transmission can improve transmission bandwidth by utilizing parallel paths and alleviate network congestion. However, multi-path transmission can cause data packet out-of-order, and when the receiving side finds that the packet PSN is discontinuous, it will trigger packet loss detection and retransmission, which will affect the transmission rate. Therefore, in order to support multi-path transmission, the RDMA protocol needs to be designed additionally.

[0004] Existing multi-path or RDMA methods supporting out-of-order packets have great deficiencies. For example, the Chinese application patent with the publication number CN109936510A and the name “Multi-path RDMA transmission” proposes a multi-path RDMA transmission method. This method embeds path ID, synchronization packet flag bit, retransmission packet flag bit and other fields in the sending packet or ACK packet, and realizes functions such as multi-path congestion window update, path detection and elimination, and synchronization packet delay DMA. The patent mentions “in response to receiving the first packet, determining an address for storing the first packet in a target memory, storing the first packet at the address”, and “for RDMA read or write operations, the RoCEv2 data header already includes the target address in the memory to be written, and the data is directly stored at the target address”. However, in the multi-path out-of-order transmission scenario, the “first packet” in the patent may not arrive at the response end first. Moreover, the target address field (address length is 64B) is inserted in the header of each read or write operation RoCEv2 data packet, which additionally increases the length of the data packet header and reduces the transmission efficiency of the actual data payload.

[0005] Therefore, the existing RDMA transmission protocol does not support out-of-order packets, and the existing multi-path RDMA transmission method has certain functional defects, and the address field additionally inserted in each read-write operation data packet also affects the improvement of data transmission efficiency. SUMMARY

[0006] The purpose of the present application is to overcome the defects of the existing RDMA transmission protocol which does not support out-of-order packet receiving, and further overcome the deficiencies of the existing multi-path RDMA transmission method in function and transmission performance, and propose an RDMA out-of-order direct writing method, which relates to a data sending end and a data receiving end.

[0007] When the data receiving end does not cache the start address of the RDMA message memory, the data sending end embeds the payload address in the sent data packet to form a data packet and sends it.

[0008] When the data receiving end receives the data packet, if it does not cache the start address of the RDMA message memory, it writes the data packet into the embedded memory address in the data packet, then calculates the start memory address of the RDMA message and caches it; if it has cached the start memory address of the RDMA message, it calculates the memory address of the data packet and writes the data packet directly into the memory.

[0009] As an improvement of the above method, the formula for calculating the memory address pkt_addr of the data packet by the data receiving end is:

[0010] pkt_addr = rdma_msg_start_addr + (pkt_psn - rdma_msg_start_psn) x PMTU

[0011] Wherein, pkt_psn is the PSN of the data packet; rdma_msg_start_psn is the start PSN of the RDMA message; rdma_msg_start_addr is the start memory address of the RDMA message; PMTU is the path maximum transmission unit.

[0012] As an improvement of the above method, the formula for calculating the start memory address rdma_msg_start_addr of the RDMA message by the data receiving end is:

[0013] rdma_msg_start_addr = pkt_addr - (pkt_psn - rdma_msg_start_psn) x PMTU

[0014] Wherein, pkt_addr is the payload memory address embedded in the data packet; pkt_psn is the PSN of the data packet; rdma_msg_start_psn is the start PSN of the RDMA message.

[0015] As an improvement of the above method, after the data receiving end calculates the start memory address of the RDMA message and caches it, it informs the data sending end in the reply acknowledgement packet that the start memory address has been cached.

[0016] As an improvement of the above method, the data receiving end records the received data packet PSN to the bitmap after writing the data packet to the memory.

[0017] As an improvement of the above method, the data receiving end records the received data packet PSN to the bitmap, and triggers the ACK confirmation when the bitmap continuously receives the data packet, and triggers the retransmission of the data packet not received in the interval when the out-of-order distance of the received data packet PSN exceeds the preset network out-of-order distance.

[0018] As an improvement of the above method, when the data length of the RDMA message is greater than 2*PMTU, the data sending end generates a plurality of OOO RDMAWRITE Only packets and an OOO RDMAWRITE Only Last packet; the OOO RDMAWRITE Only Last packet is used to indicate that the data packet is the last packet of the RDMA message; wherein, PMTU is the path maximum transmission unit.

[0019] The application also provides an RDMA out-of-order direct writing system based on the above method, and the system comprises:

[0020] The data sending end is used for sending the data packet; when the data receiving end does not cache the memory start address of the RDMA message, the embedded payload address is embedded in the sent data packet.

[0021] The data receiving end is used for receiving the data packet; if the data receiving end does not cache the start memory address of the RDMA message, the data packet is written to the embedded memory address in the data packet, and then the start memory address of the RDMA message is calculated and cached; if the start memory address of the RDMA message is cached, the memory address of the data packet is calculated, and the data packet is directly written to the memory.

[0022] As an improvement of the above system, the data receiving end further comprises:

[0023] The bitmap module is used for recording the received data packet PSN of the data receiving end, and triggering the retransmission of the data packet not received in the interval when the out-of-order distance of the data packet PSN exceeds the preset network out-of-order distance.

[0024] As an improvement of the above system, in the bitmap module, the bit width of the bitmap is at least the preset network out-of-order distance.

[0025] Compared with the prior art, the application has the following advantages:

[0026] 1. The application supports direct writing of the payload of out-of-order RDMA datagrams into memory without packet sorting, and the memory write address of the payload can be obtained from the embedded fields of the datagram or by calculation;

[0027] 2. The application embeds the memory write address of the payload into the sending datagram only when the RDMA message receiving end does not cache the starting memory address, and only the first few packets need to embed the address during the write operation, and no address needs to be embedded during the read operation, so the additional fields of the header are less, and the bandwidth utilization of the data payload is high;

[0028] 3. The application maintains a packet receiving bitmap at the RDMA data receiving end, is compatible with read and write operations, and only needs to instantiate a bitmap of the depth of a QP on a network card, the bitmap storage occupies less space, and when the out-of-order distance of the data packet is greater than a threshold, the corresponding bitmap interval is sent to the retransmission initiator, and the performance of the packet loss detection and retransmission process is high. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 Fig. 1 shows a flow chart of the RDMA out-of-order direct writing method;

[0030] Figure 2 Fig. 2 shows a system block diagram of the RDMA out-of-order direct writing method;

[0031] Figure 3 Fig. 3 shows a structure diagram of the RDMA WRITE message related datagram;

[0032] Figure 4 Fig. 4 shows a structure diagram of the RDMA READ message related datagram;

[0033] Figure 5 Fig. 5 shows a schematic diagram of directly writing the payload of the RDMA datagram into the corresponding position in the memory;

[0034] Figure 6 Fig. 6 shows a structure diagram of the RDMA packet receiving bitmap. DETAILED DESCRIPTION

[0035] The technical solutions of the application will be described in detail below with reference to the accompanying drawings.

[0036] As shown in Fig. 1, the RDMA out-of-order direct writing method comprises the following steps: Figure 1As shown, the present application provides an RDMA out-of-order direct write method and system, which involves a data sending end and a data receiving end. When sending data, the memory write address of the payload does not need to be carried in each out-of-order packet. For an RDMA write message, the data sending end is the request end. When the data receiving end does not cache the memory start address of the RDMA message, the payload address is embedded in the data packet. When the data receiving end calculates and caches the memory start address of the RDMA message, the data sending end is informed in the reply acknowledgement packet, and the subsequent data packet does not need to embed the payload address. For an RDMA read message, the data sending end is the response end, and the data receiving end caches the memory start address of the RDMA message. The data receiving end can calculate or obtain the memory address of the payload from the packet, and directly write the payload into the memory without sorting the out-of-order packets. The data receiving end records the received data packet PSN to a bitmap. When the bitmap continuously receives packets, an ACK confirmation is triggered. When the out-of-order distance of the received packet PSN exceeds the preset network out-of-order distance, the corresponding bitmap interval is sent to the retransmission initiator to trigger the retransmission of the data packets in the interval that have not been received.

[0037] For an RDMA read message or a write message, the calculation formula of the memory address pkt_addr of the data packet payload calculated by the data receiving end is as follows:

[0038] pkt_addr = rdma_msg_start_addr + (pkt_psn - rdma_msg_start_psn) × PMTU

[0039] Wherein, pkt_psn is the packet PSN, rdma_msg_start_psn is the start PSN of the RDMA message, rdma_msg_start_addr is the start memory address of the RDMA message, and PMTU is the path maximum transmission unit.

[0040] For an RDMA write message, the calculation formula of the start memory address rdma_msg_start_addr of the RDMA message calculated by the data receiving end is as follows:

[0041] rdma_msg_start_addr = pkt_addr - (pkt_psn - rdma_msg_start_psn) × PMTU

[0042] Wherein, pkt_addr is the payload memory address embedded in the packet, pkt_psn is the packet PSN, and rdma_msg_start_psn is the start PSN of the RDMA message.

[0043] Figure 2Figure 1 is a system diagram of the method of the present application. The request side and response side hosts of RDMA are interconnected through a multi-path network. The left side of the figure is the RDMA request side, and the right side is the RDMA response side. The RDMA function is implemented by the network interface card (NIC) of the request side and the response side. The RDMA NIC can bypass the central processing unit (CPU) and directly implement read and write to the memory (Memory) storage, thereby realizing large-scale data network transmission across hosts.

[0044] The native RDMA NIC supports only single-path transmission according to the RDMA related transmission protocol specification, such as RoCEv2.

[0045] After the RDMA out-of-order direct write system of the present application is applied, the request side and the response side of RDMA are simultaneously deployed with an RDMA NIC supporting out-of-order (OOO) direct write, i.e., OOO RDMA NIC. The OOO RDMA NIC is deployed with an RDMA data sending end and an RDMA data receiving end. For an RDMA write message, the RDMA data sending end is the request side; for an RDMA read message, the RDMA data sending end is the response side.

[0046] For the OOO RDMA NIC, the payload of the out-of-order RDMA data packet is directly written into the memory, and there is no packet sorting at the receiving end.

[0047] The process of the native RDMA NIC processing the RDMA WRITE message is as follows. When the user application program issues an RDMA WRITE message, for the native RDMA NIC, the request side NIC parses the RDMA WRITE message, and when the data length of the RDMA WRITE message is greater than 3*PMTU, the request side NIC will manufacture an RDMA WRITE First packet, a plurality of RDMA WRITE Middle packets, and an RDMA WRITE Last packet to send to the response side. Among them, Figure 3 101 is the structure of the RDMA WRITE First packet, Figure 3 102 is the structure of the RDMA WRITE Middle and RDMA WRITE Last packets. Figure 3The MAC, IP, UDP, BTH, RETH, ICRC and Payload in the figure are respectively a header of a data link layer, a header of a network layer, a header of a user datagram protocol, a header related to RDMA, a check bit related to RDMA, and a data load of RDMA. The RDMAWRITE First packet has a RETH header, and the RETH header contains a write start address, a length and a key of the data load of the RDMAWRITE message in the memory of the response end. The network card of the response end sequentially receives the RDMAWRITE First packet, the RDMAWRITE Middle packet and the RDMAWRITE Last packet, and then sequentially writes the load of the packet into the specific address of the memory of the response end.

[0048] However, the original RDMA NIC cannot perform the RDMAWRITE operation in the out-of-order transmission scenario. In the out-of-order transmission scenario, the original RDMA NIC considers that packet loss occurs in the network and triggers packet loss retransmission because the sequence of the RDMAWRITE First packet, the RDMAWRITE Middle packet and the RDMAWRITE Last packet arriving at the response end is chaotic and the packet number (PSN) of the received packet is discontinuous. Thus, the RDMA transmission efficiency is greatly affected, and the RDMAWRITE First packet does not necessarily arrive at the response end RDMANIC first, and the out-of-order RDMA WRITE Middle packet and RDMAWRITE Last packet cannot be written into the memory of the response end and are directly discarded.

[0049] The RDMA out-of-order direct writing method provided in the application relates to an RDMA data sending end and an RDMA data receiving end. The RDMA data sending end is a requestor for an RDMA write message and is a responder for an RDMA read message. The specific process is as follows.

[0050] The RDMA data sending end embeds the write address of the data packet load in the memory of the RDMA data receiving end in a packet and sends the packet when the corresponding RDMA data receiving end does not cache the start memory address of the RDMA message.

[0051] The RDMA data receiving end receives the packet, writes the packet load into the embedded memory address in the packet, calculates the start memory address of the RDMA message and caches the start memory address, sets the cached flag to 1 in the reply acknowledgement packet, and writes the load directly into the memory if the start memory address of the RDMA message is cached.

[0052] The RDMA data receiving end writes the load of the out-of-order packet directly into the memory and records the received packet PSN to a bitmap.

[0053] The present application supports RDMA WRITE operation in out-of-order transmission scenario. After the RDMA out-of-order write system applying the present application, when the user application issues the RDMA WRITE message, if the data length of the RDMA message is greater than 2*PMTU, the request end OOO RDMA NIC will generate a plurality of OOO RDMA WRITE Only packets and one OOO RDMA WRITE Only Last packet. The OOO RDMA WRITE Only Last packet is used to indicate that the data packet is the last packet of the RDMA message. When the response end does not cache the memory address of the RDMA message, the request end network card will embed the write address of the packet payload in the memory of the response end in each OOO RDMA WRITE Only packet and OOO RDMA WRITE Only Last packet. The packet structure is shown in FIG. 101, wherein the RETH header stores the write address of the packet payload in the memory of the response end. Therefore, when the response end receives the out-of-order OOO RDMA WRITE Only packet and OOO RDMA WRITE Only Last packet, if the starting memory address of the RDMA message has not been cached, the response end OOO RDMA NIC can directly write the payload into the memory according to the embedded write address of the payload in the memory, without the need of packet sorting. Figure 3

[0054] In particular, the response end OOO RDMA NIC of the present application has an address caching mechanism, which reduces the number of data packets embedding the write address of the payload in the memory and improves the transmission efficiency. When the response end OOO RDMA NIC has not cached the starting memory address of the RDMA message, after receiving the OOO RDMA WRITE Only packet and OOO RDMA WRITE Only Last packet embedding the write address of the payload in the memory, the starting memory address of the RDMA message can be calculated and cached. Subsequently, the response end OOO RDMA NIC sets the cached flag to 1 in the reply acknowledgement packet. After the request end OOO RDMA NIC receives the acknowledgement packet with the cached flag set to 1, the subsequent OOO RDMA WRITE Only packet and OOO RDMA WRITE Only Last packet for the RDMA message will no longer embed the write address of the payload in the memory. The packet structure is shown in FIG. 102, so as to achieve the effect of reducing the header length and improving the bandwidth utilization of the data payload. Figure 3

[0055] The above-mentioned response end of the RDMA write message calculates the starting memory address rdma_msg_start_addr of the RDMA write message, and the calculation formula is as follows:

[0056] ​​rdma_msg_start_addr=pkt_addr-(pkt_psn-rdma_msg_start_psn)×PMTU

[0057] Where pkt_addr is the memory address of the payload embedded in the message, pkt_psn is the message PSN, and rdma_msg_start_psn is the starting PSN of the RDMA message.

[0058] The responding end OOO RDMANIC can calculate the memory write address of the message payload and write the data payload directly to memory. After the responding end OOO RDMANIC calculates and caches the starting memory address of the RDMAWRITE message, for subsequent incoming data packets, the responding end OOO RDMANIC can directly write the message payload to a specific memory location. The memory write address of the message payload can be calculated from the data packet number, the starting memory address of the RDMA message, and the starting packet number, using the same formula as for calculating the memory write address of the data packet payload in the RDMAREAD message scenario.

[0059] The process of a native RDMA NIC handling an RDMA READ message is as follows: When a user application sends an RDMAREAD message, the requesting network card of the native RDMA NIC parses the RDMAREAD message and creates an RDMAREAD REQ packet, which is then sent to the responding end. The structure of the RDMAREAD REQ packet is as follows: Figure 4 As shown in section 201, the RETH header contains the address and length of the data payload of the RDMAREAD message in the responding network interface card's memory. After receiving the RDMAREAD REQ packet, if the RDMAREAD message data length is greater than 2 * PMTU, the responding network interface card will reply with an RDMAREAD RESPONSE First packet, several RDMAREAD RESPONSE Middle packets, and one RDMAREAD RESPONSE Last packet. The structures of the RDMAREAD RESPONSE First and RDMAREAD RESPONSE Last packets are as follows... Figure 4 As shown in Figure 202, the structure of the RDMAREAD RESPONSE Middle packet is as follows: Figure 4 As shown in 203, the AETH header records relevant information such as the RDMA message sequence number (MSN).

[0060] However, the native RDMANIC cannot perform RDMAREAD operations in out-of-order transmission scenarios. In out-of-order transmission scenarios, because the order in which RDMA READ RESPONSE First, RDMA READ RESPONSE Middle, and RDMAREAD RESPONSE Last packets are returned to the requesting end is disordered, the requesting RDMANIC will trigger packet loss retransmission due to the discontinuous PSN numbers received, which greatly affects transmission efficiency.

[0061] This application supports RDMAREAD operation in out-of-order transmission scenarios. After applying the RDMA out-of-order direct write system of this application, when the user application sends an RDMAREAD message, the requesting end OOO RDMA NIC parses the RDMAREAD message and sends an OOO RDMAREAD REQ packet to the responding end, with the packet structure as follows: Figure 4 The 201 response is the same. Upon receiving the OOO RDMAREADREQ packet, if the message data length is greater than 2 * PMTU, the responding end's OOO RDMANIC will generate several OOO RDMA READOnly packets and one OOO RDMAREAD Only Last packet. The OOO RDMAREAD Only Last packet indicates that this data packet is the last segment of the RDMA message. The message structure of OOO RDMAREAD Only is as follows: Figure 4 As shown in 202, the message structure of OOORDMAREAD Only Last is as follows: Figure 4 As shown in Figure 203, when the requesting end OOO RDMANIC receives out-of-order OOO RDMAREAD Only or OOO RDMAREAD Only Last packets, the network card can calculate the memory write address of the packet payload based on the packet number, the starting memory address of the RDMA message, and the starting PSN, and write the data payload to memory via DMA without reordering the packets.

[0062] The formula for calculating the memory write address pkt_addr of the data packet payload is as follows:

[0063] pkt_addr=rdma_msg_start_addr+(pkt_psn-rdma_msg_start_psn)×PMTU

[0064] Where pkt_psn is the message PSN, rdma_msg_start_psn is the starting PSN of the RDMA message, rdma_msg_start_addr is the starting memory address of the RDMA message, and PMTU is the path maximum transmission unit.

[0065] For RDMAREAD messages, or RDMAWRITE messages where the starting memory address has been cached at the responding end, the calculation of the memory write address for the packet payload and the process of DMA writing the packet payload to memory are as follows: Figure 5 As shown.

[0066] The bitmap module is used to record the packet number (PSN) of the data packets received by the RDMA data receiver. When the out-of-order distance of the received packet PSN exceeds the preset network out-of-order distance, the retransmission of the packet corresponding to the PSN that has not been received within the relevant bitmap interval is triggered.

[0067] The bitmap module is located at the RDMA data receiver and uses a BITMAP to record packet reception. The bit width of the BITMAP is the size of the transmission window or the network out-of-order distance. The first bit of the bitmap corresponds to the expected PSN of the received packet, and the PSNs of subsequent bits increase sequentially.

[0068] The bitmap module is instantiated on the RDMA data receiver. Specifically, for an RDMAWRITE message, the bitmap module of the corresponding QP on the responding end is activated to record the received out-of-order packet numbers: 000 RDMAWRITE Only and 000 RDMAWRITE Only Last. For an RDMAREAD message, the bitmap module of the corresponding QP on the requesting end is activated to record the received out-of-order packet numbers: 0000 RDMAREAD RESPONSE Only and 0000 RDMAREAD RESPONSE Only Last. Therefore, an 0 ...

[0069] The bitmap module's bit width should be at least the preset network out-of-order distance. Placing the bitmap module at the RDMA data receiver allows for real-time recording of packet reception at the destination of out-of-order RDMA data packets, reducing the bitmap width and lowering resource consumption.

[0070] The receive bitmap is compatible with read and write operations. That is, on the same OOO RDMANIC, only one bitmap with a QP depth needs to be instantiated. Located at the out-of-order receiver, the bitmap's storage space is small.

[0071] Preferably, the bit width of the bitmap should be at least the preset network out-of-order distance to enable acknowledgment of out-of-order packets and retransmission of lost packets. When hardware resources are sufficient, the bit width of the bitmap can be expanded to the size of the transmission window of the RDMA data transmitter, which helps to prevent packet loss caused by the bitmap being too small when packet loss occurs.

[0072] like Figure 6This is an example bitmap structure. The total bitmap width is 128, which is divided into four sub-bitmap regions with a width of 32 for ease of description. In an RDMAWRITE scenario, the bitmap is located at the RDMA response end. Assuming the response end expects packet number 1 before sending the RDMAWRITE message, the first bit of the response end's bitmap (e.g., ...) is... Figure 5 The packet number corresponding to the first_bit identifier is 1, that is, bit_0 corresponds to packet number 1, bit_1 corresponds to packet number 2, bit_2 corresponds to packet number 3, and so on.

[0073] The response end's OOO RDMANIC bitmap receives out-of-order OOO RDMAWRITE Only and OOO RDMA WRITE OnlyLast packets, and has the ability to record the number of incoming packets and perform deduplication. When the corresponding bit of the received packet's PSN in the bitmap is 0, the bit is set to 1 and the message payload is written to memory. When the corresponding bit of the received packet's PSN is already 1, the packet is considered a duplicate and is lost. Let... Figure 5 The packet to be received is number 1, and no packet has arrived yet. When packet number 2 arrives, since bit_1 is 0, the payload of the packet is written to memory by DMA and bit_1 is set to 1. When packet number 2 arrives again, since bit_1 is 1, the packet is a duplicate packet and is lost.

[0074] The responding end's OOO RDMANIC bitmap receives out-of-order OOO RDMAWRITE Only and OOO RDMA WRITE OnlyLast packets, and has sliding acknowledgment capability. When the received packet PSN equals the expected received packet PSN, the bitmap slides one consecutive bit forward from the first bit (representing received packets), updates the expected received packet PSN, and triggers an acknowledgment (ACK) for the received packet PSN. Let... Figure 5 In the bitmap, bits 0, 1, 2, and 3 are 0, 1, 0, and 1 respectively, indicating that the expected packet number is 1. When the packet with packet number 1 arrives, the first bit of the bitmap is updated to the position corresponding to the original bit 2, that is, the bitmap is shifted two bits to the left, and the expected packet number is updated to 3, triggering the ACK reception confirmation for packets with packet numbers 1 and 2.

[0075] The responding end, OOO RDMA NIC bitmap, receives OOO RDMA WRITE Only and OOO RDMA WRITE OnlyLast packets, and has the capability to detect packet loss and trigger retransmission based on the out-of-order distance. When the out-of-order distance of the received packet PSN exceeds the preset network out-of-order distance, the corresponding bitmap interval is sent to the retransmission initiator, triggering the retransmission of packets with 0 bits (representing unreceived packets) of PSN within the interval. Let... Figure 5The preset network out-of-order distance is 32. Therefore, when the received packet number falls within bitmap area 2, retransmission of the packet number corresponding to all 0 bits in bitmap area 0 will be triggered; similarly, when the received packet number falls within bitmap area 3, retransmission of the packet number corresponding to all 0 bits in bitmap area 1 will be triggered. Packet loss is determined by the received packet out-of-order distance being greater than the preset network out-of-order distance.

[0076] Preferably, in the RDMAWRITE scenario, when the responding end detects packet loss according to the above process, it can send the BITMAP range containing the lost packets (such as sub-bitmap region 0 or sub-bitmap region 1) back to the requesting end via a BITMAP-NAK packet. The requesting end, by scanning the positions of 0 bits in the BITMAP, triggers the retransmission of the packet with the corresponding data packet number (retransmitting OOO RDMAWRITEOnly or OOO RDMAWRITE Only Last). In the RDMAREAD scenario, when the requesting end detects packet loss according to the above process, it can send the BITMAP range containing the lost packets (such as sub-bitmap region 0 or sub-bitmap region 1) to the retransmission packet creation module. The retransmission packet creation module, by scanning the 0 bits in the BITMAP, triggers the retransmission of the packet with the corresponding data packet number (sending several OOO RDMAREAD REQ packets, the data range requested by these packets being a subset of the original RDMAREAD message data range).

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of this application do not depart from the spirit and scope of the technical solutions of this application, and should all be covered within the scope of the claims of this application.

Claims

1. A method for RDMA out-of-order direct write, involving a data transmitter and a data receiver; the method includes: When the data receiver does not cache the memory start address of the RDMA message, the data sender embeds the payload address into the data packet to form a data packet and sends it. When the data receiver receives a data packet, if the starting memory address of the RDMA message is not cached, it writes the data packet to the memory address embedded in the data packet, then calculates and caches the starting memory address of the RDMA message. If the starting memory address of the RDMA message has been cached, the memory address of the data packet is calculated, and the data packet is written directly into memory.

2. The RDMA out-of-order direct write method according to claim 1, characterized in that, The formula for calculating the memory address pkt_addr of a data packet at the data receiver is: pkt_addr=rdma_msg_start_addr+(pkt_psn-rdma_msg_start_psn)×PMTU Where pkt_psn is the PSN of the data packet; rdma_msg_start_psn is the starting PSN of the RDMA message; rdma_msg_start_addr is the starting memory address of the RDMA message; and PMTU is the path maximum transmission unit.

3. The RDMA out-of-order direct write method according to claim 1, characterized in that, The formula for calculating the starting memory address rdma_msg_start_addr of the RDMA message at the data receiver is: rdma_msg_start_addr=pkt_addr-(pkt_psn-rdma_msg_start_psn)×PMTU Where pkt_addr is the memory address of the payload embedded in the data packet; pkt_psn is the PSN of the data packet; and rdma_msg_start_psn is the starting PSN of the RDMA message.

4. The RDMA out-of-order direct write method according to claim 1, characterized in that, After the data receiver calculates and caches the starting memory address of the RDMA message, it informs the data sender that the starting memory address has been cached in the reply acknowledgment packet.

5. The RDMA out-of-order direct write method according to claim 1, characterized in that, After the data receiving end writes the data packet into memory, it records the received data packet PSN in the bitmap.

6. The RDMA out-of-order direct write method according to claim 5, characterized in that, After the data receiving end records the PSN of the received data packets in the bitmap, it triggers ACK confirmation when the bitmap receives packets continuously. When the out-of-order distance of the PSN of the received data packets exceeds the preset network out-of-order distance, it sends the corresponding bitmap interval to the retransmission initiator to trigger the retransmission of data packets that have not been received within the interval.

7. The RDMA out-of-order direct write method according to claim 1, characterized in that, When the data length of an RDMA message is greater than 2×PMTU, the data sender generates several OOO RDMAWRITE Only packets and one OOO RDMAWRITE Only Last packet; the OOO RDMAWRITE Only Last packet is used to indicate that the data packet is the last packet of the RDMA message; where PMTU is the path maximum transmission unit.

8. An RDMA out-of-order direct write system, implemented based on the method of any one of claims 1-7, the system comprising: The data sender is used to send data packets; when the data receiver does not cache the memory start address of the RDMA message, the data packet being sent contains the payload address. and The data receiver is used to receive data packets. If the data receiver does not cache the starting memory address of the RDMA message, it writes the data packet into the memory address embedded in the data packet, and then calculates and caches the starting memory address of the RDMA message. If the starting memory address of the RDMA message has been cached, the memory address of the data packet is calculated, and the data packet is written directly into memory.

9. The RDMA out-of-order direct write system according to claim 8, characterized in that, The data receiving end also includes: The bitmap module is used to record the PSN of data packets received by the data receiver. When the out-of-order distance of the data packet PSN exceeds the preset network out-of-order distance, the corresponding bitmap interval is sent to the retransmission initiator to trigger the retransmission of data packets not received within the interval.

10. The RDMA out-of-order direct write system according to claim 9, characterized in that, In the bitmap module, the bit width of the bitmap is at least the preset network out-of-order distance.

Images