Packet receiving bitmap management method for out-of-order RDMA (Remote Direct Memory Access)

Abstract

In a transmission scene with large network delay or large out-of-order distance, in order to solve the problems that the bit width of a packet receiving bitmap of an RDMA data receiving end is too large and storage resources are occupied too much, the invention provides a packet receiving bitmap management method of out-of-order RDMA. According to the method, an RDMA queue does not apply for a storage space in advance for an annular packet receiving bitmap of Q), the annular packet receiving bitmap is segmented into a plurality of sub-bitmaps, and a sub-bitmap space is dynamically applied or released from an available sub-bitmap address pool according to needs; the available sub-bitmap address pool is a queue, all sub-bitmap addresses are queued into the address pool during initialization, when the QP needs a sub-bitmap, one sub-bitmap address is dequeued from the address pool, and after a certain sub-bitmap of the QP is used, the used sub-bitmap address is queued back into the address pool; when a certain sub-bitmap of the QP is 1, the sub-bitmap space can be released in advance. According to the invention, the RDMA data receiving end supports the packet receiving bitmap with a large bit width, the occupied bitmap storage resources are small, and the utilization rate is high.

Description

Technical Field

[0001] This application belongs to the field of communication technology, specifically relating to a method for managing the receive bitmap of out-of-order RDMA. Background Technology

[0002] Remote Direct Memory Access (RDMA) technology can bypass the central processing unit (CPU) to enable large-scale memory data read and write operations between network nodes, and its application in data center networks is becoming increasingly widespread. To fully utilize parallel path bandwidth between RDMA nodes and reduce network congestion, the RDMA transmission protocol needs to expand from supporting only single paths to supporting multiple paths. To achieve large-scale data transmission between long-distance nodes, RDMA applications need to expand from data center networks to wide area networks (WANs).

[0003] However, multipath RDMA transmission can cause RDMA data receivers to receive messages out of order. In order to record the status of arriving data packets at the receiving end, a bitmap data structure is required. In the bitmap, 0 bits represent that the data packet has not been received, and 1 bit represents that the data packet has been received.

[0004] In transmission scenarios with high network latency or long out-of-order distances, the bitmap width of the RDMA data receiver is relatively large. Furthermore, because each QP (Queries Per Instance) requires an independent receive bitmap space, the receive bitmap of the RDMA network card occupies excessive hardware storage space, posing challenges to hardware design. Therefore, how to efficiently implement and manage the session bitmap within limited resource space has become an important research problem.

[0005] Existing methods, such as the Chinese invention patent CN113726486A entitled "A Method, System, and Storage Medium for Deduplication of Packets in a Parallel Redundant Network," achieve packet deduplication and sorting through boundary interval windows and bitmaps. In this method, the bitmaps for each session are stored in a single bitmap, with sufficient space reserved for each session. Because the space for the bitmap needs to be pre-allocated, resource consumption is significant as the number of sessions increases and the bitmap width required per session grows, making hardware implementation difficult.

[0006] Therefore, the aforementioned bitmap management method for tracking out-of-order data packets has significant loopholes and shortcomings, and cannot be applied to the hardware implementation of large-width receive bitmaps in out-of-order RDMA transmission scenarios. Summary of the Invention

[0007] The purpose of this application is to overcome the drawback of excessive storage space consumption of large-width packet receiving bitmaps in RDMA data receivers.

[0008] To achieve the above objectives, this application proposes a packet receive bitmap management method for out-of-order RDMA, comprising:

[0009] The packet receiving bitmap is divided into several sub-bitmaps. Each sub-bitmap does not request storage space in advance, and the addresses of all sub-bitmaps are recorded in the sub-bitmap address pool.

[0010] The RDMA queue dynamically requests or releases sub-bitmap space from the available sub-bitmap address pool as needed;

[0011] When a QP needs a sub-bitmap, it dequeues a sub-bitmap address from the address pool. When a QP finishes using a sub-bitmap, it enqueues the used sub-bitmap address back into the address pool.

[0012] When a sub-bitmap of a QP is filled with received flags, release the space of that sub-bitmap.

[0013] As an improvement to the above method, the sub-bitmap address pool is a first-in-first-out queue.

[0014] As an improvement to the above method, the receive bitmap of a QP is a circular array that stores sub-bitmap nodes; each sub-bitmap node contains the address, valid bits, and all receive flag bits of the sub-bitmap; the head pointer of the receive bitmap points to the bit position of the bitmap corresponding to the expected receive packet number of the QP.

[0015] As an improvement to the above method, when the valid bit and all receive flag bits of a certain sub-bitmap node are all marked as received, the sub-bitmap is regarded as a fully received sub-bitmap when the receive bitmap header pointer moves.

[0016] As an improvement to the above method, the operation mechanism of the packet receiving bitmap includes:

[0017] When receiving a data packet, if the PSN of the data packet is equal to the expected packet number, the receiving bitmap head pointer moves the valid receiving bits forward continuously until it encounters an unreceived bit. The sub-bitmap space crossed by the receiving bitmap head pointer can be released. If the PSN of the data packet is greater than the expected packet number, the corresponding bit position of the PSN in the receiving bitmap is set to a received flag. If the valid bit of the sub-bitmap node to which the PSN of the data packet falls is an unreceived flag, a sub-bitmap address is requested from the available sub-bitmap address pool.

[0018] As an improvement to the above method, the operation mechanism of the packet receiving bitmap also includes:

[0019] When all bits in a sub-bitmap are marked as received, all receive flags of that sub-bitmap node are set to received, allowing the sub-bitmap address to be released back to the address pool ahead of schedule.

[0020] As an improvement to the above method, the operation mechanism of the packet receiving bitmap also includes:

[0021] When the out-of-order distance of the received packet PSN exceeds the preset network out-of-order distance, the retransmission of the packet corresponding to the packet number with the unreceived packet identifier bit in the corresponding bitmap interval is triggered, and the corresponding bitmap interval is sent to the retransmission initiator.

[0022] As an improvement to the above method, all sub-bitmaps are stored in a sub-bitmap resource pool; sub-bitmaps are indexed by sub-bitmap addresses; free sub-bitmap addresses are stored in an available sub-bitmap address pool; all QPs share the sub-bitmap resource pool and apply for and release resources as needed.

[0023] Compared with existing technologies, the advantages of this application are:

[0024] 1. This application supports maintaining a large-width ring-shaped packet receiving bitmap at the data receiver in RDMA. All QPs share the sub-bitmap resource pool, resulting in low bitmap storage resource consumption and high utilization efficiency.

[0025] 2. When all bits of a sub-bitmap are 1, setting the all-1 flag to 1 can release the sub-bitmap space in advance, further saving the resource usage of the sub-bitmap;

[0026] 3. This application utilizes the received bitmap and the out-of-order distance of data packets to perform packet loss detection, and sends the corresponding bitmap range to the retransmission initiator. The packet loss detection and retransmission algorithm has high performance. Attached Figure Description

[0027] Figure 1 The diagram shown is a block diagram of the receive bitmap management method for out-of-order RDMA.

[0028] Figure 2 The diagram shown is a block diagram of the packet receiving bitmap for each QP;

[0029] Figure 3 The diagram shows the relationship between the sub-bitmap resource pool and the sub-bitmap address.

[0030] Figure 4 The diagram shown is a block diagram of the available sub-bitmap address pool. Detailed Implementation

[0031] The technical solution of this application will be described in detail below with reference to the accompanying drawings.

[0032] like Figure 1 As shown, this application provides a method for receiving bitmap management in out-of-order RDMA, the method comprising:

[0033] The RDMA queue pair (QP) does not pre-allocate storage space for its circular receive bitmap. Instead, it is segmented into several sub-bitmaps and dynamically allocates or releases sub-bitmap space from the available sub-bitmap address pool as needed. The available sub-bitmap address pool uses a first-in-first-out queue. During initialization, all sub-bitmap addresses are enqueued into the address pool. When the QP needs a sub-bitmap, a sub-bitmap address is dequeued from the address pool. When a sub-bitmap of the QP is finished using it, the used sub-bitmap address is enqueued back into the address pool. When a sub-bitmap of the QP is all 1s, the space of that sub-bitmap can be released in advance.

[0034] A QP's receive bitmap uses a circular array of sub-bitmap nodes. Each node contains the address, valid bits, and all-1 flag of the sub-bitmap. The head pointer of the receive bitmap points to the bit position of the bitmap corresponding to the expected receive packet number of the QP.

[0035] All sub-bitmaps are stored in the sub-bitmap resource pool. Sub-bitmaps can be indexed by their addresses. Idle sub-bitmap addresses are stored in the available sub-bitmap address pool. All QPs share the sub-bitmap resource pool and can apply for and release resources as needed.

[0036] The operation mechanism of the packet receiving bitmap is as follows:

[0037] When receiving a data packet, if the data packet PSN is equal to the expected packet number, the receiving bitmap head pointer can move forward continuously by valid receiving bits until it encounters an unreceived bit. The sub-bitmap space crossed by the bitmap head pointer can then be released.

[0038] When the PSN of a data packet is greater than the expected packet number, the corresponding bit of the PSN in the receiving bitmap is set to 1. If the valid bit of the sub-bitmap node to which the PSN of the data packet falls is 0, a sub-bitmap address is requested from the available sub-bitmap address pool.

[0039] When all bits in a sub-bitmap are 1, the all-1 flag of that sub-bitmap node is set to 1, and the sub-bitmap address can be released back to the address pool in advance.

[0040] When the valid bit and the all-1 flag of a certain sub-bitmap node are both 1, the sub-bitmap is regarded as a sub-bitmap with all 1s when the receiving bitmap header pointer moves.

[0041] When the PSN out-of-order distance of received packets exceeds the preset network out-of-order distance, the retransmission of the packet corresponding to the 0-bit (representing unreceived packet) packet number in the corresponding bitmap interval is triggered, and the corresponding bitmap interval is sent to the retransmission initiator.

[0042] The packet receiving bitmap involved in this application method is deployed on the network card of the RDMA data receiver. Figure 2This is a block diagram of the receive bitmap for each QP in the method of this application. Because RDMA network cards use QPs to distinguish different sessions, and different QPs can perform data transmission in parallel and independently, the receive bitmaps of each QP are also independent of each other. Figure 2 As shown, suppose the QP ID of the RDMA network card is from 0, 1, ..., X-1, and there are X QPs in total. Then the receive bitmap of these QPs corresponds to the entries stored in recv_bitmap_QP_0, recv_bitmap_QP_1, ..., recv_bitmap_QP_X-1 in sequence.

[0043] In RDMA transmission scenarios with high network latency or long out-of-order distances, the bitmap width requirement at the receiving end is relatively large. Therefore, the RDMA queue pair (QP) does not pre-allocate storage space for its circular receive bitmap. Instead, it segments the bitmap into several sub-bitmaps and dynamically allocates or releases sub-bitmap space from the available sub-bitmap address pool as needed. The available sub-bitmap address pool uses a first-in-first-out queue. During initialization, all sub-bitmap addresses are written to the address pool. When the QP needs a sub-bitmap, it pops a sub-bitmap address from the address pool. When a sub-bitmap of the QP is finished using it, its address is written back to the address pool. When a sub-bitmap of the QP is all 1s, its space can be released early. Therefore, Figure 2 Each bitmap in a QP is actually a circular array of pointers to child bitmap nodes, such as... Figure 1 As shown.

[0044] Figure 1 This is a schematic diagram of the receive bitmap of a QP according to the method of this application. At the RDMA data receiver, the receive bitmap recv_bitmap of a QP consists of multiple sub-bitmaps. The receive bitmap of a QP is a circular array of sub-bitmap nodes. Each sub-bitmap node (as shown in the figure, sub_bitmap_node) contains the address sub_bitmap_addr, the valid bit, and the all-1 flag.

[0045] The head pointer of the receive bitmap points to the bit position corresponding to the expected receive packet number. When the received packet number equals the expected receive packet number, the head pointer can move forward by 1 consecutive bit. For example... Figure 1 If the expected packet number is 1, then the position pointed to by the bitmap's header_ptr is as follows: Figure 1 As shown.

[0046] If the receiving bitmap has N child bitmap nodes and the bit width of each child bitmap is M, then the bit width of the receiving bitmap is N*M, which is a circular buffer.

[0047] In the initial state, the sub_bitmap_addr of all sub-bitmap nodes in QP is 0, the valid bit is 0, and the all_1_flag bit is 0.

[0048] The operation mechanism of the receive bitmap in this RDMA receive bitmap management method is as follows:

[0049] (1) Receive data packets. When the data packet PSN is equal to the expected packet number, the packet receiving bitmap head pointer can move the valid receiving bits continuously backward until it encounters an unreceived bit. The sub-bitmap space crossed by the bitmap head pointer can be released.

[0050] (2) Receive data packets. When the PSN of the data packet is greater than the expected packet number, set the corresponding bit of the PSN of the data packet in the receiving bitmap to 1. If the valid bit of the sub-bitmap node into which the PSN of the data packet falls is 0, then apply for a sub-bitmap address from the available sub-bitmap address pool.

[0051] (3) When all bits in a sub-bitmap are 1, set the all-1 flag of the sub-bitmap node to 1, and the sub-bitmap address can be released back to the address pool in advance.

[0052] (4) When the PSN out-of-order distance of the received packet exceeds the preset network out-of-order distance, the retransmission of the packet corresponding to the 0-bit (representing the unreceived packet) packet number in the corresponding bitmap interval is triggered, and the corresponding bitmap interval is sent to the retransmission initiator.

[0053] like Figure 1 Initially, assuming the expected packet number is 1, the sub-bitmap width M is 32, and the number of sub-bitmap nodes N per QP is 16, when the RDMA data packet with packet number 32 arrives, the packet falls into the first bit position of sub_bitmap_node_1. Since sub_bitmap_node_1 has not been used before, the valid bit of this sub-bitmap is 0. Therefore, the RDMA network card needs to request a free sub-bitmap from the sub-bitmap resource pool. After obtaining the sub-bitmap space, the first bit of the sub-bitmap is set to 1, and then sub_bitmap_addr of sub_bitmap_node_1 is updated to the address of the sub-bitmap, the valid bit is set to 1, and the all-1 flag is set to 0.

[0054] Subsequently, when RDMA packets with packet numbers 33, 34, ..., 63 arrive sequentially, since the valid bit of sub_bitmap_node_1 is 1, the packet number can be directly set to 1 in the corresponding bit of the corresponding sub-bitmap. Specifically, when packet number 63 arrives, after setting the corresponding bit to 1, since all bits of the sub-bitmap of sub_bitmap_node_1 are 1, setting the all-1 flag of sub_bitmap_node_1 to 1 allows the RDMA network card to release the sub-bitmap space in advance, that is, release the sub-bitmap address sub_bitmap_addr of sub_bitmap_node_1 back to the sub-bitmap resource pool. In subsequent processing, because the valid bit and the all-1 flag of sub_bitmap_node_1 are both 1, sub_bitmap_node_1 will be treated as an all-1 interval. This mechanism of releasing the sub-bitmap space in advance helps to further save hardware resources occupied by the sub-bitmap.

[0055] Subsequently, when the RDMA message with packet number 64 arrives, assuming the preset network out-of-order distance is 32, since the packet out-of-order distance exceeds the preset network out-of-order distance (i.e., packet number 64 falls in sub_bitmap_node_2), and the out-of-order distance between this packet and all packet numbers in sub_bitmap_node_0 is greater than 32, the RDMA network card triggers the retransmission of all packets corresponding to 0-bit packet numbers in the sub_bitmap_node_0 sub-bitmap (excluding the range before the bitmap header pointer). Preferably, when the RDMA network card applies the method of this application, the above-mentioned sub-bitmap partitioning and block determination method can be used to reduce the hardware implementation difficulty of packet out-of-order distance calculation and retransmission triggering.

[0056] Preferably, in the above-mentioned retransmission triggering process, for RDMA write operations, the receive bitmap is located at the responding end. The responding end RDMA network card can use the sub-bitmap of sub_bitmap_node_0 to construct a BITMAP-NACK packet and send it to the requesting end RDMA network card, requesting retransmission of the packet corresponding to the 0-bit number in the BITMAP; for RDMA read operations, the receive bitmap is located at the requesting end. The requesting end RDMA network card can send the sub-bitmap of sub_bitmap_node_0 to the retransmission packet generation module, triggering the module to send a read request for the payload of the packet corresponding to the 0-bit number in the BITMAP.

[0057] like Figure 1Subsequently, when the RDMA message with packet number 1 arrives, since the packet number equals the expected received packet number of the QP (i.e., the received packet number falls at the bit position pointed to by the bitmap's head pointer), the head pointer bit is set to 1, and the head pointer can then scan forward, sliding over consecutive 1-bit bits until it encounters a 0-bit bit. The packet number corresponding to the expected received packet number of bit 0 is updated, triggering an ACK reception confirmation with the packet number corresponding to bit 0 decremented by 1. When the bitmap's head pointer moves forward due to consecutive 1-bit bits and crosses a sub-bitmap, the crossed sub-bitmap space can be released back to the sub-bitmap resource pool. The sub-bitmap address of the corresponding sub-bitmap node in the sub-bitmap node circular array is cleared, the valid bit is set to 0, and the all-1 flag is set to 0. The sub-bitmap node crossed by the head pointer can be reused for packet reception.

[0058] All sub-bitmaps are stored in the sub-bitmap resource pool. Sub-bitmaps can be indexed by their addresses. Idle sub-bitmap addresses are stored in the available sub-bitmap address pool. All QPs share the sub-bitmap resource pool and can apply for and release resources as needed.

[0059] A diagram illustrating the relationship between the sub-bitmap resource pool and the sub-bitmap address, as shown below. Figure 3 As shown. The sub-bitmap resource pool is the storage space for all sub-bitmaps, such as... Figure 3 In the RDMA network card's sub-bitmap resource pool, a total of Y sub-bitmaps are instantiated: sub_bitmap_0, sub_bitmap_1, ..., sub_bitmap_Y-1. Assuming each sub-bitmap has a bit width of M, the sub-bitmap resource pool occupies a total of Y*M bits of storage space. The address of each sub-bitmap is its index, which can be the starting storage address of that sub-bitmap, i.e., the physical address of its first bit, such as... Figure 3 The values ​​of sub_bitmap_0_addr, sub_bitmap_1_addr, ..., sub_bitmap_Y-1_addr are shown in the figure.

[0060] A block diagram of the sub-bitmap address pool can be used, such as... Figure 4 As shown. Preferably, the available sub-bitmap address pool uses a first-in-first-out queue. The available sub-bitmap address pool is controlled by write and read pointers to write (QP releases a sub-bitmap address) and pop (QP occupies a sub-bitmap address). Initially, the read and write pointers are at the same position. Subsequently, the RDMA network card writes all sub-bitmap addresses into the address pool, and the write pointer moves up sequentially. When a QP needs to use a sub-bitmap, the address pool moves the read pointer up and pops the sub-bitmap address that the read pointer has crossed, making it available for the QP to use, thus completing the popping of the sub-bitmap address; when a QP has finished using a sub-bitmap and needs to release it, the address pool clears the used sub-bitmap to zero, writes the address of the sub-bitmap to the position pointed to by the write pointer, and moves the write pointer up, thus completing the writing of the sub-bitmap address.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of this application do not depart from the spirit and scope of the technical solutions of this application, and should all be covered within the scope of the claims of this application.

Claims

1. A method for managing the receive bitmap in out-of-order RDMA, comprising: The packet receiving bitmap is divided into several sub-bitmaps. Each sub-bitmap does not request storage space in advance, and the addresses of all sub-bitmaps are recorded in the sub-bitmap address pool. The RDMA queue dynamically requests or releases sub-bitmap space from the available sub-bitmap address pool as needed; When a QP needs a sub-bitmap, it dequeues a sub-bitmap address from the address pool. When a QP finishes using a sub-bitmap, it enqueues the used sub-bitmap address back into the address pool. When a sub-bitmap of a QP is filled with received flags, release the space of that sub-bitmap.

2. The packet receive bitmap management method for out-of-order RDMA according to claim 1, characterized in that, The sub-bitmap address pool is a first-in-first-out queue.

3. The packet receive bitmap management method for out-of-order RDMA according to claim 1, characterized in that: A QP's receive bitmap is a circular array that stores sub-bitmap nodes; each sub-bitmap node contains the address, valid bits, and all receive flag bits of that sub-bitmap; the head pointer of the receive bitmap points to the bit position of the bitmap corresponding to the expected receive packet number of that QP.

4. The packet receive bitmap management method for out-of-order RDMA according to claim 1, characterized in that, When the valid bit and all receive flag bits of a certain sub-bitmap node are all marked as received, the sub-bitmap is regarded as a fully received sub-bitmap when the receive bitmap header pointer moves.

5. The packet receive bitmap management method for out-of-order RDMA according to claim 1, characterized in that, The operating mechanism of the packet receiving bitmap includes: When receiving a data packet, if the PSN of the data packet is equal to the expected packet number, the receiving bitmap head pointer moves the valid receiving bits forward continuously until it encounters an unreceived bit. The sub-bitmap space crossed by the receiving bitmap head pointer can be released. If the PSN of the data packet is greater than the expected packet number, the corresponding bit position of the PSN in the receiving bitmap is set to a received flag. If the valid bit of the sub-bitmap node to which the PSN of the data packet falls is an unreceived flag, a sub-bitmap address is requested from the available sub-bitmap address pool.

6. The packet receive bitmap management method for out-of-order RDMA according to claim 5, characterized in that, The operation mechanism of the packet receiving bitmap also includes: When all bits in a sub-bitmap are marked as received, all receive flags of that sub-bitmap node are set to received, allowing the sub-bitmap address to be released back to the address pool ahead of schedule.

7. The packet receive bitmap management method for out-of-order RDMA according to claim 5 or 6, characterized in that, The operation mechanism of the packet receiving bitmap also includes: When the out-of-order distance of the received packet PSN exceeds the preset network out-of-order distance, the retransmission of the packet corresponding to the packet number with the unreceived packet identifier bit in the corresponding bitmap interval is triggered, and the corresponding bitmap interval is sent to the retransmission initiator.

8. The packet receive bitmap management method for out-of-order RDMA according to claim 1, characterized in that, All sub-bitmaps are stored in the sub-bitmap resource pool; sub-bitmaps are indexed by sub-bitmap addresses; free sub-bitmap addresses are stored in the available sub-bitmap address pool, and all QPs share the sub-bitmap resource pool, which is requested and released as needed.

Images