Software and hardware asynchronous RDMA transmission packet loss recovery system and method

Abstract

The invention discloses a software and hardware asynchronous RDMA (Remote Direct Memory Access) transmission packet loss recovery system, which is characterized in that a bitmap is arranged on a software kernel, and retransmission parameters are arranged in context state information of each connection; the method comprises the following steps of: caching a preset part of actively connected bitmaps to a network card along with a context state, directly updating a hardware cache when a serial number of a bitmap to be updated is within a cache range, and initiating an asynchronous bitmap updating request to a software kernel if the serial number is not within the range; the network card judges whether a packet loss event occurs or not according to the PSN and the state parameters of the data packet; and the network card or the software updates the bitmap information and the state parameter of the data packet corresponding to the packet loss event and re-issues the bitmap information and the state parameter to the network card. According to the software and hardware asynchronous RDMA transmission packet loss recovery system, only lost data packets need to be retransmitted, repeated transmission of arrived data packets is avoided, and high throughput and low delay are maintained; therefore, the network utilization rate is improved in a lossy environment, and the existing expansibility scheme is compatible.

Description

Technical Field

[0001] This invention relates to an RDMA transmission protocol, and more particularly to a hardware and software asynchronous RDMA transmission packet loss recovery system and method. Background Technology

[0002] Currently, RDMA protocols suffer from severe limitations in connection scalability due to limited on-chip storage. Furthermore, historical issues in the development of RoCE protocols, particularly the lossless mechanisms and simple packet loss recovery mechanisms in earlier versions, further restrict network scalability. Existing solutions address certain scalability issues to some extent, but most solutions, while focusing on solving one scalability problem, often exacerbate another. Some affect transmission performance, some flow fairness, and some are not adaptable to all environments, only suitable for specific scenarios. For example, the PFC mechanism constructs a lossless network environment for RDMA through switch backpressure strategies; however, PFC introduces victim flows and PFC storms, impacting network availability and fairness. As the network scales, these problems become more severe and unacceptable, significantly limiting the scalability of RDMA networks. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a hardware and software asynchronous RDMA transmission packet loss recovery system and method that only needs to retransmit lost data packets, avoids repeated transmission of already arrived data packets, and improves network utilization.

[0004] To solve the above-mentioned technical problems, the technical solution proposed by this invention is: a hardware and software asynchronous RDMA transmission packet loss recovery system, characterized in that: it includes a bitmap, retransmission parameters, a ReqQ that loads retransmission requests, and a FlushQ that loads bitmap update requests; the bitmap is set on the software kernel, and the retransmission parameters are set in the context state information of each connection; the bitmaps of some pre-set active connections are cached on the network card along with the context state; when the bitmap sequence number to be updated is within the cache range, the hardware cache is directly updated; if it is not within the range, an asynchronous bitmap update request is initiated to the software kernel; the retransmission parameters include SendOutOfOrder / RecvOutOfOrder, sackHigh / recvHigh, eACK / ePSN, sendLeft / recvLeft, and sendRight / recvRight; The network interface card (NIC) determines whether a packet loss event has occurred by using the PSN and the status parameters of the data packets. The NIC directly updates the bitmap and status parameters, or the software updates the bitmap information and status parameters of the data packets corresponding to the packet loss event and retransmits them to the NIC. The NIC completes asynchronous interaction with the software kernel through ReqQ and FlushQ. The NIC sends a request to update the bitmap or status information through FlushQ and obtains the latest bitmap cache, the updated status parameters, and the sequence number of the data packets that need to be retransmitted through ReqQ.

[0005] In the aforementioned asynchronous hardware and software RDMA transmission packet loss recovery system, preferably, when in a lost and retransmitted state, after receiving a data packet with the expected sequence number, the hardware bitmap corresponding to the data packet needs to be updated to a transmitted state immediately. Then, the hardware bitmap is traversed to find the first untransmitted sequence number. If every subsequent bit is found to be in a transmitted state, the ePSN is temporarily updated to the largest sequence number in the cached bitmap + 1. Then, an update bitmap request is sent to the software bitmap. The software kernel needs to traverse the bitmap after the updated ePSN in the software bitmap to find the first gap packet. Then, the latest bitmap of the corresponding size that needs to be cached, along with the latest gap sequence number, is updated to the hardware network card as the next expected data packet.

[0006] In the aforementioned asynchronous hardware and software RDMA transmission packet loss recovery system, preferably, the ReqQ record contains the sequence number of the data packet that needs to be retransmitted, the updated ePSN sequence number, and the latest bitmap that needs to be cached; the FlushQ stores the retransmission logic of the network card hardware and the sequence number of the currently received data packet.

[0007] In the aforementioned asynchronous hardware and software RDMA transmission packet loss recovery system, preferably, the size of the preset bitmap of some active connections is greater than or equal to 8 bits, and it can be configured via parameter K.

[0008] This invention also provides a method for recovering packet loss during asynchronous RDMA transmission using both hardware and software, comprising the following steps: 1) The network card polls the doorbell register, receives a send request, obtains WQE requests in batches from the SQ Buffr and caches them on the network card, and parses the WQE to obtain the send request, obtains the virtual address and length of the data to be sent, and converts the virtual address into the corresponding physical address through the MTT mapping relationship. The DMA engine retrieves the data to be sent from memory according to the corresponding physical address and caches it on the network card for protocol stack processing. 2) After the data packet transmission is complete, the network card generates a CQE to notify the host application that the transmission is complete; 3) When the network card receives a data packet, it determines whether packet loss has occurred by checking the PSN and recording parameters, and updates the bitmap and status parameters accordingly based on whether packet loss has occurred. When updating the bitmap, the hardware directly updates the bitmap and the latest parameters based on the bitmap cache, or the hardware needs to send a request to the software. After the software updates the bitmap and the new parameters, it sends a retransmission data packet and the latest parameters to the hardware. 3.1) The receiving end's network card determines the data packet reception status; ① When the PSN of the data packet received on the receiving end's network card is greater than the ePSN, it enters the packet loss retransmission state, modifies the parameter RecvOutOfOrder to True, and replies to the sending end with a sack PSN. ② When PSN = ePSN, it indicates that the expected data packets arrive in order; directly receive the data packets and reply with an ack PSN to the sender; ③ If the PSN of the currently received data packet is less than the sequence number of the expected data packet, it means that the data packet has already been received and should be discarded. 3.2) The sending end receives the ack PSN or sack PSN from the receiving end and decides whether to retransmit; When the sending end receives the reply ack PSN in step 3.1) and SendOutOfOrder is False, it indicates that the receiving end has correctly received the expected data packet and no packet loss event has occurred. II. When the sending end receives the reply ack in step 3.1) and SendOutOfOrder is True, it indicates that after the receiving end enters the select retransmission state, the receiving end receives the expected data packet in the retransmission. III. When the sending end receives the reply in step 3.1) as sack and SendOutOfOrder as False, it indicates that no packet loss event has occurred, but the reply to the currently received data packet is that the receiving end has perceived packet loss. IV: When the sender receives a sack response from step 3.1) and SendOutOfOrder is True, it indicates that packet loss has occurred. Determine the relationship between PSN and eACK: 4) If the sending end detects packet loss according to step 3), it will add the data packet corresponding to the packet loss event back to the retransmission queue for retransmission; until the sending end receives an ack reply and SendOutOfOrder is updated to True, it will exit the packet loss retransmission state.

[0009] The aforementioned asynchronous hardware and software RDMA transmission packet loss recovery method, preferably, includes the following specific steps in step II: If PSN < eACK, the receiving end has received a data packet that was previously acknowledged, and no action will be taken in this case. If PSN >= eACK, the receiving end has received the data packet PSN in order. It needs to determine whether the PSN exceeds the range of the hardware bitmap buffer. If it does not exceed the range, it needs to change all bitmaps before PSN in the hardware buffer bitmap to the "transmitted" state and trim them, and start traversing the bitmap gaps after PSN, finding the first gap and updating eACK to the corresponding sequence number. If no gap is found or PSN exceeds the hardware buffer bitmap, it submits a request to the software to update the bitmap, changes all bitmaps before PSN to the "transmitted" state and trims them, and starts traversing the bitmap gaps after PSN, finding the first gap and updating eACK to the corresponding sequence number, and then transmits the updated eACK to the hardware network card. If the updated eACK > sackHigh, it means that the receiver has received all the retransmitted data packets, and SendOutOfOrder should be changed to False; if eACK < sackHigh, it means that the retransmitted data packets have not been received yet and the receiver is still in the lost and retransmitted state, so no other changes are needed.

[0010] The aforementioned asynchronous hardware and software RDMA transmission packet loss recovery method, preferably, includes the following specific steps in step III: If PSN < eACK, no action is taken; If PSN = eACK, then eACK = PSN + 1, sackHigh = PSN; If PSN > eACK, sackHigh = PSN, and SendOutOfOrder is changed to True. If PSN is within the hardware bitmap cache range, the PSN position is directly changed to the transmitted state in the hardware cache; otherwise, PSN is sent to the software kernel to update the bitmap, and the position of PSN in the bitmap is changed to the transmitted state. All data packets between eACK and PSN are added to the retransmission queue for retransmission.

[0011] The above-mentioned asynchronous hardware and software RDMA transmission packet loss recovery method, preferably, the specific steps of step IV are as follows: If PSN < eACK, no action is taken; If PSN=eACK, it indicates that the eACK of the currently retransmitted data packet has been received. The corresponding PSN in the hardware bitmap is updated to the transmitted state and then trimmed. The next gap after the hardware bitmap is found and eACK is updated to the corresponding sequence number. If no gap is found, eACK is temporarily updated to the maximum bit of the hardware bitmap buffer + 1, and the software kernel is uploaded to update the bitmap sequence number to the transmitted state and trimmed. eACK is updated to the first gap in the bitmap and then transmitted to the hardware network card. If eACK > sackHigh, it indicates that all currently retransmitted data packets have been received, and SendOutOfOrder is changed to False. If eACK < sackHigh, it indicates that there are still retransmitted data packets in transmission, which need to be maintained without further modification. If eACK < PSN < sackHigh, it indicates that a retransmitted data packet was lost again, and only the PSN retransmission was received. If the PSN is within the hardware bitmap buffer range, the PSN position is directly updated to the transmitted state, and the bitmap is traversed to add all gap data packets between eACK and PSN to the retransmission queue; otherwise, all gap data packets in the hardware bitmap are added to the retransmission queue, and then the PSN is sent to the software kernel to update the bitmap, changing the position of the bitmap with PSN to the transmitted state; and all gap data packets between the highest bit + 1 of the hardware bitmap and PSN are added to the retransmission queue again for retransmission. If sackHigh < PSN, it indicates that a new data packet has been lost. Set sackHigh to PSN. If PSN does not exceed the hardware bitmap buffer range, directly update the corresponding position of PSN in the hardware bitmap to the transmitted state. Otherwise, send PSN to the software kernel to update the bitmap and change the position of PSN in the bitmap to the transmitted state. In addition, all data packets between sackHigh and PSN are added to the retransmission queue for retransmission.

[0012] In the above-mentioned asynchronous hardware and software RDMA transmission packet loss recovery method, preferably, in step 4), when in the loss retransmission state, after receiving the data packet with the expected sequence number, the corresponding bitmap of the hardware data packet needs to be updated to the transmitted state immediately. Then, the hardware bitmap is traversed to find the first untransmitted sequence number. If it is found that every subsequent bit is in the transmitted state, the ePSN is temporarily updated to the largest sequence number in the cached bitmap + 1. Then, the software bitmap initiates a bitmap update request. The software kernel needs to traverse the bitmap after the updated ePSN in the software bitmap to find the first gap packet. Then, the latest bitmap of the corresponding size that needs to be cached, and the latest gap sequence number are updated together with the latest next expected data packet to be received on the hardware network card.

[0013] Compared with the prior art, the advantages of the present invention are: the hardware and software asynchronous RDMA transmission packet loss recovery system of the present invention only needs to retransmit the lost data packets, avoiding repeated transmission of data packets that have already arrived, thus maintaining high throughput and low latency; thereby improving network utilization in lossy environments, while being compatible with existing scalability solutions. Attached Figure Description

[0014] Figure 1 This is a framework diagram of the asynchronous hardware and software RDMA transmission packet loss recovery system in Example 1.

[0015] Figure 2 This is a flowchart of the data packet processing at the receiving end of the asynchronous hardware and software RDMA transmission packet loss recovery system in Example 1.

[0016] Figure 3 This is a flowchart of the data packet processing at the sending end of the asynchronous hardware and software RDMA transmission packet loss recovery system in Example 1.

[0017] Figure 4 This is a schematic diagram of the bitmap overhead required in the worst case when a retransmission is successful in Example 1.

[0018] Figure 5 This is a bitmap diagram illustrating hardware and software updates in Example 1.

[0019] Figure 6 This is a schematic diagram of directly updating the ePSN using hardware cache in Example 1.

[0020] Figure 7 This is a schematic diagram illustrating the need for software bitmap updates to the ePSN in Example 1.

[0021] Figure 8 This is a schematic diagram illustrating the need to update the software bitmap again during the software update process in Example 1.

[0022] Figure 9 This is a schematic diagram of the bitmap that needs to be merged after the software update for example 1, where the ePSN does not fall within the range.

[0023] Figure 10 This is a schematic diagram of the bitmap updated using the right parameter after the ePSN falls within the range in Example 1 following a software update.

[0024] Figure 11 The throughput test results are for each scheme under different packet loss rates.

[0025] Figure 12 The results of FCT tests are shown for each scheme under different packet loss rates.

[0026] Figure 13 The results show the bitmap overhead test results for each scheme during data transmission.

[0027] Figure 14 Test results for measuring bitmap overhead (100 packets, 100 connections) for each scheme in the simulator. Detailed Implementation

[0028] To facilitate understanding of the present invention, the present invention will be described more fully and in detail below with reference to preferred embodiments, but the scope of protection of the present invention is not limited to the following specific embodiments.

[0029] It should be noted that when a component is described as being "fixed to, attached to, connected to or connected to" another component, it can be directly fixed to, attached to, connected to or connected to the other component, or it can be indirectly fixed to, attached to, connected to or connected to the other component through other intermediate connectors.

[0030] Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by those skilled in the art. The technical terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the scope of the invention. Example

[0031] like Figure 1The illustrated hardware-software asynchronous RDMA transmission packet loss recovery system includes a bitmap, retransmission parameters, a ReqQ that loads retransmission requests, and a FlushQ that loads bitmap update requests. The bitmap is set in the software kernel, and the retransmission parameters are set in the context state information of each connection. A preset set of bitmaps for some active connections is cached on the network interface card (NIC) along with the context state. The preset size of the bitmap for some active connections is 8 bits, and the size can be increased as needed. This can be configured via parameter K, for example, K=16 bits. Parameter K can be set as needed, but this will consume more NIC resources. When the bitmap sequence number to be updated is within the cache range, the hardware cache is updated directly; otherwise, an asynchronous bitmap update request is initiated to the software kernel. The retransmission parameters include SendOutOfOrder / RecvOutOfOrfer, sackHigh / recvHigh, eACK / ePSN, sendLeft / recvLeft, and sendRight / recvRight. SendOutOfOrder / RecvOutOfOrfer is used to mark whether the current state has entered retransmission mode. It is a boolean type and has a bit value of 1. sackHigh / recvHigh is used to record the maximum sequence number of the currently received ACK / SACK packet. It is an int type and has a bit value of 32. eACK / ePSN is used to record the sequence number of the data packet that is expected to be received. It is an int type and has a bit value of 32. sendLeft / recvLeft and sendRight / recvRight are used to record the range of consecutive data packets received by the hardware during the software processing when a request to update eACK / ePSN is sent. They are int types and have a bit value of 32.

[0032] The network interface card (NIC) determines whether a packet loss event has occurred by using the PSN (packet sequence number) and packet status parameters. The NIC directly updates the bitmap and status parameters, or the software updates the bitmap information and status parameters of the packet corresponding to the packet loss event and retransmits it to the NIC. The NIC obtains the software update results and sends update requests to the software via ReqQ and FlushQ. ReqQ records the sequence number of the packet that needs to be retransmitted, the updated ePSN sequence number, and the latest bitmap that needs to be cached. FlushQ stores the retransmission logic communicated by the NIC hardware to the software and the sequence number of the currently received packet.

[0033] In this embodiment, when in a lost-and-retransmitted state, upon receiving a data packet with the expected sequence number, the corresponding bitmap of the hardware data packet needs to be immediately updated to the transmitted state. Then, the hardware bitmap is traversed to find the first untransmitted sequence number. In this embodiment, the transmitted state is marked as 1, and the untransmitted state is marked as 0. If every subsequent bit is found to be 1, the ePSN is temporarily updated to the largest sequence number in the cached bitmap + 1. Then, an update bitmap request is sent to the software bitmap. The software kernel needs to traverse the bitmap after the updated ePSN in the software bitmap to find the first gap packet. Then, the latest bitmap of the corresponding size that needs to be cached, along with the latest gap sequence number, is updated to the hardware network card as the next expected data packet.

[0034] This embodiment also provides a method for recovering packet loss during asynchronous RDMA transmission, including the following steps: 1) The network card polls the doorbell register. Upon receiving a send request, it retrieves WQE (Work Queue Element) requests in batches from the SQ Buffr and caches them on the network card. It also parses the WQE (Work Queue Element) to obtain the send request, gets the virtual address and length of the data to be sent, and converts the virtual address to the corresponding physical address through the MTT (Virtual Address to Physical Address) mapping relationship. The DMA engine retrieves the data to be sent from memory according to the corresponding physical address and caches it on the network card for protocol stack processing.

[0035] 2) After the data packet transmission is complete, the network card generates a CQE (Completion Queue Element) to notify the host application that the transmission is complete; 3) When the network card receives a data packet, it uses the PSN (Package Sequence Number) and recorded parameters to determine whether packet loss has occurred, and updates the bitmap and status parameters accordingly based on whether packet loss has occurred. When updating the bitmap, the hardware directly updates the bitmap and the latest parameters based on the bitmap cache, or the hardware needs to send a request to the software. After the software updates the bitmap and the new parameters, it sends a retransmission data packet and the latest parameters to the hardware. 3.1) The receiving end's network card determines the data packet reception status; ① When the PSN (Package Sequence Number) of the data packet received on the receiving end's network card is greater than the ePSN (Expected Packet Sequence Number); it enters the packet loss retransmission state, modifies the parameter SendOutOfOrder to True, and replies to the sending end with a sack PSN; ② When PSN (Package Sequence Number) = ePSN (Expected Package Sequence Number), it indicates that the expected data packets have arrived in order; directly receive the data packets and reply with an ack PSN to the sender; ③ If the PSN (Package Sequence Number) of the currently received data packet is less than the ePSN (Expected Packet Sequence Number), it means that the data packet has already been received and should be discarded. 3.2) The sending end receives the ack PSN or sack PSN from the receiving end and decides whether to retransmit; When the sending end receives the reply ack PSN in step 3.1) and SendOutOfOrder is False, it indicates that the receiving end has correctly received the expected data packet and no packet loss event has occurred. II. When the sending end receives the reply ack in step 3.1) and SendOutOfOrder is True, it indicates that after the receiving end enters the select retransmission state, the receiving end receives the expected data packet in the retransmission. III. When the sending end receives the reply in step 3.1) as sack and SendOutOfOrder as False, it indicates that no packet loss event has occurred, but the reply to the currently received data packet is that the receiving end has perceived packet loss. IV. When the sender receives a sack response from step 3.1) and SendOutOfOrder is True, it indicates that packet loss has occurred. The relationship between PSN (Package Sequence Number) and eACK (Expected Acknowledgment Sequence Number) is then determined: 4) If the sending end detects packet loss according to step 3), it will add the data packet corresponding to the packet loss event back to the retransmission queue for retransmission. This continues until the sending end receives an ack response and SendOutOfOrder is True. Then, the software kernel updates the bitmap information.

[0036] In this embodiment, the specific steps of step II are as follows: If PSN (Package Sequence Number) < eACK (Expected Acknowledgment Sequence Number), the received data packet has already been acknowledged, and no action will be taken. If PSN >= eACK, the receiver has received the data packet PSN in order. It needs to determine if the PSN exceeds the hardware bitmap buffer range. If it does not exceed the range, the bitmap before PSN in the hardware buffer needs to be changed to the "transmitted" state and clipped. Then, it starts traversing the bitmap gaps after PSN, finds the first gap, and updates the eACK with the corresponding sequence number. If no gap is found or the PSN exceeds the hardware buffer bitmap, it submits a request to the software to update the bitmap, changes the bitmap before PSN to the "transmitted" state and clips it. Then, it starts traversing the bitmap gaps after PSN, finds the first gap, updates the eACK with the corresponding sequence number, and then transmits the updated eACK to the hardware network card. If the updated eACK (expected acknowledgment sequence number) > sackHigh (maximum acknowledgment sequence number received), it means that the receiving end has received all retransmitted data packets, and SendOutOfOrder should be changed to False; if eACK < sackHigh, it means that the retransmitted data packets have not been completely received and the receiver is still in the lost and retransmitted state, requiring no further changes.

[0037] Step 1) in this embodiment includes two stages: request issuance and data transmission. The first stage involves initiating a WQE (Work Queue Element) request. After the application creates a QP (Connection) and both ends successfully establish a connection, the application needs to call the WR (Work Request) issuance interface to initiate a transmission request. Depending on the data transmission operation type, four interfaces can be called to complete the data transmission operation: ibv_post_send, ibv_post_receive, ibv_post_read, and ibv_post_write. After the application calls the required interface, the user-space driver completes the logic of the corresponding interface. The driver packages the data address, length, and operation type to be transmitted, constructs a WR (Work Request) request, and after further processing, converts it into a WQE (Work Queue Element) and loads it into the SQ (Send Queue). It also modifies the network card's doorbell register to inform the network card to read the WQE (Work Queue Element) request from the SQ (Send Queue) queue.

[0038] Data transmission and reception phases. The network interface card (NIC) polls the doorbell register. Upon receiving a transmission request, it retrieves a batch of WQE (Work Queue Elements) requests from the SQ (Transmit Queue) Buffr and buffers them on the NIC. It then parses the WQEs to obtain the transmission request, acquiring the virtual address and length of the data to be transmitted. Using the MTT (Multi-Telegraphics Table), it translates the virtual address into the corresponding physical address. The DMA engine retrieves the data to be transmitted from memory based on the corresponding physical address and buffers it on the NIC for protocol stack processing. The transport layer, based on the corresponding congestion control logic, determines the current transmission rate or window and performs data transmission operations. The details of the data transmission process differ slightly depending on the operation.

[0039] (1) Send and Receive operations: The receiving end needs to send a Receive WQE first before the sending end can send a Send WQE (work queue element). After data transmission, the receiving end receives the data, parses the data packet to obtain information such as the connection number (Queue Pair Number, QPN) and the packet sequence number (PSN), performs a series of checks and judges whether the data meets expectations, and then processes the data packets that pass the checks according to the corresponding protocol stack. It finds the Receive WQE (work queue element) of the corresponding connection through the QPN, and obtains the physical address of the corresponding memory through address translation of the address MTT (virtual address to physical address mapping table) according to the virtual address of the receiving memory indicated by the Receive WQE (work queue element). The DMA engine then transfers the data payload to the corresponding physical address for storage.

[0040] (2) Read Operation: Before the sending end initiates a Read request, after the connection is established, the receiving end sends the memory address of the data to be read and the corresponding access Rkey to the sending end. The sending end puts the address and Rkey into ReadWR and sends it to the network card. After the sending application initiates a Read request, the sending end network card parses the WQE (Work Queue Element) Read request, encapsulates a Read request data packet, which does not contain the data payload, but only contains the target QPN, the remote virtual address to be read, the corresponding Rkey, the data length, and other information. After the receiving end receives the Read request, it parses the QPN to find the corresponding context information, uses the Rkey to verify the read permission to the remote VA, and after successful verification, the receiving end network card directly accesses the physical address corresponding to the VA, retrieves the data, encapsulates it into one or more RDMA Read response packets, and transmits them to the sending end. The sending end receives the data packet, parses the corresponding QPN, finds the local memory address indicated in the original Read WQE, and writes the data to the corresponding location.

[0041] (3) Write operation: Similar to the Read operation, before the sending end initiates a Write request, it needs to communicate with both ends to obtain the VA and the corresponding Rkey that have been written into the memory of the receiving end, and then transfer the obtained data to the receiving end.<VA,Rkey> and the memory where the data to be transmitted is located on this end.<VA,Lkey> Together, they are assembled into a Write WR (Work Request). After the WR (Work Request) is sent to the sending network card, the network card retrieves the data from the corresponding physical address in memory according to the local VA specified in the request, and then sends the data to the receiving end.<VA,Rkey> The data is encapsulated together into a Write data packet and sent to the receiving end. After receiving the data packet, the receiving end parses the data packet header and verifies the storage permission. If the verification is successful, the data is written to the corresponding physical address according to the memory address carried in the data packet header.

[0042] In step 2) of this embodiment, the CQE generation phase is requested to be completed. CQE generation occurs after data transmission is complete; the network interface card (NIC) notifies the host application that transmission is complete so that subsequent operations can proceed. The method of generating CQE differs for different operation types.

[0043] (1) For Send and Receive operations, when the sending operation is completed and the receiving end receives the tail packet, the receiving end network card generates a CQE and loads it into the CQ, notifying the receiving end application that the data reception is complete. The application continuously polls the CQ to read the WC (Work Complete Mark, which is CQE for the network card), and reads the received data from the corresponding registered memory into user space according to the WC indication information. The receiving end generates a CQE and also replies with an ACK to the sending end. The ACK carries information to notify the sending end that the data transmission is successfully completed. After receiving the reply, the sending end network card generates a CQE and puts it into its local CQ. The application continuously polls the CQ to read the WC and continues to perform subsequent data transmission operations.

[0044] (2) For the Read operation, when the sending end receives all Read response packets, the sending end network card generates a WC and loads it into the CQ of the corresponding QP, and notifies the application that the Read operation is complete.

[0045] (3) For Write operation, after the receiving end receives all Write data packets, it will send an ack confirmation packet to the sending end. After the sending end network card receives the ack confirmation packet, it considers the Write operation to be completed, generates a WC in the CQ corresponding to the local QP, and notifies the local application that the Write operation is complete and continues to perform subsequent operations.

[0046] In this embodiment, the general process of initiating data transmission is disclosed in step 1). After the control plane is prepared, after the application initiates a data plane request, the data is transmitted to the network card for protocol stack processing. The processing process includes operations such as congestion control and packet loss recovery. Congestion control and packet loss recovery are two relatively independent processing modules. Congestion control is to control the sending frequency of data packets, and packet loss recovery is responsible for ensuring that data packets arrive as expected. Therefore, the two solutions can often be orthogonally combined.

[0047] The network card hardware protocol stack is responsible for processing the data packets for data transmission in the data plane process. The connection establishment process of the control plane includes two methods: socket and CM connection establishment, which are respectively ensured by the traditional TCP / IP kernel protocol stack and the protocol logic corresponding to the CM interface for reliable transmission.

[0048] In step 3) of this embodiment, the receiving end mainly performs reliable transmission processing on the received data packets, including receiving out-of-order data packets and feedbacking the receiving situation to the sending end. After the network card transmission engine receives the data packets, the hardware protocol stack analyzes the data packet header to obtain the IB partial header parameters, obtains the target QPN (connection queue pair number) and PSN (data packet sequence number), and first finds the corresponding QPC (connection queue context information) according to QPN. Judge whether the current connection is in the state of packet loss retransmission: judge whether the parameter ReceiveOutOfOrder is true. After judging the loss state, judge the situation of the received data packet according to PSN and related parameters. First, the protocol processing process without considering the position of the bitmap will be described below, focusing on describing the details of each step of the protocol; later, the hardware bitmap cache will be added to describe the protocol processing process again, emphasizing the cache bitmap synchronization process.

[0049] 3.1) The receiving end network card judges the receiving state of the data packet; such as Figure 2 The receiving end logic is: ① If ReceiveOutOfOrder is False, it indicates that the packet loss retransmission state has not been entered yet, Judge that if the current received data packet PSN (data packet sequence number) < ePSN (expected received data packet sequence number), it indicates that the received data packet has been received before, and directly discard the data packet.

[0050] If PSN = ePSN, it indicates that the expected received data packet arrives in order. Directly receive the data packet, update eACK to ACK + 1, receiveHigh = PSN; DMA the data to the corresponding expected memory; finally reply with a deterministic reply ack (Acknowledge Character) data packet carrying the current PSN, that is, ack PSN.

[0051] If PSN > ePSN, it indicates that all preceding data packets have been lost. The sender needs to retransmit all data packets between ePSN and PSN. In this case, the parameter receiveHigh = PSN needs to be modified; ReceiveOutOfOrder=false needs to be modified to enter the packet loss retransmission state; the PSN position in the bitmap needs to be filled to 1; and a Selective Acknowledged Character (sack) data packet needs to be sent with the current PSN, i.e., sack PSN, to inform the other party that the out-of-order data packet PSN has been received.

[0052] ② If ReceiveOutOfOrder is True, it indicates that the system has entered the packet loss retransmission state, meaning that packet loss has already occurred. If PSN < ePSN, it means the data packet has already been received and should be discarded.

[0053] If PSN = ePSN, it indicates that the current data packet was received as a retransmitted packet after being lost. In this case, an ACK PSN should be sent first; then, the bitmap should be traversed, and the ePSN updated to the first gap after the ePSN. If the updated ePSN > receiveHigh, it means there are no more lost packets, and ReceiveOutOfOrder should be updated to false. If ePSN < receiveHigh, it means the packet loss recovery is still in progress. No further modifications are needed.

[0054] If ePSN < PSN < receiveHigh, it indicates that a retransmitted data packet PSN has been received, and all gaps (bitmap value of 0, i.e., lost) in the previously retransmitted data packet ePSN to PSN have been lost again. Therefore, the sender needs to be instructed to retransmit these gap packets. Reply with sack PSN (data packet sequence number); and update the PSN bitmap position to 1.

[0055] If PSN > receiveHigh, it indicates that the packet in the gap between receiveHigh+1 and PSN was lost. It is necessary to update receiveHigh = PSN; reply with sack PSN; and update the PSN position in the bitmap to 1.

[0056] 3.2) The sending end receives the ack PSN or sack PSN from the receiving end and, based on the current status value, decides whether to retransmit the data packet; for example... Figure 3 The sending logic is as follows: The sending end is responsible for sending data and processing received ACK and SACK packets, including determining and sending retransmission packets. Data transmission involves the network interface card (NIC) reading the WQE request, encapsulating the data according to congestion control and other constraints, and then placing it into a transmission queue for transmission. Upon receiving an ACK / SACK packet from the receiving end, the NIC first parses the packet to obtain the QPN, packet type (ACK or SACK), and PSN. First, it finds the corresponding QPC (Connection Context Information) based on the QPN. In the parsed packet type, ACK indicates that the received packet is the expected packet, while SACK indicates that the received packet is an unexpected packet. The NIC then combines the packet type, the current value of the SendOutOfOrder parameter (whether the sending end is currently in a lost-and-retransmit state), and the PSN parameter to make a joint judgment, update the parameters, and initiate a retransmission operation.

[0057] ① If the data packet type is ack and SendOutOfOrder is False, it indicates that the receiving end has correctly received the expected data packet and no packet loss event has occurred.

[0058] If PSN (Package Sequence Number) < eACK (Expected Acknowledgment Packet), it indicates that the receiving end has received the packet that has already been received and no further processing is required.

[0059] If PSN >= eACK, it indicates that the receiver has received the expected data packet. eACK = PSN+1, sackHigh (maximum sequence number of the received acknowledgment packet) = PSN.

[0060] ② If the data packet type is ack and SendOutOfOrder is True, it indicates that after the receiving end enters the select retransmission state, the receiving end receives the expected data packet in the retransmission.

[0061] If PSN < eACK, the data packet received by the receiving end has been previously acknowledged. According to the protocol, this situation will not occur, and no action will be taken in such cases.

[0062] If PSN >= eACK, the receiver has received the data packets in order. It needs to update the bitmap by setting all bits before PSN to 1 and cropping them. Then, it iterates through the gaps after PSN, finding the first gap and updating eACK to the corresponding sequence number. If the updated eACK > sackHigh, the receiver has received all retransmitted data packets, and SendOutOfOrder is changed to False. If eACK < sackHigh, the retransmitted data packets have not been fully received and the receiver remains in a lost-and-retransmitted state, requiring no further changes.

[0063] ③ If the data packet type is sack and SendOutOfOrder is False, it indicates that no packet loss event has occurred, but the response of the currently received data packet is that the receiving end has perceived packet loss.

[0064] If PSN < eACK, the protocol will not generate this situation and will not take any action.

[0065] If PSN=eACK, the protocol will not produce this situation. If it does occur, eACK = PSN+1 and sackHigh = PSN.

[0066] If PSN > eACK, set sackHigh = PSN and SendOutOfOrder to True; then update the bitmap, setting the position of PSN to 1; finally, add all data packets between eACK and PSN to the retransmission queue for retransmission.

[0067] ④ If the packet type is sack and SendOutOfOrder is True, it indicates that packet loss has occurred. Determine the relationship between PSN and eACK: If PSN < eACK, the protocol will not generate this situation and will not take any action.

[0068] If PSN=eACK, the protocol will not produce this situation. If it does occur, theoretically it means that the eACK of the currently retransmitted data packet has been received, and eACK will be temporarily updated to eACK+1; the value of the eACK position in the bitmap will be updated to 1, and then the bitmap will be traversed, updating eACK to the first gap in the bitmap; if eACK > sackHigh, it means that all currently retransmitted data packets have been received, and SendOutOfOrder will be changed to False; if eACK < sackHigh, it means that there are still retransmitted data packets in transmission, which need to be maintained without any other modifications.

[0069] If eACK < PSN < sackHigh, it indicates that a retransmitted data packet was lost, and only the PSN retransmission was received. Therefore, the bitmap is updated, and the position of PSN in the bitmap is changed to 1. The bitmap is then traversed to add all gap data packets between eACK and PSN back to the retransmission queue for retransmission.

[0070] If sackHigh < PSN, it indicates that new data packets have been lost. Set sackHigh = PSN; update the bitmap, change the bitmap position of PSN to 1; and add all data packets between sackHigh and PSN to the retransmission queue for retransmission.

[0071] To enable the asynchronous selective repeat protocol to operate more efficiently within the RDMA protocol stack and avoid repeatedly uploading data packets to the software kernel for processing, this invention's hardware-software asynchronous RDMA transmission packet loss recovery system, based on protocol optimization, proposes using a hardware-based bitmap cache to accelerate state parameter updates. Compared to previous solutions that cache all or none of the bitmaps required for the connection, caching the bitmap presents two new challenges: first, how to design the size of the cached bitmap; and second, how to synchronize the cached bitmap with the bitmap in the software kernel. These two aspects will be explained in detail below.

[0072] In this embodiment, the overall bitmap caching scheme is as follows: the asynchronous selective repeat strategy places all the bitmaps required for the connection in the software kernel, while the hardware caches the first few parts of the bitmap. This allows the protocol stack to quickly traverse the bitmap to obtain the sequence number of the next expected data packet when it receives the expected data packet, thus updating the status parameters. Therefore, the size of the bitmap cache should be designed to ensure both fast parameter updates and avoid excessive redundant hardware caching that leads to high connection context storage overhead and exacerbates connection scalability issues.

[0073] Firstly, for a connection, the total size of the entire bitmap required has a reference value: the Bandwidth-Delay Product (BDP), which indicates the maximum number of datagrams that can fly across the entire network link. The reason is as follows... Figure 4 As shown, assuming the bitmap can be infinitely large, the worst-case overhead of the bitmap is when the first data packet initiated by the sender is lost, but all subsequent data packets are transmitted normally. In this case, the bitmap needs to record all subsequent data packets until the first data packet is retransmitted and received normally. The interval from the detection of packet loss to the arrival of the retransmitted data packet is one RTT (Round-Trip Time). Therefore, the maximum overhead of a conventional bitmap is RTT * bandwidth / MTU, which we usually call BDP.

[0074] Therefore, the size of the cached bitmap can range from 0 to all bitmaps in the BDP range. This embodiment uses an empirical value for the specific size, verified through a simulation experiment. The experiment setup is as follows: First, we simulated the implementation of the asynchronous selective repeat protocol, designing the selective repeat logic in both the software kernel and the hardware protocol stack. Then, we designed the bitmaps cached in the hardware with different bitmap sizes: 0-bit, 8-bit, 16-bit, 32-bit, 64-bit, and 128-bit. Next, we simulated the Flow Completion Time (FCT) under the same packet loss model but with different bitmap cache sizes. Based on the comparative test data, we found that caching only a small number of bitmaps, with a value of 8 bits, is sufficient to meet the goal of high-performance transmission. Of course, setting it to 16-bit, 32-bit, 64-bit, or 128-bit is also possible, but it will consume network card resources.

[0075] In this embodiment, the optimal size of the cache bitmap was confirmed to be 8 bits, and it also supports configuration using the parameter K, which defaults to 8. The bitmap cache synchronization method is as follows: Similar scenarios exist in computer memory's L1, L2, and L3 caches. Data is cached in locations that are more easily accessible. When data is read, it is read directly from the cache. When data needs to be written, it is written to the cache, and the cache location is marked as dirty. When the cache needs to be replaced to memory, the data is written back to memory. Cache entries not marked as dirty are directly replaced without needing to be written back to memory. The hardware network card bitmap and software bitmap synchronization mechanism of RaLL is not exactly the same as computer memory synchronization. The goal of the bitmap is to record unreceived data packets, so its first position must record unreceived data packets. The overhead of the bitmap is released when retransmitted data packets are received. The overall structure of the software bitmap is designed as a circular array. The array index represents the data packet sequence number, and the values ​​in the array use 1 and 0 to indicate whether the data packet has been received. The starting position of the bitmap is controlled by pointers. For the receiving end, the starting position of the bitmap is the ePSN, and the pointer indicating the sequence number needs to take the remainder of the ePSN with respect to the bitmap length. For the sending end, the starting position of the bitmap is the eACK, which is the first gap packet when a packet is lost. The bitmap cached on the hardware is 8 bits starting from the beginning position, which can be controlled by the parameter K, and its mechanism is as follows.

[0076] Receiver buffer bitmap logic. Initially, the buffer bitmap has K bits, all of which are 0, indicating that no data packets have been received yet. After receiving a data packet, it is necessary to determine the relationship between the received data packet's PSN and ePSN: ① If PSN=ePSN and ReceiveOutOfOrder = false, the receiving end will always receive packets in sequence and will not enter the packet loss retransmission state. The bitmap is not enabled, and the network card protocol stack will receive data packets normally.

[0077] ② If PSN ≠ ePSN, regardless of whether it has entered the lost and retransmitted state, it is necessary to first determine whether the position of the bitmap to be filled is within the buffer range: the determination method is the relationship between the PSN and ePSN+K of the currently received data packet. If PSN < ePSN+K, then directly update the hardware cache, update the position of the corresponding PSN to 1, and update the receiveHigh and ReceiveOutOfOrder parameters; reply to the sender with sack PSN.

[0078] If PSN >= ePSN+K, then parameters such as receiveHigh and ReceiveOutOfOrder need to be updated first; then sack PSN in response; finally, the PSN is uploaded to the software bitmap for updating. The above two methods of updating the bitmap are as follows: Figure 5 As shown.

[0079] ③ If PSN = ePSN and ReceiveOutOfOrder = true, it indicates that the current state is in packet loss retransmission and the expected retransmission data packet has been received. First, modify the corresponding ePSN position in the hardware buffer bitmap to 1; then traverse the hardware buffer bitmap to find the sequence number of the next bitmap position that is 0, such as... Figure 6 As shown. If as Figure 7 As shown, if every bit after ePSN is found to be 1, then the ePSN is temporarily updated to ePSN + K, and recvLeft = recvHigh and recvRight = recvHigh are recorded; then the PSN is transmitted to the software bitmap, and a software bitmap update request is initiated; finally, the sender is replied with ack PSN.

[0080] like Figure 7 After receiving the update request, the software continues to search for the new ePSN and the first bitmap value of 0, updates the ePSN to this sequence value, and then sends the updated ePSN and the K-bit bitmap starting with the ePSN to the hardware network card for parameter updates and new bitmap caching.

[0081] like Figure 8 , Figure 9 , Figure 10 As shown, special handling is required for new data packets arriving during software processing. If the packet's PSN = recvRight + 1, then update recvHigh = PSN and update recvRight = PSN. If PSN > recvHigh + 1, then update recvLeft and recvRight to both PSN. When a new bitmap is updated upon receiving a data packet, this check needs to be performed again, such as... Figure 9 , 10As shown, if PSN < ePSN+K, then the corresponding PSN position is updated to 1 in the refreshed empty hardware bitmap; for example... Figure 8 As shown, if PSN >= ePSN+K, the bitmap update request is sent to the software part for bitmap update.

[0082] like Figure 9 , Figure 10 As shown, after the software processing is complete, it sends out updated hardware network card parameters and bitmaps. If the updated ePSN falls within the range [recvLeft, recvRight], then... Figure 10 As shown, ePSN will be updated to recvRight+1, and the bitmap cache will be directly updated to all zeros. Otherwise, as... Figure 9 As shown, the hardware ePSN is updated to the software-deployed ePSN. The bitmap is aligned with both the software-deployed bitmap and the hardware bitmap, and each bit is bitwise ORed with the two bitmaps to obtain the updated hardware bitmap. After the update, the bitmap is traversed to find new gaps to become the latest ePSN. If all bits are 1 after merging, the ePSN is temporarily updated to ePSN+K.

[0083] The sending end uses a similar logic for caching bitmaps in hardware. The initial bitmap is also K bits, with each bit set to 0.

[0084] ① If the packet type is ack and SendOutOfOrder is False, it indicates that the receiver has correctly received the expected packet and no packet loss event has occurred. Figure 1 It remains in an unenabled, initialized state.

[0085] ② If the packet type is ack and SendOutOfOrder is True, it indicates that after the receiver entered the select-retransmit state, the receiver received the expected retransmitted packet. When PSN >= eACK table, eACK and bitmap information need to be updated: When PSN < eACK+K, simply update eACK in the hardware bitmap to set all bits of PSN to 1 and then trim it. Then, iterate through the hardware bitmap after PSN, find the next gap, and update eACK to that sequence number. If all hardware bitmap bits after PSN are 1, temporarily update eACK to eACK+K and then send a request to the software to find the latest eACK. Upon receiving the request, the software kernel iterates through the portion of the software bitmap after eACK+K, finds the first gap as the new eACK, and updates the eACK and parameter values ​​along with the bitmap cache in the range [eACK, eACK+K] to the hardware network card.

[0086] When PSN >= eACK+K, eACK needs to be temporarily updated to eACK+K, and sendLeft = sackHigh and sendRight = sackHigh are recorded. Then, PSN is sent to the software update bitmap. The software kernel traverses the software bitmap, finds the sequence number of the next bitmap after PSN that is 0, updates eACK to that sequence number, and sends eACK and the K-bit bitmap after eACK to the hardware update bitmap cache.

[0087] During the software bitmap update, newly arriving data packets require special handling. The process is based on a temporary parameter state (temporarily updated eACK). Since the entire bitmap is set to 1s, it is directly refreshed to 0s to describe the [eACK, eACK+K] portion of the bitmap. For arriving data packets, if their PSN < eACK+K, the corresponding position in the hardware bitmap is directly updated to 1; if their PSN >= eACK+K, an update request is uploaded to the software for bitmap updating. If PSN = sendRight+1, then sendRight = PSN; if PSN > sendRight+1, then both sendLeft and sendRight are updated to PSN.

[0088] After the software bitmap is updated, the updated eACK and bitmap are sent to the hardware network card. If the updated eACK falls within the range [sendLeft, sendRight], then eACK is updated to sendRight+1, and the bitmap is cleared. Otherwise, the software-sent bitmap and hardware bitmap are aligned, merged, and bitwise OR operations are performed on each bit. After obtaining the merged bitmap, the new bitmap is traversed to find new gaps to become the latest eACK. If each bit is 1 after merging, then eACK is temporarily updated to eACK+K. Finally, the parameter indicating whether the packet loss retransmission state is in progress is updated based on the relationship between eACK and sackHigh. If eACK > sackHigh, it indicates that all retransmitted data packets have been received, SendOutOfOrder is changed to False, and the packet loss retransmission state is exited. If eACK < sackHigh, it indicates that there are still retransmitted data packets in transmission, which need to be maintained without further modifications.

[0089] ③ If the packet type is sack and SendOutOfOrder is False, it indicates that no packet loss event has occurred, but the response to the currently received packet indicates that the receiver has detected packet loss. If PSN > eACK, first update the hardware status parameters SendOutOfOrder, sack_high, etc.; then perform bitmap updates and retransmission judgment: When PSN < eACK+K, update the corresponding position in the hardware bitmap to 1 directly, and then add all data packets between eACK and PSN to the retransmission queue.

[0090] When PSN >= eACK+K, after the hardware updates the status parameters, all data packets between eACK and PSN are added to the retransmission queue, and then the PSN parameter is sent to the software for bitmap update.

[0091] ④ If the packet type is sack and SendOutOfOrder is True, it indicates that packet loss has occurred. Determine the relationship between PSN and eACK: If PSN == eACK, it indicates that the eACK of the currently retransmitted data packet has been received. The corresponding position in the hardware bitmap is changed to 1. Then, the hardware bitmap is traversed to find the next sequence number that is 0. If every bit in the bitmap after that is 1, eACK is temporarily updated to eACK+K. SendLeft = sackHigh and sendRight = sackHigh are recorded. The bitmap update request is uploaded to the software. After receiving the request, the software kernel finds the first sequence number that is 0 after eACK+K in the bitmap, updates eACK to that sequence number, and transmits eACK to the hardware network card for parameter update.

[0092] The handling of new data packets during software updates, as well as the synchronization logic of hardware and software parameters and bitmaps after software updates, are completely consistent with ② above.

[0093] If eACK < PSN < sackHigh, it indicates that a retransmitted data packet was lost, and only the PSN retransmission was received. If PSN < eACK+K, the corresponding PSN position in the hardware bitmap is changed to 1; and the hardware bitmap is traversed to add all gap data packets between eACK and PSN back to the retransmission queue for retransmission. If PSN >= eACK+K, the hardware bitmap is first traversed, adding all gap packets in the bitmap to the retransmission queue; then, the PSN is uploaded to the software kernel, updating the corresponding position in the software bitmap to 1. The software kernel traverses the software bitmap, records all gaps from eACK+K to PSN, and sends them to the hardware, which then adds all gap packets to the retransmission queue for retransmission.

[0094] If sackHigh < PSN, it indicates that a new data packet has been lost; update sackHigh to PSN; and add all data packets between sackHigh and PSN back to the retransmission queue for retransmission. If PSN < eACK+K, change the corresponding PSN position in the hardware bitmap to 1. If PSN >= eACK+K, send the PSN to the software part and modify the corresponding position in the software bitmap to 1.

[0095] This completes the update and synchronization of the hardware bitmap. It can be observed that the hardware bitmap does not need to be synchronously written back to the software bitmap; it only needs to fetch software bitmap blocks and cache them in the hardware bitmap. Specifically, when an expected data packet is received, ePSN / eACK needs to be updated. Since the hardware bitmap is full of 1s, ePSN / eACK cannot be found. Therefore, it is necessary to traverse the software bitmap to obtain the latest eACK and the hardware bitmap cache. By caching the hardware bitmap and using asynchronous transmission logic, the protocol stack can respond quickly and parameters can be updated more rapidly.

[0096] Figure 11 The throughput of different schemes in various scenarios is shown. Three schemes are compared in the figure: (1) IRN where the bitmap and all selective retransmission logic are implemented in hardware; (2) SRNIC where the bitmap and some selective retransmission logic are placed in software, but the hardware processing logic is blocked by the software; (3) This embodiment where the bitmap and some retransmission logic are placed in software, and the hardware and software logic are completely asynchronous. Unless otherwise specified, the schemes and corresponding legends in the subsequent experiments are consistent with those in this figure. In the experiment, the size of the transmitted data packet was fixed at 1342B, and the simulated network bandwidth was 100Gbps. This paper controls the amount of data transmitted by adjusting the number of data packets to 3, 10, 100, and 1000, and controls the concurrency by adjusting the number of concurrent connections. The packet loss rate reflects the intermediate network quality, and the packet loss rate ranges from 0 to 1%.

[0097] The experiment tested the throughput of the three schemes under varying packet loss rates, with different combinations of data packets and connections. When the number of data packets was 100, each connection transmitted 100KB of data, theoretically achieving a maximum bandwidth of 100Gbps. Under low packet loss rates, all three schemes maintained almost full throughput. As the packet loss rate increased to 1%, the blocking logic IRN and SRNIC schemes, which used retransmission logic, experienced increased latency due to bitmap read / write and retransmission logic, resulting in a 4-5% decrease in throughput. SRNIC, due to PCIe round-trip latency caused by accessing the software bitmap, experienced even higher latency than IRN, leading to a greater throughput decrease. The non-blocking RaLL scheme, however, maintained 100% throughput throughout, representing a 5.2% improvement over IRN and SRNIC.

[0098] Different research schemes require different bitmap positions for selective retransmission, resulting in varying parameters and bitmap interaction logic. These different interaction modes lead to varying processing latency on the endpoint. This paper aims to compare the latency performance of different implementation modes, using Stream Completion Time (FCT) as the metric. When the data volume is fixed, average throughput data can be obtained simultaneously with the stream completion time. Experiments were conducted to test the FCT values ​​under varying data transmission volume, number of connections, and network packet loss rate.

[0099] Figure 12 This study demonstrates the average latency of different schemes under various scenarios. The data packet size transmitted between the sender and receiver is 1342 bytes. Similar to the throughput measurement process, the experiment controlled the data volume of a single stream by adjusting the number of data packets and the packet loss rate, observing the latency performance of different schemes under different packet loss environments with fixed data volume and fixed concurrency. It can be observed that, similar to the throughput trend, when the packet loss rate is 0% and 0.001%, the FCT is close to linear flow completion time. When the packet loss rate increases to 1%, the FCT of SRNIC and IRN increases by ~30 microseconds, approximately 3%. RaLL, due to its non-blocking selective retransmission protocol, still maintains a linear flow completion time.

[0100] Table 1 Storage overhead for each scheme implementing selective retransmission

[0101] Storage overhead primarily considers the overhead of information cached in hardware. Based on the above analysis, different protocol implementations employ different selective retransmission strategies, resulting in different bitmap storage locations and consequently varying bitmap overhead. However, if we assume that the bitmap overhead for each connection is fixed regardless of packet loss, then storage can be directly calculated theoretically, as shown in Table 1. Figure 13 The document demonstrates the storage overhead of selective retransmission under different schemes as the packet loss rate increases, as well as the hardware bitmap buffer size of IRN, SRNIC, and RaLL during transmission.

[0102] However, if we assume that bitmap overhead is only allocated when packet loss occurs, we need to measure the maximum number of connections experiencing simultaneous packet loss. In addition, the experiment recorded the actual bitmap overhead incurred when each connection experienced packet loss, and finally recorded the maximum sum of bitmap overhead for all connections experiencing simultaneous packet loss. If the actual bitmap cache can be dynamically controlled, this value is also the ideal value for allocating bitmap space. Figure 14We compared the maximum overhead of fixed hardware, software bitmap, bitmap request during packet loss, and actual bitmap. Specifically, we analyzed the scenario with 100 packets and 100 connections. The simulator's BDP was 60 bits (under a 100Gbps network, assuming a network RTT of 6 microseconds and an MTU of 1342B; the actual measured RTT was approximately 24 microseconds, only shortening network propagation latency without affecting end-side processing time). Therefore, the fixed hardware bitmap overhead was 60 * 100 = 6000 bits, and the software bitmap overhead was 0 bits because it wasn't cached in the hardware. Observations showed that requesting a bitmap during packet loss was very close to the ideal bitmap overhead. Requesting a fixed 60-bit bitmap only during packet loss resulted in extremely low bitmap overhead at a packet loss rate of 0.1% or less, approximately two orders of magnitude lower than the fixed hardware overhead. At a packet loss rate of 1%, the overhead during packet loss was twice that of the fixed overhead, and on the same order of magnitude as the ideal overhead. Therefore, we can conclude that there is considerable room for compression in bitmap overhead.

[0103] As can be seen, the hardware and software asynchronous RDMA transmission packet loss recovery system in this embodiment only needs to retransmit the lost data packets, avoiding the repeated transmission of data packets that have already arrived, thus maintaining high throughput and low latency; thereby improving network utilization in lossy networks, while being compatible with existing scalability solutions.

Claims

1. A hardware / software asynchronous RDMA transmission packet loss recovery system, characterized in that: This includes bitmaps, retransmission parameters, ReqQ for loading retransmission requests, and FlushQ for loading update bitmap requests; the bitmaps are set on the software kernel, and the retransmission parameters are set in the context state information of each connection. A pre-defined bitmap of some active connections is cached on the network card along with the context state. When the bitmap sequence number to be updated is within the cache range, the hardware cache is updated directly; otherwise, an asynchronous bitmap update request is sent to the software kernel. The retransmission parameters include SendOutOfOrder / RecvOutOfOrfer, sackHigh / recvHigh, eACK / ePSN, sendLeft / recvLeft, and sendRight / recvRight. The network card determines whether a packet loss event has occurred by using PSN and packet status parameters; The network interface card (NIC) can directly update the bitmap and status parameters, or the software can update the bitmap information and status parameters of the data packets corresponding to the packet loss event and resend them to the NIC. The NIC can asynchronously interact with the software kernel through ReqQ and FlushQ. The NIC can send a request to update the bitmap or status information through FlushQ and obtain the latest bitmap cache, the updated status parameters, and the sequence number of the data packets that need to be retransmitted through ReqQ.

2. The hardware and software asynchronous RDMA transmission packet loss recovery system according to claim 1, characterized in that: When in the lost and retransmitted state, after receiving a data packet with the expected sequence number, the hardware bitmap corresponding to the data packet needs to be updated to the transmitted state immediately. Then, the hardware bitmap is traversed to find the first untransmitted sequence number. If every subsequent bit is found to be transmitted, the ePSN is temporarily updated to the largest sequence number in the cached bitmap + 1. Then, an update bitmap request is sent to the software bitmap. The software kernel needs to traverse the bitmap after the updated ePSN in the software bitmap to find the first gap packet. Then, the latest bitmap of the corresponding size that needs to be cached, along with the latest gap sequence number, is updated to the hardware network card as the next expected data packet.

3. The hardware and software asynchronous RDMA transmission packet loss recovery system according to claim 1, characterized in that: The ReqQ record contains the sequence number of the data packet that needs to be retransmitted, the updated ePSN sequence number, and the latest bitmap that needs to be cached; the FlushQ record stores the retransmission logic of the network card hardware and the sequence number of the currently received data packet.

4. The hardware-software asynchronous RDMA transmission packet loss recovery system according to claim 1, characterized in that: The size of the preset bitmap of partially active connections is greater than or equal to 8 bits, and it can be configured via parameter K.

5. A method for recovering packet loss during asynchronous RDMA transmission (hardware and software), characterized in that; Includes the following steps: 1) The network card polls the doorbell register, receives a send request, obtains WQE requests in batches from the SQ Buffr and caches them on the network card, and parses the WQE to obtain the send request, obtains the virtual address and length of the data to be sent, and converts the virtual address into the corresponding physical address through the MTT mapping relationship. The DMA engine retrieves the data to be sent from memory according to the corresponding physical address and caches it on the network card for protocol stack processing. 2) After the data packet transmission is complete, the network card generates a CQE to notify the host application that the transmission is complete; 3) When the network card receives a data packet, it determines whether packet loss has occurred by checking the PSN and recording parameters, and updates the bitmap and status parameters accordingly based on whether packet loss has occurred. When updating the bitmap, the hardware directly updates the bitmap and the latest parameters based on the bitmap cache, or the hardware needs to send a request to the software. After the software updates the bitmap and the new parameters, it sends a retransmission data packet and the latest parameters to the hardware. 3.1) The receiving end's network card determines the data packet reception status; ① When the PSN of the data packet received on the receiving end's network card is greater than the ePSN, it enters the packet loss retransmission state, modifies the parameter RecvOutOfOrder to True, and replies to the sending end with a sack PSN. ② When PSN = ePSN, it indicates that the expected data packets arrive in order; directly receive the data packets and reply with an ack PSN to the sender; ③ If the PSN of the currently received data packet is less than the sequence number of the expected data packet, it means that the data packet has already been received and should be discarded. 3.2) The sending end receives the ack PSN or sack PSN from the receiving end and decides whether to retransmit; When the sending end receives the reply ack PSN in step 3.1) and SendOutOfOrder is False, it indicates that the receiving end has correctly received the expected data packet and no packet loss event has occurred. II. When the sending end receives the reply ack in step 3.1) and SendOutOfOrder is True, it indicates that after the receiving end enters the select retransmission state, the receiving end receives the expected data packet in the retransmission. III. When the sending end receives the reply in step 3.1) as sack and SendOutOfOrder as False, it indicates that no packet loss event has occurred, but the reply to the currently received data packet is that the receiving end has perceived packet loss. IV: When the sender receives a sack response from step 3.1) and SendOutOfOrder is True, it indicates that packet loss has occurred. Determine the relationship between PSN and eACK: 4) If the sending end detects packet loss according to step 3), it will add the data packet corresponding to the packet loss event back to the retransmission queue for retransmission; until the sending end receives an ack reply and SendOutOfOrder is updated to True, it will exit the packet loss retransmission state.

6. The method for recovering packet loss in asynchronous RDMA transmission using software and hardware as described in claim 5, characterized in that: The specific steps of step II are as follows: If PSN < eACK, the receiving end has received a data packet that was previously acknowledged, and no action will be taken in this case. If PSN >= eACK, the receiving end has received the data packet PSN in order. It needs to determine whether the PSN exceeds the range of the hardware bitmap buffer. If it does not exceed the range, it needs to change all bitmaps before PSN in the hardware buffer bitmap to the "transmitted" state and trim them, and start traversing the bitmap gaps after PSN, finding the first gap and updating eACK to the corresponding sequence number. If no gap is found or PSN exceeds the hardware buffer bitmap, it submits a request to the software to update the bitmap, changes all bitmaps before PSN to the "transmitted" state and trims them, and starts traversing the bitmap gaps after PSN, finding the first gap and updating eACK to the corresponding sequence number, and then transmits the updated eACK to the hardware network card. If the updated eACK > sackHigh, it means that the receiver has received all the retransmitted data packets, and SendOutOfOrder should be changed to False; if eACK < sackHigh, it means that the retransmitted data packets have not been received yet and the receiver is still in the lost and retransmitted state, so no other changes are needed.

7. The method for recovering packet loss in asynchronous RDMA transmission using software and hardware as described in claim 5, characterized in that: The specific steps of step III are as follows: If PSN < eACK, no action is taken; If PSN = eACK, then eACK = PSN + 1, sackHigh = PSN; If PSN > eACK, sackHigh = PSN, and SendOutOfOrder is changed to True. If PSN is within the hardware bitmap cache range, the PSN position is directly changed to the transmitted state in the hardware cache; otherwise, PSN is sent to the software kernel to update the bitmap, and the position of PSN in the bitmap is changed to the transmitted state. All data packets between eACK and PSN are added to the retransmission queue for retransmission.

8. The method for recovering packet loss in asynchronous RDMA transmission using software and hardware as described in claim 5, characterized in that: The specific steps of step IV are as follows: If PSN < eACK, no action is taken; If PSN=eACK, it indicates that the eACK of the currently retransmitted data packet has been received. The corresponding PSN in the hardware bitmap is updated to the transmitted state and then clipped. The next gap after the hardware bitmap is found and eACK is updated to the corresponding sequence number. If no gap is found, eACK is temporarily updated to the maximum bit of the hardware bitmap buffer + 1, and the software kernel is uploaded to update the bitmap sequence number to the transmitted state and clipped. eACK is updated to the first gap in the bitmap and then transmitted to the hardware network card. If eACK > sackHigh, it indicates that all currently retransmitted data packets have been received, and SendOutOfOrder is changed to False. If eACK < sackHigh, it indicates that there are still retransmitted data packets in transmission, which need to be maintained without further modification. If eACK < PSN < sackHigh, it indicates that a retransmitted data packet was lost again, and only the PSN retransmission was received. If the PSN is within the hardware bitmap buffer range, the PSN position is directly updated to the transmitted state, and the bitmap is traversed to add all gap data packets between eACK and PSN to the retransmission queue; otherwise, all gap data packets in the hardware bitmap are added to the retransmission queue, and then the PSN is sent to the software kernel to update the bitmap, changing the position of the bitmap with PSN to the transmitted state; and all gap data packets between the highest bit + 1 of the hardware bitmap and PSN are added to the retransmission queue again for retransmission. If sackHigh < PSN, it indicates that a new data packet has been lost. Set sackHigh to PSN. If PSN does not exceed the hardware bitmap buffer range, directly update the corresponding position of PSN in the hardware bitmap to the transmitted state. Otherwise, send PSN to the software kernel to update the bitmap and change the position of PSN in the bitmap to the transmitted state. In addition, all data packets between sackHigh and PSN are added to the retransmission queue for retransmission.

9. The method for recovering packet loss in asynchronous RDMA transmission using software and hardware as described in claim 5, characterized in that: In step 4), when in the lost and retransmitted state, after receiving the data packet with the expected sequence number, the corresponding bitmap of the hardware data packet needs to be updated to the transmitted state immediately. Then, the hardware bitmap is traversed to find the first untransmitted sequence number. If it is found that every subsequent bit is in the transmitted state, the ePSN is temporarily updated to the largest sequence number in the cached bitmap + 1. Then, the software bitmap initiates a bitmap update request. The software kernel needs to traverse the bitmap after the updated ePSN in the software bitmap to find the first gap packet. Then, the latest bitmap of the corresponding size that needs to be cached, as well as the latest gap sequence number, are updated together with the latest next expected data packet to be received and updated to the hardware network card.

Images