RDMA (Remote Direct Memory Access) reliable connection fault-oriented rapid recovery method and system

Abstract

The invention belongs to the technical field of computer network communication, and provides an RDMA-oriented reliable connection fault rapid recovery method and system, the method is executed by a first communication end, communication is established between the first communication end and a second communication end through a queue pair QP, and the queue pair QP is used for bearing service data transmission. The first communication end reserves resources after the QP enters an error state, and sends a PROBE control frame to the opposite end through the control plane queue pair; the second communication end decides to reuse the original QP or allocate a new QP (reuse an original protection domain, a memory registration area and a completion queue) according to the local QP state, and returns a QP number and a start packet serial number in an RESP frame; and the first communication end updates the local QP context and drives the state machine to switch to a ready sending state. The whole process does not need to destroy the QP and does not depend on CM handshake, the recovery time delay is less than 0.5 millisecond, and the method is transparent to the upper layer application and is suitable for the RoCE v2 network environment.

Description

Technical Field

[0001] This invention belongs to the field of computer network communication, and relates to high-performance computing networks and remote direct memory access (RDMA) technology. Specifically, it relates to a method and system for rapid recovery of reliable connection failures in RDMA. Background Technology

[0002] Remote Direct Memory Access (RDMA) technology significantly reduces network latency and improves throughput by bypassing the operating system kernel and CPU to participate in data copying, and has become a key communication infrastructure in fields such as high-performance computing (HPC), distributed storage, artificial intelligence training, and high-frequency trading in finance. Among them, the RoCE (RDMA over Converged Ethernet) protocol, derived from the InfiniBand architecture, especially RoCE v2, is widely deployed in modern data centers because of its compatibility with standard Ethernet infrastructure.

[0003] In the RDMA communication model, the Reliable Connection (RC) mode provides ordered, lossless, end-to-end data transmission services through queue pairs (QPs). Each QP maintains independent send / receive work queues (SQ / RQ) and ensures data consistency and reliability through packet sequence numbers (PSNs) and ACK / NACK mechanisms.

[0004] However, when a link interruption, switch port failure, or host hardware / software malfunction occurs, QP will enter an Error State (ERR) due to consecutive retransmission failures, and communication will be immediately interrupted. The existing recovery process follows the paradigm of "timeout detection → resource destruction → CM reconstruction," which suffers from the following serious technical bottlenecks: 1. High fault detection latency: The network card hardware relies on the ACK timeout mechanism to trigger the QP_ERR event. In common commercial RNIC configurations, the ACK timeout is usually several milliseconds to tens of milliseconds, and the number of retransmissions N_RETRY is often on the order of several times. Even if the application layer actively injects heartbeats, it is still limited by the driver reporting latency and cannot achieve microsecond-level detection. 2. High overhead of resource reclamation and reconstruction: Upper-layer applications must call ibv_destroy_qp() to release QP and related resources (including CQ, PD, MR, etc.), and then perform a three-way handshake through RDMA Connection Manager (CM) to renegotiate parameters such as QPN, PSN, and rkey. This process involves multiple layers of protocol interaction, including user-mode-kernel mode switching, ARP resolution, and path lookup, and typically takes 1–10 milliseconds. 3. Application-layer perceived interruption, business continuity impaired: After QP reconstruction, the original WQE becomes invalid, and the upper layer needs to re-register memory, construct a new WQE, and resend incomplete requests. In avionics, real-time control systems, or low-latency transaction scenarios, such interruptions may lead to task timeouts, inconsistent states, or even system-level failures.

[0005] 4. Hot standby solutions suffer from resource waste and poor scalability: Some studies propose pre-built hot-standby QPs to achieve rapid switching, but each service QP requires an additional QPN and associated resources. Under large-scale deployment (such as a multi-card cluster), this leads to rapid depletion of NIC (Network Interface Card) resources, and it is difficult to dynamically adapt to changes in the QP lifecycle. The aforementioned problems are particularly prominent in critical areas sensitive to communication determinism and latency. For example, in automotive electronic systems, RDMA is used for critical data transmission in radar, autonomous driving, and other applications, requiring rapid recovery from link interruptions; while in high-frequency quantitative trading, even millisecond-level latency differences can lead to significant economic losses. Therefore, there is an urgent need for a lightweight, fast recovery mechanism that requires no pre-built resources, does not rely on CM handshakes, and can complete parameter updates and state resets locally within the original QP context. Summary of the Invention

[0006] To address the technical problem that existing RDMA reliable connections (RC) must undergo long ACK timeouts, destroy communication resources, and re-establish the connection through a CM three-way handshake after a link failure, resulting in service interruption latency far exceeding the requirements of real-time scenarios, this invention discloses a method and system for rapid recovery of RDMA reliable connection failures for the RoCE v2 protocol. This method achieves sub-millisecond self-healing capabilities while maintaining complete transparency to existing network infrastructure and upper-layer applications.

[0007] To achieve the above objectives, the present invention provides the following technical solution: On the one hand, a method for rapid recovery from reliable connection failures in RDMA is provided. This method is executed by a first communication terminal, which has established communication with a second communication terminal through a queue pair (QP) in a reliable connection (RC) mode. The queue pair (QP) is used to carry service data transmission. The method includes the following steps: S1. After detecting that the queue pair QP has entered an error state due to a link failure or node failure, the queue pair QP and its associated protection domain, memory registration area and completion queue resources are retained, and the queue pair QP is not destroyed. S2. Send a parameter request to the second communication terminal and receive the parameter response returned by it to obtain the connection recovery parameters fed back by the second communication terminal; S3. Based on the connection recovery parameters, update the context data structure of the queue pair and drive the state machine of the queue pair to switch from the error state to the ready-to-send state to restore data transmission capability.

[0008] Furthermore, in step S1, the first communication terminal automatically injects heartbeat frames into the queue pair QP through network card hardware or host driver, and sets the queue pair QP to an error state when no acknowledgment response is received for several consecutive times.

[0009] Furthermore, the heartbeat frame is a zero-byte RDMA write operation with an immediate value; wherein, the network card hardware or the host driver configures an acknowledgment timeout threshold and a maximum number of retransmissions for the queue pair QP, and when the number of consecutively sent heartbeat frames that do not receive an acknowledgment response within the acknowledgment timeout threshold reaches the maximum number of retransmissions, an operation to set the queue pair QP to an error state is triggered.

[0010] Further, in step S2, a parameter request is sent to the second communication terminal, and a parameter response is received to obtain the connection restoration parameters fed back by the second communication terminal, including: S21. The host driver of the first communication terminal sends a PROBE control frame to the second communication terminal through the control plane; S22. After receiving the PROBE control frame, the second communication terminal returns the RESP control frame. S23. The host driver of the first communication terminal parses the RESP control frame and extracts the connection recovery parameters, which include the remote queue pair QP number and the peer start packet sequence number (PSN).

[0011] Furthermore, the control plane is implemented based on Global Service Interface Queues (GSI-QP). Both the PROBE control frame and the RESP control frame are transmitted through the Global Service Interface Queues (GSI-QP), and the encapsulation format is compatible with the RoCE v2 protocol stack.

[0012] Furthermore, the encapsulation structure of both the PROBE control frame and the RESP control frame includes: an outer layer consisting of an Ethernet header, an IP header, and a UDP header, and an inner layer consisting of an InfiniBand Transport Header (BTH), a Datagram Extension Header (DETH), and a Management Datagram Header (MAD Header).

[0013] Furthermore, in step S22, the remote queue pair QP number depends on the processing result of the second communication end on its local queue pair QP state, including: When the second communication terminal reuses its local queue pair QP, the RESP control frame returns the number of the local queue pair QP. When the second communication terminal cannot reuse its local queue pair QP, it allocates a new queue pair QP and associates the new queue pair QP with the protection domain, memory registration area and completion queue resources associated with the original queue pair in order to reuse the protection domain, memory registration area and completion queue resources, and returns the number of the new queue pair QP in the RESP control frame. The first communication terminal updates the context of its local queue pair QP only based on the remote queue pair QP number, without needing to know whether the second communication terminal reused the original queue pair QP or allocated a new queue pair QP.

[0014] Furthermore, in step S3, during the process of driving the state machine of the queue to switch from the error state to the ready to send state, the first communication terminal does not call the QP destruction and reconstruction interface throughout the entire process, and the state switch is completed within a single driving operation; the second communication terminal reuses the original QP or creates a new QP'.

[0015] On the other hand, a system for rapid recovery from reliable RDMA connection failures is also provided, including a fault detection module, a management channel module, and a QP rapid reconfiguration module.

[0016] The fault detection module is used to retain the queue pair QP and its associated protection domain, memory registration area and completion queue resources after detecting that the queue pair QP has entered an error state due to link failure or node failure, without destroying the queue pair QP. The management channel module is used to send parameter requests to the second communication terminal and receive the parameter responses returned by it, so as to obtain the connection recovery parameters fed back by the second communication terminal; The QP fast reconstruction module is used to update the context data structure of the queue pair based on the connection recovery parameters, and drive the state machine of the queue pair to switch from the error state to the ready-to-send state in order to restore data transmission capability.

[0017] Furthermore, the entire recovery process after a connection failure does not require a pre-built hot standby QP, does not rely on re-performing the complete handshake process of the RDMA connection management protocol, the end-to-end recovery latency is less than the time required to re-establish the connection, and it is completely transparent to the existing RoCEv2 network environment.

[0018] This invention addresses the inherent shortcomings of existing RDMA reliable connection fault recovery mechanisms by proposing an innovative three-in-one architecture of "fast detection—control plane negotiation—local reconnection," which has the following significant technical advantages and practical value: 1. Ultra-low recovery latency to meet stringent real-time requirements; 2. Zero resource redundancy, improving NIC resource utilization; 3. Completely transparent to upper-layer applications, seamlessly integrating with the existing ecosystem; 4. Strong protocol compatibility, supporting large-scale deployment; 5. High robustness and adaptability; 6. Applicable to key application scenarios such as avionics, financial transactions, AI / HPC clusters, and cloud-native databases. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of the method for rapid recovery from reliable RDMA connection failures according to the present invention; Figure 2 This is the execution flow of the method for rapid recovery from reliable RDMA connection failures according to the present invention; Figure 3 This is an architecture diagram of the system for rapid recovery from reliable RDMA connection failures according to the present invention; Figure 4 This is the structure of the PROBE control frame. Detailed Implementation

[0021] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0022] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features of the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] In existing RoCEv2 networks, when a link or node experiences a transient failure, the anomaly is typically triggered passively by RDMA's ACK confirmation timeout for the service data itself and continuous retransmission failures. This is because the sender only determines a connection anomaly when a number of consecutive data packets fail to receive ACKs and the retransmission timer expires. Since the lower limit of these timers is generally greater than 10 milliseconds, services still need to wait on the order of milliseconds after the fault is actually cleared before recovery can begin. Consequently, latency-sensitive applications may experience "idling" or direct timeouts.

[0024] Even when an anomaly is detected, the traditional reconstruction process remains extremely lengthy. Typically, the upper-layer application first proactively destroys the faulty QP and reclaims related memory and communication resources. Then, it re-executes RESOLVE ADDR, RESOLVEROUTE, and CONNECT via RDMA-CM to perform address resolution, route resolution, and connection establishment. Management messages rely on QP1 (GSI) transmission. Meanwhile, creating a new QP requires requesting a QPN from the network card hardware, initializing the QP context, and allocating memory, which incurs additional overhead of hundreds of microseconds to milliseconds under high load. Application threads often get stuck in the exception handling branch, requiring re-registration of memory and updating memory resources, further expanding the failure window.

[0025] To address the aforementioned issues, this invention proposes a "fast detection-parameter negotiation-local reconnection" strategy. This strategy monitors the link status via microsecond-level heartbeats through the sending network interface card (NIC), triggering a QP_ERR after a short ACK timeout. The driver then sends a PROBE control frame through the management channel GSI-QP, requesting the peer to return an available QPN and PSN. The peer either reuses the original QP or creates a new QP and feeds back the parameters. The sending end directly updates the remote QPN and bidirectional PSN within the original QP context and completes recovery through a state machine pipelined switching. The entire process requires no QP destruction, does not rely on CM handshakes, has a recovery latency of less than 0.5 milliseconds, is transparent to upper-layer applications, and can be seamlessly deployed in existing RoCE v2 networks, significantly improving system high availability and latency determinism.

[0026] Specifically, in one embodiment of the present invention, a method for rapid recovery from reliable connection failures in RDMA is disclosed. This method is executed by a first communication terminal, which has established communication with a second communication terminal through a reliable connection (RC) mode queue pair (QP). The queue pair QP is used to carry service data transmission. It should be noted that the first communication terminal is... Figure 2 The requesting end shown is the second communication end, which is also the response end. See also... Figure 1 As shown, it includes the following steps: S1. After detecting that the queue pair QP has entered an error state due to a link failure or node failure, the queue pair QP and its associated protection domain, memory registration area and completion queue resources are retained, and the queue pair QP is not destroyed. S2. Send a parameter request to the second communication terminal and receive the parameter response returned by it to obtain the connection recovery parameters fed back by the second communication terminal; S3. Based on the connection recovery parameters, update the context data structure of the queue pair and drive the state machine of the queue pair to switch from the error state to the ready-to-send state to restore data transmission capability.

[0027] Furthermore, in step S1, the first communication terminal automatically injects heartbeat frames into the queue pair QP through network card hardware or host driver, and sets the queue pair QP to an error state when no acknowledgment response is received for several consecutive times.

[0028] Furthermore, the heartbeat frame is a zero-byte RDMA write operation with an immediate value; wherein, the network card hardware or the host driver configures an acknowledgment timeout threshold and a maximum number of retransmissions for the queue pair QP, and when the number of consecutively sent heartbeat frames that do not receive an acknowledgment response within the acknowledgment timeout threshold reaches the maximum number of retransmissions, an operation to set the queue pair QP to an error state is triggered.

[0029] In specific implementation, such as Figure 2 As shown, the first communication end, i.e., the requesting end, automatically injects heartbeat frames into its queue QP at fixed microsecond intervals. It uses RDMA write with immediate values ​​as the heartbeat frame and employs a shortened ACK-Timeout and a limited retransmission count. When consecutive heartbeats of a heartbeat frame fail to receive consecutive ACKs within the window, the sending end's network card immediately sets the QP state to ERR and triggers a QP_ERR interrupt to the requesting end's network card hardware or host driver. In implementations where the hardware does not support 0-byte writes, a minimum length (e.g., 4 bytes) can be set as the heartbeat payload; this invention is not limited to a specific heartbeat payload length.

[0030] Further, in step S2, a parameter request is sent to the second communication terminal, and a parameter response is received to obtain the connection restoration parameters fed back by the second communication terminal, including: S21. The host driver of the first communication terminal sends a PROBE control frame to the second communication terminal through the control plane.

[0031] In practice, when a fault is detected, the first communication terminal driver suspends communication with the second communication terminal network card; then, it generates its own first packet sequence number (PSN) and sends a PROBE control frame (request realignment message PROBE) to the responding terminal driver via the GSI-QP of the RDMA-CM management channel, requesting the other end to return an available QPN and the other end's first packet sequence number (PSN).

[0032] S22. After receiving the PROBE control frame, the second communication terminal returns the RESP control frame.

[0033] S23. The host driver of the first communication terminal parses the RESP control frame and extracts the connection recovery parameters, which include the remote queue pair QP number and the peer start packet sequence number (PSN).

[0034] In specific implementation of steps S22 and S23, the remote queue pair QP number in the RESP control frame depends on the processing result of the second communication end on its local queue pair QP status. That is, after the second communication end receives the PROBE control frame, the driver verifies the request PROBE. If the original QP can be ERR→RESET→RTS, the old QPN is retained; otherwise, a new QP is created immediately to obtain a new number QPN′, and the peer's first packet sequence PSN is randomly generated and fed back in the RESP control frame {resp_qpn, resp_psn}.

[0035] Specifically, the judgment and corresponding operation process is as follows: S221. When the second communication terminal reuses its local queue pair QP, the RESP control frame returns the number of the local queue pair QP. S222. When the second communication terminal cannot reuse its local queue pair QP, it allocates a new queue pair QP and associates the new queue pair QP with the protection domain, memory registration area and completion queue resources associated with the original queue pair, so as to reuse the protection domain, memory registration area and completion queue resources, and returns the number of the new queue pair QP in the RESP control frame. When step S3 is executed, the first communication terminal updates the context of its local queue pair QP only based on the remote queue pair QP number, without needing to know whether the second communication terminal reused the original queue pair QP or allocated a new queue pair QP.

[0036] Furthermore, the control plane is implemented based on Global Service Interface Queues (GSI-QP). Both the PROBE control frame and the RESP control frame are transmitted through the Global Service Interface Queues (GSI-QP), and the encapsulation format is compatible with the RoCE v2 protocol stack.

[0037] Furthermore, the encapsulation structure of both the PROBE control frame and the RESP control frame includes: an outer layer consisting of an Ethernet header, an IP header, and a UDP header, and an inner layer consisting of an InfiniBand Transport Header (BTH), a Datagram Extension Header (DETH), and a Management Datagram Header (MAD Header).

[0038] Furthermore, in step S3, during the process of driving the state machine of the queue to switch from the error state to the ready-to-send state, the first communication terminal does not call the QP destruction and reconstruction interface at all, and the state switch is completed within a single driving operation; the second communication terminal reuses the original QP or creates a new QP′, which does not affect the applicability of the present invention.

[0039] Specifically, when the first communication end receives the RESP control frame and extracts the connection recovery parameters, it can sequentially execute ERR→RESET→INIT→RTR→RTS on the original QP in step 4. During this process, only dest_qp_num = resp_qpn, rq_psn = PSN_remote, and sq_psn = PSN_local are rewritten to restore the transmission capability in one go.

[0040] After the update, the first communication end replies with an ACK frame via GSI-QP, and the second communication end releases the recovery state upon receiving it; subsequently, heartbeats and service traffic return normally on the updated QP, and the connection recovery is complete.

[0041] Furthermore, the state transition time is less than a preset time threshold (e.g., 60 microseconds), and the doorbell register is only written after the state transition is complete. Connections are restored within the same PD / MR environment, eliminating the need for a pre-built hot standby QP and ensuring full compatibility with existing InfiniBand / RoCE protocol stacks. For example, in a prototype system implementation, testing showed that the measured end-to-end recovery latency is approximately sub-millisecond, consistently meeting the continuity requirements of real-time data links. This invention is not limited to this test data.

[0042] Based on the same inventive concept, this invention also provides a system for rapid recovery from RDMA reliable connection failures, as described in the following embodiments. Since the principle underlying the problem-solving of the system for rapid recovery from RDMA reliable connection failures is similar to the method for rapid recovery from RDMA reliable connection failures disclosed in the above embodiments, the implementation of the system for rapid recovery from RDMA reliable connection failures can refer to the implementation of the method for rapid recovery from RDMA reliable connection failures, and repeated details will not be elaborated further. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0043] Figure 3 This is a structural block diagram of a system for rapid recovery from reliable RDMA connection failures disclosed in an embodiment of the present invention, as shown below. Figure 3 As shown, the system is deployed on an RDMA network card or FPGA-NIC and includes a fault detection module, a management channel module, and a QP fast reconfiguration module. The structure is described below.

[0044] The fault detection module is used to retain the queue pair QP and its associated protection domain, memory registration area and completion queue resources after detecting that the queue pair QP has entered an error state due to a link failure or node failure, without destroying the queue pair QP.

[0045] More specifically, this module is integrated into the FPGA / NIC hardware and is used to monitor the heartbeat of each QP. This module automatically injects a zero-byte RDMA Write with Immediate (RWI) operation and monitors the ACK-Timeout index and the number of retransmissions. Once a certain number of consecutive heartbeats fail to receive an ACK, a fault interrupt is triggered, and the corresponding QP status is set to ERR.

[0046] The management channel module is used to send parameter requests to the second communication terminal and receive the parameter responses returned by it, so as to obtain the connection recovery parameters fed back by the second communication terminal.

[0047] More specifically, this module constructs a control channel through a resident GSI-QP. After fault detection, the management channel is responsible for sending a PROBE control frame, carrying the local QPN and requesting the latest PSN from the peer. Simultaneously, it is responsible for receiving PROBE frames and reporting them to the QP fast reconfiguration module.

[0048] The QP fast reconstruction module is used to update the context data structure of the queue pair based on the connection recovery parameters, and drive the state machine of the queue pair to switch from the error state to the ready-to-send state in order to restore data transmission capability.

[0049] More specifically, after receiving the PROBE frame, this module first verifies the integrity and validity of the control frame, and then checks the current state of the corresponding local QP: if the QP can still be safely reset from ERR, it immediately performs a QP state change: ERR → RESET → INIT → RTR → RTS to preserve the original QPN; if the QP context is corrupted or the mapping is invalid, it immediately allocates a new QP′ from the spare QP pool, generates a new QPN′, and writes it into the mapping table. Finally, it encapsulates the selected QPN / first packet PSN into a RESP control frame and sends it back to the sender.

[0050] In addition, after receiving the RESP control frame from the other end, the module binds the new QPN / PSN to the original QP and quickly switches between the states ERR→RESET→INIT→RTR→RTS through the pipeline to complete the QP state recovery.

[0051] The system does not require a pre-built hot standby QP during the entire recovery process of connection failure, does not rely on the three-way handshake process of the RDMA connection management protocol, has an end-to-end recovery latency of less than 0.5 milliseconds, and is completely transparent to the existing RoCE v2 network environment.

[0052] In one embodiment, see Figure 2 As shown, the working process of the fault detection module specifically includes: 1. Heartbeat Message Type: Zero-byte RDMA Write with Immediate (RWI) messages are preferred as heartbeat frames. Since RWI messages are the shortest, RDMA write operations do not consume CPU resources, and a custom heartbeat sequence number HB_SEQ can be carried in the Immediate field. Therefore, high-frequency health checks can be achieved without significantly increasing link bandwidth.

[0053] 2. Heartbeat Injection: The fault detection module adds a heartbeat injection FIFO to each service QP and configures an independent counter for it. When the counter reaches the preset heartbeat period HB_INTERVAL, the WQE of the heartbeat RWI is written to send the heartbeat message.

[0054] 3. ACK-Timeout Index and Retransmission Strategy: Each service QP maintains an ACK timer and a retransmission counter. If no heartbeat ACK is received within period t, a retransmission is triggered, and the counter is incremented by one. When the retransmission count reaches the threshold N_RETRY and no ACK is received, the fault detection module immediately writes the status of the QP as ERR and reports it to the host software layer via an interrupt.

[0055] In one embodiment, see Figure 2 As shown, the working process of the management channel module is as follows: To ensure isolation between the control plane and the service plane, the driver calls ib_alloc_pd() during the loading phase to request a separate control protection domain (control_pd) and statically creates a GSI-QP based on this PD. The QPN of the GSI-QP defaults to 1. When the host CPU detects a QP error state, it immediately constructs a PROBE control frame according to Table 1 below, fills in the SGE, and sends a WQE transmission operation based on the GSI_QP.

[0056] Table 1: Field Composition of the PROBE Control Frame

[0057] like Figure 4 As shown, the PROBE control frame of this invention adopts a standard RoCE format encapsulation structure: the outermost layer is an Ethernet (L2 / MAC) frame header, which can optionally insert an 802.1Q VLAN tag to carry priority information; the upper layer sequentially adds an IPv4 / IPv6 header and a UDP (RoCE v2 reserved port 4791) header to ensure interoperability with existing data center switching networks; then, the InfiniBand (IB) transmission header BTH and the datagram extension header DETH are attached to provide RDMA semantics such as QPN and PSN; the MAD header inserts a fixed 256-byte InfiniBand management datagram (MAD) field after BTH / DETH to reuse the CM class management identifier, carry control fields such as TransactionID, and reserve extension space, thereby ensuring protocol compatibility with the existing RoCE-v2 management stack. The core payload area carries the PROBEPayload defined in this invention, used to exchange control information such as the QPN / PSN and heartbeat sequence number of the faulty QP; finally, an ICRC check field is added to complete end-to-end bit error detection at the physical layer. This linear layered design is compatible with RoCE V2 MAD in frame format and can be implemented using existing RoCE v2 management paths.

[0058] In one embodiment, see Figure 2 As shown, the working process of the QP rapid refactoring module specifically includes: The network card hardware writes the PROBE packet to the GSI-QP pre-registered Recv Buffer via DMA and generates a CQE. The network card driver polls the buffer, and if wc->imm_data = 0xA1 (PROBE identifier), it enters the PROBE parsing process; otherwise, it is routed to the regular MAD processing.

[0059] The driver uses the SRC_QPN index carried in the PROBE frame to access the local qp_ctx_tbl and reads the context state and CRC checksum of the target service QP. If the QP is still in an ERR state and the context is not corrupted, the driver will trigger a reset within the FPGA in the order ERR→RESET→INIT→RTR→RTS, retaining the original QPN and adjusting the starting PSN to the requested value or a safe offset. Conversely, if the QP's state machine is not ERR or the context is invalid, the driver will immediately allocate a new QP' from the spare QP pool, assign it a brand new QPN' and a random starting PSN, and through "hot inheritance," allow the newly created QP' to directly use all the system resources already occupied by the old QP to ensure that subsequent data streams can correctly fall into the new context. In this way, the application layer does not need to be aware of the QPN change: the driver synchronously updates the qp_num field in the WQE when submitting the first restorative work request, and the receiving end starts receiving data with the new PSN after processing the first RESP frame; while the doorbell page and CQE path left by the old QP remain unchanged, avoiding the additional memory allocation and interrupt rebinding overhead caused by destruction and reconstruction. Through this "resource hot inheritance" mechanism, the new QP can take over the transmission task in a shorter time, achieving true plug-and-play fast takeover.

[0060] Once the new QPN / PSN is determined, the driver quickly encapsulates the RESP control frame in the control buffer pool. The RESP payload sets the opcode to 0xA2 and writes the old and new QPNs, the corresponding PSN, and the status code. After the buffer is ready, the driver submits the work request WQE to the GSI-QP via the MMIO Doorbell. The FPGA then pulls the SGE and automatically completes the MAD header, BTH / DETH, and ICRC concurrent packets. The payload of the RESP field is shown in Table 2 below: Table 2: Payload of the RESP field

[0061] See Figure 2 As shown, the working process of the QP rapid refactoring module will be explained below.

[0062] When the network interface card (NIC) receives a RESP control message from the peer through the management channel, the NIC hardware writes the RESP message to the GSI-QP pre-registration Recv Buffer via DMA and generates a CQE. The NIC driver polls; if wc->imm_data = 0xA2 (RESP identifier), it enters the RESP parsing process; otherwise, it is routed to regular MAD processing. The driver parses the RESP message, extracting the remote QPN and remote PSN negotiation parameters from the RESP frame. The QP information configuration unit receives the cached configuration data, writes the corresponding fields of QP_CONTEXT through the internal AXI-Lite bus, writes the updated QPN_REMOTE to the "remote QPN" field of the local context, and updates the local PSN initial value. After confirming that the parameter update is complete and the local QP can be safely reset, the QP state management unit sequentially drives the QP state machine to complete fast rebinding, switching the QP state from ERR to RESET, and then from RESET→INIT→RTR→RTS. After the QP state management unit completes writing the QP state, this unit automatically sends a write command to the doorbell register to notify the network card to update the QP context and start the service data stream. Subsequently, an interrupt or event is generated to inform the upper-layer driver that "rebinding is complete," and the driver can safely resume application data reading and writing.

[0063] This invention addresses the inherent shortcomings of existing RDMA reliable connection fault recovery mechanisms by proposing an innovative three-in-one architecture of "fast detection—control plane negotiation—local reconnection," achieving the following technical effects: 1. Ultra-low recovery latency to meet stringent real-time requirements: By injecting microsecond heartbeats at the network card hardware level and shortening the ACK timeout threshold, the fault detection time is compressed to the sub-millisecond level. Combined with the original QP context, the remote QPN and bidirectional PSN are updated locally, and the state machine is driven to switch in a pipelined manner (ERR→RESET→INIT→RTR→RTS). The actual measured latency of the end-to-end recovery process is less than 0.5 milliseconds, which is more than 10 times better than the traditional CM reconstruction scheme. It fully meets the mandatory requirement of <1ms recovery window in scenarios such as airborne avionics and industrial control.

[0064] 2. Zero resource redundancy, improving NIC resource utilization: No need to pre-build a hot standby QP, and it does not occupy scarce hardware resources such as QPN, CQ, and PD. At the same time, it supports the "resource hot inheritance" mechanism - when the peer creates a new QP, the MR / PD / CQ context associated with the original QP can be reused, avoiding repeated memory registration and permission configuration, and significantly reducing driver and kernel overhead.

[0065] 3. Completely transparent to upper-layer applications, seamlessly integrating with the existing ecosystem: The entire recovery process is completed collaboratively by the driver and network card. Upper-layer applications do not need to modify code or be aware of QP reconstruction. The original WQE can continue to be submitted after recovery. Business flow is uninterrupted and there is no intrusion of retransmission logic, which greatly simplifies the design complexity of high-availability systems.

[0066] 4. Strong protocol compatibility, supporting large-scale deployment: Control frames (PROBE / RESP / ACK) are transmitted based on the standard GSI-QP (Global Shared Interface QP), and the encapsulation format strictly follows the RoCE v2 protocol stack (Ethernet / IP / UDP +IB BTH / DETH / MAD). They can run seamlessly on existing commercial switches and smart network cards without requiring network-side modifications.

[0067] 5. High robustness and adaptability: It supports two modes: reusing or creating a new peer QP. If the original QP context is intact, it is directly reset and reused; if it is damaged, a new QP is dynamically allocated and its parameters are fed back. The local end does not need to distinguish the behavior of the peer, realizing an intelligent recovery strategy of "one-time negotiation and automatic adaptation".

[0068] 6. Applicable to key application scenarios such as avionics, financial transactions, AI / HPC clusters, and cloud-native databases. In avionics, it can meet the DO-254 / DO-178C certification requirements for high availability of communication links; in financial transactions, it can avoid order loss or delay penalties due to network jitter; for AI / HPC clusters, it can improve the fault tolerance efficiency of collective communication operations such as AllReduce; for cloud-native databases, it can ensure strong consistency and low-tail latency of distributed transactions.

[0069] In summary, the method of this invention not only solves the core pain point of RDMA reliable connection in fault recovery, but also achieves the optimal balance between resource efficiency, protocol compatibility and application transparency, providing key technical support for building next-generation highly deterministic and highly available network infrastructure.

[0070] In this embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-described methods for rapid recovery from RDMA reliable connection failures.

[0071] Specifically, the computer device can be a computer terminal, a server, or a similar computing device.

[0072] In this embodiment, a computer-readable storage medium is provided, which stores a computer program that performs any of the above-described methods for fast recovery from RDMA reliable connection failures.

[0073] Specifically, computer-readable storage media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable storage media does not include transient media, such as modulated data signals and carrier waves.

[0074] Obviously, those skilled in the art should understand that the modules or steps of the above-described embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.

[0075] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for rapid recovery from reliable RDMA connection failures, characterized in that, The method is executed by a first communication terminal, which has established communication with a second communication terminal through a queue pair QP in a reliable connection mode. The queue pair QP is used to carry service data transmission, including: After detecting that the queue pair QP has entered an error state due to a link failure or node failure, the queue pair QP and its associated protection domain, memory registration area and completion queue resources are retained, and the queue pair QP is not destroyed. Send a parameter request to the second communication terminal and receive the parameter response returned by it to obtain the connection restoration parameters fed back by the second communication terminal; Based on the connection recovery parameters, the context data structure of the queue pair is updated, and the state machine of the queue pair is driven to switch from the error state to the ready-to-send state to restore the data transmission capability.

2. The method for rapid recovery from reliable RDMA connection failures according to claim 1, characterized in that, The first communication terminal automatically injects heartbeat frames into the queue pair QP through network card hardware or host driver, and sets the queue pair QP to an error state if it fails to receive an acknowledgment response multiple times in a row.

3. The method for rapid recovery from reliable RDMA connection failures according to claim 2, characterized in that, The heartbeat frame is a zero-byte RDMA write operation with an immediate value; wherein, the network card hardware or the host driver configures the acknowledgment timeout threshold and the maximum number of retransmissions for the queue pair QP, and when the number of consecutively sent heartbeat frames that have not received an acknowledgment response within the acknowledgment timeout threshold reaches the maximum number of retransmissions, the operation of setting the queue pair QP to an error state is triggered.

4. The method for rapid recovery from reliable RDMA connection failures according to claim 1, characterized in that, Send a parameter request to the second communication terminal and receive its returned parameter response to obtain the connection restoration parameters fed back by the second communication terminal, including: The host driver of the first communication terminal sends a PROBE control frame to the second communication terminal through the control plane; After receiving the PROBE control frame, the second communication terminal returns the RESP control frame; The host driver of the first communication terminal parses the RESP control frame and extracts the connection recovery parameters, which include the remote queue pair QP number and the peer start packet sequence number.

5. The method for rapid recovery from reliable RDMA connection failures according to claim 4, characterized in that, The control plane is implemented based on a global service interface queue pair. Both the PROBE control frame and the RESP control frame are transmitted through the global service interface queue pair, and the encapsulation format is compatible with the RoCE v2 protocol stack.

6. The method for rapid recovery from reliable RDMA connection failures according to claim 5, characterized in that, The encapsulation structure of both the PROBE control frame and the RESP control frame includes: an outer layer consisting of an Ethernet header, an IP header, and a UDP header, and an inner layer consisting of an InfiniBand transport header, a datagram extension header, and a management datagram header.

7. The method for rapid recovery from reliable RDMA connection failures according to claim 4, characterized in that, The remote queue pair QP number depends on the processing result of the second communication end on its local queue pair QP state, including: When the second communication terminal reuses its local queue pair QP, the RESP control frame returns the number of the local queue pair QP. When the second communication terminal cannot reuse its local queue pair QP, it allocates a new queue pair QP and associates the new queue pair QP with the protection domain, memory registration area and completion queue resources associated with the original queue pair in order to reuse the protection domain, memory registration area and completion queue resources, and returns the number of the new queue pair QP in the RESP control frame. The first communication terminal updates the context of its local queue pair QP only based on the remote queue pair QP number, without needing to know whether the second communication terminal reused the original queue pair QP or allocated a new queue pair QP.

8. The method for rapid recovery from reliable RDMA connection failures according to claim 1, characterized in that, During the process of driving the queue to switch the state machine of QP from the error state to the ready to send state, the first communication terminal does not call the QP destruction and reconstruction interface at all, and the state switch is a single event; the second communication terminal reuses the original QP or creates a new QP'.

9. A system for rapid recovery from reliable RDMA connection failures, characterized in that, include: The fault detection module is used to retain the queue pair QP and its associated protection domain, memory registration area and completion queue resources after detecting that the queue pair QP has entered an error state due to link failure or node failure, without destroying the queue pair QP. The management channel module is used to send parameter requests to the second communication terminal and receive the parameter responses returned by it, so as to obtain the connection recovery parameters fed back by the second communication terminal; The QP fast reconstruction module is used to update the context data structure of the queue pair based on the connection recovery parameters, and drive the state machine of the queue pair to switch from the error state to the ready-to-send state in order to restore data transmission capability.

Images