Network interface card, storage device, packet receiving method and transmitting method

Abstract

The application discloses a network interface card, a storage device, a message receiving method and a message sending method, relates to the storage field, and is used for improving the reliability of a storage device using an RDMA network and improving the storage access performance of a single QP connection. The network interface card comprises a parser and at least one shared connector. The parser is used for distributing a received first remote direct memory access (RDMA) message to a first shared connector in the at least one shared connector. The first shared connector is used for establishing a QP connection with a peer network device, and determining a first queue pair (QP) corresponding to a first controller according to the first RDMA message to realize communication with the first controller. The first shared connector is connected with a plurality of controllers including the first controller, and one QP corresponding to each controller in the plurality of controllers corresponds to the first shared connector.

Description

Technical Field

[0001] This application relates to the field of storage, and more particularly to a network interface card, a storage device, a message receiving method, and a message sending method. Background Technology

[0002] Currently, much data is not stored locally, but rather on remote storage devices. For example... Figure 1 As shown in the illustration, this application provides a storage system including a storage device 11 and host clients 12. The storage device 11 is connected to the host clients 12 via a front-end fabric and can provide data storage services to multiple host clients 12. With the rise of solid-state drives (SSDs) based on the non-volatile memory express (NVMe) specification, more and more front-end fabrics are adopting NVMe overfabric (NoF) networks based on switched networks. Remote direct memory access (RDMA) is a standard in NoF networks that can reduce the latency of data access by the storage device.

[0003] like Figure 1 As shown, the storage device 11 includes multiple controllers 111 and at least one network interface card (NIC) 112. When the front-end network uses an RDMA network, the host client can establish a one-to-one queue pair (QP) connection with one controller 111. This method has low reliability; if the corresponding controller fails, the host client will be unable to access the storage device. Moreover, this method has low performance, as each host client's access request can only be processed by one controller, wasting the processing power of other controllers. The host client can also establish one-to-one QP connections with multiple controllers separately, but this method also has low reliability because each controller accesses a specific storage space. If one or more corresponding controllers fail, the host client can only access a portion of the storage space, and the host client needs to detect the fault and perform fault recovery processing. Summary of the Invention

[0004] This application provides a network interface card, a storage device, a message receiving method, and a message sending method to improve the reliability of storage devices using RDMA networks and to improve the storage access performance of a single QP connection.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, a network interface card is provided, including a parser and at least one shared connector. The parser is used to distribute a received first remote direct memory access (RDMA) message to a first shared connector among the at least one shared connector. The first shared connector is used to establish a QP connection with a peer network device and determine a first queue pair (QP) corresponding to a first controller based on the first RDMA message to enable communication with the first controller. The first shared connector is connected to multiple controllers, including the first controller, and each controller has a QP corresponding to a first shared connector.

[0007] The network interface card provided in this application includes a parser and at least one shared connector. The parser distributes received first Remote Direct Memory Access (RDMA) messages to the first shared connector among the at least one shared connector. The first shared connector establishes a QP connection with the peer network device and determines a first queue pair of QPs corresponding to the first controller based on the first RDMA message to enable communication with the first controller. The first shared connector connects to multiple controllers, including the first controller, and each controller has a QP corresponding to the first shared connector. For the host client, only the shared connectors of the network interface card on the storage device side are visible. However, the first shared connector can distribute received RDMA messages from the host client to different controllers containing different QPs for processing. Therefore, it can shield controllers that are not functioning properly. As long as one controller is functioning properly, the storage device can provide storage services, thus improving the reliability of storage devices using RDMA networks. Furthermore, by distributing IO command requests to controllers containing different QPs for parallel processing, compared to existing solutions where RDMA messages for the same QP connection can only be processed by a fixed controller, its performance can be improved by N times, where N is the number of controllers. Therefore, it can improve the storage access performance of a single QP connection.

[0008] In one possible implementation, the first shared connector is further configured to: if the first RDMA message is not an acknowledgment message, store the first RDMA message in the storage space corresponding to the first QP. This implementation achieves the storage of RDMA messages.

[0009] In one possible implementation, the first shared connector is further configured to: if the first RDMA message is a tail packet, a single packet, or an acknowledgment message, submit a completion queue entry CQE to the completion queue CQ associated with the first QP, whereby the CQE indicates that the RDMA operation corresponding to the first RDMA message has been completed. This implementation indicates that the RDMA operation has been completed.

[0010] In one possible implementation, the first shared connector is specifically configured to: if the first RDMA message is the first packet or a single packet of a SEND message, determine the first QP based on information in the Non-Volatile Memory Specification (NVME) Submission Queue Entry (SQE) in the first RDMA message; wherein the information in the NVME SQE corresponds to the first controller. This implementation illustrates how received RDMA messages are mapped to QPs.

[0011] In one possible implementation, the first shared connector is specifically configured to: if the first RDMA message is an acknowledgment message of an RDMAWRITE message, an acknowledgment message of a SEND message, or the first packet or a single packet of an RDMA READ response message, determine the first QP based on the correspondence between the message sequence number of the first RDMA message and the message sequence number of the second RDMA message and the first QP; wherein, the second RDMA message is an RDMA message sent by the first QP; the message sequence number of the first RDMA message corresponds to the message sequence number of the second RDMA message; when the first RDMA message is an acknowledgment message of an RDMA WRITE message, the second RDMA message belongs to the RDMA WRITE message; when the first RDMA message is an acknowledgment message of a SEND message, the second RDMA message belongs to the SEND message; when the first RDMA message is the first packet or a single packet of an RDMA READ response message, the second RDMA message belongs to the RDMA READ request message. This implementation illustrates how to map received RDMA messages to QPs.

[0012] In one possible implementation, the first shared connector is further configured to: if the first RDMA message is the first packet, store the correspondence between the first packet of the message to which the first RDMA message belongs and the first QP. After storing the correspondence between the first packet and the first QP, subsequent received intermediate packets or tail packets belonging to the same RDMA message as the first packet can also be mapped to the first QP, so that the first packet, intermediate packet and tail packet of the same RDMA message are processed by the same QP.

[0013] In one possible implementation, the first shared connector is further configured to: if the first RDMA message is a tail packet or an intermediate packet, determine the first packet corresponding to the first RDMA message based on the ordered incrementality of the message sequence numbers of the segmented RDMA messages and the message sequence number of the first RDMA message; and determine the first QP based on the first packet corresponding to the first RDMA message and the corresponding relationship. This implementation enables the first packet, intermediate packet, and tail packet of the same RDMA message to be processed by the same QP.

[0014] Secondly, a network interface card is provided, including a scheduler and at least one shared connector. The scheduler is used to select a second QP with messages to be sent from multiple queue pairs QPs of multiple controllers in place, corresponding to a second shared connector. The second shared connector is one of at least one shared connector. The second shared connector is used to establish a QP connection with the peer network device and send messages in the storage space corresponding to the second QP to the host client in the form of a second remote direct memory access (RDMA) message. The second shared connector is connected to multiple controllers respectively, and each controller has a QP corresponding to a second shared connector.

[0015] The network interface card and storage device provided in this application embodiment include a scheduler and at least one shared connector. The scheduler selects a second QP with messages to be sent from multiple QPs of multiple controllers in place, corresponding to a second shared connector. The second shared connector is one of the at least one shared connector. The shared connector establishes a QP connection with the peer network device and sends messages in the storage space corresponding to the second QP to the host client in the form of a second RDMA message. The second shared connector is connected to multiple controllers respectively, and each controller has a corresponding QP and a second shared connector. The scheduler can select one QP from multiple QPs, and the corresponding second shared connector processes its RDMA messages to be sent, so that storage services can be provided as long as one controller is working properly, thus improving the reliability of storage devices using RDMA networks. Furthermore, controllers containing different QPs can process different IO command requests in parallel. Compared to existing solutions where RDMA messages of the same QP connection can only be processed by a fixed controller, the performance can be improved by N times, where N is the number of controllers, thus improving the storage access performance of a single QP connection.

[0016] In one possible implementation, the second shared connector is further configured to: if the second RDMA message belongs to an RDMAWRITE message, a SEND message, or an RDMA READ request message, then save the correspondence between the message sequence number in the second RDMA message and the second QP. After storing these correspondences between command messages and the second QP, subsequent data messages corresponding to the command message can also be mapped to the second QP, so that the command message and the corresponding data message are processed by the same QP.

[0017] In one possible implementation, the scheduler is specifically configured to: select a second QP from a plurality of QPs corresponding to the second shared connector via round-robin scheduling. This implementation determines how the second QP is selected.

[0018] Thirdly, a message receiving method is provided, applied to a network interface card as described in the first aspect and any embodiment thereof. The method includes: a parser distributing a received first remote direct memory access (RDMA) message to a first shared connector among at least one shared connector; the first shared connector establishing a QP connection with a peer network device and determining a first queue pair (QP) corresponding to a first controller based on the first RDMA message to enable communication with the first controller, wherein the first shared connector is connected to multiple controllers, including the first controller, and each controller has a QP corresponding to a first shared connector.

[0019] In one possible implementation, the method further includes: if the first RDMA message is not an acknowledgment message, the first shared connector stores the first RDMA message in the storage space corresponding to the first QP.

[0020] In one possible implementation, the method further includes: if the first RDMA message is a tail packet, a single packet, or an acknowledgment message, the first shared connector submits a completion queue entry CQE to the completion queue CQ associated with the first QP, whereby the CQE indicates that the RDMA operation corresponding to the first RDMA message has been completed.

[0021] In one possible implementation, determining the first queue pair (QP) corresponding to the first controller based on the first RDMA message includes: if the first RDMA message is the first packet or a single packet of a SEND message, the first shared connector determines the first QP based on the information in the Non-Volatile Memory Specification (NVME) Submission Queue Entry (SQE) in the first RDMA message; wherein the information in the NVME SQE corresponds to the first controller.

[0022] In one possible implementation, determining the first queue pair (QP) corresponding to the first controller based on the first RDMA message includes: if the first RDMA message is an acknowledgment message of an RDMA WRITE message, an acknowledgment message of a SEND message, or the first packet or a single packet of an RDMA READ response message, then the first shared connector determines the first QP based on the correspondence between the message sequence number of the first RDMA message and the message sequence number of the second RDMA message and the first QP; wherein, the second RDMA message is an RDMA message sent by the first QP; the message sequence number of the first RDMA message corresponds to the message sequence number of the second RDMA message; when the first RDMA message is an acknowledgment message of an RDMA WRITE message, the second RDMA message belongs to the RDMA WRITE message; when the first RDMA message is an acknowledgment message of a SEND message, the second RDMA message belongs to the SEND message; when the first RDMA message is the first packet or a single packet of an RDMAREAD response message, the second RDMA message belongs to the RDMA READ request message.

[0023] In one possible implementation, the method further includes: if the first RDMA message is the first packet, the first shared connector stores the correspondence between the first packet of the message to which the first RDMA message belongs and the first QP.

[0024] In one possible implementation, the method further includes: if the first RDMA message is a tail packet or an intermediate packet, the first shared connector determines the first packet corresponding to the first RDMA message based on the ordered incrementality of the message sequence number of the segmented RDMA message and the message sequence number of the first RDMA message; and determines the first QP based on the first packet corresponding to the first RDMA message and the corresponding relationship.

[0025] Fourthly, a message sending method is provided, applied to a network interface card as described in the second aspect and any embodiment thereof. The method includes: a scheduler selecting a second QP with messages to be sent from multiple queue pairs (QPs) of multiple controllers in place, the second shared connector being one of at least one shared connector; the second shared connector establishing a QP connection with a peer network device and sending messages in the storage space corresponding to the second QP to a host client in the form of a second remote direct memory access (RDMA) message; wherein the second shared connector is connected to multiple controllers respectively, and each controller in the multiple controllers corresponds to a QP with the second shared connector.

[0026] In one possible implementation, the method further includes: if the second RDMA message belongs to an RDMA WRITE message, a SEND message, or an RDMA READ request message, the second shared connector stores the correspondence between the message sequence number in the second RDMA message and the second QP.

[0027] In one possible implementation, the scheduler selects a second QP with messages to be sent from multiple queue pairs QPs of multiple in-place controllers, which corresponds to the second shared connector, including: the scheduler selects the second QP from the multiple QPs corresponding to the second shared connector through round-robin scheduling.

[0028] Fifthly, a storage device is provided, including a network interface card as described in the first aspect and any embodiment thereof, or as described in the second aspect and any embodiment thereof, and a plurality of controllers, wherein the network interface card is connected to each of the plurality of controllers respectively.

[0029] Regarding the technical effects of the third to fifth aspects, refer to the technical effects of the first and second aspects. Attached Figure Description

[0030] Figure 1 A schematic diagram of the architecture of a storage system provided in an embodiment of this application;

[0031] Figure 2 A schematic diagram of a QP provided in an embodiment of this application;

[0032] Figure 3 A schematic diagram of a reliable QP connection provided in an embodiment of this application;

[0033] Figure 4 A schematic diagram illustrating the order-preserving incrementing PSN of an RDMA message, provided for an embodiment of this application;

[0034] Figure 5A A schematic diagram illustrating the structure of a SEND message and its acknowledgment message based on the IB protocol, provided for an embodiment of this application;

[0035] Figure 5B A schematic diagram illustrating the structure of a SEND message and its acknowledgment message based on the RoCEv2 protocol, provided for embodiments of this application;

[0036] Figure 6 A schematic diagram illustrating the interaction process between a requesting end and a responding end in a reliable connection, provided in an embodiment of this application;

[0037] Figure 7A A schematic diagram illustrating the structure of an RDMAWRITE message and its acknowledgment message based on the IB protocol, provided for embodiments of this application;

[0038] Figure 7B A schematic diagram illustrating the structure of an RDMAWRITE message and an acknowledgment message based on the RoCEv2 protocol, provided for embodiments of this application;

[0039] Figure 8 A schematic diagram illustrating the interaction process between the requesting and responding ends of another reliable connection provided in this application embodiment;

[0040] Figure 9A A schematic diagram illustrating the structure of an RDMAREAD request message and an RDMAREAD response message based on the IB protocol, provided in an embodiment of this application;

[0041] Figure 9B A schematic diagram illustrating the structure of an RDMAREAD request message and an RDMAREAD response message based on the RoCEv2 protocol, provided in an embodiment of this application;

[0042] Figure 10 A schematic diagram illustrating the interaction process between the requesting and responding ends of another reliable connection provided in this application embodiment;

[0043] Figure 11This is a schematic diagram illustrating the process of a host client and a storage device performing I / O read / write operations using RDMA, as provided in an embodiment of this application.

[0044] Figure 12 This is a schematic diagram of another storage system architecture provided in an embodiment of this application;

[0045] Figure 13 This is a schematic diagram of the architecture of another storage system provided in an embodiment of this application;

[0046] Figure 14 A flowchart illustrating a message receiving method provided in an embodiment of this application;

[0047] Figure 15 A flowchart illustrating another message receiving method provided in an embodiment of this application;

[0048] Figure 16 A flowchart illustrating a message sending method provided in an embodiment of this application;

[0049] Figure 17 A schematic diagram illustrating scheduling based on the RR algorithm provided in an embodiment of this application;

[0050] Figure 18 This is a schematic diagram illustrating another process by which a host client and a storage device utilize RDMA operations to complete I / O read / write operations, as provided in an embodiment of this application. Detailed Implementation

[0051] As used herein, the terms “component,” “module,” “system,” etc., are intended to refer to a computer-related entity, which may be hardware, firmware, a combination of hardware and software, software, or running software. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, an executable file, a running thread, a program, and / or a computer. As an example, an application running on a computing device and the computing device itself can both be components. One or more components may reside in a running process and / or thread, and components may be located in a single computer and / or distributed among two or more computers. Furthermore, these components are capable of execution from various computer-readable media having various data structures thereon. These components may communicate locally and / or remotely via signals, such as based on one or more data packets (e.g., data from a component that interacts with a local system, another component in a distributed system, and / or signals that interact with other systems via a network such as the Internet).

[0052] First, let me describe the concepts involved in this application:

[0053] The storage device involved in this application may be referred to as a storage server, storage array, blade server, etc., and the name is not limited. The storage device may include at least one controller and at least one network interface card (NIC). The controller may include, but is not limited to, a processor and memory, and the processor runs an operating system and applications.

[0054] The processor involved in the embodiments of this application can be a chip. For example, it can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0055] The memory involved in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0056] The NIC involved in the embodiments of this application can also be called a network interface controller, network adapter, network card, or local area network adapter. It is a piece of computer hardware designed to allow computers to communicate on a computer network.

[0057] To improve system reliability, storage devices often employ a multi-controller architecture, supporting multiple controllers to provide data storage services to one or more host clients. Mid-to-high-end storage devices typically use a multi-host NIC to implement the front-end network interface. Multiple controllers share a single NIC through different high-speed serial computer extended bus (PCIe) channels, with each controller acting as an independent network node.

[0058] Storage devices connect to host clients via a front-end network, providing data storage services to multiple host clients. With the rise of NVMe SSDs, the protocol for host clients to access storage devices has transitioned from Small Computer System Interface (SCSI) to NVMe. Correspondingly, the front-end network has also migrated from Fibre Channel (FC) networks and Transmission Control Protocol / Internet Protocol (TCP / IP) networks to NoF networks.

[0059] The NoF protocol supports both memory and message type operations. Specifically, PCIe supports memory type operations, FC supports message type operations, and RDMA supports both memory and message type operations.

[0060] RDMA can transmit data from the storage space of the storage device as messages to the storage space of the host client via the front-end network in the form of RDMA messages, or vice versa. These transmission and copy operations are primarily performed by the network interface cards (e.g., RDMA network interface cards, RNICs) of the storage device and the host client, and do not require processor execution, thus not affecting the performance of the host client.

[0061] Currently, hardware implementations of RDMA include Infiniband (IB), RDMA over converged Ethernet (RoCE) (including RoCEv1 and RoCEv2), and the Internet Wide Area RDMA protocol (iWARP). Infiniband is a network specifically designed for RDMA, guaranteeing reliable transmission at the hardware level. RoCE and iWARP are both Ethernet-based RDMA technologies, supporting corresponding verb interfaces; RoCE also supports hardware-level guarantees of reliable transmission.

[0062] like Figure 2As shown, RDMA is a transport layer network protocol that uses queue pairs (QPs) in the channel adapter (CA) to send and receive data (as messages) from the consumer. Each QP contains a send queue (SQ) and a receive queue (RQ), where the SQ is responsible for sending messages and the RQ is responsible for receiving messages. Each QP's SQ and RQ can be associated with a completion queue (CQ). Each QP has its own connection context to maintain its connection state, and the SQ, RQ, and CQ also have their own queue contexts to maintain their queue usage state.

[0063] In this application's embodiments, the consumer refers to the entity that uses QP to send and receive messages, generally referring to a software process, such as NoF protocol software.

[0064] When sending a message, the consumer posts a work queue element (WQE) to the SQ in the QP of this node (acting as the requester). This WQE includes the RDMA operation type, the address of the storage space pointing to the message to be transmitted, and the size of the message to be transmitted. The data to be transmitted is treated as a message, with a length supporting 0 bytes to 4GB. The network interface card segments the data pointed to by the WQE into packets according to the path maximum transmission unit (PMTU) and sends them to the network. When the network interface card has completed transmitting all the contents of the message pointed to by a WQE, it posts a completion queue element (CQE) to the CQ associated with the SQ to notify the consumer that the RDMA operation indicated by the posted WQE has been completed.

[0065] Before receiving a SEND message, the consumer submits a WQE to the RQ in the QP (responder) of this node. This WQE includes a pointer to the storage space capable of storing data and the size of that storage space. After receiving data from the front-end network, the network interface card writes it to the storage space pointed to by the WQE of the RQ. Once the network interface card has finished receiving data, it submits a CQE to the CQ associated with the RQ to notify the consumer that it has completed receiving an RDMA message. For receiving an RDMA READ response message, the network interface card writes the received data to the storage space pointed to by the WQE of the corresponding RDMA READ operation. Furthermore, for the tail packet or single packet of the RDMA READ response message, once the network interface card has finished receiving data, it submits a CQE to the CQ associated with the SQ to notify the consumer that it has completed the RDMA READ operation indicated by a specific WQE it submitted. Upon receiving an acknowledgment message for an RDMA WRITE or RDMA SEND message, the network interface card submits a CQE to the CQ associated with the SQ to notify the consumer that the RDMA WRITE or RDMA SEND operation indicated by the submitted WQE has been completed. Acknowledgment messages for SEND, RDMA READ response, and RDMA WRITE messages are described later.

[0066] RDMA supports both reliable and unreliable QP connections. Reliable QP connections only support a one-to-one communication model, meaning a QP on one node can only establish a QP connection with one QP on other nodes; no QP can simultaneously establish QP connections with multiple QPs on other nodes. For example, ... Figure 3 As shown, consumer A's QPx in node 1 establishes a QP connection with consumer B's QP a in node 2. Consumer A's QPx in node 1 cannot simultaneously establish QP connections with other QPs (e.g., QP b, QP c). NoF uses RDMA that supports reliable QP connections for communication.

[0067] In RDMA, messages received from a reliable QP connection must adhere to strict ordering. The responding end determines whether to accept a message based on whether the packet sequence numbers (PSNs) of the received messages are strictly ordered and incremented. For example... Figure 4 As shown in Figure A, the responding end will only accept a message if the message's PSN is strictly incremented (e.g., the message's PSN is successively equal to n, n+1, n+2, n+3). Figure 4 As shown in Figure B, when the PSN of a message changes (for example, the PSN of a message is missing n+2), the responding end will discard all subsequent messages with PSNs that are not strictly incrementing for that QP. Figure 4 As shown in C, when the PSN of a message is the same as the PSN of a received message (for example, a message with PSN n+1 is received again), the responding end will discard the duplicate message.

[0068] RDMA supports message types including the SEND operation, which involves the processor and is applicable to Transmission Control messages (or command messages).

[0069] RDMA supports memory operations including RDMA read and RDMA write operations. These operations do not involve the processor; data reading or writing is accomplished through the network interface card's operation of the storage space, making them suitable for transmitting data packets (or data messages).

[0070] The following sections describe the transmit operation, RDMA read operation, and RDMA write operation respectively:

[0071] Send operation

[0072] Sending operations can also be called push operations or having channel semantics. The requester pushes data to the responder without needing to know where the data is stored in the responder's memory. The responder's channel adapter places the data into the next available memory space in its QP, indicated by the WQE in the header of the QP's RQ.

[0073] The requesting end can initiate a sending operation by sending a SEND message, which includes the data to be pushed to the responding end. The data length can be 0 bytes to 2 bytes. 31Between (4G) bytes, if the data length exceeds the path maximum transmission unit (PMTU), it will be segmented into multiple messages according to the PMTU size. The responding end then reassembles these messages to obtain the data. For a reliable connection, if the data is a short message (i.e., it does not need to be segmented into multiple messages), the responding end sends an acknowledgment (ACK) message to the requesting end for each message. If the data is a long message (i.e., segmented into multiple messages), the responding end can send an acknowledgment (ACK) message to the requesting end for each message, or send an acknowledgment message to the requesting end for multiple consecutive messages of the same message, or send an acknowledgment message to the requesting end for the last packet of the message. In addition, regardless of whether the data is a short message or a long message, the responding end can send an acknowledgment (ACK) message for multiple previously received messages. For example, an acknowledgment message for a SEND message with PSN X can be used to confirm that messages with PSNs less than X preceding this SEND message have been successfully received by the responding end.

[0074] Each SEND message may include a four-byte immediate data (ImmDt). If an immediate data is to be included, it is included in the additional header field of the last segment of the SEND message (either the Immediate Extended Transport Header or the ImmDt field).

[0075] For example, assuming the SEND message is segmented into three messages, the structure of the SEND message and its acknowledgment message based on the IB protocol is as follows: Figure 5A As shown, the structure of the SEND message and its acknowledgment message based on the RoCEv2 protocol is as follows: Figure 5B As shown, the interaction process between the requesting and responding ends of a reliable connection is as follows: Figure 6 As shown.

[0076] The local route header (LRH) field is used for routing through switches in the subnet.

[0077] The global route header (GRH) field is used to route to a destination outside the sender's local subnet.

[0078] The invariant CRC (ICRC) field covers all fields that should not be changed in the message and performs CRC checks on these fields.

[0079] The variable CRC (VCRC) field covers all fields of the message and performs CRC checks on these fields.

[0080] The ACK extended transport header (AETH) includes additional transport fields for the acknowledgment message, which are included in each acknowledgment message of the SEND message.

[0081] The Media Access Control (MAC) field is used for Layer 2 forwarding through switches in an Ethernet subnet.

[0082] The Internet Protocol (IP) field is used for Layer 3 forwarding through switches.

[0083] The User Datagram Protocol (UDP) field is used to indicate that the payload of the message is an RDMA message.

[0084] The base transport header (BTH) field indicates the destination QP, operation code, packet sequence numbers (PSN), and partition. The opCode field within the BTH field determines the start and end of the SEND message.

[0085] For SEND messages, if the data length is less than or equal to PMTU, the BTH opcode "SENDOnly" or "SEND Only with Immediate" is used to indicate that the message cannot be split (i.e., it is a single packet).

[0086] If the data length is zero, the BTH opcode "SEND Only" or "SEND Only with Immediate" is used, and the message does not contain a data payload field. For a SEND message with the BTH opcode "SEND Only", other fields are as follows: Figure 5A or Figure 5B As shown in message 1; for a SEND message with BTH opcode "SEND Only with Immediate", other fields are as follows: Figure 5A or Figure 5B The message is shown in message 3.

[0087] If the length of the data is greater than the PMTU, the BTH opcode of the first message is "SEND First", indicating that the message is the first packet; the BTH opcode of the middle messages is "SEND Middle", indicating that the message is the middle packet; and the BTH opcode of the last message is "SEND Last" or "SEND Last with Immediate", indicating that the message is the last packet.

[0088] For the acknowledgment message of the SEND message, the BTH opcode is "Acknowledge".

[0089] For a given requesting QP, once a multi-packet send operation is initiated, no other SEND messages, RDMA WRITE messages, or RDMA READ request messages can be sent before the message with the opcode "SEND Last" or "SEND Last with Immediate".

[0090] The responding end does not know the final length of the data until the last message with the "SEND Last" or "SEND Last with Immediate" opcode arrives.

[0091] It should be noted that after the responding end completes the reception of the single packet or tail packet of the SEND message, it reports the CQE to the CQ associated with the RQ of this node. After the requesting end completes the sending process of the sending operation (receiving the acknowledgment message corresponding to the single packet or tail packet of the SEND message), it submits the CQE to the CQ associated with the SQ of this node.

[0092] RDMA write operation

[0093] RDMA write operations are used by the requesting end to write data to the storage space of the responding end.

[0094] Before allowing the requesting end to perform an RDMA write operation, the responding end first allocates a memory space for access by its QP (or QP group). The responding end's channel adapter associates a 32-bit M_Key key with the virtual address of this memory space. The responding end sends the virtual address, length, and M_Key key of the memory space to the requesting end that can access the memory region. For example, this information can be sent to the requesting end using the send operation described above.

[0095] The requesting end can initiate an RDMA write operation by sending an RDMA WRITE message, which includes the data to be written to the responding end, the virtual address of the responding end's memory space, the length of the data, and the M_Key key. The data length can be 0 bytes to 2 bytes. 31Similar to the sending operation, if the data length is greater than the PMTU, it will be segmented into multiple messages according to the PMTU size. The responding end then reassembles these messages to obtain the data. For reliable connections, if the data is a short message (i.e., it does not need to be segmented into multiple messages), the responding end sends an acknowledgment (ACK) message to the requesting end for each message. If the data is a long message (i.e., segmented into multiple messages), the responding end can send an acknowledgment (ACK) message to the requesting end for each message, or send an acknowledgment message to the requesting end for multiple consecutive messages of the same data, or send an acknowledgment message to the requesting end for the end packet of the message. In addition, regardless of whether the data is a short message or a long message, the responding end can send an acknowledgment (ACK) message for multiple previously received messages. For example, an acknowledgment message for an RDMA WRITE message with PSN X can be used to confirm that messages with PSNs less than X preceding this RDMA WRITE message have been successfully received by the responding end.

[0096] For example, assuming the RDMA WRITE message is segmented into three packets, the structure of the RDMA WRITE message and its acknowledgment packet based on the IB protocol is as follows: Figure 7A As shown, the structure of the RDMA WRITE message and the acknowledgment message of the RDMA WRITE message based on the RoCEv2 protocol is as follows: Figure 7B As shown. The interaction process between the requesting and responding ends of a reliable connection is as follows: Figure 8 As shown.

[0097] The RDMA extended transport header (RETH) includes additional transport fields for the RDMA operation. These include, for example, the virtual address of the target storage space, the length of the data for the RDMA operation, and the M_Key key. The responding end determines, based on the M_Key, that the requesting end has permission to access the corresponding virtual address and stores the data indicated by that length in the storage space pointed to by the virtual address.

[0098] Similar to the SEND operation, the OpCode field in the BTH field determines the start and end of the RDMA WRITE message:

[0099] For RDMA WRITE messages, if the data length is less than or equal to PMTU, the BTH opcode "RDMA WRITE Only" or "RDMA WRITE Only with Immediate" is used to indicate that the message is not split (i.e., it is a single packet).

[0100] If the data length is zero, the BTH opcode "RDMA WRITE Only" or "RDMA WRITE Only with Immediate" is used, and the data payload field is not present in the message. For an RDMA WRITE message with the BTH opcode "RDMA WRITE Only", other fields are as follows: Figure 7A or Figure 7B As shown in message 1; for the RDMAWRITE message with BTH opcode "RDMA WRITE Only with Immediate", in such a case... Figure 7A or Figure 7B In message 1, the ImmDt field is added after the RETH field, and the other fields are as shown in message 1.

[0101] If the length of the data is greater than the PMTU, the BTH opcode of the first message is "RDMA WRITE first", indicating that the message is the first packet; the BTH opcode of the middle message is "RDMAWRITE Middle", indicating that the message is the middle packet; and the BTH opcode of the last message is "RDMA WRITE Last" or "RDMAWRITE Last with Immediate", indicating that the message is the last packet.

[0102] For the acknowledgment message of the RDMAWRITE message, the BTH opcode is "Acknowledge".

[0103] RDMA read operation

[0104] RDMA read operation is the reverse of RDMA write operation, used by the requesting end to read data from the storage space of the responding end.

[0105] Before allowing the requesting end to perform an RDMA read operation, the responding end first grants the requesting end access to its virtual address, that is, sends the virtual address, length, and M_Key key of the data to be read by the RDMA read operation to the requesting end. For example, this information can be sent to the requesting end using the sending operation described above.

[0106] The requesting end can initiate an RDMA read operation by sending an RDMA READ Request message, which includes the virtual address of the responding end's memory space, the length of the data to be read in the RDMA operation, and the M_Key key. The responding end sends an RDMA READ Response message to the requesting end, which includes the data to be read. The length of the data to be read can be 0 bytes to 2 bytes. 31If the length between bytes exceeds the PMTU, it will be segmented into multiple packets according to the PMTU size for transmission via multiple RDMAREAD response messages. The requesting end then reassembles these packets to obtain the data. For reliable connections, the RDMAREAD response message does not have a corresponding acknowledgment message. If the requesting end detects packet loss in the RDMA READ response message, it will re-initiate the RDMA read operation.

[0107] For example, assuming the RDMA READ response message is segmented into three messages, the structure of the RDMA READ request message and RDMA READ response message based on the IB protocol is as follows: Figure 9A As shown, the structure of the RDMA READ request message and RDMA READ response message based on the RoCEv2 protocol is as follows: Figure 9B As shown. The interaction process between the requesting and responding ends of a reliable connection is as follows: Figure 10 As shown.

[0108] The opcode field in the BTH field determines the start and end of the message (RDMA READ request message or RDMA READ response message):

[0109] For an RDMA READ request message, the BTH opcode is "RDMA READ Request".

[0110] For RDMA READ response messages, if the data length is less than or equal to PMTU, the BTH opcode "RDMA READ Response Only" is used, indicating that the message cannot be split (i.e., it is a single packet).

[0111] If the data length is zero, the BTH opcode "RDMA READ Response Only" is used, and the RDMA read response message does not contain a data payload field, but all other fields are as follows: Figure 9A or Figure 9B The message is shown in message 1.

[0112] If the length of the data is greater than the PMTU, the BTH opcode of the first message is "RDMA READ Response first", indicating that the message is the first packet; the BTH opcode of the middle message is "RDMA READ Response Middle", indicating that the message is the middle packet; and the BTH opcode of the last message is "RDMA READ Response Last", indicating that the message is the last packet.

[0113] The host client and storage device use the aforementioned RDMA operations to complete the IO read / write process as follows: Figure 11 As shown:

[0114] The host client first discovers the storage device on the network and establishes a connection with it.

[0115] During the I / O write process from the host client to the storage device, the host client sends a SEND message to the storage device. The data payload of this message is the I / O write command. For a reliable connection, the storage device can send an acknowledgment message to the host client with the SEND message, indicating that the I / O write command has been received. Then, the storage device (as the requesting end) sends an RDMA READ request message to the host client (as the responding end), and the host client sends an RDMA READ response message to the storage device. The data payload of this message is the I / O data. After the storage device persistently stores the I / O data, i.e., stores the I / O data on a hard disk drive (HDD) or solid-state drive (SSD) or other storage media, it sends a SEND message to the host client. This message includes the I / O response, indicating that the I / O write process is complete. The host client then sends an acknowledgment message to the storage device with the SEND message, indicating that the I / O response has been received.

[0116] During an I / O read operation from the host client to the storage device, the host client sends a SEND message to the storage device. The data payload of this message is the I / O read command. For a reliable connection, the storage device can send an acknowledgment message to the host client via the SEND message, indicating that the I / O read command has been received. After the storage device acquires the I / O data (i.e., after obtaining the I / O data from a storage medium such as an HDD or SSD), the storage device (as the requesting end) sends an RDMA WRITE message to the host client (as the responding end). The data payload of this message is the I / O data. The host client then sends an acknowledgment message to the storage device via the RDMA WRITE message. Finally, the storage device sends a SEND message to the host client, which includes the I / O response, indicating that the I / O read process is complete. The host client then sends an acknowledgment message to the storage device via the SEND message, indicating that the I / O response has been received.

[0117] This shows that a complete IO write process uses the SEND operation to transmit command messages (such as command requests and command responses) and the RDMAREAD operation to transmit data messages; a complete IO read process uses the SEND operation to transmit command messages and the RDMA WRITE operation to transmit data messages.

[0118] based on Figure 1 The storage system shown, such as Figure 12 As shown, the architecture for implementing the front-end network using RDMA between storage device 11 and host client 12 is as follows: Figure 12As shown.

[0119] Each controller 111 of the storage device 11 implements the NoF protocol software (also known as the NoF protocol stack) and driver. The host client 12 also implements the NoF protocol software and driver. The NoF protocol software and driver run in the processor. The NoF protocol software acts as an application program, and the driver provides the NoF protocol software with an interface to control the network interface card.

[0120] Each controller 111 of the storage device 11 corresponds to at least one QP as a node, and multiple controllers 111 share a first network interface card (e.g., RNIC) 112. The host client 12 includes a second network interface card (e.g., RNIC) 121, which can establish a reliable connection with the QP of one controller 111 of the storage device 11 using one QP as needed, or can establish reliable connections with the QPs of multiple controllers 111 of the storage device 11 using multiple QPs respectively.

[0121] As mentioned earlier, this method has low reliability; if the corresponding controller fails, the host client will be unable to access the storage device. Furthermore, this method has low performance, as each host client's access request can only be processed by one controller, wasting the processing power of other controllers. The host client can also establish a one-to-many QP connection with multiple controllers, but this method also has low reliability because each controller accesses a specific storage space. If one or more corresponding controllers fail, the host client can only access a portion of the storage space. Moreover, this method has a small network scale because the number of QP connections is limited; the more QP connections each host client uses, the fewer host clients the storage device can support simultaneously.

[0122] Therefore, such as Figure 13 As shown in the embodiments of this application, another network interface card, storage device, and storage system are provided.

[0123] The storage device 11 includes a plurality of controllers 111 and at least one network interface card (e.g., RNIC) 112.

[0124] The network interface card 112 includes a parser 1121, a scheduler 1122, at least one shared connector 1123, and a QP cache 1124. The parser 1121, scheduler 1122, and at least one shared connector 1123 can be application-specific integrated circuit (ASIC) chip modules, firmware burned into a programmable logic device (PLD), or software modules running on a processor, microprocessor, or embedded processor; this application does not limit the scope of these components.

[0125] At least one QP (also referred to as an SQ / RQ pair, the name is not limited) and a CQ associated with at least one QP are established on each controller 111. A shared connector 1123 is connected to each of the multiple controllers 111, and each controller 111 corresponds to one QP with the shared connector 1123. The network interface card 112 operates on the QPs by caching the WQEs (WQEs of SQs and RQs) of the QPs in the multiple controllers 111 to the QP cache 1124 of the network interface card 112.

[0126] The host client 12 can initiate a request to establish at least one QP connection with at least one shared connector 1123 of the storage device 11. This QP connection can be a reliable or unreliable connection. That is, the storage device 11 establishes at least one QP connection with the host client 12 (at least one QP) through at least one shared connector 1123. Each QP connection corresponds to one shared connector 1123 and one QP for each controller 111. The aforementioned shared connector 1123 and one QP for each controller 111 can be collectively referred to as a QP group; that is, one QP connection corresponds to one QP group, and within each QP group, one shared connector 1123 and one QP for each controller 111. The number of controllers corresponding to one QP connection can dynamically increase or decrease as the controllers are powered on or off.

[0127] The shared connector 1123 is visible to the host client 12, while each QP in the controller 111 is not visible to the host client 12. That is, the first network interface card 112 of the storage device 11 communicates with the second network interface card 121 of the host client 12 through the shared connector 1123, and establishes a QP connection with one of the QPs of the host client 12.

[0128] The queue context from the standard RDMA QP can be carried on the QP, while the connection context can be carried on the shared connector 1123. All QPs within a QP group use the same connection context and a set of ordered, incrementing PSNs.

[0129] For storage device 11 receiving the first RDMA message from host client 12:

[0130] Parser 1121 is used to distribute the received first RDMA message to one of the at least one shared connector 1123.

[0131] The shared connector 1123 is used to establish a QP connection with a peer network device (e.g., a host client) and determine the first QP corresponding to the first controller based on the first RDMA message, so as to communicate with the first controller and notify the corresponding consumer of the first RDMA message. The plurality of controllers 111 includes the first controller.

[0132] Specifically, whether it's a command message or a data message, the same message can be segmented into multiple packets, but the information available for distribution generally only exists in the first packet of the message. The shared connector 1123, based on load balancing principles, distributes the first packet of the IO command request to a normally functioning controller 111 for storage and maintains the correspondence between the first packet and the controller 111. This allows different IO command requests and corresponding data messages from a single host client 12 to be distributed to different controllers 111 for parallel processing, improving processing performance. If subsequent packets of the same message are received, the shared connector 1123 can utilize the order-preserving characteristic of RDMA transmission to map the subsequent packets to the first packet, and distribute them to the corresponding controller 111 based on the corresponding first packet and the correspondence between the first packet and the controller 111, maintaining consistency in the distribution of packets of the same message.

[0133] For different messages, if a command message is received, the shared connector 1123 distributes it to the normally operating controller 111 for storage according to the load balancing principle, and saves the correspondence between the command message and the controller 111. This allows multiple command messages from a single host client 12 to be distributed to different controllers 111 for parallel processing, improving processing performance. After processing the IO read / write request message, the controller initiates an IO data transmission operation, that is, the controller submits the corresponding RDMA write operation or RDMA read operation WQE to the corresponding QP. After receiving the acknowledgment message of the RDMA write operation or the response message of the RDMA read operation, the first shared connector distributes it to the first QP of the first controller that initiated the operation according to the PSN of the first RDMA message, maintaining the consistency of command messages and data messages in distribution.

[0134] For storage device 11 sending a second RDMA message to host client 12:

[0135] Scheduler 1122 is used to select a second QP with a message to be sent from among the multiple QPs of the multiple controllers 111 in place.

[0136] Shared connector 1123 is used to send the second RDMA message in the storage space corresponding to the second QP to the host client 12.

[0137] Specifically, after the scheduler 1122 determines that a controller 111 needs to send a message, it selects the corresponding shared connector to send the message according to the scheduling algorithm. The shared connector 1123 then sends the message to the corresponding host client 12 through the corresponding QP connection, thereby achieving fair transmission. Furthermore, for messages from different controllers 111 on the same QP connection, the PSN is guaranteed to be ordered and incremented before being sent to the host client 12, so that the host client 12 believes that it is communicating with a QP of the storage device 11.

[0138] In summary, multiple controllers 111 can work in parallel to provide storage services for the same QP connection. When one controller 111 fails, as long as one controller 111 is functioning normally, subsequent messages can be redirected and distributed to the working controller 111 for processing. This ensures that the host client 12 can access the storage space in the storage device 11, improving reliability and realizing the active-active (AA) function of the controllers in the storage device 11. Furthermore, different IO command requests received through the same QP connection can be distributed to controllers of different QPs for parallel processing, thus improving the storage access performance of a single QP connection.

[0139] For receiving messages, the network interface card of the storage device can perform actions such as... Figure 14 The message receiving method shown includes:

[0140] S1401, The parser receives the first RDMA message from the host client and distributes the first RDMA message to the first shared connector in at least one shared connector.

[0141] The peer network device (e.g., the host client) discovers the storage device in the front-end network and establishes at least one QP connection with the storage device via at least one QP connection to its first shared connector. In this way, the host client can exchange data with the storage device through the QP connection.

[0142] The first RDMA message can be a command message or a data message. The command message includes a SEND message and an acknowledgment message for the SEND message. The data message includes an acknowledgment message for the RDMA WRITE message and an RDMA READ response message.

[0143] The parser can determine the corresponding QP connection and the corresponding first shared connector based on the queue pair number (QPN) in the first RDMA message, and then distribute the first RDMA to the first shared connector. The QPN is a unique identifier for the QP connection, and therefore can be used to determine the corresponding QP connection.

[0144] As mentioned above, a QP connection corresponds to a QP group, and within the QP group, there is a shared connector and a QP corresponding to each controller. Therefore, based on the QPN, not only can the corresponding QP connection be determined, but also the QP group corresponding to that QP connection, as well as the shared connector and QP within the QP group.

[0145] S1402, the first shared connector determines the first QP corresponding to the first controller according to the first RDMA message to realize communication with the first controller.

[0146] like Figure 15 As shown, step S1402 includes S14021-S14025:

[0147] S14021, The first shared connector performs a validity check on the first RDMA message.

[0148] The validity verification includes, but is not limited to, ICRC check, determining whether a packet has been lost, and determining whether a message has been received repeatedly. Specifically, for acknowledgment messages, validity verification includes ICRC check; for other messages, validity verification includes ICRC check, determining whether a packet has been lost, and determining whether a message has been received repeatedly.

[0149] For a reliable QP connection, the PSN of RDMA packets within the same QP connection is ordered sequentially. If the PSN of an RDMA packet received by the first shared connector is not ordered sequentially, it can be determined that packet loss has occurred. If the PSN of an RDMA packet received by the first shared connector is duplicated, it can be determined that a packet has been received repeatedly.

[0150] If the ICRC check fails or packet loss occurs, the first RDMA packet is discarded and a negative acknowledgment (NACK) is sent to the host client to notify the client to retransmit. If a duplicate packet is received, the first RDMA packet is discarded. Only RDMA packets that pass the validity check will proceed to the next step.

[0151] S14022. If the first RDMA message is the first packet or a single packet of a SEND message, the first shared connector determines the first QP corresponding to the first RDMA message based on the information in the non-volatile memory express (NVME) submission queue entry (SQE) in the first RDMA message.

[0152] If the first network interface card connects to a client, and the first RDMA message is the first packet or a single packet of a SEND message, the first shared connector can determine the first QP corresponding to the first RDMA message based on the information in the NVME SQE within the first RDMA message. If the first network interface card connects to multiple clients, and the first RDMA message is the first packet or a single packet of a SEND message, the first shared connector can determine the first QP corresponding to the first RDMA message based on the information in the NVME SQE within the first RDMA message and the client address or QP connection. The first network interface card can perform load balancing on the received IO command requests from different clients based on the client address or QP connection.

[0153] In this embodiment, the RDMA message is segmented to obtain multiple RDMA packets. The first packet refers to the first RDMA packet among the multiple RDMA packets, the last packet refers to the last RDMA packet among the multiple RDMA packets, and the middle packets refer to the other RDMA packets among the multiple RDMA packets. A single packet refers to an RDMA packet that is not segmented, that is, the RDMA packet is a complete RDMA message.

[0154] For example, the first packet refers to the BTH opcode of the RDMA message being "SEND First", "RDMA WRITE first", or "RDMA READ Response first", etc.

[0155] A single packet refers to an RDMA message whose BTH opcode is "SEND Only", "SEND Only with Immediate", "RDMA WRITE Only", "RDMA WRITE Only with Immediate", or "RDMA READ Response Only", etc.

[0156] The information in NVME SQE includes namespace ID (NSID) and logical block address (LBA).

[0157] For example, if the first RDMA message is a command message (e.g., a SEND message), the first shared connector determines the first QP corresponding to the first RDMA message based on the NVME SQE information (e.g., NSID and / or LBA) in the first RDMA message. This is because the information in the NVME SQE is many-to-one with the controller; that is, multiple NVME SQE messages correspond to one controller, thus the corresponding first controller can be determined based on the information in the NVME SQE. Furthermore, a QP belonging to a specific controller within a QP group is unique; therefore, the first slave QP in the first controller can be determined, meaning the information in the NVME SQE is also many-to-one with the QPs in the controller.

[0158] S14023. If the first RDMA message is an acknowledgment message of an RDMA WRITE message, an acknowledgment message of a SEND message, or the first packet or a single packet of an RDMA READ response message, then the first shared connector determines the first QP corresponding to the first RDMA message based on the correspondence between the PSN of the first RDMA message and the PSN of the second RDMA message and the first QP.

[0159] The intermediate packet refers to the BTH opcode of the RDMA message, which is "SEND Middle", "RDMA WRITE Middle" or "RDMAREAD Response Middle", etc.

[0160] The tail packet refers to the BTH opcode of the RDMA message, which is "SEND Last", "SEND Last with Immediate", "RDMA WRITE Last" or "RDMA READ Response Last", etc.

[0161] The second RDMA message is the RDMA message sent by the first QP; the PSN of the first RDMA message corresponds to the PSN of the second RDMA message:

[0162] When the first RDMA message is an acknowledgment message of an RDMA WRITE message, the second RDMA message is also an RDMA WRITE message, and the PSN of the first RDMA message is the same as that of the second RDMA message.

[0163] When the first RDMA message is an acknowledgment message of the SEND message, the second RDMA message is also a SEND message, and the PSN of the first RDMA message is the same as that of the second RDMA message.

[0164] When the first RDMA message is the first packet or a single packet of an RDMA READ response message, and the second RDMA message is an RDMAREAD request message, the PSN of the first RDMA message is incremented in order relative to the PSN of the second RDMA message. For example, assuming the PSN of the second RDMA message is 1, the PSN of the first RDMA message will increment in order starting from 1.

[0165] In other words, when a QP requests to send a second RDMA message belonging to an IO command request or command response (such as an RDMA WRITE message, SEND message, or RDMA READ request message), the first shared connector can store the correspondence between the PSN of the second RDMA message and the QP (hereinafter referred to as the second QP). Subsequently, when the first shared connector receives a first RDMA message belonging to the same IO operation (an acknowledgment message for an RDMA WRITE message, an acknowledgment message for a SEND message, or an RDMA READ response message), the PSN of the first RDMA message corresponds to the PSN of the second RDMA message. The first shared connector determines the PSN of the corresponding second RDMA message using the PSN of the first RDMA message, and then, based on the aforementioned correspondence between the PSN of the second RDMA message and the second QP, it can determine the second QP requesting the sending of the second RDMA message. At this point, the second QP and the first QP are the same QP.

[0166] Additionally, for steps S14022-S14023, if the first RDMA message is the first packet, the first shared connector can save the correspondence between the first packet of the message to which the first RDMA message belongs and the first QP, that is, save the correspondence between the first packet of the message to which the first RDMA message belongs and the first controller to which the first QP belongs, for example, save the correspondence between the PSN of the first RDMA message and the identifier of the first controller to which the first QP belongs. This allows the first shared connector to match subsequent RDMA messages (intermediate packets or tail packets) belonging to the same message as the first RDMA message to the corresponding first controller and first QP.

[0167] S14024. If the first RDMA message is a tail packet or an intermediate packet, the first shared connector determines the first packet corresponding to the first RDMA message based on the order-preserving incrementality of the PSN of the segmented RDMA message and the PSN of the first RDMA message; and determines the first QP based on the first packet corresponding to the first RDMA message and the above correspondence between the first packet and the first QP.

[0168] like Figure 4 As described, the PSN of RDMA messages after a message segmentation is ordered and incremented. Therefore, for RDMA messages in the tail or middle packets, the first packet of the same message can be determined by its PSN. The first QP can be determined by combining the correspondence between the first packet and the first QP.

[0169] S14025, the first shared connector stores the first RDMA message in the storage space corresponding to the first QP, and / or submits a CQE to the CQ associated with the first QP.

[0170] After determining the first QP corresponding to the first RDMA message, the process of the first shared connector communicating with the first controller may include: the first shared connector storing the first RDMA message in the storage space corresponding to the first QP, and / or submitting a CQE to the CQ associated with the first QP.

[0171] CQE is used to indicate the completion of the RDMA operation (e.g., SEND, RDMA WRITE, RDMA READ) corresponding to the first RDMA message. The first QP is the QP corresponding to the QP connection of one of the at least one controller with the first shared connector. A QP belonging to a particular controller within a QP group is unique, and a QP group corresponds to a single QP connection; therefore, the first QP can correspond to the QP connection of the first shared connector.

[0172] The storage space corresponding to the first QP can also refer to the storage space corresponding to the controller where the first QP resides. For example, an RDMA SEND message carrying an IO command can be stored in the storage space pointed to by the WQE of the first QP's RQ. An RDMA READ response message carrying IO write data can be stored in the storage space requested by the controller where the first QP resides when sending the RDMA READ message, rather than in the storage space pointed to by the WQE of the QP's RQ.

[0173] Specifically, if the first RDMA message is not an acknowledgment message, the first shared connector stores the first RDMA message in the storage space corresponding to the first QP. If the first RDMA message is a tail packet, a single packet, or an acknowledgment message, the first shared connector submits a CQE to the CQ associated with the first QP. If the first RDMA message is a tail packet or a single packet, the first shared connector stores the first RDMA message in the storage space corresponding to the first QP and submits a CQE to the CQ associated with the first QP.

[0174] In other words, for a first RDMA message that is an acknowledgment (ACK) message, the first shared connector does not store it in the storage space corresponding to the first QP, but only submits a CQE to the CQ associated with the first QP, indicating that the corresponding RDMA operation is complete. For the first packet or intermediate packet where the first RDMA message is a SEND message or an RDMA READ response message, since the RDMA operation has not yet been completed, the first shared connector only stores the first RDMA message in the storage space corresponding to the first QP, and does not submit a CQE to the CQ associated with the first QP. For a single packet or tail packet where the first RDMA message is a SEND message or an RDMA READ response message, since the corresponding RDMA operation has been completed, the first RDMA message is both stored in the storage space corresponding to the first QP and a CQE is submitted to the CQ associated with the first QP.

[0175] For example, for the first RDMA message which is the tail packet or single packet of a SEND message, the first shared connector stores the tail packet or single packet of the SEND message in the aforementioned storage space and submits a CQE to the CQ associated with the first QP. The CQE is used to indicate the completion of receiving the SEND message.

[0176] For example, if the first RDMA message is the first packet or intermediate packet of a SEND message, the first shared connector stores the first packet or intermediate packet of the SEND message in the aforementioned storage space without submitting a CQE.

[0177] For example, for the first RDMA message which is the tail packet or single packet of the RDMA READ response message, the first shared connector stores the tail packet or single packet of the RDMA READ response message in the aforementioned storage space and submits a CQE to the CQ associated with the first QP. The CQE is used to indicate the completion of the RDMA read operation.

[0178] For example, for the first RDMA message which is the first packet or intermediate packet of the RDMA READ response message, the first shared connector stores the first packet or intermediate packet of the RDMA READ response message in the aforementioned storage space without submitting a CQE.

[0179] For example, for a first RDMA message that is an acknowledgment message of a SEND message or an acknowledgment message of an RDMAWRITE message, the storage device submits a CQE to the CQ associated with the first QP. The CQE is used to indicate that the SEND operation or the RDMA WRITE operation is completed.

[0180] The network interface card and storage device provided in this application embodiment include a parser and at least one shared connector. The parser distributes received first Remote Direct Memory Access (RDMA) messages to a first shared connector among the at least one shared connector. The first shared connector establishes a QP connection with the peer network device and determines a first queue pair (QP) corresponding to a first controller based on the first RDMA message to enable communication with the first controller. The first shared connector connects to multiple controllers, including the first controller, and each controller has a corresponding QP with the first shared connector. The storage device is only visible to the host client through the shared connectors. However, the first shared connector can distribute received RDMA messages from the host client to different controllers containing different QPs for processing. Therefore, it can shield controllers that are not functioning properly. As long as one controller is functioning correctly, the storage device can provide storage services, thus improving the reliability of storage devices using RDMA networks. Furthermore, by distributing different IO command requests to controllers located in different QPs for parallel processing, the performance can be improved by N times compared to the existing solution where RDMA messages of the same QP connection can only be processed by a fixed controller, where N is the number of controllers. Therefore, the storage access performance of a single QP connection can be improved.

[0181] For sending messages, the network interface card of the storage device can perform actions such as... Figure 16 The message sending method shown includes:

[0182] S1601, The scheduler selects a second QP with a message to be sent from among the multiple QPs of the multiple controllers in place, which corresponds to the second shared connector.

[0183] The second shared connector is one of at least one shared connector.

[0184] The scheduler first obtains the doorbell signal of at least one QP from multiple controllers.

[0185] As mentioned above, each controller corresponds to a QP, and each QP includes an SQ, which includes a WQE. The doorbell signal is used to indicate that there is a message (data message or command message) to be sent in the storage space pointed to by the WQE in the SQ of the corresponding controller's QP.

[0186] When a controller has a message to be transmitted, one or more WQEs are added to the SQ of the QP of the controller. The WQE points to the storage space of the message to be transmitted. The QP generates the doorbell signal mentioned above, and the scheduler obtains the doorbell signal.

[0187] The scheduler aggregates doorbell signals from various QPs, but does not directly trigger the sending of RDMA messages. Instead, it selects one doorbell signal to respond to according to preset rules. The corresponding QP is the second QP, and the controller where the second QP is located must work normally.

[0188] This application does not limit specific preset rules. In one possible implementation, the scheduler can select the second QP from multiple QPs (located in different controllers) corresponding to the second shared connector through round robin (RR) scheduling.

[0189] For example, such as Figure 17 As shown, assuming that QPa of controller 1, QPb of controller 2, QPc of controller 3, and shared connector a belong to the same QP group, and QPa of controller 1 has three WQEs (i.e., messages in three storage spaces are waiting to be transmitted), then the depth of the doorbell signal DB1 of QPa by the scheduler is 3; QPb of controller 2 has one WQE (i.e., messages in one storage space are waiting to be transmitted), then the depth of the doorbell signal DB2 of QPb by the scheduler is 1; QPc of controller 3 has two WQEs (i.e., messages in two storage spaces are waiting to be transmitted), then the depth of the doorbell signal DB3 of QPc by the scheduler is 2.

[0190] According to the Recursive Response (RR) principle, in the first round, the WQEs of QPa, QPb, and QPc are scheduled sequentially. That is, QPa, QPb, and QPc are used as the second QPs in turn, and messages in the storage spaces pointed to by their respective WQEs are sent. Therefore, the depth of DB1 becomes 2, the depth of DB2 becomes 0, and the depth of DB3 becomes 1. Since the depth of DB2 is 0, indicating that there are no more messages to send, in the second round, QPa and QPc are used as the second QPs in turn, and messages in the storage spaces pointed to by their respective WQEs are sent. Therefore, the depth of DB1 becomes 1, and the depth of DB3 becomes 0, indicating that there are no more messages to send. In the third round, QPa is used as the second QP, and messages in the storage spaces pointed to by its corresponding WQEs are sent. Therefore, the depth of DB1 becomes 0, indicating that there are no more messages to send.

[0191] Preset rules can also include other methods, such as weighted round-robin (WRR) scheduling and dynamic weighted round-robin (DWRR). Regardless of the preset rule, QPs of controllers that are not in place or idle QPs can be skipped during scheduling.

[0192] S1602, the second shared connector sends the messages in the storage space corresponding to the second QP to the host client in the form of a second RDMA message.

[0193] Similar to step S1401, before the second shared connector sends the RDMA message, the peer network device (e.g., the host client) discovers the storage device in the front-end network and establishes at least one QP connection with the storage device's second shared connector via at least one QP connection. In this way, the host client can exchange data with the storage device through the QP connection.

[0194] Specifically, the storage space corresponding to the second QP refers to the storage space pointed to by WQE in the SQ of the second QP. The second QP can be the same as or different from the first QP.

[0195] As mentioned earlier, the second shared connector segments messages into RDMA packets before sending them. The scheduling granularity between QPs of different controllers within the same QP group is based on WQE (Wide Memory Equivalent). This means that once a QP is scheduled, it must continuously send all messages stored in the memory space pointed to by the WQE without being interrupted by other QPs within the same QP group. Scheduling between different QP groups is independent and does not affect each other.

[0196] When the second shared connector sends RDMA messages for multiple QPs within the same QP group, it uses the same set of PSNs. That is, the PSNs carried by the RDMA messages of different QPs within the same QP group are obtained by incrementing the same counter, so that the network interface card of the host client thinks that these RDMA messages come from a QP of the storage device.

[0197] Additionally, if the second RDMA message is an RDMA WRITE message, SEND message, or RDMA READ request message, the second shared connector stores the correspondence between the PSN of the second RDMA message and the second QP, as described in step S14023, so that the network interface card can determine the first QP based on the PSN of the first RDMA message when receiving the first RDMA message.

[0198] The network interface card and storage device provided in this application embodiment include a scheduler and at least one shared connector. The scheduler selects a second QP with messages to be sent from multiple QPs of multiple controllers in place, corresponding to a second shared connector. The second shared connector is one of the at least one shared connector. The shared connector establishes a QP connection with the peer network device and sends messages in the storage space corresponding to the second QP to the host client in the form of a second RDMA message. The second shared connector is connected to multiple controllers respectively, and each controller has a corresponding QP and a second shared connector. The scheduler can select one QP from multiple QPs, and the corresponding second shared connector processes its RDMA messages to be sent, so that storage services can be provided as long as one controller is working properly, thus improving the reliability of storage devices using RDMA networks. Furthermore, controllers containing different QPs can process different IO command requests in parallel. Compared to existing solutions where RDMA messages of the same QP connection can only be processed by a fixed controller, the performance can be improved by N times, where N is the number of controllers, thus improving the storage access performance of a single QP connection.

[0199] Similar to Figure 11 The I / O read / write process involves the host client and storage device utilizing the aforementioned message receiving and sending methods to implement the I / O read / write process, as follows: Figure 18 As shown:

[0200] S1801. The host client discovers the storage device in the front-end network and establishes a QP connection with a shared connector in the network interface card of the storage device through a QP.

[0201] S1802, The host client sends a first RDMA message (SEND message) to the storage device via the QP connection. The network interface card parser of the storage device receives the first RDMA message from the host client.

[0202] The SEND message can include either an IO write command or an IO read command; that is, the data payload of the SEND message can be either an IO write command or an IO read command.

[0203] This step refers to step S1401.

[0204] S1803. The parser in the network interface card of the storage device determines the corresponding first shared connector based on the QPN in the first RDMA message, and distributes the first RDMA message to the first shared connector.

[0205] This step refers to step S1401.

[0206] S1804, The first shared connector performs a validity check on the first RDMA message.

[0207] This step is the same as step S14021.

[0208] S1805. If the first RDMA message is the first packet or a single packet of the SEND message, the first shared connector determines the first QP corresponding to the first RDMA message based on the NVME SQE information (e.g., NSID and / or LBA) in the first RDMA message, and saves the correspondence between the first packet of the message to which the first RDMA belongs and the first QP.

[0209] This step refers to steps S14022 and S14023.

[0210] S1806. If the first RDMA message is an intermediate or final packet of a SEND message, the first shared connector determines the first packet corresponding to the first RDMA message based on the ordered and incremental nature of the PSN of the segmented RDMA message and the PSN of the first RDMA message; and determines the first QP based on the first packet corresponding to the first RDMA message and the correspondence between the first packet and the first QP.

[0211] This step refers to step S14024.

[0212] S1807, the first shared connector stores the first RDMA message in the storage space corresponding to the first QP, and / or submits a CQE to the CQ associated with the first QP.

[0213] This step is the same as step S14025.

[0214] S1808, the consumer of the controller where the first QP is located (e.g., NoF protocol software) queries the CQ to find that a new CQE has been submitted, parses the NVME SQE information and initiates data interaction.

[0215] If the NVME SQE message includes an IO read command, the consumer, after reading IO data from the back-end storage medium, submits an RDMA WRITE operation WQE to the QP's SQ and generates a doorbell signal to notify the network interface card that there is a message to be transmitted. Alternatively, the consumer can submit a SEND operation WQE after submitting an RDMA WRITE operation WQE to the QP's SQ.

[0216] If the NVME SQE message includes an IO write command, the consumer submits an RDMA READ operation WQE to the QP's SQ and generates a doorbell signal to notify the network interface card that there is an RDMA READ message to be transmitted.

[0217] If the NVME SQE message includes an IO write command with an immediate value, the consumer constructs a NoF command response message after completing the persistent storage of the IO data, submits the WQE of the SEND operation to the SQ of the QP, and generates a doorbell signal to notify the network interface card that there is a SEND message to be transmitted.

[0218] S1809, The scheduler gathers doorbell signals from each QP and selects a second QP with a message to be sent from among the multiple QPs of the multiple controllers in place, corresponding to the second shared connector.

[0219] This step is the same as step S1601.

[0220] S1810, the second shared connector sends the messages in the storage space corresponding to the second QP to the host client in the form of a second RDMA message.

[0221] This step is the same as step S1602.

[0222] S1811, The second shared connector submits a CQE to the CQ associated with the second QP.

[0223] If the WQE of the SQ of the second QP indicates a SEND operation or an RDMA WRITE operation, then after the second shared connector receives an acknowledgment message from the host client, it submits a CQE to the CQ associated with the second QP.

[0224] If the WQE of the SQ of the second QP indicates an RDMA READ operation, then after the second shared connector receives the tail packet or single packet of the RDMA READ response message from the host client, it submits a CQE to the CQ associated with the second QP.

[0225] Specifically, if the WQE of the SQ of the second QP indicates an RDMA READ operation, the second shared connector, when sending a second RDMA message belonging to the RDMA READ request message, saves the correspondence between the PSN of the second RDMA message and the second QP. If the first RDMA message received by the second shared connector is the first packet or a single packet of an RDMA READ response message, the second shared connector determines the second QP (i.e., the first QP) based on the PSN of the first RDMA message and the correspondence between the PSN of the second RDMA message and the second QP, and saves the correspondence between the first packet of the message to which the first RDMA belongs and the first QP. If the first RDMA message is the last packet or a middle packet of an RDMA READ response message, the second shared connector determines the first packet corresponding to the first RDMA message based on the ordered incremental nature of the PSN of the segmented RDMA messages and the PSN of the first RDMA message; and determines the first QP (i.e., the second QP) based on the first packet corresponding to the first RDMA message and the correspondence between the first packet and the first QP.

[0226] S1812, The consumer of the controller where the second QP is located (e.g., NoF protocol software) queries the CQ and finds that a new CQE has been submitted.

[0227] If the newly submitted CQE indicates that the sending process in the SEND operation is complete, it means that an IO operation has been completed at this time, and the consumer reclaims the WQE resources in the associated SQ and the CQE resources in the CQ.

[0228] If the newly submitted CQE indicates that the RDMA WRITE operation is complete, the consumer reclaims the WQE resources in the relevant SQ and the CQE resources in the CQ.

[0229] If the newly submitted CQE indicates that the RDMA READ operation is complete, the consumer, after completing the persistent storage of IO data, constructs a NoF command response message, submits a WQE for the SEND operation to the SQ of the QP, and generates a doorbell signal to notify the network interface card that there is a message to be transmitted. Proceed to step S1809.

[0230] Figure 18 In this context, the first QP refers to the QP during the message receiving process, and the second QP refers to the QP during the message sending process. The first QP and the second QP can be the same QP.

[0231] Figure 18 The first shared connector and the second shared connector can be the same shared connector.

[0232] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0233] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0234] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0235] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0236] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0237] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0238] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0239] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A network interface card, characterized in that, Includes a parser and at least one shared connector. The parser is used to distribute the received first RDMA message to the first shared connector among the at least one shared connector; The first shared connector is used to establish a QP connection with the peer network device and determine the first QP corresponding to the first controller according to the first RDMA message to realize communication with the first controller. The first shared connector is connected to multiple controllers, including the first controller, and each QP of the multiple controllers corresponds to the first shared connector.

2. The network interface card according to claim 1, characterized in that, The first shared connector is also used for: If the first RDMA message is not an acknowledgment message, then the first RDMA message is stored in the storage space corresponding to the first QP.

3. The network interface card according to claim 1 or 2, characterized in that, The first shared connector is also used for: If the first RDMA message is a tail packet, a single packet, or an acknowledgment message, then a completion queue entry CQE is submitted to the completion queue CQ associated with the first QP. The CQE is used to indicate that the RDMA operation corresponding to the first RDMA message is complete.

4. The network interface card according to claim 1, characterized in that, The first shared connector is specifically used for: If the first RDMA message is the first packet or a single packet of a SEND message, then the first QP is determined based on the information in the Non-Volatile Memory Specification (NVME) Submission Queue Entry (SQE) in the first RDMA message. The information in the NVME SQE corresponds to the first controller.

5. The network interface card according to claim 1, characterized in that, The first shared connector is specifically used for: If the first RDMA message is an acknowledgment message of an RDMA WRITE message, an acknowledgment message of a SEND message, or the first packet or a single packet of an RDMAREAD response message, then the first QP is determined according to the correspondence between the message sequence number of the first RDMA message and the message sequence number of the second RDMA message and the first QP. Wherein, the second RDMA message is the RDMA message sent by the first QP; the message sequence number of the first RDMA message corresponds to the message sequence number of the second RDMA message; when the first RDMA message is an acknowledgment message of an RDMA WRITE message, the second RDMA message belongs to an RDMA WRITE message; when the first RDMA message is an acknowledgment message of a SEND message, the second RDMA message belongs to a SEND message; when the first RDMA message is the first packet or a single packet of an RDMA READ response message, the second RDMA message belongs to an RDMA READ request message.

6. The network interface card according to claim 4 or 5, characterized in that, The first shared connector is also used for: If the first RDMA message is the first packet, then the correspondence between the first packet of the message to which the first RDMA message belongs and the first QP is saved.

7. The network interface card according to claim 6, characterized in that, The first shared connector is also used for: If the first RDMA message is a tail packet or an intermediate packet, the first packet corresponding to the first RDMA message is determined based on the order-preserving incrementality of the message sequence number of the segmented RDMA message and the message sequence number of the first RDMA message. The first QP is determined based on the first packet corresponding to the first RDMA message and the corresponding relationship.

8. A network interface card, characterized in that, Includes a scheduler and at least one shared connector. The scheduler is configured to select a second QP with a message to be sent from multiple queue pairs QP of multiple controllers in place, the second shared connector being one of the at least one shared connector; The second shared connector is used to establish a QP connection with the peer network device and send the messages in the storage space corresponding to the second QP to the host client in the form of a second RDMA message; wherein, the second shared connector is connected to the plurality of controllers respectively, and each QP of the plurality of controllers corresponds to the second shared connector.

9. The network interface card according to claim 8, characterized in that, The second shared connector is also used for: If the second RDMA message is an RDMA WRITE message, SEND message, or RDMA READ request message, then the correspondence between the message sequence number in the second RDMA message and the second QP is saved.

10. The network interface card according to claim 8 or 9, characterized in that, The scheduler is specifically used for: The second QP is selected from multiple QPs corresponding to the second shared connector through a round-robin scheduling.

11. A message receiving method, characterized in that, Applied to a network interface card as described in any one of claims 1-7, the method includes: The parser distributes the received first RDMA message to the first shared connector in at least one shared connector; The first shared connector establishes a QP connection with the peer network device and determines the first QP corresponding to the first controller according to the first RDMA message to realize communication with the first controller. The first shared connector is connected to multiple controllers, including the first controller, and each of the multiple controllers has a QP corresponding to the first shared connector.

12. The method according to claim 11, characterized in that, Also includes: If the first RDMA message is not an acknowledgment message, the first shared connector will store the first RDMA message in the storage space corresponding to the first QP.

13. The method according to claim 11 or 12, characterized in that, Also includes: If the first RDMA message is a tail packet, a single packet, or an acknowledgment message, the first shared connector submits a completion queue entry CQE to the completion queue CQ associated with the first QP. The CQE is used to indicate that the RDMA operation corresponding to the first RDMA message is complete.

14. The method according to claim 11, characterized in that, The step of determining the first queue pair (QP) corresponding to the first controller based on the first RDMA message includes: If the first RDMA message is the first packet or a single packet of a SEND message, the first shared connector determines the first QP based on the information in the Non-Volatile Memory Specification (NVME) Submission Queue Entry (SQE) in the first RDMA message. The information in the NVME SQE corresponds to the first controller.

15. The method according to claim 11, characterized in that, The step of determining the first queue pair (QP) corresponding to the first controller based on the first RDMA message includes: If the first RDMA message is an acknowledgment message of an RDMA WRITE message, an acknowledgment message of a SEND message, or the first packet or a single packet of an RDMAREAD response message, then the first shared connector determines the first QP based on the message sequence number of the first RDMA message and the correspondence between the message sequence number of the second RDMA message and the first QP. Wherein, the second RDMA message is the RDMA message sent by the first QP; the message sequence number of the first RDMA message corresponds to the message sequence number of the second RDMA message; when the first RDMA message is an acknowledgment message of an RDMA WRITE message, the second RDMA message belongs to an RDMA WRITE message; when the first RDMA message is an acknowledgment message of a SEND message, the second RDMA message belongs to a SEND message; when the first RDMA message is the first packet or a single packet of an RDMA READ response message, the second RDMA message belongs to an RDMA READ request message.

16. The method according to claim 14 or 15, characterized in that, Also includes: If the first RDMA message is the first packet, the first shared connector stores the correspondence between the first packet of the message to which the first RDMA message belongs and the first QP.

17. The method according to claim 16, characterized in that, Also includes: If the first RDMA message is a tail packet or an intermediate packet, the first shared connector determines the first packet corresponding to the first RDMA message based on the order-preserving incrementality of the message sequence number of the segmented RDMA message and the message sequence number of the first RDMA message. The first QP is determined based on the first packet corresponding to the first RDMA message and the corresponding relationship.

18. A message transmission method, characterized in that, Applied to a network interface card as described in any one of claims 8-10, the method includes: The scheduler selects a second QP with messages to be sent from multiple queue pairs QPs of multiple controllers in place, which corresponds to a second shared connector, and the second shared connector is one of at least one shared connector; The second shared connector establishes a QP connection with the peer network device and sends the messages in the storage space corresponding to the second QP to the host client in the form of a second RDMA message; wherein, the second shared connector is connected to the plurality of controllers respectively, and each QP of the plurality of controllers corresponds to the second shared connector.

19. The method according to claim 18, characterized in that, Also includes: If the second RDMA message is an RDMA WRITE message, SEND message, or RDMA READ request message, the second shared connector stores the correspondence between the message sequence number in the second RDMA message and the second QP.

20. The method according to claim 18 or 19, characterized in that, The scheduler selects a second QP with messages to be sent from multiple queue pairs (QPs) of multiple in-place controllers, corresponding to the second shared connector, including: The scheduler selects the second QP from multiple QPs corresponding to the second shared connector through round-robin scheduling.

21. A storage device, characterized in that, It includes a network interface card as described in any one of claims 1-10 and a plurality of controllers, wherein the network interface card is connected to each of the plurality of controllers respectively.

Images