Network interface card, controller, storage device and message sending method

Abstract

The present application discloses a network interface card, a controller, a storage device, and a message sending method, relating to the field of storage and used to reduce the latency of RDMA read operations. The network interface card includes: a second processing module for obtaining a first work queue element (WQE) from a first send queue (SQ) corresponding to the first controller in response to a doorbell signal from a first controller, wherein the doorbell signal is used to indicate that there is a message to be sent in the storage space pointed to by at least one WQE in the first SQ; the first WQE includes indication information, the indication information is used to indicate the type of a first remote direct memory access (RDMA) message scheduled by the first WQE; and a second transceiver module for sending the first RDMA message to a host client before the second processing module polls the SQ of the next controller if the indication information indicates that the type of the first RDMA message is a tail packet or a single packet of IO data.

Description

Technical Field

[0001] The present application relates to the field of storage, and in particular to a network interface card, a controller, a storage device, and a message sending method. Background Art

[0002] Currently, much data is not stored locally, but in remote storage devices. Figure 1 As shown, an embodiment of the present application provides a storage system, including a storage device 11, at least one host client 12, and a switch 13. The storage device 11 is connected to the switch 13 via a front-end fabric, and the switch 13 is connected to the at least one host client 12. The storage device 11 can provide data storage services for multiple host clients 12.

[0003] The storage device 11 includes multiple controllers 111, at least one network interface card (NIC) 112, and a disk frame 113. The controller 111 includes a processor, memory, and software. Multiple controllers can operate in load sharing mode to improve the throughput and reliability of the storage system. The disk frame 113 is mounted with storage media such as hard disk drives (HDD) or solid state drives (SSD). The NIC 112 has multiple high-speed serial computer expansion bus standard (Peripheral Component Interconnect, PCIe) interfaces, which are connected to each controller 111 respectively, and can distribute read and write requests from the host client 12 to multiple controllers 111 to achieve parallel processing. The controller 111 can obtain a read request from the host client 12 through the NIC 112, then read the IO data from the hard disk of the disk frame 113, and then the NIC 112 returns the IO data to the host client 12; the controller 111 can also obtain a write request from the host client 12 through the NIC 112, and write the IO data obtained by the NIC 112 to the hard disk of the disk frame 113.

[0004] With the rise of solid-state drives (SSDs) based on the non-volatile memory express (NVMe) specification, more and more front-end networks are adopting NVMe over fabric (NoF) networks based on switching fabrics. Remote direct memory access (RDMA), a standard in NoF networks, can reduce the latency of data access to storage devices.

[0005] like Figure 2 As shown in the figure, in an RDMA read operation, the host client, as the requester, sends an IO read command to the storage device. The storage device, as the responder, receives the IO read command, which is then distributed by the storage device's NIC to different controllers for parallel processing. The storage device's NIC sequentially sends the IO data and IO responses to the host client. To ensure fairness, when sending messages to the host client, the NIC polls each controller to see if it has any messages to send. That is, after sending IO data to one controller, it begins sending other messages to the next controller. The IO response from that controller is not sent until the next poll to that controller. This results in a longer interval between the IO data and IO response from the same controller, increasing the latency of the RDMA read operation. Summary of the Invention

[0006] Embodiments of the present application provide a network interface card, a controller, a storage device, and a message sending method for reducing the latency of an RDMA read operation.

[0007] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0008] In a first aspect, a network interface card is provided, comprising: a second processing module configured to obtain a first work queue element (WQE) from a first send queue (SQ) corresponding to the first controller in response to a doorbell signal from a first controller, wherein the doorbell signal indicates that there is a message to be sent in the storage space pointed to by at least one WQE in the first SQ; the first WQE includes indication information indicating the type of a first remote direct memory access (RDMA) message scheduled by the first WQE; and a second transceiver module configured to send the first RDMA message and a second RDMA message scheduled by the next WQE of the first WQE in the first SQ to a host client before the second processing module polls the SQ of the next controller if the indication information indicates that the first RDMA message is a tail packet or a single packet of I / O data.

[0009] The above-mentioned network interface card provided in the embodiment of the present application, the network interface card (NIC) of the storage device obtains a first WQE from the first SQ of the first controller, and the first WQE includes indication information, and the indication information is used to indicate the type of the first RDMA message scheduled by the first WQE. If the indication information indicates that the type of the first RDMA message is the tail packet or a single packet of IO data, before polling the SQ of the next controller, the first RDMA message and the second RDMA message (i.e., the RDMA message of the IO response) scheduled by the next WQE of the first WQE in the first SQ are sent to the host client. This allows the host client to quickly receive the corresponding IO response after receiving the tail packet or the single packet of IO data, and determine that the RDMA read operation is completed without having to wait for the NIC to poll the first controller next time, thereby reducing the latency of the RDMA read operation.

[0010] In one possible implementation, the second processing module is further configured to perform a data integrity function (DIF) check on the first RDMA message. The second transceiver module is specifically configured to: if the indication information indicates that the first RDMA message type is a tail packet or a single packet of I / O data and the DIF check passes, send the first RDMA message and the second RDMA message scheduled for the next WQE of the first WQE in the first SQ to the host client before the second processing module polls the next controller SQ. The DIF check can protect data integrity and consistency by appending information such as checksum data to the data.

[0011] In a possible implementation, the second transceiver module is further configured to: if the DIF check fails, send a third RDMA message to the host client, the third RDMA message indicating that the first RDMA message fails the DIF check, so that the host client can be notified of an IO data acquisition error.

[0012] In a possible implementation, the type of the first remote direct memory access (RDMA) packet includes a first packet, a middle packet, a last packet, or a single packet of IO data. This implementation discloses several possible types of the first RDMA packet.

[0013] In a second aspect, a controller is provided, comprising: a first processing module configured to submit a first work queue element (WQE) to a first send queue (SQ), the first WQE including indication information indicating the type of a first remote direct memory access (RDMA) message scheduled by the first WQE; and a first transceiver module configured to send a doorbell signal to a network interface card, the doorbell signal indicating that at least one storage space pointed to by the WQE in the first SQ has a message to be sent.

[0014] The controller provided in an embodiment of the present application adds indication information in the WQE to indicate the type of the first remote direct memory access (RDMA) message scheduled by the first WQE, that is, whether it is the first packet, middle packet, last packet, or single packet of IO data, and notifies the network interface card. If the indication information indicates that the type of the first RDMA message is the last packet or single packet of IO data, the controller sends the first RDMA message and the second RDMA message (that is, the RDMA message of the IO response) scheduled by the next WQE of the first WQE in the first SQ to the host client before polling the SQ of the next controller. This allows the host client to quickly receive the corresponding IO response after receiving the last packet or single packet of IO data, and determine that the RDMA read operation is completed without having to wait for the NIC to poll the first controller next time. Therefore, the latency of the RDMA read operation can be reduced.

[0015] In a possible implementation, the type of the first remote direct memory access (RDMA) packet includes a first packet, a middle packet, a last packet, or a single packet of IO data. This implementation discloses several possible types of the first RDMA packet.

[0016] In a third aspect, a storage device is provided, comprising the network interface card as described in the first aspect and any embodiment thereof and the controller as described in the second aspect.

[0017] In a fourth aspect, a message sending method is provided, comprising: in response to a doorbell signal from a first controller, obtaining a first work queue element (WQE) from a first send queue (SQ) corresponding to the first controller, wherein the doorbell signal indicates that at least one storage space pointed to by the WQE in the first SQ has a message to be sent; the first WQE includes indication information indicating the type of a first remote direct memory access (RDMA) message scheduled by the first WQE. If the indication information indicates that the first RDMA message is a tail packet or a single packet of I / O data, before polling the SQ of the next controller, sending the first RDMA message to a host client, as well as a second RDMA message scheduled by a subsequent WQE of the first WQE in the first SQ.

[0018] In one possible implementation, if the indication information indicates that the type of the first RDMA message is the tail packet or a single packet of IO data, then before polling the SQ of the next controller, the first RDMA message and the second RDMA message scheduled by the next WQE of the first WQE in the first SQ are sent to the host client, including: performing a data integrity function DIF check on the first RDMA message; if the indication information indicates that the type of the first RDMA message is the tail packet or a single packet of IO data, and the DIF check passes, then before polling the SQ of the next controller, the first RDMA message and the second RDMA message scheduled by the next WQE of the first WQE in the first SQ are sent to the host client.

[0019] In a possible implementation, the method further includes: if the DIF check fails, sending a third RDMA message to the host client, where the third RDMA message indicates that the first RDMA message fails the DIF check.

[0020] In a possible implementation, the type of the first remote direct memory access (RDMA) packet includes a first packet, a middle packet, a last packet, or a single packet of IO data.

[0021] In a fifth aspect, a message sending method is provided, including: submitting a first work queue element WQE to a first sending queue SQ, the first WQE including indication information, the indication information being used to indicate the type of a first remote direct memory access (RDMA) message scheduled by the first WQE; sending a doorbell signal to a network interface card, the doorbell signal being used to indicate that there is a message to be sent in the storage space pointed to by at least one WQE in the first SQ.

[0022] In a possible implementation, the type of the first remote direct memory access (RDMA) packet includes a first packet, a middle packet, a last packet, or a single packet of IO data.

[0023] In the sixth aspect, a computer-readable storage medium is provided, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method described in the fourth aspect and any one of its embodiments, or the fifth aspect and any one of its embodiments.

[0024] In the seventh aspect, a computer program product comprising instructions is provided, which, when the instructions are executed on a computer or a processor, enables the computer or the processor to execute the method as described in the fourth aspect and any one of its embodiments, or the fifth aspect and any one of its embodiments.

[0025] Regarding the technical effects of the third to seventh aspects, refer to the first aspect and any embodiment thereof and the technical effects of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic diagram of the architecture of a storage system provided in an embodiment of the present application;

[0027] Figure 2 A flowchart of an RDMA read operation provided in an embodiment of the present application;

[0028] Figure 3 A schematic diagram of a QP provided in an embodiment of the present application;

[0029] Figure 4 A schematic diagram of a QP connection for reliable connection provided in an embodiment of the present application;

[0030] Figure 5 A schematic diagram of an RDMA message with PSN increasing in order according to an embodiment of the present application;

[0031] Figure 6A A schematic diagram of the structure of a SEND message and a confirmation message of the SEND message based on the IB protocol provided in an embodiment of the present application;

[0032] Figure 6B A schematic diagram of the structure of a SEND message and a confirmation message of the SEND message based on the RoCEv2 protocol provided in an embodiment of the present application;

[0033] Figure 7 A schematic diagram of the interaction process between a requesting end and a responding end of a reliable connection provided in an embodiment of the present application;

[0034] Figure 8A A schematic diagram of the structure of an RDMA WRITE message and a confirmation message of the RDMA WRITE message based on the IB protocol provided in an embodiment of the present application;

[0035] Figure 8B A schematic diagram of the structure of an RDMA WRITE message and a confirmation message of the RDMA WRITE message based on the RoCEv2 protocol provided in an embodiment of the present application;

[0036] Figure 9 A schematic diagram of the interaction process between a requesting end and a responding end of another reliable connection provided by an embodiment of the present application;

[0037] Figure 10A A schematic diagram of the structure of an RDMA READ request message and an RDMA READ response message based on the IB protocol provided in an embodiment of the present application;

[0038] Figure 10B A schematic diagram of the structure of an RDMA READ request message and an RDMA READ response message based on the RoCEv2 protocol provided in an embodiment of the present application;

[0039] Figure 11 A schematic diagram of the interaction process between a requesting end and a responding end of another reliable connection provided in an embodiment of the present application;

[0040] Figure 12 A schematic diagram of a process in which a host client and a storage device use RDMA operations to complete IO reading and writing, provided in an embodiment of the present application;

[0041] Figure 13 A schematic diagram of the latency of an RDMA read operation provided in an embodiment of the present application;

[0042] Figure 14 A schematic diagram of the architecture of another storage system provided in an embodiment of the present application;

[0043] Figure 15 A flowchart of a message sending method provided in an embodiment of the present application;

[0044] Figure 16 A schematic diagram of the effect of reducing the latency of RDMA read operations provided by an embodiment of the present application;

[0045] Figure 17 A flowchart of another message sending method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] As used in this application, the terms "component", "module", "system" and the like are intended to refer to a computer-related entity, which can be hardware, firmware, a combination of hardware and software, software, or software in operation. For example, a component can be, but is not limited to: a process running on a processor, a processor, an object, an executable file, a thread in execution, a program and / or a computer. As an example, both an application running on a computing device and the computing device can be a component. One or more components can exist in an executing process and / or thread, and a component can be located in a computer and / or distributed between two or more computers. In addition, these components can be executed from various computer-readable media having various data structures thereon. These components can communicate in the form of local and / or remote processes, such as based on signals having one or more data packets (e.g., data from a component that interacts with another component in a local system, a distributed system, and / or interacts with other systems in the form of signals over a network such as the Internet).

[0047] First, the concepts involved in this application are described:

[0048] The storage devices referred to in this application may be referred to as storage servers, storage arrays, blade servers, etc., without limitation. The storage devices may include at least one controller and at least one network interface card (NIC). The controller may include, but is not limited to, a processor and memory, and the processor runs an operating system and application programs.

[0049] The processor involved in the embodiments of the present application may be a chip. For example, it may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0050] The memory involved in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0051] The NIC involved in the embodiments of the present application can also be called a network interface controller (NIC), a network adapter (network adapter), a network card, etc., or a local area network adapter (LAN adapter). It is a piece of computer hardware designed to allow computers to communicate on a computer network.

[0052] To improve system reliability, storage devices often employ multi-controller architectures, supporting multiple controllers providing data storage services to one or more host clients. Mid-range and high-end storage devices employ multi-host NICs for front-end network interfaces. Multiple controllers share a single NIC through different PCIe channels, with each controller acting as an independent network node.

[0053] Storage devices connect to host clients through a front-end network and can provide data storage services to multiple host clients. With the rise of NVMe SSDs, the protocol for host clients to access storage devices has transitioned from small computer system interface (SCSI) to NVMe. Correspondingly, the front-end network has also migrated from Fibre Channel (FC) networks and Transmission Control Protocol / Internet Protocol (TCP / IP) networks to NoF networks.

[0054] The NoF protocol supports memory type and message type operations. PCIe supports memory type operations, FC supports message type operations, and RDMA supports memory type and message type operations.

[0055] RDMA can transfer data from a storage device's storage space to a host client's storage space as messages in the form of RDMA packets over a front-end network. Conversely, data from a host client's storage space can be transferred to a storage device's storage space as messages in the form of RDMA packets over a front-end network. These transfer and copy operations are primarily performed by the network interface cards (e.g., RDMA network interface cards (RNICs)) of the storage device and the host client, without requiring a processor to execute them. Therefore, they do not affect the performance of the host client.

[0056] Currently, RDMA hardware implementations include Infiniband (IB), RDMA over converged Ethernet (RoCE) (including RoCEv1 and RoCEv2), and the Internet Wide Area RDMA Protocol (iWARP). Infiniband is a network designed specifically for RDMA, guaranteeing reliable transmission at the hardware level. Both RoCE and iWARP are Ethernet-based RDMA technologies that support corresponding verb interfaces. RoCE also provides hardware-level guarantees for reliable transmission.

[0057] like Figure 3 As shown in the figure, RDMA is a transport layer network protocol that uses queue pairs (QPs) in channel adapters (CAs) to send and receive consumer data (as messages). Each QP contains a send queue (SQ) and a receive queue (RQ), where the SQ is responsible for sending messages and the RQ is responsible for receiving messages. Each QP's SQ and RQ can be associated with a completion queue (CQ) respectively. Each QP has its own connection context to maintain its connection status, and the SQ, RQ, and CQ also have their own queue context to maintain their queue usage status.

[0058] The consumer involved in the embodiments of the present application refers to the subject that uses QP to send and receive messages, generally referring to a software process, such as NoF protocol software.

[0059] When sending a message, a consumer submits a work request (WR). The controller driver converts the WR into a work queue element (WQE) in the SQ of the QP of the local node (acting as the requester). The WQE includes the RDMA operation type, the address of the storage space for the message to be transmitted, and the size of the message to be transmitted. The data to be transmitted is a message with a length range of 0 bytes to 4GB. The network interface card segments the data pointed to by the WQE into packets according to the path maximum transmission unit (PMTU) and sends them to the network. When the network interface card completes the transmission of all the contents of the message pointed to by a WQE, it submits a completion queue element (CQE) to the CQ associated with the SQ to notify the consumer that the RDMA operation indicated by the submitted WQE has been completed.

[0060] For receiving SEND messages, before receiving the message, the consumer submits a WQE to the RQ in the QP of this node (as the responder) in advance. The WQE includes a pointer to the storage space that can store data and the size of the storage space that can store data. After the network interface card receives the data from the front-end network, it writes it to the storage space pointed to by the WQE of the RQ. When the network interface card completes receiving the data, it submits a CQE to the CQ associated with the RQ to notify the consumer that it has completed receiving an RDMA message. For receiving RDMA READ response messages, the network interface card writes the received data to the storage space pointed to by the WQE of the corresponding RDMA READ operation, and for receiving the tail packet or single packet of the RDMA READ response message, when the network interface card completes receiving the data, it submits a CQE to the CQ associated with the SQ to notify the consumer that it has completed the RDMA READ operation indicated by a certain WQE it submitted. For the confirmation message of receiving an RDMA WRITE message or RDMA SEND message, the network interface card submits a CQE to the CQ associated with the SQ to notify the consumer that the RDMA WRITE operation or RDMA SEND operation indicated by the submitted WQE has been completed. For the confirmation message of SEND message, RDMA READ response message and RDMA WRITE message, please refer to the following description.

[0061] RDMA supports both reliable and unreliable QP connections. Reliable QP connections only support a one-to-one communication model, meaning a QP on a node can only establish a QP connection with a QP on another node. Any QP cannot simultaneously establish QP connections with multiple QPs on other nodes. For example, Figure 4 As shown, QPx of consumer A on node 1 establishes a QP connection with QPa of consumer B on node 2. QPx of consumer A on node 1 can no longer establish a QP connection with other QPs (e.g., QPb, QPc) at the same time. NoF uses RDMA that supports reliable QP connections for communication.

[0062] The QP connection messages of the reliable connection received in RDMA should comply with the strict order preservation requirement. The responder decides whether to accept the message based on whether the packet sequence numbers (PSN) in the messages received in sequence are strictly increasing in order. Figure 5 As shown in Figure A, the responder will only receive messages when the PSN of the message is strictly increasing (for example, the PSN of the message is equal to n, n+1, n+2, and n+3 in sequence). Figure 5 As shown in Figure B, when the PSN of a message changes (for example, the PSN of a message lacks n+2), the responder will discard all subsequent messages of the QP whose PSN is not strictly increasing. Figure 5 As shown in C, when the PSN of a message is repeated with the PSN of a received message (for example, a message with a PSN of n+1 is received repeatedly), the responding end will discard the repeated message.

[0063] The message type operations supported by RDMA include a send operation (SEND operation), which involves a processor and is applicable to the transmission of control messages (or command messages).

[0064] RDMA supports memory operations including RDMA read and write operations. These operations do not involve the processor; data reading or writing is accomplished by operating the storage space on the network interface card. They are suitable for transmitting data packets (also known as data messages).

[0065] The following describes the send operation, RDMA read operation, and RDMA write operation respectively:

[0066] Send Operation

[0067] Send operations can also be called push operations or having channel semantics. The requester pushes data to the responder without knowing where the data is stored in the responder. The responder's channel adapter places the data into the next available storage space in its queue, as indicated by the WQE at the head of the queue's RQ.

[0068] The requester can initiate a send operation by sending a SEND message, which includes the data to be pushed to the responder. The length of the data can be from 0 bytes to 2 31 (4G) bytes, if the length of the data is greater than the path maximum transmission unit (PMTU), it will be segmented into multiple messages according to the PMTU size, and the responder will then reassemble these messages to obtain the data. For a reliable connection, if the data is a short message (i.e., it does not need to be segmented into multiple messages), the responder sends an acknowledgment (ACK) message to the requester for each message; if the data is a long message (i.e., it is segmented into multiple messages), the responder can send an acknowledgment (ACK) message to the requester for each message, or send an acknowledgment message to the requester for multiple consecutive messages of the same message, or send an acknowledgment message to the requester for the last packet of the message; in addition, regardless of whether the data is a short message or a long message, the responder can send an acknowledgment (ACK) message for multiple messages previously received. For example, an acknowledgment message for a SEND message with a PSN of X can be used to confirm that the message with a PSN less than X before the SEND message has been successfully received by the responder.

[0069] Each SEND message may include a four-byte immediate data (ImmDt). If an immediate data is to be included, it is included in the additional header field (immediate extended transmission header or ImmDt field) of the last message of the SEND message.

[0070] For example, assuming that the SEND message is segmented into three messages, the structure of the SEND message and the confirmation message of the SEND message based on the IB protocol is as follows: Figure 6A As shown, the structure of the SEND message and the confirmation message of the SEND message based on the RoCEv2 protocol is as follows Figure 6B As shown, the interaction process between the requester and responder of a reliable connection is as follows Figure 7 shown.

[0071] The local route header (LRH) field is used for routing through switches in the subnet.

[0072] The global route header (GRH) field is used for routing to destinations outside the sender's local subnet.

[0073] The invariant CRC (ICRC) field covers all fields of the message that should not be changed and performs CRC checks on these fields.

[0074] The variable CRC (VCRC) field covers all fields of the message and performs CRC check on these fields.

[0075] The ACK extended transport header (AETH) includes additional transport fields for the ACK message, which is included in each ACK message of the SEND message.

[0076] The media access control (MAC) field is used for Layer 2 forwarding through switches in the Ethernet subnet.

[0077] The Internet Protocol (IP) field is used for Layer 3 forwarding through the switch.

[0078] The User Datagram Protocol (UDP) field is used to indicate that the payload of the message is an RDMA message.

[0079] The base transport header (BTH) field is used to indicate the destination QP, operation code, packet sequence number (PSN), and partition. The OpCode field in the BTH field determines the start and end of the SEND message:

[0080] For a SEND message, if the data length is less than or equal to the PMTU, the BTH opcode "SENDOnly" or "SEND Only with Immediate" is used, indicating that the message cannot be split (i.e., a single packet).

[0081] If the length of the data is zero, the BTH opcode "SEND Only" or "SEND Only with Immediate" is used, and the data payload field does not exist in the message. For a SEND message with the BTH opcode "SEND Only", other fields such as Figure 6A or Figure 6B As shown in message 1 in the BTH operation code for the SEND message "SEND Only with Immediate", other fields are as follows Figure 6A or Figure 6B As shown in message 3 in .

[0082] If the data length is greater than the PMTU, the BTH operation code of the first packet is "SEND First", indicating that the packet type is the first packet; the BTH operation code of the middle packet is "SEND Middle", indicating that the packet type is the middle packet; the BTH operation code of the last packet is "SEND Last" or "SEND Last with Immediate", indicating that the packet type is the last packet.

[0083] For the confirmation message of the SEND message, the BTH operation code is "Acknowledge".

[0084] For a given requester's QP, once a multi-packet send operation is initiated, no other SEND messages, RDMA WRITE messages, or RDMA READ request messages can be sent before a message with the operation code "SEND Last" or "SEND Last with Immediate".

[0085] The responder does not know the final length of the data until the last message with the "SEND Last" or "SEND Last with Immediate" opcode arrives.

[0086] It should be noted that after the responding end completes the reception of a single packet or the last packet of the SEND message, it reports the CQE to the CQ associated with the RQ of this node. After the requesting end completes the sending process of the sending operation (receiving the confirmation message corresponding to the single packet or the last packet of the SEND message), it submits the CQE to the CQ associated with the SQ of this node.

[0087] RDMA write operations

[0088] The RDMA write operation is used by the requesting side to write data into the storage space of the responding side.

[0089] Before allowing the requester to perform an RDMA write operation, the responder first allocates a storage space for access by the responder's QP (or QP group). The responder's channel adapter associates a 32-bit M_Key with the virtual address of this storage space. The responder sends the virtual address, length, and M_Key of this storage space to the requester, which is authorized to access the memory area. For example, this information can be sent to the requester using the send operation described above.

[0090] The requesting end can initiate an RDMA write operation by sending an RDMA WRITE message, which includes the data to be written to the responding end, the virtual address of the responding end's storage space, the length of the data, and the M_Key key. The length of the data can be from 0 bytes to 2 31 (4G) bytes, similar to the sending operation, if the length of the data is greater than the PMTU, it will be segmented into multiple messages according to the PMTU size, and the responder will then reassemble these messages to obtain the data. For a reliable connection, if the data is a short message (i.e., it does not need to be segmented into multiple messages), the responder sends an acknowledgment (ACK) message to the requester for each message; if the data is a long message (i.e., it is segmented into multiple messages), the responder can send an acknowledgment (ACK) message to the requester for each message, or send an acknowledgment message to the requester for multiple consecutive messages of the same data, or send an acknowledgment message to the requester for the last packet; in addition, regardless of whether the data is a short message or a long message, the responder can send an acknowledgment (ACK) message for multiple messages received previously. For example, an acknowledgment message for an RDMA WRITE message with a PSN of X can be used to confirm that the message with a PSN less than X before the RDMA WRITE message has been successfully received by the responder.

[0091] For example, assuming that the RDMA WRITE message is segmented into three messages, the structures of the RDMA WRITE message and the confirmation message of the RDMA WRITE message based on the IB protocol are as follows: Figure 8A As shown in the figure, the structure of the RDMA WRITE message and the confirmation message of the RDMA WRITE message based on the RoCEv2 protocol is as follows Figure 8B The interaction process between the requester and responder of a reliable connection is as follows. Figure 9 shown.

[0092] The RDMA extended transport header (RETH) includes additional transport fields for RDMA operations. For example, these include the virtual address of the target storage space, the length of the RDMA operation data, and the M_Key key. The responder uses the M_Key key to determine whether the requester has permission to access the corresponding virtual address and stores the data indicated by the length in the storage space pointed to by the virtual address.

[0093] Similar to the SEND operation, the OpCode field in the BTH field determines the start and end of the RDMA WRITE message:

[0094] For an RDMA WRITE message, if the data length is less than or equal to the PMTU, the BTH operation code "RDMA WRITE Only" or "RDMA WRITE Only with Immediate" is used, indicating that the message is not split (i.e., a single packet).

[0095] If the length of the data is zero, the BTH opcode "RDMA WRITE Only" or "RDMA WRITE Onlywith Immediate" is used, and the data payload field does not exist in the message. For the RDMA WRITE message with the BTH opcode "RDMA WRITE Only", other fields such as Figure 8A or Figure 8B As shown in message 1 in FIG; For the RDMA WRITE message with the BTH operation code of "RDMA WRITEOnly with Immediate", as shown in FIG Figure 8A or Figure 8B The ImmDt field is added after the RETH field of message 1, and the other fields are as shown in message 1.

[0096] If the data length is greater than the PMTU, the BTH operation code of the first packet is "RDMA WRITE First", indicating that the packet type is the first packet; the BTH operation code of the middle packet is "RDMA WRITE Middle", indicating that the packet type is the middle packet; the BTH operation code of the last packet is "RDMA WRITE Last" or "RDMA WRITE Last with Immediate", indicating that the packet type is the last packet.

[0097] For the confirmation message of the RDMA WRITE message, the BTH operation code is "Acknowledge".

[0098] RDMA read operations

[0099] The RDMA read operation is the reverse operation of the RDMA write operation, and is used by the requesting side to read data from the storage space of the responding side.

[0100] Before allowing the requester to perform an RDMA read operation, the responder first allows the requester to access its virtual address, that is, sends the virtual address, length, and M_Key of the data to be read by the RDMA read operation to the requester. For example, the above information can be sent to the requester through the send operation described above.

[0101] The requesting end can initiate an RDMA read operation by sending an RDMA READ Request message, which includes the virtual address of the responding end's storage space, the length of the RDMA operation data, and the M_Key key. The responding end sends an RDMA READ Response message to the requesting end, which includes the read data. The length of the data to be read can be from 0 bytes to 2 31 If the length of a packet is greater than the PMTU (4G), it will be segmented into multiple packets according to the PMTU size for transmission via multiple RDMA READ response messages. The requester then reassembles these packets to obtain the data. For reliable connections, the RDMA READ response message does not have a corresponding acknowledgment message. If the requester detects that the RDMA READ response message has been lost, it will re-initiate the RDMA read operation.

[0102] For example, assuming that the RDMA READ response message is segmented into three messages, the structures of the RDMA READ request message and the RDMA READ response message based on the IB protocol are as follows: Figure 10A As shown in the figure, the structure of the RDMA READ request message and RDMA READ response message based on the RoCEv2 protocol is as follows: Figure 10B The interaction process between the requester and responder of a reliable connection is as follows. Figure 11 shown.

[0103] The OpCode field in the BTH field determines the start and end of the message (RDMA READ request message or RDMAREAD response message):

[0104] For an RDMA READ request message, the BTH operation code is "RDMA READ Request".

[0105] For an RDMA READ response message, if the data length is less than or equal to the PMTU, the BTH operation code "RDMA READ Response Only" is used, indicating that the message cannot be split (ie, a single packet).

[0106] If the length of the data is zero, the BTH opcode "RDMA READ Response Only" is used, and the data payload field is not present in the RDMA read response message, but all other fields are present as shown in the following example: Figure 10A or Figure 10B As shown in message 1 in .

[0107] If the data length is greater than the PMTU, the BTH operation code of the first packet is "RDMA READ ResponseFirst", indicating that the packet type is the first packet; the BTH operation code of the middle packet is "RDMA READ ResponseMiddle", indicating that the packet type is the middle packet; the BTH operation code of the last packet is "RDMA READ ResponseLast", indicating that the packet type is the last packet.

[0108] The host client and storage device use the above RDMA operation to complete the IO reading and writing process as follows Figure 12 As shown:

[0109] The host client first discovers the storage device in the network and establishes a connection with the storage device.

[0110] During the IO write process between the host client and the storage device, the host client sends a SEND message to the storage device. The data load of the message is the IO write command. For a reliable connection, the storage device can send a confirmation message of the SEND message to the host client, indicating that the IO write command has been received. Then the storage device (as the requesting end) sends an RDMA READ request message to the host client (as the responding end), and the host client sends an RDMA READ response message to the storage device. The data load of the message is the IO data. After the storage device performs persistent storage on the IO data, that is, after storing the IO data on a storage medium such as a hard disk drive (HDD) or a solid state drive (SSD), it sends a SEND message to the host client. The message includes an IO response, indicating that the IO write process is completed. The host client sends a confirmation message of the SEND message to the storage device, indicating that the IO response has been received.

[0111] During the IO read process between the host client and the storage device, the host client sends a SEND message to the storage device. The data payload of the message is the IO read command. For a reliable connection, the storage device can send a confirmation message of the SEND message to the host client, indicating that the IO read command has been received. After the storage device obtains the IO data, that is, after obtaining the IO data from a storage medium such as an HDD or SSD, the storage device (as the requesting end) sends an RDMA WRITE message to the host client (as the responding end). The data payload of the message is the IO data. The host client sends a confirmation message of the RDMA WRITE message to the storage device. The storage device sends a SEND message to the host client. The message includes an IO response, indicating that the IO read process is completed. The host client sends a confirmation message of the SEND message to the storage device, indicating that the IO response has been received.

[0112] As can be seen from this, the complete IO write process uses the SEND operation to transmit command messages (such as command requests and command responses) and the RDMA READ operation to transmit data messages. The complete IO read process uses the SEND operation to transmit command messages and the RDMA WRITE operation to transmit data messages.

[0113] As previously mentioned, in an RDMA read operation, the host client, acting as the requester, sends an I / O read command to the storage device. The storage device, acting as the responder, receives the I / O read command and distributes it to different controllers for parallel processing via the storage device's NIC. Each controller then sends the I / O data and I / O response to the host client via the NIC.

[0114] Each controller 111 corresponds to a Query Processor (QP), each QP includes a SQ, and the SQ includes a WQE. When a controller has a message (IO data, IO response) to send, the controller adds one or more WQEs to the SQ of the corresponding QP. The WQE points to the storage space of the message to be transmitted. If the message to be transmitted is large, it will be split into multiple WQEs, so multiple scheduling is required to complete the transmission. In addition, the controller generates a doorbell (DB) signal and sends it to the NIC. The DB signal is used to indicate that there is a message to be sent in the storage space pointed to by at least one WQE of the SQ corresponding to the controller.

[0115] The NIC aggregates doorbell signals from various controllers and does not directly trigger the sending of messages. Instead, it first filters the DBs of the same SQ, ultimately retaining only one DB for each SQ. It then selects a doorbell signal from these DBs to respond to based on preset rules. This application does not limit the specific preset rules. For example, the NIC can poll various controllers using round robin (RR), weighted round robin (WRR), dynamic weighted round robin (DWRR), and other methods.

[0116] For example, Figure 13 As shown, assuming that SQ0 of controller 0 has four WQEs (i.e., there are messages in four storage spaces to be transmitted), the depth of the NIC doorbell signal DB0 for SQ0 is 4; SQ1 of controller 1 has three WQEs (i.e., there are messages in three storage spaces to be transmitted), the depth of the NIC doorbell signal DB0 for SQ1 is 3; SQ2 of controller 2 has two WQEs (i.e., there are messages in two storage spaces to be transmitted), the depth of the NIC doorbell signal DB2 for SQ2 is 2.

[0117] According to the RR principle, in the first round of scheduling, the NIC schedules the WQEs of controllers 0, 1, and 2 in turn, and sends messages 0-1, 1-1, and 2-1 in the storage spaces pointed to by the corresponding WQEs. As a result, the depth of DB0 becomes 3, the depth of DB1 becomes 2, and the depth of DB2 becomes 1.

[0118] In the second round of scheduling, the NIC schedules the WQEs of controller 0, controller 1, and controller 2 in turn, and sends messages 0-2, 1-2, and 2-2 in the storage space pointed to by the corresponding WQEs. Then, the depth of DB0 becomes 2, the depth of DB1 becomes 1, and the depth of DB2 becomes 0.

[0119] Since the depth of DB2 is 0, indicating that there are no messages to be sent, in the third round of scheduling, the NIC schedules the WQEs of controller 0 and controller 1 in turn, and sends messages 0-3 and 1-3 in the storage space pointed to by the corresponding WQEs. Then the depth of DB0 becomes 1 and the depth of DB1 becomes 0.

[0120] Since the depth of DB1 is 0, indicating that there are no messages to be sent, in the fourth round of scheduling, the NIC schedules the WQE of controller 0 and sends messages 0-4 in the storage space pointed to by the corresponding WQE. Then the depth of DB0 becomes 0, indicating that there are no messages to be sent.

[0121] In the above polling scheduling method, since the IO data and IO response of the same control are not transmitted in one round of scheduling, the IO response can only be sent out after the next round. Figure 13 In the example, SQ0's IO data 0-1 and IO response 0-2 are separated by SQ1's IO data 1-1 and SQ2's IO data 2-1. The host client can only determine that the RDMA read operation is completed after receiving the IO response, which increases the latency of the RDMA read operation and causes system performance degradation.

[0122] The embodiment of the present application provides a storage device, such as Figure 14 As shown, in Figure 1 Based on the storage device shown in FIG, the controller 111 of the storage device 11 includes a first processing module 1111 and a first transceiver module 1112, and the NIC 112 of the storage device 11 includes a second processing module 1121 and a second transceiver module 1122. Figure 1 The relevant description of is not repeated here. The storage device 11 is used to execute Figure 15 The message sending method shown in FIG. 3 includes:

[0123] S1501 : The host client 12 sends an IO read command to the storage device 11 .

[0124] Accordingly, the storage device 11 receives an IO read command from the host client 12 . The IO read command is used to request IO data. Specifically, the command may be executed by the second transceiver module 1112 of the NIC 112 .

[0125] S1502 : The NIC 112 of the storage device 11 distributes the IO read command to the first controller 111 among the multiple controllers 111 .

[0126] Accordingly, the first controller 111 of the storage device 11 receives the IO read command from the NIC 112 .

[0127] Specifically, the second transceiver module 1122 of the NIC 112 of the storage device 11 can distribute the IO read command to the first controller 111 based on the load balancing principle to achieve parallel processing. Accordingly, the first transceiver module 1112 of the first controller 111 receives the IO read command from the NIC 112. For example, if the first controller is processing two IO read commands and the second controller is processing three IO read commands, the NIC 112 can distribute the new IO read command to the first controller that is processing two IO read commands.

[0128] S1503 : The first controller 111 submits a first WQE to the first SQ of the first controller 111 according to the IO read command, and sends a doorbell signal DB to the NIC 112 .

[0129] Specifically, the first processing module 1111 of the first controller 111 submits a first WQE to the first send queue SQ according to the IO read command, and the first transceiver module 1112 of the first controller 111 sends a doorbell signal DB to the NIC 112. The doorbell signal is used to indicate that there is a message to be sent in the storage space pointed to by at least one WQE in the first SQ of the first controller 111.

[0130] The first processing module 1111 can search the storage medium for IO data requested by the host client based on the IO read command, segment the IO data based on the size of the IO data to be sent, and then submit a first WQE to the first SQ. IO data with a length greater than the PMTU is segmented into three types of RDMA packets: a first packet, a middle packet, and a tail packet. IO data with a length less than or equal to the PMTU is not segmented.

[0131] Accordingly, the NIC 112 receives the doorbell signal DB from the first controller 111 , and when polling the first controller 111 , obtains the first WQE from the first SQ corresponding to the first controller in response to the doorbell signal DB of the first controller 111 .

[0132] Specifically, the second transceiver module 1122 of the NIC 112 receives the doorbell signal DB from the first controller 111 . When polling the first controller 111 , the second processing module 1121 of the NIC 112 obtains the first WQE from the first SQ.

[0133] The first WQE includes indication information, where the indication information is used to indicate a type of the first RDMA packet scheduled by the first WQE. The type of the first RDMA packet may include a first packet, a middle packet, a last packet, or a single packet of IO data.

[0134] In the embodiment of the present application, an RDMA message is segmented to obtain multiple RDMA packets. The first packet refers to the first RDMA packet in the multiple RDMA packets, the last packet refers to the last RDMA packet in the multiple RDMA packets, and the middle packet refers to the other RDMA packets in the multiple RDMA packets. A single packet refers to an RDMA packet that is not segmented, that is, the RDMA packet is a complete RDMA message.

[0135] For example, the first packet refers to an RDMA message whose BTH operation code is "SEND First", "RDMAWRITE First", or "RDMAREAD Response First".

[0136] An intermediate packet refers to an RDMA message with a BTH operation code such as "SEND Middle", "RDMA WRITE Middle", or "RDMAREAD Response First".

[0137] The tail packet refers to an RDMA message with a BTH operation code such as "SEND Last", "SEND Last with Immediate", "RDMA WRITE Last", "RDMA WRITE Last with Immediate", or "RDMA READ Response Last".

[0138] Single packet refers to the RDMA message with the BTH operation code of "SEND Only", "SEND Only with Immediate", "RDMA WRITE Only", "RDMA WRITE Only with Immediate", or "RDMA READ Response Only".

[0139] Exemplarily, the indication information is first data WR, which can be used to indicate that the type of the first RDMA message is the first packet of IO data; the indication information is middle data WR, which can be used to indicate that the type of the first RDMA message is the middle packet of IO data; the indication information is last data WR, which can be used to indicate that the type of the first RDMA message is the first packet of IO data; and the indication information is only data WR, which can be used to indicate that the type of the first RDMA message is a single packet of IO data.

[0140] S1504: If the indication information indicates that the type of the first RDMA packet is the first packet or the intermediate packet of IO data, the NIC 112 sends the first RDMA packet to the host client.

[0141] This step may be performed by the second transceiver module 1112 of the NIC 112 .

[0142] S1505. If the indication information indicates that the type of the first RDMA packet is a tail packet or a single packet of IO data, then before polling the SQ of the next controller, the NIC 112 sends the first RDMA packet to the host client, as well as the second RDMA packet scheduled by the next WQE of the first WQE in the first SQ.

[0143] Specifically, if the indication information indicates that the type of the first RDMA message is the tail packet or single packet of IO data, then before the second processing module 1121 polls the SQ of the next controller, the second transceiver module 1112 sends the first RDMA message to the host client, as well as the second RDMA message scheduled by the next WQE of the first WQE in the first SQ.

[0144] For the same controller, after submitting a WQE for the tail or single packet of IO data to the first SQ, the first controller will immediately submit a WQE for the IO response to the first SQ. Therefore, if the indication information in the first WQE indicates that the type of the first RDMA packet scheduled by the first WQE is the tail or single packet of IO data, then the second RDMA packet scheduled by the next WQE after the first WQE in the first SQ is the IO response. In other words, before the NIC polls the next SQ, the IO response of the first SQ is sent to the host client immediately following the IO data of the first SQ, allowing the host client to quickly confirm the completion of the RDMA read operation.

[0145] S1506: The host client receives the IO response and completes the RDMA read operation.

[0146] like Figure 16 As shown, assuming that the length of each RDMA message is calculated as 8KB, and assuming that the sending time of the RDMA message of IO data is 3 times that of the RDMA message of IO response, that is, the sending time of the RDMA message of IO response is 1, and the sending time of the RDMA message of IO data is 3.

[0147] For example, in relation to Figure 13 Before the improvement, the total sending time from controller 0's IO data (0-1) to IO response (0-2) was 10. After the improvement using the message sending method according to the embodiment of the present application, the total sending time is reduced to 4. It can be seen that the completion time of the improved RDMA read operation is significantly reduced. The average delay before the improvement is 15.25, and the average delay after the improvement is 12.25, with a performance improvement of 20%. Similarly, the total sending time from controller 2's IO data (2-1) to IO response (2-2) is 8. After the improvement, the total sending time is reduced to 4. The total sending time from controller 1's IO data (1-1) and IO data (1-2) to IO response (3-3) is 15. After the improvement, the total sending time is 15. The total sending time from controller 0's IO data (0-3) to IO response (0-4) is 5. After the improvement, the total sending time is reduced to 4.

[0148] The more controllers there are and the longer the RDMA message length is, the more obvious the performance improvement will be.

[0149] The above-mentioned network interface card, controller, storage device and message sending method provided by the embodiments of the present application are as follows: the network interface card (NIC) of the storage device obtains a first WQE from the first SQ of the first controller, and the first WQE includes indication information, and the indication information is used to indicate the type of the first RDMA message scheduled by the first WQE. If the indication information indicates that the type of the first RDMA message is the tail packet or single packet of IO data, before polling the SQ of the next controller, the first RDMA message and the second RDMA message (i.e., the RDMA message of the IO response) scheduled by the next WQE of the first WQE in the first SQ are sent to the host client. This allows the host client to quickly receive the corresponding IO response after receiving the tail packet or single packet of IO data, and determine that the RDMA read operation is completed without having to wait for the NIC to poll the first controller next time, thereby reducing the latency of the RDMA read operation.

[0150] In addition, each WQE submitted by the controller to the SQ contains the verification information DIF of the RDMA message. Therefore, the NIC must verify the data integrity function (DIF) of the RDMA message before sending the RDMA message.

[0151] Optional, such as Figure 17 As shown, step S1505 may include steps S15051-S15053.

[0152] S15051. The NIC performs a DIF check on the first RDMA message.

[0153] DIF verification is to protect the integrity and consistency of data by adding verification data and other information to the data.

[0154] Specifically, this step may be performed by the second processing module 1121 of the NIC 112 .

[0155] S15052. If the indication information indicates that the type of the first RDMA message is the tail packet or single packet of IO data, and the DIF check passes, the NIC sends the first RDMA message and the second RDMA message scheduled by the next WQE of the first WQE in the first SQ to the host client before polling the SQ of the next controller.

[0156] Specifically, if the indication information indicates that the type of the first RDMA message is the tail packet or single packet of IO data, and the second processing module 1121 passes the DIF check, the second transceiver module 1122 sends the first RDMA message to the host client before the second processing module 1121 polls the SQ of the next controller, as well as the second RDMA message scheduled by the next WQE of the first WQE in the first SQ.

[0157] S15053: If the DIF check fails, the NIC sends a third RDMA message to the host client, or does not send the first RDMA message and the second RDMA message.

[0158] The third RDMA message indicates that the first RDMA message fails the DIF check.

[0159] Specifically, if the second processing module 1121 fails the DIF check, the second transceiver module 1122 sends a third RDMA message to the host client. The third RDMA message indicates that the first RDMA message failed the DIF check, meaning that the NIC directly notifies the host client of the failure of the RDMA read operation. Alternatively, the second transceiver module 1122 does not send either the first or second RDMA message to the host client, and the host client timer times out, thereby determining that the RDMA read operation has failed.

[0160] This optional implementation may perform a DIF check on the RDMA message before sending the RAMD message.

[0161] The present invention also provides a computer-readable storage medium in which a computer program is stored. When the computer-readable storage medium is run on a computer or a processor, the computer or processor executes Figure 15 The method corresponding to the NIC or the first controller.

[0162] The present application also provides a computer program product comprising instructions, which, when executed on a computer or processor, causes the computer or processor to execute Figure 15 The method corresponding to the NIC or the first controller.

[0163] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0164] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0165] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0166] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0167] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0168] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0169] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode to another website, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more media that can be integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0170] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A network interface card, characterized in that: include: a second processing module, configured to obtain a first WQE from a first SQ corresponding to the first controller in response to a doorbell signal from the first controller, wherein the doorbell signal is used to indicate that there is a message to be sent in a storage space pointed to by at least one WQE in the first SQ; the first WQE includes indication information, wherein the indication information is used to indicate a type of a first RDMA message scheduled by the first WQE; a second transceiver module, configured to send the first RDMA message and a second RDMA message scheduled by a next WQE of the first WQE in the first SQ to the host client before the second processing module polls the SQ of the next controller if the indication information indicates that the type of the first RDMA message is a tail packet or a single packet of IO data.

2. The network interface card according to claim 1, wherein: The second processing module is further configured to perform a data integrity function (DIF) check on the first RDMA message; The second transceiver module is specifically configured to: if the indication information indicates that the type of the first RDMA message is the tail packet or a single packet of the IO data, and the DIF check passes, then before the second processing module polls the SQ of the next controller, send the first RDMA message to the host client, as well as the second RDMA message scheduled by the next WQE of the first WQE in the first SQ.

3. The network interface card according to claim 2, wherein: The second transceiver module is further configured to: If the DIF check fails, a third RDMA message is sent to the host client, where the third RDMA message indicates that the first RDMA message fails the DIF check.

4. The network interface card according to any one of claims 1 to 3, wherein: The type of the first RDMA packet includes a first packet, a middle packet, a last packet or a single packet of IO data.

5. A controller, characterized in that: include: A first processing module, configured to submit a first WQE to a first SQ, where the first WQE includes indication information, where the indication information is used to indicate a type of a first RDMA packet scheduled by the first WQE; The first transceiver module is further configured to send a doorbell signal to the network interface card, where the doorbell signal is configured to indicate that there is a message to be sent in the storage space pointed to by at least one WQE in the first SQ; the doorbell signal is further configured to instruct the second processing module of the network interface card to obtain a first WQE from the first SQ; wherein, when the indication information indicates that the type of the first RDMA message is a tail packet or a single packet of IO data, the second transceiver module of the network interface card is configured to send the first RDMA message to the host client, as well as a second RDMA message scheduled by the next WQE of the first WQE in the first SQ, before the second processing module polls the SQ of the next controller.

6. The controller according to claim 5, characterized in that The type of the first RDMA packet includes a first packet, a middle packet, a last packet or a single packet of IO data.

7. A storage device, characterized in that: The method comprises the network interface card according to any one of claims 1 to 4 and the controller according to any one of claims 5 to 6.

8. A message sending method, characterized in that: include: Responding to a doorbell signal from a first controller, obtaining a first WQE from a first SQ corresponding to the first controller, wherein the doorbell signal is used to indicate that a message is to be sent in a storage space pointed to by at least one WQE in the first SQ; the first WQE includes indication information, wherein the indication information is used to indicate a type of a first RDMA message scheduled by the first WQE; If the indication information indicates that the type of the first RDMA message is a tail packet or a single packet of IO data, before polling the SQ of the next controller, the first RDMA message and the second RDMA message scheduled by the next WQE of the first WQE in the first SQ are sent to the host client.

9. The method according to claim 8, characterized in that If the indication information indicates that the type of the first RDMA message is a tail packet or a single packet of IO data, before polling the SQ of the next controller, sending the first RDMA message to the host client, and a second RDMA message scheduled by the next WQE of the first WQE in the first SQ, including: Performing a data integrity function (DIF) check on the first RDMA message; If the indication information indicates that the type of the first RDMA message is the tail packet or a single packet of the IO data, and the DIF check passes, before polling the SQ of the next controller, the first RDMA message and the second RDMA message scheduled by the next WQE of the first WQE in the first SQ are sent to the host client.

10. The method according to claim 9, characterized in that Also includes: If the DIF check fails, a third RDMA message is sent to the host client, where the third RDMA message indicates that the first RDMA message fails the DIF check.

11. The method according to any one of claims 8 to 10, characterized in that: The type of the first RDMA packet includes a first packet, a middle packet, a last packet or a single packet of IO data.

12. A message sending method, characterized in that: include: Submitting a first WQE to a first SQ, where the first WQE includes indication information, where the indication information is used to indicate a type of a first RDMA packet scheduled by the first WQE; Sending a doorbell signal to the network interface card, where the doorbell signal is used to indicate that there is a message to be sent in the storage space pointed to by at least one WQE in the first SQ, and the doorbell signal is also used to instruct the second processing module of the network interface card to obtain a first WQE from the first SQ; wherein, when the indication information indicates that the type of the first RDMA message is a tail packet or a single packet of IO data, the second transceiver module of the network interface card is used to send the first RDMA message to the host client before the second processing module polls the SQ of the next controller, as well as a second RDMA message scheduled by the next WQE of the first WQE in the first SQ.

13. The method according to claim 12, characterized in that The type of the first RDMA packet includes a first packet, a middle packet, a last packet or a single packet of IO data.

Images