NVMe-over-RDMA connection establishment methods, devices, equipment and media

Abstract

This disclosure relates to a method, apparatus, device, and medium for establishing an NVMe-over-RDMA connection. The method includes: creating a management queue and creating a first queue pair based on the software resources of a network interface card (NIC), wherein management commands stored in the first queue pair are sent and / or received through the NIC based on a preset source port number; configuring the NIC and the first queue pair based on the preset source port number; initiating a first connection establishment request to a server based on the first queue pair; creating an IO queue and creating a second queue pair based on RDMA hardware resources, wherein IO commands stored in the second queue pair are sent and / or received through RDMA hardware; and initiating a second connection establishment request to the server based on the second queue pair. This disclosure, by using the software resources of the NIC to create the RDMA queue pair corresponding to the management queue, avoids the waste caused by the management queue occupying RDMA hardware resources, thus improving hardware stability.

Description

Technical Field

[0001] This disclosure relates to the field of NVMe-over-RDMA technology, and in particular to a method, apparatus, device and medium for establishing NVMe-over-RDMA connection. Background Technology

[0002] NVMe (Non-Volatile Memory Express) is an interface protocol that uses the PCIe bus to communicate directly with fast, non-volatile storage devices, improving data transfer rates and storage performance to meet the high-performance storage needs of modern data centers. RDMA (Remote Direct Memory Access) is a direct memory access technology that allows hosts to directly access remote memory without the need for remote CPU intervention, reducing access latency. NVMe-over-RDMA is a network protocol that combines the two, using RDMA network technology to enable hosts to access external storage resources while retaining the low latency and high performance characteristics of NVMe, enhancing system flexibility and scalability.

[0003] In the NVMe-over-RDMA protocol, clients and servers establish a connection via RDMA network technology to enable remote access to NVMe storage resources. Within this connection framework, the management queue handles only a small number of admin commands, while the IO queue handles intensive IO commands. However, both the management and IO queues require RDMA QPs (Queue Pairs) and consume RDMA hardware resources, resulting in inefficient and insufficient utilization of these resources. This is particularly problematic in scenarios providing network storage for large-scale, small-sized virtual machines, where the management queue consumes a significant proportion of the RDMA hardware resources in the entire system, leading to substantial resource waste. Furthermore, excessive RDMA hardware resource consumption can negatively impact RDMA hardware stability. Therefore, optimizing the connection method between clients and servers to reduce RDMA hardware resource waste and improve hardware stability is a critical technical challenge that needs to be addressed. Summary of the Invention

[0004] To address the aforementioned technical issues, this disclosure provides an NVMe-over-RDMA connection establishment method, apparatus, device, and medium.

[0005] A first aspect of this disclosure provides an NVMe-over-RDMA connection establishment method, applicable to a client, the method comprising:

[0006] In response to the detection of a connection command, a management queue is created, and a first queue pair is created based on the network card's software resources. The first queue pair is an RDMA queue pair corresponding to the management queue. The management commands stored in the first queue pair are sent and / or received through the network card based on a preset source port number.

[0007] Configure the network card and the first queue pair based on the preset source port number;

[0008] Based on the first queue, the first connection establishment request is initiated to the server to negotiate and establish a communication connection corresponding to the management queue;

[0009] Create an IO queue and create a second queue pair based on RDMA hardware resources. The second queue pair is the RDMA queue pair corresponding to the IO queue. The IO commands stored in the second queue pair are sent and / or received through RDMA hardware.

[0010] Based on the second queue, a second connection establishment request is initiated to the server to negotiate and establish a communication connection corresponding to the IO queue.

[0011] A second aspect of this disclosure provides an NVMe-over-RDMA connection establishment apparatus, which is suitable for a client and includes:

[0012] The first creation module is used to create a management queue in response to the detection of a connection command, and to create a first queue pair based on the network card's software resources. The first queue pair is an RDMA queue pair corresponding to the management queue. The management commands stored in the first queue pair are sent and / or received through the network card based on a preset source port number.

[0013] The configuration module is used to configure the network interface card and the first queue pair based on the preset source port number;

[0014] The first initiation module is used to initiate a first connection establishment request to the server based on the first queue, and negotiate to establish a communication connection corresponding to the management queue.

[0015] The second creation module is used to create an IO queue and create a second queue pair based on RDMA hardware resources. The second queue pair is the RDMA queue pair corresponding to the IO queue. The IO commands stored in the second queue pair are sent and / or received through RDMA hardware.

[0016] The second initiation module is used to initiate a second connection establishment request to the server based on the second queue, and negotiate to establish a communication connection corresponding to the IO queue.

[0017] A third aspect of this disclosure provides a computer device including a memory and a processor, and a computer program, wherein the memory stores the computer program, and when the computer program is executed by the processor, it implements the NVMe-over-RDMA connection establishment method as described in the first aspect above.

[0018] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the NVMe-over-RDMA connection establishment method as described in the first aspect above.

[0019] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0020] In the NVMe-over-RDMA connection establishment method, apparatus, device, and medium provided in this disclosure embodiment, a management queue is created in response to the detection of a connection command, and a first queue pair is created based on the network interface card (NIC) software resources. The first queue pair is an RDMA queue pair corresponding to the management queue. The management commands stored in the first queue pair are sent and / or received through the NIC based on a preset source port number. The NIC and the first queue pair are configured based on the preset source port number. A first connection establishment request is initiated to the server based on the first queue pair to negotiate and establish a communication connection corresponding to the management queue. An IO queue is created, and a second queue pair is created based on RDMA hardware resources. The second queue pair is an RDMA queue pair corresponding to the IO queue. The IO commands stored in the second queue pair are sent and / or received through the RDMA hardware resources. When sending and / or receiving data, a second connection establishment request is initiated to the server based on the second queue pair. The communication connection corresponding to the IO queue is negotiated and established. The network card software resources can be used to create the RDMA queue pair corresponding to the management queue, and the RDMA hardware resources can be used to establish the RDMA queue corresponding to the IO queue. The network card and the RDMA queue pair corresponding to the management queue are configured through the preset source port number. Finally, the communication connection corresponding to the management queue and the IO queue between the client and the server is established. In this way, while ensuring efficient data interaction of the IO queue, the RDMA hardware resources occupied by the management queue are released and can continue to be used as the IO queue, thereby improving the effective utilization rate of RDMA hardware resources and reducing the overall occupancy rate of RDMA hardware resources, thus improving the stability of RDMA hardware. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0022] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of an NVMe-over-RDMA connection establishment method provided in an embodiment of this disclosure;

[0024] Figure 2 This is a flowchart of a method for creating a first queue pair provided in an embodiment of this disclosure;

[0025] Figure 3 This is a flowchart of a method for forwarding data packets provided in an embodiment of this disclosure;

[0026] Figure 4 This is a schematic diagram of the structure of an NVMe-over-RDMA connection establishment device provided in an embodiment of this disclosure;

[0027] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure. Detailed Implementation

[0028] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0029] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0030] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0031] Figure 1 This is a flowchart illustrating an NVMe-over-RDMA connection establishment method provided in this disclosure. This method can be executed by an NVMe-over-RDMA connection establishment device, which can be implemented in software and / or hardware. This NVMe-over-RDMA connection establishment device can be configured in an electronic device acting as a client. Figure 1 As shown, the NVMe-over-RDMA connection establishment method provided in this embodiment includes the following steps:

[0032] S101. In response to the detection of a connection command, a management queue is created, and a first queue pair is created based on the network card's software resources. The first queue pair is the RDMA queue pair corresponding to the management queue. The management commands stored in the first queue pair are sent and / or received through the network card based on a preset source port number.

[0033] The management queue in this embodiment can be understood as a queue space used to submit management commands to the controller in the Non-Volatile Memory Express (NVMe) specification, including a submission queue (SQ) and a completion queue (CQ).

[0034] The RDMA queue pair in this embodiment can be understood as the basic unit of communication in Remote Direct Memory Access (RDMA), which may include a send queue and a receive queue. In NVMe over RDMA, the NVMe management queue and IO queue are carried by the RDMA queue pair.

[0035] The preset source port number in this embodiment can be understood as a port number selected in advance from port numbers that are not commonly used in daily life. For example, the preset source port number can be 65535.

[0036] In this embodiment of the present disclosure, the NVMe-over-RDMA connection establishment device can create a management queue in NVMe after detecting a connection command issued by a user, and use the network card's software resources to call the RDMA queue creation interface to create an RDMA queue pair corresponding to the management queue, and determine it as the first queue pair. The first queue pair may include a first transmit queue and a first receive queue. The management commands stored in the first transmit queue and the first receive queue are sent or received based on a preset source port number. Specifically, the source port number when the management commands stored in the first transmit queue and the first receive queue are sent and / or received may be a preset source port number.

[0037] S102. Configure the network card and the first queue pair based on the preset source port number.

[0038] In this embodiment of the present disclosure, the NVMe-over-RDMA connection establishment device can configure the network card according to a preset source port number, so that the network card can determine the processing method of the data packet according to the source port number in the data packet when sending and receiving data packets. Specifically, the network card can be configured to send data packets in the send queue of the first queue pair using the preset source port number, and / or, when the source port number of the data packet sent by the server is the preset source port number, the data packet is stored in the receive queue of the first queue pair. The NVMe-over-RDMA connection establishment device can also configure the first queue pair according to the preset source port number. Specifically, the send queue of the first queue pair can be configured to set the source port number of the management command in the first queue pair to the preset source port number.

[0039] It should be noted that, in this embodiment of the disclosure, the execution order of creating the management queue and creating the first queue pair based on the network card's software resources in step S101 and step S102 is not limited. They can be executed simultaneously, or the creation of the management queue and the first queue pair can be executed before step S102, or the creation of the management queue and the first queue pair can be executed after step S102.

[0040] S103. Based on the first queue, initiate the first connection establishment request to the server and negotiate to establish a communication connection corresponding to the management queue.

[0041] In this embodiment of the disclosure, the NVMe-over-RDMA connection establishment device can initiate a first connection establishment request to the server based on the first queue pair after the management queue and the first queue pair are created and the network card and the first queue pair are configured. Specifically, the first queue pair can be used to send various parameters required to establish the communication connection corresponding to the management queue to the server. The parameters are packaged and stored in the sending queue of the first queue pair and sent to the server through the sending queue. The data packet returned by the server is then stored in the receiving queue of the first queue pair and parsed. Finally, the communication connection corresponding to the management queue is negotiated and established.

[0042] S104. Create an IO queue and create a second queue pair based on RDMA hardware resources. The second queue pair is the RDMA queue pair corresponding to the IO queue. The IO commands stored in the second queue pair are sent and / or received through RDMA hardware.

[0043] The IO queue in this disclosure embodiment can be understood as a queue space used to submit IO commands to the controller in the Non-Volatile Memory Express (NVMe) specification, including a Submission Queue (SQ) and a Completion Queue (CQ).

[0044] In this embodiment of the disclosure, the NVMe-over-RDMA connection establishment device can create an IO queue in NVMe after the communication connection corresponding to the management queue is successfully established, and use RDMA hardware resources to call the RDMA queue creation interface to create an RDMA queue pair corresponding to the IO queue, and determine it as the second queue pair. The second queue may include a second transmit queue and a second receive queue. The IO commands stored in the second transmit queue and the second receive queue are transmitted and / or received through RDMA hardware. Specifically, they can be transmitted or received based on port numbers other than preset source port numbers. Specifically, the source port number when the management commands stored in the second transmit queue and the second receive queue are transmitted or received is other port numbers.

[0045] Optionally, the NVMe-over-RDMA connection establishment device can use the standard RDMA interface to receive and send IO commands on the second queue pair in the standard NVMe over RDMA manner when sending and receiving IO commands through RDMA hardware after the second queue pair is created.

[0046] S105. Based on the second queue, initiate a second connection establishment request to the server and negotiate to establish a communication connection corresponding to the IO queue.

[0047] In this embodiment of the disclosure, the NVMe-over-RDMA connection establishment device can initiate a second connection establishment request to the server based on the second queue pair after the IO queue and the second queue pair are created. Specifically, it can use the second queue pair to send various parameters required to establish the communication connection corresponding to the IO queue to the server, package the various parameters and store them in the sending queue of the second queue pair, and then send them to the server. Then, it can store the data packet returned by the server in the receiving queue of the second queue pair, parse it, and finally negotiate to establish the communication connection corresponding to the IO queue.

[0048] This embodiment of the disclosure, in response to a detected connection command, creates a management queue and a first queue pair based on the network interface card's (NIC) software resources. The first queue pair is an RDMA queue pair corresponding to the management queue. Management commands stored in the first queue pair are sent and / or received via the NIC based on a preset source port number. The NIC and the first queue pair are configured based on the preset source port number. A first connection establishment request is initiated to the server based on the first queue pair to negotiate and establish a communication connection corresponding to the management queue. An IO queue is created, and a second queue pair is created based on RDMA hardware resources. The second queue pair is an RDMA queue pair corresponding to the IO queue. IO commands stored in the second queue pair are sent and / or received via RDMA hardware. The client initiates a second connection request to the server to negotiate and establish a communication connection corresponding to the IO queue. It can use the network card's software resources to create an RDMA queue pair corresponding to the management queue and use RDMA hardware resources to establish an RDMA queue corresponding to the IO queue. It configures the network card and the RDMA queue pair corresponding to the management queue through a preset source port number, and finally establishes a communication connection between the client and the server corresponding to the management queue and the IO queue. In this way, while ensuring efficient data interaction of the IO queue, the RDMA hardware resources occupied by the management queue are released and can continue to be used as the IO queue, improving the effective utilization rate of RDMA hardware resources. At the same time, it reduces the overall occupancy rate of RDMA hardware resources and improves the stability of RDMA hardware.

[0049] Optionally, the server can create a management queue using the same method as S101 and S102, and create an RDMA queue pair corresponding to the management queue based on the software resources of the network card in the server. Then, the network card and the RDMA queue pair are configured based on the preset source port number. The management commands stored in the RDMA queue pair corresponding to the management queue of the server are also sent and / or received based on the preset source port number.

[0050] Optionally, the server can use the same method as S104 to create the server's IO queue, and create an RDMA queue pair corresponding to the IO queue based on the RDMA hardware resources of the network card in the server. The IO commands stored in the RDMA queue pair corresponding to the server's IO queue are sent and / or received.

[0051] In some embodiments, the communication connection may include a transport layer connection and a protocol layer link. After the server creates and configures the management queue and the corresponding RDMA queue pair using the same method as in S101 and S102, the NVMe-over-RDMA connection establishment device on the client side can initiate a first link establishment request when establishing the communication connection corresponding to the management queue, establishing an RDMA transport layer connection for the management queue. After the server network card receives the data packet corresponding to the first link establishment request, it determines that the source port number is a preset source port number, hands over the data packet to the RDMA queue pair corresponding to the management queue for processing, completes the response to the first link establishment request, the transport layer connection corresponding to the management queue is established, and the establishment of a protocol layer connection begins, and negotiation takes place at the protocol layer. The NVMe-over-RDMA connection establishment device on the client side sends commands such as Fabric Connect, Property Get, and Property Set through the first queue pair. The server-side network card recognizes the source port number of the data packet as a preset source port number and hands it over to the RDMA queue pair corresponding to the management queue on the server for processing. The server responds to these commands and replies, also specifying the preset source port number in the reply. This allows the client's first queue pair to receive the command reply and complete the negotiation of information such as the subsystem name (SUBNQN), host name (HOSTNQN), and controller ID, thereby establishing a protocol layer connection for the management queue. After the protocol layer connection is established, the first queue pair begins to exchange NVMe management commands. The client sends commands such as Idenify and Get Log Page to obtain NVMe controller information and prepare to execute the step of creating the IO queue in S104.

[0052] After the server creates the IO queue and its corresponding RDMA queue pair using the same method as S104, the NVMe-over-RDMA connection establishment device on the client side can initiate a second connection establishment request when establishing the communication connection corresponding to the IO queue. This establishes the RDMA transport layer connection for the IO queue. Upon receiving the data packet corresponding to the second connection establishment request, the server's network card determines that the source port number is not the preset source port number and hands the data packet over to the RDMA queue pair corresponding to the management queue for processing, completing the response to the second connection establishment request. The transport layer connection corresponding to the IO queue is then established, and the protocol layer connection is established and negotiated at the protocol layer. The NVMe-over-RDMA connection establishment device on the client side sends a FabricConnect command to the second queue pair. Upon receiving the packet, the server responds by initiating an RDMA read operation, obtaining the HOSTNQN, and replying whether it agrees to the connection. After the IO queue protocol layer connection is successfully established, RDMA read and RDMA write operations can be initiated on the IO queue using the RDMA driver to complete NVMe IO read and write commands.

[0053] Figure 2 This is a flowchart of a method for creating a first queue pair provided in an embodiment of this disclosure, such as... Figure 2 As shown, based on the above embodiments, the first queue pair can be created using the following method.

[0054] S201. The RDMA operation interface is extended based on preset flag bits. The RDMA operation interface is used to create RDMA queue pairs.

[0055] In this embodiment of the disclosure, the NVMe-over-RDMA connection establishment device can extend the functionality of the RDMA operation interface, so that the attribute parameters of the RDMA queue pair created by calling the extended RDMA operation interface include a preset flag bit, thereby identifying the resource type used to create the RDMA queue pair through the preset flag bit.

[0056] S202. Create a first queue pair based on the first value of the flag bit. The first value is used to indicate that the first queue pair is created based on the network card's software resources.

[0057] In this embodiment of the disclosure, the NVMe-over-RDMA connection establishment device can create a first queue pair based on the first value of the flag bit after extending the RDMA operation interface. Specifically, it can call the RDMA operation interface based on the first value of the flag bit and create the first queue pair through the network card's software resources. The value of the flag bit in the attribute parameters of the first queue pair is the first value, thereby indicating that the first queue pair is an RDMA queue pair corresponding to the management queue created based on the network card's software resources, allowing the network card device to receive or send data packets on the management queue.

[0058] Optionally, the NVMe-over-RDMA connection establishment device can create a second queue pair based on a second value of a flag bit when creating the second queue pair. The second value is used to indicate that the second queue pair is created based on RDMA hardware resources.

[0059] Specifically, after extending the RDMA operation interface, a second queue pair is created based on the second value of the flag bit. Specifically, the RDMA operation interface can be called based on the second value of the flag bit to create the second queue pair through RDMA hardware resources. The value of the flag bit in the attribute parameters of the second queue pair is the second value, thus indicating that the second queue pair is an RDMA queue pair corresponding to the IO queue created based on RDMA hardware resources, allowing the RDMA hardware device to receive or send data packets on the IO queue.

[0060] This embodiment extends the RDMA operation interface based on preset flag bits. The RDMA operation interface is used to create RDMA queue pairs. A first queue pair is created based on a first value of the flag bits, which indicates that the first queue pair is created based on the network card's software resources. A second queue pair is created based on a second value of the flag bits, which indicates that the second queue pair is created based on RDMA hardware resources. Different types of RDMA queue pairs can be created through a unified interface without cumbersome configuration and switching. At the same time, it facilitates subsequent processing of data packets in the RDMA queue pairs based on the flag bit values.

[0061] Figure 3 This is a flowchart of a method for forwarding data packets provided in an embodiment of this disclosure, such as... Figure 3 As shown, based on the above embodiments, data packets can be forwarded using the following method.

[0062] S301. In response to receiving an RDMA data packet sent by the server based on the network card, the source port number corresponding to the RDMA data packet is parsed, and it is determined whether the RDMA data packet is the first RDMA type data packet received.

[0063] In this embodiment of the disclosure, the NVMe-over-RDMA connection establishment device can parse the RDMA data packet sent by the server after receiving it through the network card, obtain the source port number corresponding to the RDMA data packet, and determine whether the received RDMA data packet is the first RDMA type data packet received after initiating the first connection establishment request to the server.

[0064] Specifically, if the RDMA data packet is the first RDMA type data packet received after the first connection establishment request is initiated to the server, it means that the RDMA data packet is a reply packet received from the server when establishing a communication connection corresponding to the management queue with the server. The source port number of the RDMA data packet is the source port number that the server will use when sending or replying to management commands later.

[0065] S302. If the RDMA data packet is the first RDMA type data packet received, then the target source port number is determined as the source port number corresponding to the RDMA data packet, the target source port number is recorded, and the RDMA data packet is forwarded to the network card driver so that the network card driver stores the RDMA data packet in the first receive queue of the first queue pair and updates the management queue based on the RDMA data packet.

[0066] The target source port number in this embodiment can be understood as the source port number used by the server when sending or replying to management commands.

[0067] In this embodiment, the NVMe-over-RDMA connection establishment device can, after parsing the source port number corresponding to the RDMA data packet, determine whether the RDMA data packet is the first RDMA type data packet received. If so, it indicates that the RDMA data packet is a response packet received from the server when establishing a communication connection with the server corresponding to the management queue. This allows the device to determine that the target source port number used by the server when sending or replying to management commands is the source port number corresponding to the RDMA data packet. The target source port number is recorded for future reference, facilitating the determination of whether the data packet is related to a management command based on the source port number of the data packet sent or replied by the server and the target source port number. Simultaneously, the NVMe-over-RDMA connection establishment device can determine that the RDMA data packet corresponds to a management command and needs to be stored in a first queue pair created based on the network card's software resources. Therefore, the RDMA data packet is forwarded to the network card driver, allowing the network card driver to store the RDMA data packet in the first receive queue of the first queue pair and update the management queue based on the RDMA data packet. Specifically, the RDMA data packet stored in the first receive queue can be processed. After processing, based on the processed data packet information, the data packet is updated in the management queue (usually the completion queue). Insert a corresponding completion queue element (CQE) into the queue. This CQE can contain key information related to processing RDMA data packets, such as whether the data packet was successfully received, the length of the received data, and whether any errors occurred.

[0068] S303. If the RDMA data packet is not the first RDMA type data packet received, then the source port number corresponding to the RDMA data packet is compared with the target source port number. If the source port number corresponding to the RDMA data packet matches the target source port number, the RDMA data packet is forwarded to the network card driver so that the network card driver stores the data packet in the first receive queue of the first queue pair and updates the management queue based on the data packet. If the source port number corresponding to the RDMA data packet does not match the target source port number, the RDMA data packet is forwarded to the RDMA driver so that the RDMA driver stores the data packet in the second receive queue of the second queue pair and updates the corresponding IO queue based on the RDMA data packet.

[0069] In this embodiment, the NVMe-over-RDMA connection establishment device can, after parsing the source port number corresponding to the RDMA data packet, determine whether the RDMA data packet is the first RDMA type data packet received. If not, it indicates that the RDMA data packet is not a server response packet received when establishing a communication connection with the server corresponding to the management queue. At this time, the target source port number used by the server when sending or replying to management commands has been recorded. The source port number corresponding to the RDMA data packet can be further compared with the target source port number. If they match, it indicates that the RDMA data packet is the data packet corresponding to the management command. The RDMA data packet is then forwarded to the network card driver so that the network card driver stores the data packet in the first queue. The first receive queue in the RDMA data packet is updated based on the data packet, and the specific processing method is the same as in S302, so it will not be repeated here. If the two are inconsistent, it means that the RDMA data packet is a data packet corresponding to the IO command and needs to be stored in the second queue pair created based on the RDMA hardware resources. Therefore, the RDMA data packet is forwarded to the RDMA driver so that the RDMA driver stores the RDMA data packet in the second receive queue of the second queue pair corresponding to the RDMA data packet and updates the IO queue corresponding to the RDMA data packet based on the RDMA data packet. Specifically, the RDMA data packet stored in the second receive queue can be processed. After processing, according to the processed data packet information, a corresponding completion queue element (CQE) is inserted into the IO queue (usually the completion queue). This CQE can contain key information related to the processing of the RDMA data packet, such as whether the data packet was successfully received, the length of the received data, and whether any errors occurred.

[0070] This embodiment of the disclosure responds to the network interface card (NIC) receiving an RDMA data packet sent by the server, parses the source port number corresponding to the RDMA data packet, and determines whether the RDMA data packet is a newly received RDMA type data packet. If the RDMA data packet is a newly received RDMA type data packet, the target source port number is determined as the source port number corresponding to the RDMA data packet, the target source port number is recorded, and the RDMA data packet is forwarded to the NIC driver. The NIC driver then stores the RDMA data packet in the first receive queue of the first queue pair and updates the management queue based on the RDMA data packet. If the RDMA data packet is not a newly received RDMA type data packet, the source port number corresponding to the RDMA data packet is compared with the target source port number, and the result is updated accordingly. When the RDMA data packet is received, it is forwarded to the network card driver so that the network card driver can store the data packet in the first receive queue of the first queue pair and update the management queue based on the data packet. When the source port number and the destination source port number of the RDMA data packet are inconsistent, the RDMA data packet is forwarded to the RDMA driver so that the RDMA driver can store the data packet in the second receive queue of the second queue pair and update the corresponding IO queue based on the RDMA data packet. It can determine and record the destination source port number of the data packet corresponding to the management command sent by the server, and distinguish the data packets corresponding to the management command and IO command according to the destination source port number. Different data packets are forwarded and written to different receive queues, which facilitates the subsequent processing of data packets in different receive queues using software and hardware resources, and further reduces the overall occupancy of RDMA hardware resources.

[0071] In some embodiments, the NVMe-over-RDMA connection establishment device can send the first data packet to the server based on a preset source port number if the network card driver detects that there is a first data packet to be sent in the first transmission queue included in the first queue pair.

[0072] Specifically, the NVMe-over-RDMA connection establishment device can periodically detect the first transmission queue in the first queue pair based on the network card driver. When a first data packet to be sent is detected in the first transmission queue, the preset source port number is determined as the source port number of the first data packet, and the first data packet is sent to the server through the network card. If the server has the same configuration as the client, the server can also determine the preset source port number as the source port number of the returned data packet. This allows the client to determine the returned data packet as the data packet corresponding to the management command based on the source port number of the returned data packet after receiving the server's returned data packet, and then use the network card's software resources to process the returned data packet, further reducing the overall occupancy of RDMA hardware resources.

[0073] In other embodiments, the NVMe-over-RDMA connection establishment device can detect, based on the RDMA driver, that there is a second data packet to be sent in the second transmission queue contained in the second queue pair, determine the source port number corresponding to the second data packet based on the queue number of the second transmission queue, and send the second data packet to the server.

[0074] Specifically, the NVMe-over-RDMA connection establishment device can periodically detect the second transmission queue in the second queue pair based on the RDMA driver. When a second data packet to be sent is detected in the second transmission queue, the queue number of the second transmission queue is obtained, and the source port number corresponding to the queue number of the second transmission queue is determined based on the pre-obtained correspondence or conversion method between the queue number of the second transmission queue and the source port number. This source port number is then determined as the source port number of the second data packet, and the second data packet is sent to the server.

[0075] Figure 4 This is a schematic diagram of the structure of an NVMe-over-RDMA connection establishment device provided in an embodiment of this disclosure. Figure 4 As shown, the NVMe-over-RDMA connection establishment device 400 is suitable for clients and includes: a first creation module 410, a configuration module 420, a first initiation module 430, a second creation module 440, and a second initiation module 450. The first creation module 410 is used to create a management queue in response to a detected connection command, and to create a first queue pair based on the network interface card's (NIC) software resources. The first queue pair is an RDMA queue pair corresponding to the management queue. The management commands stored in the first queue pair are sent and / or received through the NIC based on a preset source port number. The configuration module 420 is used to... The port number is configured for the network interface card and the first queue pair; the first initiation module 430 is used to initiate a first connection establishment request to the server based on the first queue pair to negotiate and establish a communication connection corresponding to the management queue; the second creation module 440 is used to create an IO queue and create a second queue pair based on RDMA hardware resources, the second queue pair being an RDMA queue pair corresponding to the IO queue, and the IO commands stored in the second queue pair are sent and / or received through RDMA hardware; the second initiation module 450 is used to initiate a second connection establishment request to the server based on the second queue pair to negotiate and establish a communication connection corresponding to the IO queue.

[0076] Optionally, the first creation module 410 includes: an extension unit for extending the RDMA operation interface based on a preset flag bit, the RDMA operation interface being used to create RDMA queue pairs; and a first creation unit for creating the first queue pair based on a first value of the flag bit, the first value being used to indicate that the first queue pair is created based on the network interface card's software resources.

[0077] Optionally, the second creation module 440 includes: a second creation unit, used to create the second queue pair based on a second value of the flag bit, wherein the second value is used to indicate that the second queue pair is created based on RDMA hardware resources.

[0078] Optionally, the NVMe-over-RDMA connection establishment device 400 further includes: a parsing module, configured to, in response to receiving an RDMA data packet sent by the server based on the network interface card (NIC), parse the source port number corresponding to the RDMA data packet and determine whether the RDMA data packet is a newly received RDMA type data packet; and a first forwarding module, configured to, if the RDMA data packet is a newly received RDMA type data packet, determine the target source port number as the source port number corresponding to the RDMA data packet, record the target source port number, and forward the RDMA data packet to the NIC driver, so that the NIC driver stores the RDMA data packet in the first receive queue of the first queue pair and updates the management queue based on the RDMA data packet; The second forwarding module is configured to, if the RDMA data packet is not a first-time received RDMA type data packet, compare the source port number corresponding to the RDMA data packet with the target source port number, and if the source port number corresponding to the RDMA data packet matches the target source port number, forward the RDMA data packet to the network card driver so that the network card driver stores the data packet in the first receive queue of the first queue pair and updates the management queue based on the data packet; if the source port number corresponding to the RDMA data packet does not match the target source port number, forward the RDMA data packet to the RDMA driver so that the RDMA driver stores the data packet in the second receive queue of the second queue pair and updates the corresponding IO queue based on the RDMA data packet.

[0079] Optionally, the NVMe-over-RDMA connection establishment device 400 further includes: a first sending module, configured to send the first data packet to the server based on the preset source port number if the network card driver detects that there is a first data packet to be sent in the first sending queue included in the first queue pair.

[0080] Optionally, the NVMe-over-RDMA connection establishment device 400 further includes: a second sending module, configured to, based on the RDMA driver detecting that there is a second data packet to be sent in the second sending queue included in the second queue pair, determine the source port number corresponding to the second data packet based on the queue number of the second sending queue, and send the second data packet to the server.

[0081] Optionally, the communication connection includes a transport layer connection and a protocol layer link.

[0082] The NVMe-over-RDMA connection establishment device provided in this embodiment can execute the method described in any of the above embodiments. Its execution method and beneficial effects are similar, and will not be repeated here.

[0083] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure.

[0084] like Figure 5 As shown, the computer device may include a processor 510 and a memory 520 storing computer program instructions.

[0085] Specifically, the processor 510 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0086] Memory 520 may include a large-capacity storage for information or instructions. For example, and not limitingly, memory 520 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 520 may include removable or non-removable (or fixed) media. Where appropriate, memory 520 may be internal or external to the integrated gateway device. In a particular embodiment, memory 520 is a non-volatile solid-state memory. In a particular embodiment, memory 520 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (Electrically Programmable ROM, EPROM), an electrically erasable programmable PROM (EEPROM), an electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0087] The processor 510 reads and executes computer program instructions stored in the memory 520 to perform the steps of the NVMe-over-RDMA connection establishment method provided in this embodiment of the disclosure.

[0088] In one example, the computer device may also include a transceiver 530 and a bus 540. Wherein, as... Figure 5 As shown, the processor 510, memory 520 and transceiver 530 are connected via bus 540 and communicate with each other.

[0089] Bus 540 may include hardware, software, or both. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 540 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0090] This disclosure also provides a computer-readable storage medium that can store a computer program. When the computer program is executed by a processor, the processor enables the processor to implement the NVMe-over-RDMA connection establishment method provided in this disclosure.

[0091] The aforementioned storage medium may, for example, include a memory 520 containing computer program instructions, which can be executed by the processor 510 of the NVMe-over-RDMA connection establishment device to complete the NVMe-over-RDMA connection establishment method provided in this embodiment of the disclosure. Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), compact disc ROM (CD-ROM), magnetic tape, floppy disk, and optical data storage device. The aforementioned computer program may be written in any combination of one or more programming languages ​​to perform the operations of this embodiment of the disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code may be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0092] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0093] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for establishing an NVMe-over-RDMA connection, characterized in that, The method is applicable to the client, and the method includes: In response to the detection of a connection command, a management queue is created, and a first queue pair is created based on the network card's software resources. The first queue pair is an RDMA queue pair corresponding to the management queue. The management commands stored in the first queue pair are sent and / or received through the network card based on a preset source port number. Configure the network card and the first queue pair based on the preset source port number; Based on the first queue, the first connection establishment request is initiated to the server to negotiate and establish a communication connection corresponding to the management queue; Create an IO queue and create a second queue pair based on RDMA hardware resources. The second queue pair is the RDMA queue pair corresponding to the IO queue. The IO commands stored in the second queue pair are sent and / or received through RDMA hardware. Based on the second queue, a second connection establishment request is initiated to the server to negotiate and establish a communication connection corresponding to the IO queue; The creation of the first queue pair based on the network interface card (NIC) software resources includes: The RDMA operation interface is extended based on preset flag bits, and the RDMA operation interface is used to create RDMA queue pairs. The first queue pair is created based on the first value of the flag bit, wherein the first value is used to indicate that the first queue pair is created based on the network card's software resources. The creation of the second queue pair based on RDMA hardware resources includes: The second queue pair is created based on the second value of the flag bit, whereby the second value is used to indicate that the second queue pair is created based on RDMA hardware resources; After initiating the first connection establishment request to the server based on the first queue, the method further includes: In response to receiving an RDMA data packet sent by the server based on the network card, the source port number corresponding to the RDMA data packet is parsed, and it is determined whether the RDMA data packet is the first RDMA type data packet received; If the RDMA data packet is the first RDMA type data packet received, then the target source port number is determined as the source port number corresponding to the RDMA data packet, the target source port number is recorded, and the RDMA data packet is forwarded to the network card driver so that the network card driver stores the RDMA data packet in the first receive queue of the first queue pair and updates the management queue based on the RDMA data packet. If the RDMA data packet is not the first RDMA type data packet received, the source port number corresponding to the RDMA data packet is compared with the target source port number. If the source port number corresponding to the RDMA data packet matches the target source port number, the RDMA data packet is forwarded to the network card driver so that the network card driver stores the data packet in the first receive queue of the first queue pair and updates the management queue based on the data packet. If the source port number corresponding to the RDMA data packet does not match the target source port number, the RDMA data packet is forwarded to the RDMA driver so that the RDMA driver stores the data packet in the second receive queue of the second queue pair and updates the corresponding IO queue based on the RDMA data packet.

2. The method according to claim 1, characterized in that, After initiating a second connection establishment request to the server based on the second queue and negotiating the establishment of a communication connection corresponding to the IO queue, the method further includes: If the network card driver detects that there is a first data packet to be sent in the first sending queue contained in the first queue pair, then the first data packet is sent to the server based on the preset source port number.

3. The method according to claim 1, characterized in that, After initiating a second connection establishment request to the server based on the second queue and negotiating the establishment of a communication connection corresponding to the IO queue, the method further includes: If the RDMA driver detects that there is a second data packet to be sent in the second transmission queue contained in the second queue pair, then the source port number corresponding to the second data packet is determined based on the queue number of the second transmission queue, and the second data packet is sent to the server.

4. The method according to claim 1, characterized in that, The communication connection includes a transport layer connection and a protocol layer link.

5. An NVMe-over-RDMA connection establishment device, characterized in that, The device is applicable to a client, and the device includes: The first creation module is used to create a management queue in response to the detection of a connection command, and to create a first queue pair based on the network card's software resources. The first queue pair is an RDMA queue pair corresponding to the management queue. The management commands stored in the first queue pair are sent and / or received through the network card based on a preset source port number. The configuration module is used to configure the network interface card and the first queue pair based on the preset source port number; The first initiation module is used to initiate a first connection establishment request to the server based on the first queue, and negotiate to establish a communication connection corresponding to the management queue. The second creation module is used to create an IO queue and create a second queue pair based on RDMA hardware resources. The second queue pair is the RDMA queue pair corresponding to the IO queue. The IO commands stored in the second queue pair are sent and / or received through RDMA hardware. The second initiation module is used to initiate a second connection establishment request to the server based on the second queue, and negotiate to establish a communication connection corresponding to the IO queue. The first creation module is specifically used to extend the RDMA operation interface based on a preset flag bit, wherein the RDMA operation interface is used to create RDMA queue pairs; The first queue pair is created based on the first value of the flag bit, wherein the first value is used to indicate that the first queue pair is created based on the network card's software resources. The second creation module is specifically used to create the second queue pair based on the second value of the flag bit, wherein the second value is used to indicate that the second queue pair is created based on RDMA hardware resources; After the first connection establishment request is initiated to the server based on the first queue, the device further includes: The parsing module is used to respond to the RDMA data packet sent by the server received by the network card, parse the source port number corresponding to the RDMA data packet, and determine whether the RDMA data packet is the first RDMA type data packet received. If the RDMA data packet is the first RDMA type data packet received, then the target source port number is determined as the source port number corresponding to the RDMA data packet, the target source port number is recorded, and the RDMA data packet is forwarded to the network card driver so that the network card driver stores the RDMA data packet in the first receive queue of the first queue pair and updates the management queue based on the RDMA data packet. If the RDMA data packet is not the first RDMA type data packet received, the source port number corresponding to the RDMA data packet is compared with the target source port number. If the source port number corresponding to the RDMA data packet matches the target source port number, the RDMA data packet is forwarded to the network card driver so that the network card driver stores the data packet in the first receive queue of the first queue pair and updates the management queue based on the data packet. If the source port number corresponding to the RDMA data packet does not match the target source port number, the RDMA data packet is forwarded to the RDMA driver so that the RDMA driver stores the data packet in the second receive queue of the second queue pair and updates the corresponding IO queue based on the RDMA data packet.

6. A computer device, characterized in that, include: Memory; processor; And a computer program; wherein the computer program is stored in the memory and configured to be executed by the processor to implement the NVMe-over-RDMA connection establishment method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the NVMe-over-RDMA connection establishment method as described in any one of claims 1-4.

Images