Connection establishment method and device of NVMe-over-RDMA, equipment and medium

Abstract

The invention relates to an NVMe-over-RDMA connection establishment method and device, equipment and a medium, and the method comprises the steps: creating a management queue, and creating a first queue pair based on software resources of a network card, a management command stored in the first queue pair being sent and / or received through the network card based on a preset source port number, configuring the network card and the first queue pair based on a preset source port number; initiating a first link establishment request to the server based on the first queue pair; an IO queue is created, a second queue pair is created based on RDMA hardware resources, and IO commands stored in the second queue pair are sent and / or received through RDMA hardware; and initiating a second link establishment request to the server based on the second queue pair. According to the method, the RDMA queue pair corresponding to the management queue is created by adopting the software resources of the network card, so that waste caused by the fact that the management queue occupies RDMA hardware resources can be avoided, and the hardware stability is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of NVMe-over-RDMA technology, and in particular to a method, apparatus, device, and medium for establishing an NVMe-over-RDMA connection. Background Art

[0002] NVMe (Non-Volatile Memory Express) is an interface protocol that uses the PCIe bus to communicate directly with fast non-volatile storage devices. It can improve data transfer rates and storage performance, meeting the needs of modern data centers for high-performance storage. RDMA (Remote Direct Memory Access) is a direct memory access technology that allows the host to directly access remote memory without the need for remote CPU participation, reducing access latency. NVMe-over-RDMA is a network protocol that combines the two. It uses RDMA network technology to enable host access to off-node storage resources while retaining the low latency and high performance characteristics of NVMe, enhancing system flexibility and scalability.

[0003] In the NVMe-over-RDMA protocol, the client and server establish a connection through RDMA network technology to achieve remote access to NVMe storage resources. In this connection framework, the management queue is only responsible for processing a small number of admin commands, while the IO queue is responsible for processing intensive IO commands. However, both the management queue and the IO queue need to be carried by RDMA QP (Queue Pairs) and occupy RDMA hardware resources, resulting in the RDMA hardware resources not being effectively and fully utilized. Especially in the scenario of providing network storage for large-scale small-scale virtual machines, the RDMA hardware resources occupied by the management queue account for a large proportion of the entire system, and the problem of resource waste is particularly prominent. In addition, excessive occupation of RDMA hardware resources will also affect the stability of the RDMA hardware. Therefore, how to optimize the connection method between the client and the server to reduce the waste of RDMA hardware resources and improve hardware stability is a technical problem that needs to be solved. Summary of the Invention

[0004] In order to solve the above technical problems, the present disclosure provides a method, device, equipment and medium for establishing an NVMe-over-RDMA connection.

[0005] A first aspect of an embodiment of the present disclosure provides a method for establishing an NVMe-over-RDMA connection, which is applicable to a client and includes:

[0006] In response to detecting the connection instruction, creating a management queue and creating a first queue pair based on software resources of the network card, the first queue pair being an RDMA queue pair corresponding to the management queue, and management commands stored in the first queue pair being sent and / or received through the network card based on a preset source port number;

[0007] Configuring the network card and the first queue pair based on the preset source port number;

[0008] Initiating a first link establishment request to the server based on the first queue, and negotiating to establish a communication connection corresponding to the management queue;

[0009] Creating an IO queue and creating a second queue pair based on RDMA hardware resources, where the second queue pair is an RDMA queue pair corresponding to the IO queue, and IO commands stored in the second queue pair are sent and / or received through the RDMA hardware;

[0010] Based on the second queue, a second link establishment request is initiated to the server to negotiate and establish a communication connection corresponding to the IO queue.

[0011] A second aspect of an embodiment of the present disclosure provides an NVMe-over-RDMA connection establishment device, which is applicable to a client and includes:

[0012] a first creation module, configured to create a management queue in response to detecting a connection instruction, and to create a first queue pair based on software resources of the network card, wherein the first queue pair is an RDMA queue pair corresponding to the management queue, and management commands stored in the first queue pair are sent and / or received through the network card based on a preset source port number;

[0013] A configuration module, configured to configure the network card and the first queue pair based on the preset source port number;

[0014] A first initiating module is configured to initiate a first link establishment request to a server based on the first queue, and negotiate to establish a communication connection corresponding to the management queue;

[0015] a second creation module, configured to create an IO queue and create a second queue pair based on the RDMA hardware resources, wherein the second queue pair is an RDMA queue pair corresponding to the IO queue, and the IO commands stored in the second queue pair are sent and / or received through the RDMA hardware;

[0016] The second initiating module is used to initiate a second link establishment request to the server based on the second queue, and negotiate to establish a communication connection corresponding to the IO queue.

[0017] A third aspect of an embodiment of the present disclosure provides a computer device, comprising a memory and a processor, and a computer program, wherein the computer program is stored in the memory, and when the computer program is executed by the processor, the NVMe-over-RDMA connection establishment method as described in the first aspect above is implemented.

[0018] A fourth aspect of an embodiment of the present disclosure provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the NVMe-over-RDMA connection establishment method as described in the first aspect above is implemented.

[0019] The technical solution provided by the embodiments of the present disclosure has the following advantages over the prior art:

[0020] In the NVMe-over-RDMA connection establishment method, apparatus, device and medium provided by the embodiments of the present disclosure, a management queue is created in response to detecting a connection instruction, and a first queue pair is created based on the software resources of the network card. The first queue pair is the RDMA queue pair corresponding to the management queue. The management commands stored in the first queue pair are sent and / or received through the network card based on a preset source port number. The network card and the first queue pair are configured based on the preset source port number. A first link establishment request is initiated to the server based on the first queue pair, and a communication connection corresponding to the management queue is established through negotiation. An IO queue is created, and a second queue pair is created based on the RDMA hardware resources. The second queue pair is the RDMA queue pair corresponding to the IO queue. The IO commands stored in the second queue pair are sent and / or received through the RDMA hardware. The client and the server can send and / or receive files, initiate a second link establishment request to the server based on the second queue, negotiate to establish a communication connection corresponding to the IO queue, use the software resources of the network card to create an RDMA queue pair corresponding to the management queue, use the RDMA hardware resources to establish the RDMA queue corresponding to the IO queue, and configure the RDMA queue pair corresponding to the network card and the management queue through the preset source port number, and finally establish a communication connection corresponding to the management queue and the IO queue between the client and the server. Under the premise of ensuring efficient data interaction of the IO queue, the RDMA hardware resources occupied by the management queue are released and continue to be used as the IO queue, thereby improving the effective utilization of the RDMA hardware resources, and at the same time reducing the overall occupancy of the RDMA hardware resources, and improving the stability of the RDMA hardware. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0022] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0023] Figure 1 This is a flowchart of a method for establishing an NVMe-over-RDMA connection provided by an embodiment of the present disclosure;

[0024] Figure 2 is a flowchart of a method for creating a first queue pair provided by an embodiment of the present disclosure;

[0025] Figure 3 is a flow chart of a method for forwarding data packets provided by an embodiment of the present disclosure;

[0026] Figure 4 Schematic diagram of a device for establishing an NVMe-over-RDMA connection provided by an embodiment of the present disclosure;

[0027] Figure 5 It is a structural diagram of a computer device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0029] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0030] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0031] Figure 1 This is a flow chart of a method for establishing a connection of NVMe-over-RDMA provided by an embodiment of the present disclosure. This method can be executed by an NVMe-over-RDMA connection establishment device. The NVMe-over-RDMA connection establishment device can be implemented in software and / or hardware. The NVMe-over-RDMA connection establishment device can be configured in an electronic device as a client. Figure 1 As shown, the NVMe-over-RDMA connection establishment method provided in this embodiment includes the following steps:

[0032] S101. In response to detecting a connection instruction, a management queue is created, and a first queue pair is created based on the software resources of the network card. The first queue pair is an RDMA queue pair corresponding to the management queue. The management commands stored in the first queue pair are sent and / or received through the network card based on a preset source port number.

[0033] The management queue in the embodiment of the present disclosure can be understood as a queue space used to submit management commands to the controller in the Non-Volatile Memory Express (NVMe) host controller interface specification, including a submission queue (SQ) and a completion queue (CQ).

[0034] The RDMA queue pair in the embodiment of the present disclosure can be understood as the basic unit of communication in Remote Direct Memory Access (RDMA), which may include a send queue and a receive queue. In NVMe over RDMA, the management queue and IO queue of NVMe are carried by the RDMA queue pair.

[0035] The preset source port number in the embodiment of the present disclosure may be understood as a port number pre-selected from port numbers that are not commonly used in daily use. For example, the preset source port number may be 65535.

[0036] In an embodiment of the present disclosure, the NVMe-over-RDMA connection establishment device can create a management queue in NVMe after detecting a connection instruction issued by a user, and use the software resources of the network card to call the RDMA queue creation interface to create an RDMA queue pair corresponding to the management queue, and determine it as a first queue pair, wherein the first queue pair may include a first sending queue and a first receiving queue, and the management commands stored in the first sending queue and the first receiving queue are sent or received based on a preset source port number. Specifically, the source port number when the management commands stored in the first sending queue and the first receiving queue are sent and / or received can be a preset source port number.

[0037] S102: Configure the network card and the first queue pair based on a preset source port number.

[0038] In the embodiment of the present disclosure, the NVMe-over-RDMA connection establishment device can configure the network card according to the preset source port number, so that the network card can determine the processing method of the data packet according to the source port number in the data packet when sending and receiving the data packet. Specifically, the network card can be configured to use the preset source port number when sending the data packet in the sending queue of the first queue pair, and / or, when the source port number of the data packet sent by the received server is the preset source port number, the data packet is stored in the receiving queue of the first queue pair; the NVMe-over-RDMA connection establishment device can also configure the first queue pair according to the preset source port number. Specifically, the sending queue in the first queue pair can be configured to set the source port number of the management command in the first queue pair to the preset source port number.

[0039] It should be noted that in the embodiment of the present disclosure, the execution order of creating a management queue and creating a first queue pair based on the software resources of the network card in step S101 and step S102 is not limited. The two can be executed at the same time, or the creation of the management queue and the first queue pair can be executed before step S102, or the creation of the management queue and the first queue pair can be executed after step S102.

[0040] S103: Initiate a first link establishment request to the server based on the first queue, and negotiate to establish a communication connection corresponding to the management queue.

[0041] In the embodiment of the present disclosure, the NVMe-over-RDMA connection establishment device can initiate a first link establishment request to the server based on the first queue pair after the management queue and the first queue pair are created and the network card and the first queue pair are configured. Specifically, the first queue pair can be used to send the server various parameters required for establishing the communication connection corresponding to the management queue, and the various parameters are packaged and stored in the sending queue in the first queue pair, and then sent to the server through the sending queue. The data packet returned by the server is stored in the receiving queue in the first queue pair, and is parsed, and finally the communication connection corresponding to the management queue is negotiated and established.

[0042] S104: Create an IO queue, and create a second queue pair based on the RDMA hardware resources. The second queue pair is an RDMA queue pair corresponding to the IO queue. The IO commands stored in the second queue pair are sent and / or received through the RDMA hardware.

[0043] The IO queue in the embodiment of the present disclosure can be understood as a queue space used to submit IO commands to the controller in the Non-Volatile Memory Express (NVMe) host controller interface specification, including a submission queue (SQ) and a completion queue (CQ).

[0044] In an embodiment of the present disclosure, the NVMe-over-RDMA connection establishment device can create an IO queue in NVMe after the communication connection corresponding to the management queue is successfully established, and use RDMA hardware resources to call the RDMA queue creation interface to create an RDMA queue pair corresponding to the IO queue, and determine it as a second queue pair, wherein the second queue may include a second sending queue and a second receiving queue, and the IO commands stored in the second sending queue and the second receiving queue are sent and / or received through the RDMA hardware, and can be specifically sent or received based on other port numbers other than the preset source port number. Specifically, the source port number when the management commands stored in the second sending queue and the second receiving queue are sent or received is other port numbers.

[0045] Optionally, after the second queue pair is created, the NVMe-over-RDMA connection establishment device can use the standard RDMA interface to receive and send IO commands on the second queue pair in a standard NVMe over RDMA manner when sending and receiving IO commands through RDMA hardware.

[0046] S105: Initiate a second link establishment request to the server based on the second queue, and negotiate to establish a communication connection corresponding to the IO queue.

[0047] In the disclosed embodiment, the NVMe-over-RDMA connection establishment device can initiate a second link establishment request to the server based on the second queue pair after the IO queue and the second queue pair are created. Specifically, the second queue pair can be used to send the various parameters required to establish the communication connection corresponding to the IO queue to the server, and the various parameters are packaged and stored in the sending queue in the second queue pair, and then sent to the server. The data packet returned by the server is stored in the receiving queue in the second queue pair, and is parsed, and finally the communication connection corresponding to the IO queue is negotiated and established.

[0048] The embodiment of the present disclosure creates a management queue in response to detecting a connection instruction, and creates a first queue pair based on the software resources of the network card, the first queue pair is the RDMA queue pair corresponding to the management queue, the management commands stored in the first queue pair are sent and / or received through the network card based on a preset source port number, the network card and the first queue pair are configured based on the preset source port number, a first link establishment request is initiated to the server based on the first queue pair, a communication connection corresponding to the management queue is established through negotiation, an IO queue is created, and a second queue pair is created based on the RDMA hardware resources, the second queue pair is the RDMA queue pair corresponding to the IO queue, the IO commands stored in the second queue pair are sent and / or received through the RDMA hardware, and based on the second queue A second link establishment request is initiated to the server to negotiate and establish a communication connection corresponding to the IO queue. The software resources of the network card can be used to create an RDMA queue pair corresponding to the management queue, and the RDMA hardware resources can be used to establish the RDMA queue corresponding to the IO queue. The RDMA queue pair corresponding to the network card and the management queue is configured through the preset source port number, and finally a communication connection corresponding to the management queue and the IO queue is established between the client and the server. Under the premise of ensuring efficient data interaction of the IO queue, the RDMA hardware resources occupied by the management queue are released and continue to be used as the IO queue, thereby improving the effective utilization of the RDMA hardware resources. At the same time, the overall occupancy of the RDMA hardware resources is reduced, and the stability of the RDMA hardware is improved.

[0049] Optionally, the server can use the same method as S101 and S102 to create a management queue on the server side, and create an RDMA queue pair corresponding to the management queue based on the software resources of the network card in the server side, and then configure the network card and the RDMA queue pair based on the preset source port number. The management commands stored in the RDMA queue pair corresponding to the management queue of the server side are also sent and / or received based on the preset source port number.

[0050] Optionally, the server can use the same method as S104 to create an IO queue on the server side, and create an RDMA queue pair corresponding to the IO queue based on the RDMA hardware resources of the network card in the server side, and send and / or receive IO commands stored in the RDMA queue pair corresponding to the IO queue on the server side.

[0051] In some embodiments, the communication connection may include a transport layer connection and a protocol layer link. After the server uses the same method as S101 and S102 to create and configure the management queue and the corresponding RDMA queue pair, the NVMe-over-RDMA connection establishment device on the client side can initiate a first link establishment request when establishing the communication connection corresponding to the management queue, and establish the RDMA transport layer connection of the management queue. After the server network card receives the data packet corresponding to the first link establishment request, it determines that the source port number is the preset source port number, and hands the data packet to the RDMA queue pair corresponding to the management queue for processing, completing the response to the first link establishment request. The transport layer connection corresponding to the management queue is established, and the protocol layer connection is established, and negotiation is performed at the protocol layer. The NVMe-over-RDMA connection establishment device on the client side sends commands such as Fabric Connect, Property Get, and Property Set through the first queue pair. The server network card recognizes that the source port number of the data packet is the preset source port number and hands it over to the RDMA queue pair corresponding to the management queue on the server for processing. It responds to these commands and replies, and also specifies the source port number as the preset source port number when replying. This allows the first queue pair on the client to obtain the reply to the command and complete the negotiation of information such as the subsystem name (SUBNQN), host name (HOSTNQN), and controller ID, thereby establishing a protocol layer connection for the management queue. After the protocol layer connection is established, the first queue pair begins to interact with NVMe management commands. The client sends commands such as identify (idenify) and get log page (get log page) to obtain NVMe controller information and prepare to execute the step of creating an IO queue in S104.

[0052] After the server uses the same method as S104 to create an IO queue and a corresponding RDMA queue pair, the NVMe-over-RDMA connection establishment device on the client side can initiate a second link establishment request when establishing the communication connection corresponding to the IO queue, and establish the RDMA transport layer connection of the IO queue. After the server network card receives the data packet corresponding to the second link establishment request, it determines that the source port number is not the preset source port number, and hands the data packet to the RDMA queue pair corresponding to the management queue for processing, completing the response to the second link establishment request. The transport layer connection corresponding to the IO queue is established, and the protocol layer connection is established and negotiated at the protocol layer. The NVMe-over-RDMA connection establishment device on the client side sends a FabricConnect command on the second queue pair. After receiving the packet, the server responds by initiating an RDMA read operation, obtaining the HOSTNQN, and replying whether it agrees to the connection. After the IO queue protocol layer connection is successfully established, the RDMA driver can be used on the IO queue to initiate RDMA read and RDMA write operations to complete the NVMe IO read and write commands.

[0053] Figure 2 is a flowchart of a method for creating a first queue pair provided by an embodiment of the present disclosure, such as Figure 2 As shown, based on the above embodiment, the first queue pair can be created by the following method.

[0054] S201. Expand an RDMA operation interface based on a preset flag bit, where the RDMA operation interface is used to create an RDMA queue pair.

[0055] In an embodiment of the present disclosure, the NVMe-over-RDMA connection establishment device can expand the functions of the RDMA operation interface so that the attribute parameters of the RDMA queue pair created by calling the RDMA operation interface with the expanded functions include a preset flag bit, thereby identifying the resource type used to create the RDMA queue pair through the preset flag bit.

[0056] S202: Create a first queue pair based on a first value of the flag bit, where the first value is used to indicate that the first queue pair is created based on software resources of the network card.

[0057] In the embodiment of the present disclosure, the NVMe-over-RDMA connection establishment device can create a first queue pair based on the first value of the flag bit after expanding the RDMA operation interface. Specifically, the RDMA operation interface can be called based on the first value of the flag bit, and the first queue pair can be created through the software resources of the network card. The value of the flag bit in the attribute parameter of the first queue pair is the first value, thereby indicating that the first queue pair is an RDMA queue pair corresponding to the management queue created based on the software resources of the network card, allowing the network card device to receive or send data packets on the management queue.

[0058] Optionally, the NVMe-over-RDMA connection establishment device can create a second queue pair based on a second value of the flag bit when creating the second queue pair, and the second value is used to indicate that the second queue pair is created based on RDMA hardware resources.

[0059] Specifically, after the RDMA operation interface is expanded, a second queue pair is created based on the second value of the flag bit. Specifically, the RDMA operation interface can be called based on the second value of the flag bit, and the second queue pair is created through the RDMA hardware resources. The value of the flag bit in the attribute parameter of the second queue pair is the second value, thereby indicating that the second queue pair is an RDMA queue pair corresponding to the IO queue created based on the RDMA hardware resources, allowing the RDMA hardware device to receive or send data packets on the IO queue.

[0060] The embodiment of the present disclosure extends the RDMA operation interface based on a preset flag bit. The RDMA operation interface is used to create an RDMA queue pair. A first queue pair is created based on a first value of the flag bit, and the first value is used to indicate that the first queue pair is created based on the software resources of the network card. A second queue pair is created based on a second value of the flag bit, and the second value is used to indicate that the second queue pair is created based on the RDMA hardware resources. Different types of RDMA queue pairs can be created through a unified interface without the need for cumbersome configuration and switching, and the subsequent corresponding processing of data packets in the RDMA queue pair based on the value of the flag bit is facilitated.

[0061] Figure 3 is a flow chart of a method for forwarding data packets provided by an embodiment of the present disclosure, such as Figure 3 As shown, based on the above embodiment, the data packet can be forwarded by the following method.

[0062] S301: In response to receiving an RDMA data packet sent by a server based on a network card, parse and obtain a source port number corresponding to the RDMA data packet, and determine whether the RDMA data packet is a first-received RDMA type data packet.

[0063] In the embodiment of the present disclosure, the NVMe-over-RDMA connection establishment device can parse the RDMA data packet after receiving the RDMA data packet sent by the server through the network card, obtain the source port number corresponding to the RDMA data packet, and determine whether the received RDMA data packet is the first RDMA type data packet received after the first link establishment request is initiated to the server.

[0064] Specifically, if the RDMA data packet is the first RDMA type data packet received after the first link establishment request is initiated to the server, it means that the RDMA data packet is a return packet received from the server when the communication connection corresponding to the management queue is established with the server, and the source port number of the RDMA data packet is the source port number that the server will use when sending or replying to management commands subsequently.

[0065] S302. If the RDMA data packet is the first RDMA type data packet received, the target source port number is determined to be the source port number corresponding to the RDMA data packet, the target source port number is recorded, and the RDMA data packet is forwarded to the network card driver, so that the network card driver stores the RDMA data packet in the first receive queue in the first queue pair, and updates the management queue based on the RDMA data packet.

[0066] The target source port number in the embodiment of the present disclosure can be understood as the source port number that the server will use when sending or replying to a management command.

[0067] In the embodiment of the present disclosure, the NVMe-over-RDMA connection establishment device can determine whether the RDMA data packet is the first received RDMA type data packet after parsing the source port number corresponding to the RDMA data packet. If so, it means that the RDMA data packet is a reply packet received from the server when establishing a communication connection corresponding to the management queue with the server, and then determine that the target source port number used by the server when sending or replying to the management command is the source port number corresponding to the RDMA data packet, and record the target source port number to facilitate subsequent judgment of whether the data packet is a data packet related to the management command based on the source port number of the data packet sent or replied by the server and the target source port number. At the same time, the NVMe-over-RDMA connection establishment device can determine that the RDMA data packet is a data packet corresponding to the management command and needs to be stored in the first queue pair created based on the software resources of the network card. Therefore, the RDMA data packet is forwarded to the network card driver so that the network card driver stores the RDMA data packet in the first receive queue in the first queue pair, and updates the management queue based on the RDMA data packet. Specifically, the RDMA data packet stored in the first receive queue can be processed. After the processing is completed, according to the processed data packet information, the data packet is stored in the management queue (usually the completion queue, Completion queue). The CQE can contain key information related to processing RDMA data packets, such as whether the data packet is successfully received, the length of the received data, whether an error occurs, etc.

[0068] S303. If the RDMA data packet is not a data packet of the RDMA type received for the first time, the source port number corresponding to the RDMA data packet is compared with the target source port number. When the source port number corresponding to the RDMA data packet is consistent with the target source port number, the RDMA data packet is forwarded to the network card driver, so that the network card driver stores the data packet in the first receive queue in the first queue pair and updates the management queue based on the data packet. When the source port number corresponding to the RDMA data packet is inconsistent with the target source port number, the RDMA data packet is forwarded to the RDMA driver, so that the RDMA driver stores the data packet in the second receive queue in the second queue pair and updates the corresponding IO queue based on the RDMA data packet.

[0069] In the embodiment of the present disclosure, the NVMe-over-RDMA connection establishment device can determine whether the RDMA data packet is the first received RDMA type data packet after parsing the source port number corresponding to the RDMA data packet. If not, it means that the RDMA data packet is not a reply packet received from the server when the communication connection corresponding to the management queue is established with the server. At this time, the target source port number used by the server when sending or replying to the management command has been recorded. The source port number corresponding to the RDMA data packet can be further compared with the target source port number. If the two are consistent, it means that the RDMA data packet is a data packet corresponding to the management command, and the RDMA data packet is forwarded to the network card driver, so that the network card driver stores the data packet in the first queue. The first receiving queue in the RDMA driver is stored in the first receiving queue, and the management queue is updated based on the data packet. The specific processing method is the same as that in S302 and will not be repeated here. If the two are inconsistent, it means that the RDMA data packet is a data packet corresponding to the IO command and needs to be stored in the second queue pair created based on the RDMA hardware resource. Therefore, the RDMA data packet is forwarded to the RDMA driver, so that the RDMA driver stores the RDMA data packet in the second receiving queue of the second queue pair corresponding to the RDMA data packet, and updates the IO queue corresponding to the RDMA data packet based on the RDMA data packet. Specifically, the RDMA data packet stored in the second receiving queue can be processed. After the processing is completed, a corresponding completion queue element (CQE) is inserted into the IO queue (usually the completion queue) according to the processed data packet information. This CQE can contain key information related to the processing of the RDMA data packet, such as whether the data packet is successfully received, the length of the received data, whether an error occurs, etc.

[0070] The embodiment of the present disclosure obtains the source port number corresponding to the RDMA data packet in response to the RDMA data packet sent by the server side received based on the network card, and determines whether the RDMA data packet is the first RDMA type data packet received. If the RDMA data packet is the first RDMA type data packet received, the target source port number is determined to be the source port number corresponding to the RDMA data packet, the target source port number is recorded, and the RDMA data packet is forwarded to the network card driver, so that the network card driver stores the RDMA data packet in the first receiving queue of the first queue pair, and updates the management queue based on the RDMA data packet. If the RDMA data packet is not the first RDMA type data packet received, the source port number corresponding to the RDMA data packet is compared with the target source port number, and the source port number corresponding to the RDMA data packet is compared with the target source port number. When the RDMA data packet is consistent, the RDMA data packet is forwarded to the network card driver, so that the network card driver stores the data packet in the first receiving queue of the first queue pair, and updates the management queue based on the data packet. When the source port number corresponding to the RDMA data packet is inconsistent with the target source port number, the RDMA data packet is forwarded to the RDMA driver, so that the RDMA driver stores the data packet in the second receiving queue of the second queue pair, and updates the corresponding IO queue based on the RDMA data packet. The target source port number of the data packet corresponding to the management command sent by the server can be determined and recorded, and the data packets corresponding to the management command and the IO command can be distinguished according to the target source port number, and different data packets are forwarded and written into different receiving queues, so as to facilitate the subsequent use of software resources and hardware resources to process the data packets in different receiving queues, and further reduce the overall occupancy of RDMA hardware resources.

[0071] In some embodiments, the NVMe-over-RDMA connection establishment device can detect based on the network card driver that there is a first data packet to be sent in the first sending queue included in the first queue pair, and then send the first data packet to the server based on a preset source port number.

[0072] Specifically, the NVMe-over-RDMA connection establishment device can perform regular detection on the first sending queue in the first queue pair based on the network card driver. When it is detected that there is a first data packet to be sent in the first sending queue, the preset source port number is determined as the source port number of the first data packet and the first data packet is sent to the server through the network card. If the server is configured the same as the client, the server can also determine the preset source port number as the source port number of the return data packet when returning the data packet. After receiving the return data packet from the server, if the client can determine that the return data packet is a data packet corresponding to the management command based on the source port number of the return data packet, the software resources of the network card are used to process the return data packet, thereby further reducing the overall occupancy of RDMA hardware resources.

[0073] In other embodiments, the NVMe-over-RDMA connection establishment device can detect based on the RDMA driver that there is a second data packet to be sent in the second sending queue included in the second queue pair, and then determine the source port number corresponding to the second data packet based on the queue number of the second sending queue, and send the second data packet to the server.

[0074] Specifically, the NVMe-over-RDMA connection establishment device can perform regular detection on the second sending queue in the second queue pair based on the RDMA driver. When it is detected that there is a second data packet to be sent in the second sending queue, the queue number of the second sending queue is obtained, and based on the correspondence or conversion method between the queue number of the second sending queue and the source port number obtained in advance, the source port number corresponding to the queue number of the second sending queue is determined, and it is determined as the source port number of the second data packet, and the second data packet is sent to the server.

[0075] Figure 4 This is a schematic diagram of the structure of an NVMe-over-RDMA connection establishment device provided by an embodiment of the present disclosure. Figure 4 As shown, the NVMe-over-RDMA connection establishment device 400 is applicable to the client, including: a first creation module 410, a configuration module 420, a first initiation module 430, a second creation module 440, and a second initiation module 450, wherein the first creation module 410 is used to create a management queue in response to detecting a connection instruction, and to create a first queue pair based on the software resources of the network card, the first queue pair being the RDMA queue pair corresponding to the management queue, and the management commands stored in the first queue pair being sent and / or received through the network card based on a preset source port number; the configuration module 420 is used to create a management queue based on the preset source port number. The port number configures the network card and the first queue pair; the first initiating module 430 is used to initiate a first link establishment request to the server based on the first queue pair, and negotiate to establish a communication connection corresponding to the management queue; the second creating module 440 is used to create an IO queue and create a second queue pair based on RDMA hardware resources, the second queue pair is the RDMA queue pair corresponding to the IO queue, and the IO commands stored in the second queue pair are sent and / or received through the RDMA hardware; the second initiating module 450 is used to initiate a second link establishment request to the server based on the second queue pair, and negotiate to establish a communication connection corresponding to the IO queue.

[0076] Optionally, the first creation module 410 includes: an extension unit, used to extend the RDMA operation interface based on a preset flag bit, and the RDMA operation interface is used to create an RDMA queue pair; a first creation unit, used to create the first queue pair based on a first value of the flag bit, and the first value is used to characterize that the first queue pair is created based on the software resources of the network card.

[0077] Optionally, the second creation module 440 includes: a second creation unit, configured to create the second queue pair based on a second value of the flag bit, wherein the second value is used to indicate that the second queue pair is created based on RDMA hardware resources.

[0078] Optionally, the NVMe-over-RDMA connection establishment device 400 further includes: a parsing module, configured to, in response to receiving an RDMA data packet sent by the server based on the network card, parse and obtain a source port number corresponding to the RDMA data packet, and determine whether the RDMA data packet is a data packet of the RDMA type received for the first time; a first forwarding module, configured to, if the RDMA data packet is a data packet of the RDMA type received for the first time, determine a target source port number as the source port number corresponding to the RDMA data packet, record the target source port number, and forward the RDMA data packet to the network card driver, so that the network card driver stores the RDMA data packet in the first receive queue of the first queue pair, and updates the management queue based on the RDMA data packet; The second forwarding module is configured to compare the source port number corresponding to the RDMA data packet with the target source port number if the RDMA data packet is not a data packet of the RDMA type received for the first time, and when the source port number corresponding to the RDMA data packet is consistent with the target source port number, forward the RDMA data packet to the network card driver so that the network card driver stores the data packet in the first receive queue of the first queue pair and updates the management queue based on the data packet; and when the source port number corresponding to the RDMA data packet is inconsistent with the target source port number, forward the RDMA data packet to the RDMA driver so that the RDMA driver stores the data packet in the second receive queue of the second queue pair and updates the corresponding IO queue based on the RDMA data packet.

[0079] Optionally, the NVMe-over-RDMA connection establishment device 400 also includes: a first sending module, which is used to detect based on the network card driver that there is a first data packet to be sent in the first sending queue included in the first queue pair, and then send the first data packet to the server based on the preset source port number.

[0080] Optionally, the NVMe-over-RDMA connection establishment device 400 also includes: a second sending module, which is used to detect based on the RDMA driver that there is a second data packet to be sent in the second sending queue included in the second queue pair, then determine the source port number corresponding to the second data packet based on the queue number of the second sending queue, and send the second data packet to the server.

[0081] Optionally, the communication connection includes a transport layer connection and a protocol layer link.

[0082] The NVMe-over-RDMA connection establishment device provided in this embodiment can execute the method described in any of the above embodiments. Its execution method and beneficial effects are similar and will not be repeated here.

[0083] Figure 5 It is a structural diagram of a computer device provided by an embodiment of the present disclosure.

[0084] like Figure 5 As shown, the computer device may include a processor 510 and a memory 520 storing computer program instructions.

[0085] Specifically, the processor 510 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0086] Memory 520 may include a large-capacity memory for information or instructions. By way of example, and not limitation, memory 520 may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disk, a magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 520 may include removable or non-removable (or fixed) media. Where appropriate, memory 520 may be internal or external to the integrated gateway device. In certain embodiments, memory 520 is non-volatile solid-state memory. In certain embodiments, memory 520 includes read-only memory (ROM). Where appropriate, the ROM may be mask-programmed ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0087] The processor 510 reads and executes the computer program instructions stored in the memory 520 to perform the steps of the NVMe-over-RDMA connection establishment method provided in the embodiment of the present disclosure.

[0088] In one example, the computer device may further include a transceiver 530 and a bus 540. Figure 5 As shown, the processor 510 , the memory 520 and the transceiver 530 are connected via a bus 540 and communicate with each other.

[0089] The bus 540 includes hardware, software, or both. By way of example and not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, the bus 540 may include one or more buses. Although embodiments herein describe and illustrate a particular bus, this application contemplates any suitable bus or interconnect.

[0090] An embodiment of the present disclosure also provides a computer-readable storage medium, which can store a computer program. When the computer program is executed by a processor, the processor implements the NVMe-over-RDMA connection establishment method provided by the embodiment of the present disclosure.

[0091] The above-mentioned storage medium may, for example, include a memory 520 of computer program instructions, and the above-mentioned instructions may be executed by the processor 510 of the NVMe-over-RDMA connection establishment device to complete the NVMe-over-RDMA connection establishment method provided in the embodiment of the present disclosure. Optionally, the storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a ROM, a random access memory (Random Access Memory, RAM), a compact disc read-only memory (Compact Disc ROM, CD-ROM), a tape, a floppy disk, and an optical data storage device, etc. The above-mentioned computer program may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present disclosure, and the programming languages ​​include object-oriented programming languages ​​such as Java, C++, etc., and also include conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on the remote computing device, or entirely on the remote computing device or server.

[0092] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0093] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A method for establishing an NVMe-over-RDMA connection, characterized in that: The method is applicable to a client, and includes: In response to detecting the connection instruction, creating a management queue and creating a first queue pair based on software resources of the network card, the first queue pair being an RDMA queue pair corresponding to the management queue, and management commands stored in the first queue pair being sent and / or received through the network card based on a preset source port number; Configuring the network card and the first queue pair based on the preset source port number; Initiating a first link establishment request to the server based on the first queue, and negotiating to establish a communication connection corresponding to the management queue; Creating an IO queue and creating a second queue pair based on RDMA hardware resources, where the second queue pair is an RDMA queue pair corresponding to the IO queue, and IO commands stored in the second queue pair are sent and / or received through the RDMA hardware; Based on the second queue, a second link establishment request is initiated to the server to negotiate and establish a communication connection corresponding to the IO queue.

2. The method according to claim 1, characterized in that The creating of the first queue pair based on the network card software resource includes: Extending an RDMA operation interface based on a preset flag bit, wherein the RDMA operation interface is used to create an RDMA queue pair; The first queue pair is created based on a first value of the flag bit, where the first value is used to indicate that the first queue pair is created based on software resources of a network card.

3. The method according to claim 2, characterized in that The creating a second queue pair based on the RDMA hardware resources includes: The second queue pair is created based on a second value of the flag bit, where the second value is used to indicate that the second queue pair is created based on RDMA hardware resources.

4. The method according to claim 1, wherein After initiating a first link establishment request to the server based on the first queue, the method further includes: In response to receiving an RDMA data packet sent by the server based on the network card, parsing to obtain a source port number corresponding to the RDMA data packet, and determining whether the RDMA data packet is a data packet of the RDMA type received for the first time; If the RDMA data packet is a data packet of the RDMA type received for the first time, determining the target source port number as the source port number corresponding to the RDMA data packet, recording the target source port number, and forwarding the RDMA data packet to the network card driver, so that the network card driver stores the RDMA data packet in the first receive queue of the first queue pair, and updates the management queue based on the RDMA data packet; If the RDMA data packet is not a data packet of the RDMA type received for the first time, the source port number corresponding to the RDMA data packet is compared with the target source port number. When the source port number corresponding to the RDMA data packet is consistent with the target source port number, the RDMA data packet is forwarded to the network card driver, so that the network card driver stores the data packet in the first receive queue of the first queue pair and updates the management queue based on the data packet. When the source port number corresponding to the RDMA data packet is inconsistent with the target source port number, the RDMA data packet is forwarded to the RDMA driver, so that the RDMA driver stores the data packet in the second receive queue of the second queue pair and updates the corresponding IO queue based on the RDMA data packet.

5. The method according to claim 1, characterized in that After initiating a second link establishment request to the server based on the second queue and negotiating to establish a communication connection corresponding to the IO queue, the method further includes: Based on the network card driver detecting that there is a first data packet to be sent in the first sending queue included in the first queue pair, the first data packet is sent to the server based on the preset source port number.

6. The method according to claim 1, characterized in that After initiating a second link establishment request to the server based on the second queue and negotiating to establish a communication connection corresponding to the IO queue, the method further includes: Based on the RDMA driver detecting that there is a second data packet to be sent in the second sending queue included in the second queue pair, the source port number corresponding to the second data packet is determined based on the queue number of the second sending queue, and the second data packet is sent to the server.

7. The method according to claim 1, characterized in that The communication connection includes a transport layer connection and a protocol layer link.

8. An NVMe-over-RDMA connection establishment device, characterized in that: The device is applicable to a client, and includes: a first creation module, configured to create a management queue in response to detecting a connection instruction, and to create a first queue pair based on software resources of the network card, wherein the first queue pair is an RDMA queue pair corresponding to the management queue, and management commands stored in the first queue pair are sent and / or received through the network card based on a preset source port number; A configuration module, configured to configure the network card and the first queue pair based on the preset source port number; A first initiating module is configured to initiate a first link establishment request to a server based on the first queue, and negotiate to establish a communication connection corresponding to the management queue; a second creation module, configured to create an IO queue and create a second queue pair based on the RDMA hardware resources, wherein the second queue pair is an RDMA queue pair corresponding to the IO queue, and the IO commands stored in the second queue pair are sent and / or received through the RDMA hardware; The second initiating module is used to initiate a second link establishment request to the server based on the second queue, and negotiate to establish a communication connection corresponding to the IO queue.

9. A computer device, characterized in that: include: Memory; processor; And a computer program; wherein, the computer program is stored in the memory and is configured to be executed by the processor to implement the NVMe-over-RDMA connection establishment method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by the processor, the NVMe-over-RDMA connection establishment method according to any one of claims 1 to 7 is implemented.

Images