Toe-based spdk-nvme / tcp service acceleration method and server terminal architecture

Abstract

The application discloses a kind of SPDK-NVMe / TCP service acceleration method and server terminal architecture based on TOE, in which the SPDK-NVMe / TCP service acceleration method based on TOE is applied to NVMe / TCP server, and NVMe / TCP server is provided with TOE network module, and the SPDK-NVMe / TCP service acceleration method based on TOE is to process the network transmission between NVMe / TCP client and NVMe / TCP server through TOE network module.The SPDK-NVMe / TCP service acceleration method and server terminal architecture based on TOE of the application can improve the problem that the service quality of NVMe / TCP server is sharply decreased due to the significant increase of CPU occupancy rate in high-speed transmission scenario, thereby relieving the processor stress, improving the service quality and accelerating the NVMe / TCP service response.On the other hand, the same bandwidth can be achieved using less CPU resources, thereby reducing the cost.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of network storage, and particularly relates to a SPDK-NVMe / TCP service acceleration method based on a TOE and a server terminal architecture. BACKGROUND

[0002] NVMe (Non-Volatile Memory Express) is a storage access and transmission protocol for flash memory and solid state disks. The NVMe protocol defines multiple transmission modes, and NVMe / TCP (Non-Volatile Memory Express over Transmission Control Protocol) is a transmission layer protocol based on TCP / IP. Through the NVMe / TCP protocol, a client can read and write and manage NVMe storage devices in a remote server as if it were accessing a local NVMe device. Compared with the NVMe transmission layer based on the RDMA (Remote Direct Memory Access) network, the NVMe / TCP protocol can run in the existing Ethernet, without the need to modify the network infrastructure, and has good universality.

[0003] SPDK (Storage Performance Development Kit) is a program development library running in the user mode, which implements a server supporting the NVMe / TCP protocol. Compared with the kernel version of the NVMe / TCP server, the SPDK runs in the user space, is convenient for development and debugging, and can achieve higher throughput and lower latency, so the SPDK is being selected by more and more enterprises as the NVMe / TCP server.

[0004] The NVMe / TCP protocol is based on TCP / IP for network communication. With the rapid growth of network transmission bandwidth, the CPU resources consumed by the NVMe / TCP server in processing network transmission are increasing, which occupies the CPU resources of the upper layer application program, causing the quality of service to decline; on the other hand, a large amount of CPU resources are used to process network transmission, causing resource waste. This is particularly evident in the scenario of high concurrency requests of the client.

[0005] At present, there is no effective optimization scheme for the problem that the CPU occupancy rate of the NVMe / TCP server is greatly improved in the high-speed transmission scenario, resulting in a significant decline in the quality of service. Therefore, in order to optimize the quality of service of the NVMe / TCP server in the high-speed transmission scenario and accelerate the service response, it is necessary to further study the acceleration method for the NVMe / TCP service. SUMMARY

[0006] The application provides a TOE-based SPDK-NVMe / TCP service acceleration method and server terminal architecture to solve the above-mentioned technical problems.

[0007] A TOE-based SPDK-NVMe / TCP service acceleration method is applied to an NVMe / TCP server, and the NVMe / TCP server is provided with a TOE network module.

[0008] The TOE network module processes network transmission between an NVMe / TCP client and the NVMe / TCP server.

[0009] Further, the TOE network module includes a TOE network card device, a TOE network driver and a redirected Socket API.

[0010] The TOE network card device is used to complete TCP / IP protocol stack processing, encapsulate a response into a TCP packet and send the TCP packet to a network, or receive a client request and put parsed data into a receiving buffer.

[0011] The SPDK-NVMe / TCP service calls the redirected Socket API to receive a client request or send a server response.

[0012] The redirected Socket API enters the TOE network driver to complete corresponding logical processing.

[0013] Further, the specific method for processing network transmission between the NVMe / TCP client and the NVMe / TCP server through the TOE network module is as follows:

[0014] The TOE network module offloads network protocol stack processing work of the NVMe / TCP server from a CPU to the TOE network card device.

[0015] Further, the specific method for processing network transmission between the NVMe / TCP client and the NVMe / TCP server through the TOE network module includes:

[0016] The TOE network card device receives a TCP packet sent by the NVMe / TCP client, performs analysis and puts parsed data into a corresponding receiving buffer.

[0017] The TOE network driver notifies the SPDK-NVMe / TCP service that the current TCP connection has received data through epoll;

[0018] The SPDK-NVMe / TCP service calls the redirection Socket API to receive the NVMe / TCP request of the NVMe / TCP client;

[0019] The redirection Socket API enters the TOE network driver to read the data in the receive buffer;

[0020] The SPDK-NVMe / TCP service performs subsequent data analysis and processing.

[0021] Further, the specific method for the SPDK-NVMe / TCP service to perform subsequent data analysis and processing is as follows:

[0022] The SPDK-NVMe / TCP service analyzes the data to obtain a specific NVMe / TCP command, further analyzes the NVMe / TCP command into an NVMe command, then issues the NVMe command to a local NVMe storage device, then checks the completed NVMe command result of the local NVMe storage device, and finally encapsulates the completed NVMe command result as an NVMe / TCP reply and sends it to the NVMe / TCP client.

[0023] Further, the specific method for the SPDK-NVMe / TCP service to send the NVMe / TCP reply to the NVMe / TCP client is as follows:

[0024] After the SPDK-NVMe / TCP service processes the request in the worker thread, the service sends the reply content as a server response to the NVMe / TCP client by calling the redirection Socket API;

[0025] The redirection Socket API processes specific sending logic in the TOE network driver;

[0026] The response is packaged into a TCP packet in the TOE network card device and transmitted to the network, and finally reaches the NVMe / TCP client.

[0027] Further, the specific sending logic processed by the redirection Socket API in the TOE network driver includes:

[0028] The reply content data is placed in the corresponding sending buffer, and the data address and length to be sent are written to the corresponding sending queue of the TOE network card device.

[0029] A TOE-based SPDK-NVMe / TCP server terminal architecture, comprising: a TOE network module, a processor and a storage terminal;

[0030] The TOE network module is responsible for offloading the network transmission of the NVMe / TCP server, reducing the load of the processor in the network transmission process;

[0031] The processor is responsible for running the SPDK-NVMe / TCP service process, realizing receiving client requests and sending server responses and NVMe / TCP protocol processing;

[0032] The storage terminal is used to realize the local NVMe read and write and management function of the NVMe / TCP server.

[0033] The application has the advantages that the provided TOE-based SPDK-NVMe / TCP service acceleration method and server terminal architecture can improve the problem that the service quality of the NVMe / TCP server is significantly reduced due to the significant increase of CPU occupancy rate in the high-speed transmission scene, thereby relieving the pressure of the processor, improving the service quality and accelerating the NVMe / TCP service response. On the other hand, the same bandwidth can be achieved by using less CPU resources, thereby reducing the cost. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0035] Figure 1 A system schematic diagram for implementing the TOE-based SPDK-NVMe / TCP service acceleration method of the present application;

[0036] Figure 2 A step flowchart for receiving client requests provided by an embodiment of the present application;

[0037] Figure 3 A step flowchart for sending server responses provided by another embodiment of the present application;

[0038] Figure 4 A TOE-based SPDK-NVMe / TCP server terminal architecture schematic diagram provided by the present application. DETAILED DESCRIPTION

[0039] Embodiments of the present application are described below in detail, examples of which are shown in the accompanying drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0040] As shown in Figure 1 the present application sets a TOE network module on the SPDK-NVMe / TCP server side to replace the traditional network protocol processing path. The TOE technology is used to offload the processing work of the network protocol stack of the NVMe / TCP server from the CPU to a dedicated processing unit, thereby reducing CPU occupation and relieving processor pressure, so as to achieve the purpose of accelerating NVMe / TCP service response and improving service quality.

[0041] Specifically, the present application discloses a TOE-based SPDK-NVMe / TCP service acceleration method, which is applied to an NVMe / TCP server side. The NVMe / TCP server side is provided with a TOE network module. The TOE-based SPDK-NVMe / TCP service acceleration method of the present application mainly processes the network transmission between the NVMe / TCP client and the NVMe / TCP server side through the TOE network module.

[0042] As shown in Figure 1 in the embodiment of the present application, the TOE network module includes a TOE network card device, a TOE network driver and a redirected Socket API.

[0043] The TOE network card device is used to complete TCP / IP protocol stack processing, encapsulate the response into a TCP packet and send it to the network, or receive the client request and put the parsed data into the receiving buffer. The SPDK-NVMe / TCP service calls the redirected Socket API to receive the client request or send the server response. The redirected Socket API enters the TOE network driver to complete the corresponding logical processing.

[0044] In the embodiment of the present application, the specific method of processing the network transmission between the NVMe / TCP client and the NVMe / TCP server side through the TOE network module is as follows:

[0045] The network protocol stack processing of the NVMe / TCP server is offloaded from the CPU to the TOE network card device through the TOE network module. That is, after the SPDK-NVMe / TCP service calls the redirection Socket API, it does not enter the kernel protocol stack for processing, but directly processes the corresponding function interface logic through the network driver, and the TOE control network card device completes the transmission and reception of TCP / IP messages. It should be noted that, Figure 1 The arrow direction in the figure indicates the flow direction of data, not the direction of function call.

[0046] As shown in Figure 2 In the embodiments of the present application, the specific method for processing network transmission between the NVMe / TCP client and the NVMe / TCP server through the TOE network module includes:

[0047] S11: The NVMe / TCP client sends a request. That is, the NVMe / TCP client encapsulates the NVMe / TCP request and sends it to the NVMe / TCP server based on the TCP protocol.

[0048] S12: The TOE network card device on the NVMe / TCP server receives the request sent by the NVMe / TCP client, i.e., the related TCP message, performs analysis, and puts the parsed data, i.e., the NVMe / TCP request message, into the receiving buffer of the corresponding TCP connection.

[0049] S13: The TOE network driver notifies the POSIX Socket API layer of the SPDK-NVMe / TCP service that the current TCP connection has received data through epoll.

[0050] S14: The POSIX Socket API layer of the SPDK-NVMe / TCP service obtains the corresponding information through epoll_wait, and calls the redirection Socket API to receive the NVMe / TCP request of the NVMe / TCP client.

[0051] S15: The redirection Socket API enters the TOE network driver to read the corresponding length of data in the receiving buffer.

[0052] S16: The SPDK-NVMe / TCP service performs subsequent data analysis and processing.

[0053] As a preferred embodiment, in step S16, the specific method for the SPDK-NVMe / TCP service to perform subsequent data analysis and processing is:

[0054] The SPDK-NVMe / TCP service parses data to obtain a specific NVMe / TCP command, further parses the NVMe / TCP command into an NVMe command, then transmits the NVMe command to a local NVMe storage device through a PCIe transmission layer, checks the completed NVMe command result of the local NVMe storage device, and finally encapsulates the completed NVMe command result into an NVMe / TCP reply and sends the NVMe / TCP reply to the NVMe / TCP client.

[0055] As shown in Figure 3 In the embodiments of the present application, the specific method of the SPDK-NVMe / TCP service sending the NVMe / TCP reply to the NVMe / TCP client is as follows:

[0056] S21: After the SPDK-NVMe / TCP service processes the request in the worker thread, the completed NVMe command result is taken as a server response, and a redirection Socket API is called to send the server response to the NVMe / TCP client.

[0057] S22: The redirection Socket API processes specific sending logic in the TOE network driver, including placing reply content data into the sending buffer of the corresponding TCP connection, writing the data address and length to be sent to the corresponding sending queue of the TOE network card device, and the like.

[0058] S23: The TOE network card device completes TCP protocol processing, executes the command in the sending queue, packages the response in the sending buffer into a TCP packet, and transmits the TCP packet to the network, and finally reaches the NVMe / TCP client.

[0059] S24: The NVMe / TCP client receives the NVMe / TCP server response data, and completes one NVMe / TCP request.

[0060] As shown in Figure 4 The present application also discloses a TOE-based SPDK-NVMe / TCP server terminal architecture for implementing the foregoing TOE-based SPDK-NVMe / TCP service acceleration method. The TOE-based SPDK-NVMe / TCP server terminal architecture comprises a TOE network module, a processor, and a storage terminal.

[0061] Specifically, the TOE network module is responsible for offloading network transmission of the NVMe / TCP server, reducing the load of the processor in the network transmission process. The processor is responsible for running the SPDK-NVMe / TCP service process and realizing receiving. The storage terminal is used to realize local NVMe read and write and management functions of the NVMe / TCP server.

[0062] The above shows and describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the above embodiments do not limit the present application in any form, and any technical solutions obtained by equivalent replacement or equivalent transformation fall within the protection scope of the present application.

Claims

1. A TOE-based SPDK-NVMe / TCP service acceleration method, characterized in that, The SPDK-NVMe / TCP service acceleration method based on the TOE is applied to an NVMe / TCP server, the NVMe / TCP server is provided with a TOE network module, and the SPDK-NVMe / TCP service acceleration method based on the TOE comprises the following steps of: processing network transmission between an NVMe / TCP client and the NVMe / TCP server through the TOE network module; the TOE network module comprises a TOE network card device, a TOE network driver and a redirected Socket API; the TOE network card device is used for completing TCP / IP protocol stack processing, encapsulating a response into a TCP packet and sending the TCP packet to a network, or receiving a client request and putting parsed data into a receiving buffer; the SPDK-NVMe / TCP service receives a client request or sends a server response by calling the redirected Socket API; the redirected Socket API enters the TOE network driver to complete corresponding logical processing; the specific method for processing network transmission between the NVMe / TCP client and the NVMe / TCP server through the TOE network module comprises the following steps of: offloading network protocol stack processing work of the NVMe / TCP server from a CPU to the TOE network card device through the TOE network module; the specific method for processing network transmission between the NVMe / TCP client and the NVMe / TCP server through the TOE network module comprises the following steps of: the TOE network card device receives a TCP packet sent by the NVMe / TCP client, parses the TCP packet and puts parsed data into a corresponding receiving buffer; the TOE network driver notifies the SPDK-NVMe / TCP service that a current TCP connection has received data by using an epoll; the SPDK-NVMe / TCP service receives an NVMe / TCP request of the NVMe / TCP client by calling the redirected Socket API; the redirected Socket API enters the TOE network driver to read data in the receiving buffer; the SPDK-NVMe / TCP service performs subsequent data parsing and processing; the specific method for the SPDK-NVMe / TCP service to perform subsequent data parsing and processing comprises the following steps of: the SPDK-NVMe / TCP service parses data to obtain a specific NVMe / TCP command, further parses the specific NVMe / TCP command into an NVMe command, subsequently issues the NVMe command to a local NVMe storage device, then checks a completed NVMe command result of the local NVMe storage device, and finally encapsulates the completed NVMe command result into an NVMe / TCP reply and sends the NVMe / TCP reply to the NVMe / TCP client.

2. The SPDK-NVMe / TCP service acceleration method based on the TOE according to claim 1, wherein The specific method of the SPDK-NVMe / TCP service sending a NVMe / TCP reply to the NVMe / TCP client is: After the request is processed in the worker thread, the SPDK-NVMe / TCP service sends the reply content as a server response to the NVMe / TCP client by calling the redirect Socket API; The redirect Socket API processes the specific sending logic in the TOE network driver; The reply is packaged into a TCP packet in the TOE network card device and transmitted to the network, and finally reaches the NVMe / TCP client.

3. The TOE-based SPDK-NVMe / TCP service acceleration method according to claim 2, wherein the redirect Socket API processes the specific sending logic in the TOE network driver, including: The reply content data is placed in the corresponding sending buffer, and the data address and length to be sent are written to the corresponding sending queue of the TOE network card device. It includes a TOE network module, a processor and a storage terminal.

4. A TOE-based SPDK-NVMe / TCP server, characterized by, The TOE network module is responsible for offloading the network transmission of the NVMe / TCP server and reducing the load of the processor during network transmission. The processor is responsible for running the SPDK-NVMe / TCP service process, implementing receiving client requests and sending server responses, and processing NVMe / TCP protocols. The storage terminal is used to realize the local NVMe read and write and management functions of the NVMe / TCP server. The TOE network module processes the network transmission between the NVMe / TCP client and the NVMe / TCP server. The TOE network module includes a TOE network card device, a TOE network driver and a redirect Socket API. The TOE network card device is used to complete TCP / IP protocol stack processing, encapsulate the reply into a TCP packet and send it to the network, or receive client requests and place the parsed data into a receiving buffer. The SPDK-NVMe / TCP service calls the redirect Socket API to receive client requests or send server responses. The redirect Socket API enters the TOE network driver to complete corresponding logic processing. The specific method of processing the network transmission between the NVMe / TCP client and the NVMe / TCP server through the TOE network module is: The network protocol stack processing of the NVMe / TCP server is offloaded from the CPU to the TOE network card device through the TOE network module; The specific method of processing the network transmission between the NVMe / TCP client and the NVMe / TCP server through the TOE network module includes: The TOE network card device receives the TCP packet sent by the NVMe / TCP client, parses it, and places the parsed data into the corresponding receiving buffer. ​ The TOE network driver notifies the SPDK-NVMe / TCP service that the current TCP connection has received data through epoll; The SPDK-NVMe / TCP service calls the redirection Socket API to receive the NVMe / TCP request of the NVMe / TCP client; The redirection Socket API enters the TOE network driver to read the data in the receiving buffer; The SPDK-NVMe / TCP service performs subsequent data analysis and processing; The specific method for the SPDK-NVMe / TCP service to perform subsequent data analysis and processing is as follows: The SPDK-NVMe / TCP service analyzes the data to obtain a specific NVMe / TCP command, further analyzes the NVMe command, then issues the NVMe command to a local NVMe storage device, then checks the completed NVMe command result of the local NVMe storage device, and finally encapsulates the completed NVMe command result as an NVMe / TCP reply and sends it to the NVMe / TCP client.

Images