SPDK-NVMe / TCP service acceleration method based on TOE and server terminal architecture

Abstract

The invention discloses an SPDK-NVMe / TCP service acceleration method based on TOE and a server terminal architecture, the SPDK-NVMe / TCP service acceleration method based on the TOE is applied to an NVMe / TCP server side, the NVMe / TCP server side is provided with a TOE network module, and the SPDK-NVMe / TCP service acceleration method based on the TOE is used for processing network transmission between an NVMe / TCP client side and the NVMe / TCP server side through the TOE network module. According to the SPDK-NVMe / TCP service acceleration method based on the TOE and the server terminal architecture, the problem that the service quality of an NVMe / TCP server is sharply reduced due to the fact that the CPU occupancy rate of the NVMe / TCP server is greatly increased in a high-speed transmission scene can be solved, so that the pressure of a processor is relieved, the service quality is improved, and NVMe / TCP service response is accelerated. And on the other hand, the same bandwidth can be achieved by using fewer CPU resources, so that the cost is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of network storage technology, and in particular relates to a TOE-based SPDK-NVMe / TCP service acceleration method and a server terminal architecture. Background Art

[0002] NVMe (Non-Volatile Memory Express) is a storage access and transmission protocol for flash memory and solid-state drives. The NVMe protocol defines multiple transmission methods, among which NVMe / TCP (Non-Volatile Memory Express over Transmission Control Protocol) is a transport layer protocol based on TCP / IP. Through the NVMe / TCP protocol, the client can read, write, and manage NVMe storage devices in the remote server just like accessing local NVMe devices. Compared with the NVMe transport layer based on the RDMA (Remote Direct Memory Access) network, the NVMe / TCP protocol can run on the existing Ethernet network without the need to modify the network infrastructure and has good versatility.

[0003] SPDK (Storage Performance Development Kit) is a program development library that runs in user mode and implements a server that supports the NVMe / TCP protocol. Compared with the kernel version of the NVMe / TCP server, SPDK runs in user space, which is easier to develop and debug, and can achieve higher throughput and lower latency. Therefore, SPDK is being selected by more companies as the NVMe / TCP server.

[0004] The NVMe / TCP protocol is based on TCP / IP for network communication. With the rapid growth of network transmission bandwidth, the CPU resources consumed by NVMe / TCP servers in the process of processing network transmission are increasing, which occupies the CPU resources of upper-layer applications and causes the service quality to decline. On the other hand, a large amount of CPU resources are used to process network transmission, resulting in resource waste. This is particularly obvious in the scenario of high concurrent requests from clients.

[0005] At present, there is no effective optimization solution for the problem that the CPU usage of NVMe / TCP servers in high-speed transmission scenarios increases significantly, resulting in a significant decrease in service quality. Therefore, in order to optimize the service quality of NVMe / TCP servers in high-speed transmission scenarios and accelerate service response, further research is needed on acceleration methods for NVMe / TCP services. Summary of the invention

[0006] The present invention provides a TOE-based SPDK-NVMe / TCP service acceleration method and server terminal architecture to solve the above-mentioned technical problems, and specifically adopts the following technical solutions:

[0007] A TOE-based SPDK-NVMe / TCP service acceleration method, the TOE-based SPDK-NVMe / TCP service acceleration method is applied to an NVMe / TCP server, the NVMe / TCP server is provided with a TOE network module, and the TOE-based SPDK-NVMe / TCP service acceleration method comprises:

[0008] The TOE network module processes the network transmission between the NVMe / TCP client and the NVMe / TCP server.

[0009] Furthermore, the TOE network module includes a TOE network card device, a TOE network driver and a redirection SocketAPI;

[0010] The TOE network card device is used to complete the TCP / IP protocol stack processing, encapsulate the response into a TCP message and send it to the network, or receive the client request and put the parsed data into the receiving buffer;

[0011] The SPDK-NVMe / TCP service calls the redirected Socket API to receive client requests or send server responses;

[0012] The redirected Socket API enters the TOE network driver to complete corresponding logical processing.

[0013] Furthermore, the specific method of processing the network transmission between the NVMe / TCP client and the NVMe / TCP server through the TOE network module is:

[0014] The network protocol stack processing work of the NVMe / TCP server is unloaded from the CPU to the TOE network card device through the TOE network module.

[0015] Furthermore, the specific method of processing the network transmission between the NVMe / TCP client and the NVMe / TCP server through the TOE network module includes:

[0016] The TOE network card device receives the TCP message sent by the NVMe / TCP client, parses it, and puts the parsed data into the corresponding receiving buffer;

[0017] The TOE network driver notifies the SPDK-NVMe / TCP service through epoll that the current TCP connection has received data;

[0018] The SPDK-NVMe / TCP service calls the redirected Socket API to receive the NVMe / TCP request of the NVMe / TCP client;

[0019] The redirected Socket API enters the TOE network driver to read the data in the receiving buffer;

[0020] The SPDK-NVMe / TCP service performs subsequent data parsing and processing.

[0021] Furthermore, the specific method for the SPDK-NVMe / TCP service to perform subsequent data parsing and processing is:

[0022] The SPDK-NVMe / TCP service parses the data to obtain a specific NVMe / TCP command, further parses it into an NVMe command, then sends the NVMe command to the local NVMe storage device, then checks the NVMe command results completed by the local NVMe storage device, and finally encapsulates the completed NVMe command results as an NVMe / TCP reply and sends it to the NVMe / TCP client.

[0023] Furthermore, the specific method of the SPDK-NVMe / TCP service sending the NVMe / TCP reply to the NVMe / TCP client is:

[0024] After processing the request in the working thread, the SPDK-NVMe / TCP service uses the reply content as the server response, and calls the redirection Socket API to send it to the NVMe / TCP client;

[0025] The redirection Socket API processes the specific sending logic in the TOE network driver;

[0026] In the TOE network card device, the response is packaged into a TCP message and transmitted to the network, and finally reaches the NVMe / TCP client.

[0027] Furthermore, the redirection Socket API processes the specific sending logic in the TOE network driver including:

[0028] The reply content data is placed in the corresponding sending buffer, and the address and length of the data to be sent are written into the corresponding sending queue of the TOE network card device.

[0029] A TOE-based SPDK-NVMe / TCP server terminal architecture, comprising: a TOE network module, a processor and a storage terminal;

[0030] The TOE network module is responsible for offloading the network transmission of the NVMe / TCP server and reducing the load of the processor during the network transmission process;

[0031] The processor is responsible for running the SPDK-NVMe / TCP service process to receive client requests, send server responses, and process NVMe / TCP protocols;

[0032] The storage terminal is used to implement the local NVMe read, write and management functions of the NVMe / TCP server.

[0033] The benefit of the present invention is that the TOE-based SPDK-NVMe / TCP service acceleration method and server terminal architecture provided can improve the problem that the CPU occupancy rate of the NVMe / TCP server in high-speed transmission scenarios is greatly increased, resulting in a significant decrease in its service quality, thereby alleviating processor pressure, improving service quality and accelerating NVMe / TCP service response. On the other hand, the same bandwidth can be achieved using fewer CPU resources, thereby reducing costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0035] Figure 1 A system schematic diagram of the SPDK-NVMe / TCP service acceleration method based on TOE implemented by the present invention;

[0036] Figure 2 A flowchart of the steps of receiving a client request provided by one embodiment of the present invention;

[0037] Figure 3 A flowchart of the steps of sending a server response provided by another embodiment of the present invention;

[0038] Figure 4 A schematic diagram of a TOE-based SPDK-NVMe / TCP server terminal architecture provided by the present invention. DETAILED DESCRIPTION

[0039] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0040] like Figure 1 As shown, the present invention sets a TOE network module on the SPDK-NVMe / TCP server for data communication to replace the traditional network protocol processing path. The TOE technology is used to offload the processing work of the network protocol stack of the NVMe / TCP server from the CPU to a dedicated processing unit, thereby reducing CPU occupancy and relieving processor pressure to achieve the purpose of accelerating NVMe / TCP service response and improving service quality.

[0041] Specifically, the present application discloses a TOE-based SPDK-NVMe / TCP service acceleration method, which is applied to an NVMe / TCP server, and the NVMe / TCP server is provided with a TOE network module. The TOE-based SPDK-NVMe / TCP service acceleration method of the present application mainly processes the network transmission between the NVMe / TCP client and the NVMe / TCP server through the TOE network module.

[0042] like Figure 1 As shown, in an implementation of the present application, the TOE network module includes a TOE network card device, a TOE network driver and a redirection Socket API.

[0043] The TOE network card device is used to complete the TCP / IP protocol stack processing, encapsulate the response into a TCP message and send it to the network, or receive the client request and put the parsed data into the receiving buffer. The SPDK-NVMe / TCP service calls the redirected Socket API to receive the client request or send the server response. The redirected Socket API enters the TOE network driver to complete the corresponding logical processing.

[0044] In the implementation manner of the present application, the specific method for processing the network transmission between the NVMe / TCP client and the NVMe / TCP server through the TOE network module is as follows:

[0045] The TOE network module offloads the network protocol stack processing of the NVMe / TCP server from the CPU to the TOE network card device. That is, through the TOE network module, after the SPDK-NVMe / TCP service calls the redirected Socket API, it will not enter the kernel protocol stack for processing, but directly process the corresponding function interface logic through the network driver, and TOE controls the network card device to complete the sending and receiving of TCP / IP messages. It should be noted that Figure 1 The arrows in the figure refer to the direction of data flow, not the direction of function calls.

[0046] like Figure 2 As shown, in an embodiment of the present application, the specific method for processing network transmission between the NVMe / TCP client and the NVMe / TCP server through the TOE network module includes:

[0047] S11: The NVMe / TCP client sends a request. That is, the NVMe / TCP client encapsulates the NVMe / TCP request and sends it to the NVMe / TCP server based on the TCP protocol.

[0048] S12: The TOE network card device on the NVMe / TCP server receives the request sent by the NVMe / TCP client, that is, the relevant TCP message, parses it, and puts the parsed data, that is, the NVMe / TCP request message, into the receiving buffer of the corresponding TCP connection.

[0049] S13: The TOE network driver notifies the POSIX Socket API layer of the SPDK-NVMe / TCP service through epoll that the current TCP connection has received data.

[0050] S14: The POSIX Socket API layer of the SPDK-NVMe / TCP service obtains the corresponding information through epoll_wait and calls the redirected Socket API to receive the NVMe / TCP request of the NVMe / TCP client.

[0051] S15: redirecting the Socket API to enter the TOE network driver to read data of corresponding length in the receiving buffer.

[0052] S16: SPDK-NVMe / TCP service performs subsequent data analysis and processing.

[0053] As a preferred implementation, in step S16, the specific method for the SPDK-NVMe / TCP service to perform subsequent data parsing and processing is:

[0054] The SPDK-NVMe / TCP service parses the data to obtain specific NVMe / TCP commands, further parses them into NVMe commands, and then sends the NVMe commands to the local NVMe storage device through the PCIe transport layer. It then checks the completed NVMe command results of the local NVMe storage device, and finally encapsulates the completed NVMe command results into NVMe / TCP replies and sends them to the NVMe / TCP client.

[0055] like Figure 3 As shown, in the implementation manner of the present application, the specific method by which the SPDK-NVMe / TCP service sends the NVMe / TCP reply to the NVMe / TCP client is:

[0056] S21: After the SPDK-NVMe / TCP service processes the request in the working thread, it uses the completed NVMe command result as the server response and calls the redirection Socket API to send it to the NVMe / TCP client.

[0057] S22: Redirecting the Socket API to process specific sending logic in the TOE network driver, including putting the reply content data into the corresponding TCP connection's sending buffer, writing the address and length of the data to be sent into the corresponding sending queue of the TOE network card device, etc.

[0058] S23: The TOE network card device completes the TCP protocol processing, executes the command in the sending queue, packages the response in the sending buffer into a TCP message and transmits it to the network, finally reaching the NVMe / TCP client.

[0059] S24: The NVMe / TCP client receives the response data from the NVMe / TCP server and completes an NVMe / TCP request.

[0060] like Figure 4 As shown, the present application also discloses a TOE-based SPDK-NVMe / TCP server terminal architecture to implement the aforementioned TOE-based SPDK-NVMe / TCP service acceleration method. The TOE-based SPDK-NVMe / TCP server terminal architecture includes: a TOE network module, a processor, and a storage terminal.

[0061] Specifically, the TOE network module is responsible for offloading the network transmission of the NVMe / TCP server and reducing the processor load during network transmission. The processor is responsible for running the SPDK-NVMe / TCP service process to achieve reception. The storage terminal is used to implement the local NVMe read, write and management functions of the NVMe / TCP server.

[0062] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any form, and any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A TOE-based SPDK-NVMe / TCP service acceleration method, It is characterized in that The TOE-based SPDK-NVMe / TCP service acceleration method is applied to the NVMe / TCP server, and the NVMe / TCP server is provided with a TOE network module. The TOE-based SPDK-NVMe / TCP service acceleration method includes: The TOE network module processes the network transmission between the NVMe / TCP client and the NVMe / TCP server.

2. The SPDK-NVMe / TCP service acceleration method based on TOE according to claim 1, It is characterized in that The TOE network module includes a TOE network card device, a TOE network driver and a redirection Socket API; The TOE network card device is used to complete the TCP / IP protocol stack processing, encapsulate the response into a TCP message and send it to the network, or receive the client request and put the parsed data into the receiving buffer; The SPDK-NVMe / TCP service calls the redirected Socket API to receive client requests or send server responses; The redirected Socket API enters the TOE network driver to complete corresponding logical processing.

3. The SPDK-NVMe / TCP service acceleration method based on TOE according to claim 2, It is characterized in that The specific method of processing the network transmission between the NVMe / TCP client and the NVMe / TCP server through the TOE network module is: The network protocol stack processing work of the NVMe / TCP server is unloaded from the CPU to the TOE network card device through the TOE network module.

4. The SPDK-NVMe / TCP service acceleration method based on TOE according to claim 3, It is characterized in that The specific method of processing the network transmission between the NVMe / TCP client and the NVMe / TCP server through the TOE network module includes: The TOE network card device receives the TCP message sent by the NVMe / TCP client, parses it, and puts the parsed data into the corresponding receiving buffer; The TOE network driver notifies the SPDK-NVMe / TCP service through epoll that the current TCP connection has received data; The SPDK-NVMe / TCP service calls the redirected Socket API to receive the NVMe / TCP request of the NVMe / TCP client; The redirected Socket API enters the TOE network driver to read the data in the receiving buffer; The SPDK-NVMe / TCP service performs subsequent data parsing and processing.

5. The SPDK-NVMe / TCP service acceleration method based on TOE according to claim 4, It is characterized in that The specific method for the SPDK-NVMe / TCP service to perform subsequent data parsing and processing is: The SPDK-NVMe / TCP service parses the data to obtain a specific NVMe / TCP command, further parses it into an NVMe command, then sends the NVMe command to the local NVMe storage device, then checks the NVMe command results completed by the local NVMe storage device, and finally encapsulates the completed NVMe command results as an NVMe / TCP reply and sends it to the NVMe / TCP client.

6. The SPDK-NVMe / TCP service acceleration method based on TOE according to claim 5, It is characterized in that The specific method by which the SPDK-NVMe / TCP service sends the NVMe / TCP reply to the NVMe / TCP client is: After processing the request in the working thread, the SPDK-NVMe / TCP service uses the reply content as the server response, and calls the redirection Socket API to send it to the NVMe / TCP client; The redirection Socket API processes the specific sending logic in the TOE network driver; In the TOE network card device, the response is packaged into a TCP message and transmitted to the network, and finally reaches the NVMe / TCP client.

7. The SPDK-NVMe / TCP service acceleration method based on TOE according to claim 6, It is characterized in that The redirection Socket API processes the specific sending logic in the TOE network driver including: The reply content data is placed in the corresponding sending buffer, and the address and length of the data to be sent are written into the corresponding sending queue of the TOE network card device.

8. A TOE-based SPDK-NVMe / TCP server terminal architecture, It is characterized in that Includes: TOE network module, processor and storage terminal; The TOE network module is responsible for offloading the network transmission of the NVMe / TCP server and reducing the load of the processor during the network transmission process; The processor is responsible for running the SPDK-NVMe / TCP service process to receive client requests, send server responses, and process NVMe / TCP protocols; The storage terminal is used to implement the local NVMe read, write and management functions of the NVMe / TCP server.

Images