A storage method and system based on intelligent network card

Through the intelligent network card receiving and processing key-value requests on the server, the problem of high latency and bilateral processing resource occupation caused by RDMA unilateral primitives is solved, and low-latency and efficient key-value storage operations are achieved.

CN117896436BActive Publication Date: 2024-12-03YUSUR TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311668429.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-06
Publication Date
2024-12-03
Estimated Expiration
2043-12-06

AI Technical Summary

Technical Problem

In the prior art, the server-side CPU has no perception characteristics caused by the RDMA unilateral primitive, resulting in high latency, and bilateral processing requires occupancy of the server's processing resources, making it difficult to reduce latency without occupancying resources.

Method used

The storage method based on the intelligent network card is adopted, including the key-value request analysis module, the key-value operation processing module and the RDMA module. The intelligent network card receives the key-value request data packet, analyzes and processes the request, and feedbacks the results after completing the data processing to avoid server-side computing resources. Occupy and multiple return time.

Benefits of technology

It reduces processing delay, improves data processing efficiency, avoids server-side CPU usage, and realizes the rapid processing of key-value storage requests without occupying server resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117896436B_ABST
    Figure CN117896436B_ABST
Patent Text Reader

Abstract

The present invention provides a storage method and system based on a smart network card, the steps of the method include: the smart network card receives a key-value request data packet, and parses the key-value request data packet based on a key-value request parsing module to obtain a key-value storage address; based on a key-value operation processing module, the key-value data is queried from a hash table of a memory module through the key-value storage address; the RDMA module processes the processing request in the key-value request data packet corresponding to the data stored in the server based on the key-value data; the RDMA module feeds back the processing result to the client after completing the data processing. This solution first performs data processing through a smart network card connected to the server, without occupying the computing resources of the server. On the other hand, this solution does not require multiple round trip times because the smart network card can parse the key-value data and directly process the data, thereby reducing processing delays and improving data processing efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a storage method and system based on a smart network card. Background Art

[0002] RDMA provides unilateral and bilateral primitives based on the queue mechanism. The Send / Recv operations that are perceived by both server CPUs are called bilateral primitives, and the RDMAWrite and RDMARead operations that are not perceived by the target server CPU are called unilateral primitives. Bilateral primitives provide more efficient message primitives than sockets and can be used to build remote procedure call (RPC) services based on RDMA networks; unilateral primitives provide memory read and write semantics, allowing upper-layer applications to read and write remote node memory directly through the network based on virtual addresses. Unilateral operations do not require the participation of the target server CPU and operating system at all. In terms of the transmission layer, unilateral operations can provide higher throughput than bilateral operations. However, the server-side CPU-unawareness feature brought by unilateral operations leads to more complex application layer protocol design, and systems built based on unilateral operations cannot ensure the lowest latency. The high performance of RDMA read or write comes partly from the server-side CPU-unawareness, but the CPU-unawareness makes the server-side unable to actively parse requests, so the task of parsing requests has to be handled by the client. The process of sending, parsing and executing requests by the client requires at least two round-trip times (RTT).

[0003] In summary, the unilateral processing method of the prior art has a high delay because it requires at least two round-trip times; while the bilateral processing method of the prior art requires occupying the processing resources of the server. The prior art is difficult to reduce the processing delay without occupying the processing resources of the server. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a storage method based on a smart network card to eliminate or improve one or more defects existing in the prior art.

[0005] One aspect of the present invention provides a storage method based on a smart network card, the method is applied to a smart network card connected to a server, the smart network card includes a key value request parsing module, a key value operation processing module, a memory module and an RDMA module, and the steps of the method include:

[0006] The smart network card receives a key-value request data packet, and parses the key-value request data packet based on the key-value request parsing module to obtain a key-value storage address;

[0007] Based on the key value operation processing module, query the key value data from the hash table of the memory module through the key value storage address;

[0008] The RDMA module processes a processing request in a key-value request data packet corresponding to data stored in the server based on the key-value data, wherein the processing request includes a read request and a write request;

[0009] The RDMA module feeds back the processing result to the client after completing the data processing.

[0010] Using the above solution, this solution first performs data processing through the smart network card connected to the server, without occupying the computing resources of the server. On the other hand, since the smart network card can parse the key value data and directly process the data, it does not require multiple round trips, thereby reducing processing delays and improving data processing efficiency.

[0011] In some embodiments of the present invention, in the step of parsing the key-value request data packet based on the key-value request parsing module, it is determined whether the key-value request data packet is a UDP protocol data packet. If the key-value request data packet is a UDP protocol data packet, the target port number of the UDP protocol data packet is determined.

[0012] In some embodiments of the present invention, the step of parsing the key-value request data packet based on the key-value request parsing module also includes parsing the key-value request data packet through the target port number of the UDP protocol data packet and the target port corresponding to the target port number.

[0013] In some embodiments of the present invention, in the step of querying key value data from the hash table of the memory module through the key value storage address based on the key value operation processing module, key value data is set corresponding to each key value storage address in the hash table, and the key value data is obtained by matching the key value storage address in the hash table.

[0014] In some embodiments of the present invention, in the step in which the RDMA module processes a processing request in a key-value request data packet corresponding to data stored in the server based on the key-value data, if the processing request is a read request, the RDMA module determines a data address from which data is to be read from data stored in the server based on the key-value data, and reads data at the corresponding data address.

[0015] In some implementations of the present invention, if the processing request is a read request, in the step of the RDMA module feeding back the processing result to the client after completing the data processing, the read data is fed back to the client.

[0016] In some embodiments of the present invention, in the step in which the RDMA module processes a processing request in a key-value request data packet corresponding to data stored in the server based on the key-value data, if the processing request is a write request, the RDMA module determines a data address to write data from data stored in the server based on the key-value data, obtains the data to be written from the client based on the RDMA module, and writes the data to be written into the corresponding data address.

[0017] In some implementations of the present invention, if the processing request is a write request, in the step of the RDMA module feeding back the processing result to the client after completing data processing, the RDMA module sends a feedback response of successful writing to the client.

[0018] In some embodiments of the present invention, if the processing request is a write request, in the step in which the RDMA module determines the data address of writing data from the data stored in the server based on the key value data, the memory module releases the memory based on the data address of the written data.

[0019] The second aspect of the present invention also provides a storage system based on an intelligent network card, the system includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, when the computer instructions are executed by the processor, the system implements the steps implemented by the method described above.

[0020] The third aspect of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps implemented by the aforementioned storage method based on the smart network card.

[0021] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention can be specifically pointed out and obtained in the specification and the accompanying drawings.

[0022] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0024] Figure 1A schematic diagram of an implementation of a storage method based on a smart network card of the present invention;

[0025] Figure 2 It is a schematic diagram of the architecture of the storage method based on the smart network card of the present invention;

[0026] Figure 3 This is a diagram of data interaction when processing a GET request;

[0027] Figure 4 This is a diagram of data interaction when processing a SET request. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0029] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0030] Introduction to existing technology:

[0031] In the prior art, systems that use RDMA unilateral primitives to implement key-value storage GET operations in 1 RTT include KV-Direct and StRoM. Both systems redefine or expand the existing RDMA protocol. KV-Direct directly designs native key-value storage RDMA semantics for key-value storage, including core GET / PUT / DELETE / UPDATE commands and REDUCE / FILTER data processing commands, and implements them on Microsoft Smart NIC Catapult. However, KV-Direct uses a custom private protocol, which is currently only suitable for use within private data centers and is difficult to directly apply to existing clients. StRoM expands RPC based on the RoCEv2 RDMA protocol and abstracts the process in RPC into a hardware module, thereby offloading RPC from the server CPU to the RDMA network card side. KV-Direct and StRoM use a custom RDMA protocol, which can fundamentally optimize RDMA's support for key-value storage systems to some extent. However, it requires replacing the network cards of all client servers. Considering the large scale of existing data center servers, the cost of deploying KV-Direct and StRoM is high, and it is difficult to apply them directly on existing servers.

[0032] The steps of the present invention specifically include:

[0033] like Figure 1 and 2 As shown, the present invention proposes a storage method based on a smart network card, the method is applied to a smart network card connected to a server, the smart network card includes a key value request parsing module, a key value operation processing module, a memory module and an RDMA module, and the steps of the method include:

[0034] Step S100, the smart network card receives a key-value request data packet, and parses the key-value request data packet based on the key-value request parsing module to obtain a key-value storage address;

[0035] In the specific implementation process, the smart network card is a device with storage capacity and processing capacity. Specifically, the smart network card is provided with a smart chip, and the smart chip can be a field programmable gate array chip (Field Programmable Gate Array, FPGA) and a system-on-chip (System on Chip, SoC) and the like.

[0036] In a specific implementation process, the key-value request data packet may be a data packet sent from a client to a server. The smart network card is connected to the server. The server receives the data packet sent by the client through the smart network card. The server may be a server.

[0037] In a specific implementation process, the server and the smart network card may be connected via a high-speed peripheral component interconnect bus (Peripheral Component Interconnect Express, PCIe).

[0038] Step S200, querying key value data from a hash table of a memory module through a key value storage address based on the key value operation processing module;

[0039] In some embodiments of the present invention, the data storage method of the present solution is based on key-value storage (Key-ValueStore, KVS), which is a non-relational database structure. Each data item is called a key-value pair (Key-Value Pair, KVP), and a key-value pair can be uniquely identified by a key. The principle of key-value pair storage is to access or store a specific set of data through a unique key. In computer science, this data structure is widely used in various applications, such as databases, cache systems, message queues, etc. In key-value pair storage, there is a one-to-one correspondence between keys and values. Keys are usually used to locate and retrieve specific values, while values ​​can contain any type of data, such as strings, numbers, images, videos, etc. In storage systems, key-value pair storage usually uses hash functions to achieve mapping from keys to values. A hash function is an algorithm that converts an input (such as a string) into a fixed-length output. Through a hash function, a key can be mapped to a unique address that corresponds to a value in a storage space.

[0040] In a specific implementation process, the key-value data corresponds to a specific address of data storage.

[0041] Step S300, the RDMA module processes a processing request in a key-value request data packet corresponding to data stored in the server based on the key-value data, wherein the processing request includes a read request and a write request;

[0042] In some implementations of the present invention, the read request corresponds to a GET request, and the write request corresponds to a SET request.

[0043] Step S400: After completing data processing, the RDMA module feeds back the processing result to the client.

[0044] In some embodiments of the present invention, the smart network card pre-establishes an RDMA-based connection with the client. After the client and the server smart network card establish an RDMA connection, the server smart network card can use RDMA write (Write) and RDMA read (Read) to operate the client.

[0045] In some embodiments of the present invention, the client includes a local client and an RDMA network card connected to the local client. When the client is started, the driver registers the corresponding memory space with the local RDMA network card and provides remote RDMA access permissions. After the client RDMA network card and the server smart network card establish an RDMA connection, the server smart network card can use RDMA write (Write) and RDMA read (Read) to operate on this memory. The client and the server agree on the size of the key-value pair in advance, including the size of the key (bytes) and the size of the value (bytes).

[0046] Specifically, the network layer protocols of RDMA include InifiniBand, iWARP and RoCE v1 / v2. RoCEv2 is a network layer protocol based on UDP protocol encapsulation and supports routing of three-layer IP networks.

[0047] Using the above solution, this solution first performs data processing through the smart network card connected to the server, without occupying the computing resources of the server. On the other hand, since the smart network card can parse the key-value data and directly process the data, it does not require multiple round-trip times, thereby reducing processing delays and improving data processing efficiency. In addition, this solution is based on the existing mature network protocol design, does not need to introduce private network protocols, and has lower deployment costs.

[0048] Specifically, traditional key-value storage servers are implemented based on x86 architecture CPUs and Linux operating systems. Data processing frequently falls into the TCP / IP protocol stack and kernel, resulting in a decrease in the CPU instruction cache hit rate, an increase in virtual memory translation overhead, and a decrease in branch prediction accuracy, greatly reducing system performance. Secondly, with the continuous increase in data center network bandwidth, the existing server-side CPU can no longer handle the load faced by key-value storage. Remote Direct Memory Access (RDMA) provides network transmission services with low latency, high bandwidth, and kernel bypass advantages based on lossless networks and full hardware unloading protocol stacks. RDMA-based read and write operations can improve key-value storage performance. Smart NICs integrate a series of programmable computing resources on traditional NICs. The computing resources on the Smart NIC can offload I / O-intensive computing related to network packet processing on the host CPU and process data directly on the Smart NIC. On the one hand, it avoids the delay overhead of interacting with the host CPU, and on the other hand, it reduces the network protocol stack's occupancy of the host CPU, allowing the server-side CPU to serve more clients.

[0049] In some embodiments of the present invention, in the step of parsing the key-value request data packet based on the key-value request parsing module, it is determined whether the key-value request data packet is a UDP protocol data packet. If the key-value request data packet is a UDP protocol data packet, the target port number of the UDP protocol data packet is determined.

[0050] In the specific implementation process, in the step of determining whether the key-value request data packet is a UDP protocol data packet, the source IP address and target IP address of the key-value request data packet can be used to determine whether the source IP address and target IP address are the expected addresses.

[0051] In some embodiments of the present invention, the key-value request parsing module determines whether the received key-value request belongs to the smart network card processing based on the information in the key-value request data packet and the key-value pair size threshold. If it does not belong to the smart network card processing, it is forwarded to the network protocol stack of the server and processed by the server software. Specifically, if the size of the key-value request data packet is greater than the preset threshold, the key-value request data packet is forwarded to the network protocol stack of the server and processed by the server software.

[0052] In some embodiments of the present invention, the step of parsing the key-value request data packet based on the key-value request parsing module also includes parsing the key-value request data packet through the target port number of the UDP protocol data packet and the target port corresponding to the target port number.

[0053] In the specific implementation process, taking RoCEv2 data packets as an example, the destination port number in its UDP header is 4791, while the UDP header of Memcached (a typical key-value storage system) data packets is 11211. Based on the UDP destination port, Memcached and RoCEv2 data packets can be easily distinguished. Therefore, by using the client to transmit Memcached requests using UDP and implementing UDP data packet parsing on the smart network card, the smart network card side can perceive the requests carried by UDP.

[0054] In some embodiments of the present invention, in the step of querying key value data from the hash table of the memory module through the key value storage address based on the key value operation processing module, key value data is set corresponding to each key value storage address in the hash table, and the key value data is obtained by matching the key value storage address in the hash table.

[0055] In some embodiments of the present invention, in the step of querying key-value data from a hash table of a memory module through a key-value storage address based on the key-value operation processing module, the key-value operation processing module calculates a hash value for the key-value storage address, determines a hash table query address based on the calculated hash value, and then reads a corresponding hash table item in the hash table. The content of the hash table item indicates that the corresponding key-value data exists in the memory of the smart network card, and the storage address of the corresponding key-value data in the smart network card is obtained.

[0056] like Figure 3 As shown, in some embodiments of the present invention, in the step in which the RDMA module processes a processing request in a key-value request data packet corresponding to the data stored in the server based on the key-value data, if the processing request is a read request, the RDMA module determines a data address for reading data from the data stored in the server based on the key-value data, and reads the data at the corresponding data address.

[0057] In some implementations of the present invention, if the processing request is a read request, in the step of the RDMA module feeding back the processing result to the client after completing the data processing, the read data is fed back to the client.

[0058] In some embodiments of the present invention, in the step of feeding back the read data to the client, the RDMA module writes the read data into the client's memory. The specific RDMA write operation additionally carries a user-defined 32-bit immediate number (Immediate). The client RDMA network card writes the data directly into the client's memory and writes the immediate number into the completion queue.

[0059] By adopting the above scheme, during the processing of the above GET request, the CPU on the server side does not need to participate in the processing of the GET request at all, and all processing is completed on the smart network card. Therefore, the CPU occupancy rate of the GET request on the server side is 0, and the processing of the GET request is completed within one round trip time, avoiding the time overhead of multiple round trips.

[0060] like Figure 4 As shown, in some embodiments of the present invention, in the step in which the RDMA module processes a processing request in a key-value request data packet corresponding to the data stored in the server based on the key-value data, if the processing request is a write request, the RDMA module determines a data address to write data from the data stored in the server based on the key-value data, obtains the data to be written from the client based on the RDMA module, and writes the data to be written into the corresponding data address.

[0061] In some implementations of the present invention, in the step of obtaining the data to be written from the client based on the RDMA module, the RDMA module initiates a read data request to the client, and feeds back the data to be written based on the read request responded by the client.

[0062] In some implementations of the present invention, if the processing request is a write request, in the step of the RDMA module feeding back the processing result to the client after completing data processing, the RDMA module sends a feedback response of successful writing to the client.

[0063] In some implementations of the present invention, if the processing request is a write request, in the step of the RDMA module sending a write success feedback response to the client, the smart network card sends a SET completion response to the client through the RDMA module.

[0064] In some embodiments of the present invention, if the processing request is a write request, in the step in which the RDMA module determines the data address of writing data from the data stored in the server based on the key value data, the memory module releases the memory based on the data address of the written data.

[0065] By adopting the above scheme, compared with the prior art, the present invention applies programmable hardware on the smart network card to process the GET / SET operation in the key-value storage, which greatly reduces the CPU occupancy of the server and reduces the delay in processing the GET / SET request. At the same time, the present invention uses RDMA technology to reduce the delay in the transmission of key-value pairs, and designs a UDP / RDMA hybrid communication protocol, so that the GET operation can be completed within one round trip time, without the need for server CPU polling overhead, without relying on private RDMA protocols, and can interoperate with RDMA protocols widely deployed in existing data centers (such as RoCEv2).

[0066] An embodiment of the present invention also provides a storage system based on an intelligent network card, which includes a computer device, wherein the computer device includes a processor and a memory, wherein the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps implemented by the method described above.

[0067] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps implemented by the aforementioned storage method based on the smart network card are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0068] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0069] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0070] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0071] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A storage method based on a smart network card, characterized in that: The method is applied to a smart network card connected to a server, the smart network card comprising a key value request parsing module, a key value operation processing module, a memory module and an RDMA module, and the steps of the method include: The smart network card receives a key-value request data packet, and parses the key-value request data packet based on the key-value request parsing module to obtain a key-value storage address; Based on the key value operation processing module, query the key value data from the hash table of the memory module through the key value storage address; The RDMA module processes a processing request in a key-value request data packet corresponding to data stored in the server based on the key-value data, wherein the processing request includes a read request and a write request; The RDMA module feeds back the processing result to the client after completing the data processing.

2. The storage method based on the smart network card according to claim 1, characterized in that: In the step of parsing the key-value request data packet based on the key-value request parsing module, it is determined whether the key-value request data packet is a UDP protocol data packet. If the key-value request data packet is a UDP protocol data packet, the target port number of the UDP protocol data packet is determined.

3. The storage method based on smart network card according to claim 2, characterized in that: The step of parsing the key-value request data packet based on the key-value request parsing module also includes parsing the key-value request data packet through the target port number of the UDP protocol data packet and the target port corresponding to the target port number.

4. The storage method based on smart network card according to claim 1, characterized in that: In the step of querying key value data from the hash table of the memory module through the key value storage address based on the key value operation processing module, key value data is set corresponding to each key value storage address in the hash table, and the key value data is obtained by matching the key value storage address in the hash table.

5. The storage method based on smart network card according to claim 1, characterized in that: In the step where the RDMA module processes a processing request in a key-value request data packet corresponding to data stored in the server based on the key-value data, if the processing request is a read request, the RDMA module determines a data address from which data is read from data stored in the server based on the key-value data, and reads data at the corresponding data address.

6. The storage method based on the smart network card according to claim 5, characterized in that: If the processing request is a read request, in the step of the RDMA module feeding back the processing result to the client after completing the data processing, the read data is fed back to the client.

7. The storage method based on smart network card according to claim 1, characterized in that: In the step in which the RDMA module processes a processing request in a key-value request data packet corresponding to data stored in the server based on the key-value data, if the processing request is a write request, the RDMA module determines a data address to write data from data stored in the server based on the key-value data, obtains the data to be written from the client based on the RDMA module, and writes the data to be written into the corresponding data address.

8. The storage method based on the smart network card according to claim 7, characterized in that: If the processing request is a write request, in the step of the RDMA module feeding back the processing result to the client after completing data processing, the RDMA module sends a feedback response of successful writing to the client.

9. The storage method based on smart network card according to claim 7 or 8, characterized in that: If the processing request is a write request, in the step where the RDMA module determines a data address for writing data from data stored in the server based on the key value data, the memory module releases memory based on the data address for writing data.

10. A storage system based on a smart network card, characterized in that: The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions. The processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps implemented by the method as described in any one of claims 1 to 9.