LSM Tree key value type database access method based on RDMA (Remote Direct Memory Access) Chain

Abstract

The invention discloses an RDMA Chain-based LSM Tree key value type database access method. The method comprises the following steps: a database client sends a request containing a version number, a key and an operation type to a database server main node; the server main node analyzes the request and processes the request according to the operation type; during write operation, a new serial number is generated, and data is inserted into a skip list of a memory table MemTable; when reading operation is carried out and the version number is smaller than 0, whether a key exists or not is checked through a Bloom filter, if yes, query is carried out at a secondary node through the RDMA technology, and if not, an SSTable file in a disk is directly accessed; and when the reading operation is performed and the version number is greater than or equal to 0, preferentially searching the data of the corresponding version in the MemTable, and if the data of the corresponding version fails, querying the SSTable. Through the RDMA technology, CPU occupation and network round-trip time are reduced, and the response speed and concurrent processing capacity of the system are improved.

Description

Technical Field

[0001] The present invention relates to the field of database management, and particularly to a method for accessing a key-value database of an LSM Tree based on an RDMA Chain. Background Art

[0002] Key-value storage databases based on LSM-Tree are widely used in web systems, such as Bigtable, LevelDB, and RocksDB. The core design point of LSM-Tree is to solve the performance problem of traditional storage structures under high write loads. It first performs all write operations (including insertions, updates, and deletions) in a memory structure (MemTable), and then writes the accumulated data to disk in batches in sequence to form SSTable files, converting random writes into sequential writes and greatly improving write performance.

[0003] RDMA Chain (Remote Direct Memory Access Chain) is a technical concept mainly used to optimize the data transmission process in network communication. It combines the advantages of RDMA technology and achieves efficient data transmission and processing through a series of predefined operation steps.

[0004] LSM Tree, that is, Log Structured Merge Tree, is a data structure used for storing and retrieving large amounts of data. It is particularly suitable for scenarios that require efficient write operations and is widely used in modern database systems, such as Cassandra, LevelDB, RocksDB, etc.

[0005] Web key-value storage systems based on LSM-Tree face huge challenges. If the traditional request processing model is used, performance bottlenecks may occur. The application layer only sends remote procedure call (RPC) requests as a client and hands over all data processing logic to the storage layer (server) for execution. This is because the access operations of Skiplist may involve multi-level access to multiple nodes and are CPU-intensive operations. In particular, when receiving a large number of concurrent requests, the CPU overhead of the server will further increase. This will cause the CPU resources of the server to be quickly exhausted, thereby affecting the performance of the key-value storage system.

[0006] RDMA (Remote Direct Memory Access) is an advanced network technology that allows a client to directly access the memory of a remote server without involving the server's CPU. In scenarios with frequent read and write operations, such as in key-value stores, one-sided RDMA operations are more efficient than two-sided RDMA operations. This is because one-sided operations can be executed without the intervention of the server CPU, resulting in lower latency and higher throughput. Therefore, by leveraging one-sided RDMA operations, optimized data transfer paths and reduced CPU overhead significantly improve the performance of key-value stores deployed in modern data centers. This is particularly beneficial for web applications with high read and write workloads.

[0007] RDMA is widely adopted in key-value stores to improve system performance. By enabling direct memory access for remote nodes, RDMA significantly reduces network communication latency and improves bandwidth utilization. For GET operations, systems like Pilaf use multiple RDMA read operations to achieve efficient data retrieval. This approach shifts the burden of hash calculation from the server to the client, thus reducing server-side computational overhead. This client-driven model enables the server to handle more concurrent requests by minimizing CPU overhead. However, due to hash collisions, GET operations may require multiple reads to locate the target key, increasing network round-trips and performance degradation.

[0008] Most RDMA-based key-value store systems are built on hash tables, mainly supporting operations on individual key-value pairs. These systems reduce data access latency and improve system throughput by optimizing data structures and communication protocols. However, their performance is still limited by the inherent characteristics of hash tables, such as hash collisions and lack of support for range queries.

[0009] Due to the dynamic structure of Skiplist and its effective support for range queries, it has gradually been used in key-value stores to replace hash tables. The Skiplist widely used in the MemTable of key-value stores based on LSM Tree does not fully utilize the capabilities of RDMA. In these systems, the client must traverse the Skiplist from the head node to the target node, accessing each node in turn. Since a single RDMA operation can only retrieve data from one node, traversing the Skiplist requires multiple network round-trip times (rtt), resulting in increased latency and potential performance bottlenecks.

[0010] RS-store is based on Skiplist and it uses RDMA to optimize key-value storage. To further improve the efficiency of remote access, RS-store proposes a new type of skiplist, namely R-Skiplist, which organizes multiple skiplist nodes into a single data block. In R-Skiplist, one RDMA operation can retrieve the entire data block and obtain multiple nodes at once, reducing the number of network round trips and improving data access efficiency. RS-store also implements a lock-free concurrent control strategy. Lock-free concurrent control reduces thread contention and context switching by avoiding the use of traditional lock mechanisms, thus improving the system's throughput. This strategy is particularly suitable for environments that handle a large number of concurrent requests. In addition, the lock-free concurrent control strategy allows local access and remote access operations to coexist on the same system without conflicts. According to relevant research and evaluations, the performance of RS-store is 0.6 to 1 times higher than that of traditional key-value storage based on Skiplist. However, this technology still requires multiple RDMA operations, which affects the system performance.

[0011] The RedN framework proposes a new method of offloading complex operations to the NIC. RedN introduces the concept of self-modifying RDMAChain, enabling the RNIC to autonomously perform conditional, branching, and loop operations.

[0012] In RedN, two RDMA operations, WAIT and ENABLE, are introduced to enhance the synchronization and control mechanisms of RDMA. WAIT ensures that the previous RDMA operation is completed before the next operation is executed, which is crucial for maintaining the correct operation order and preventing data inconsistency. ENABLE manages the prefetch function of RDMA operations, ensuring that the data modified by the previous operation remains consistent during execution. This fine-grained control of the RDMA Chain allows the RedN framework to effectively manage complex operations on the NIC.

[0013] The RedN framework utilizes existing RDMA sorting and CAS primitives to construct conditionals and loops, thereby demonstrating the Turing completeness of RDMA operations. This means that RDMA is not only a data transfer technology but also a general-purpose computing platform capable of performing arbitrary computational tasks. By self-modifying RDMA operations, RedN can achieve operations that are difficult to implement with traditional RDMA operations. For example, using the CAS primitive, RedN can dynamically edit subsequent verbs in an RDMA program and modify the execution path of the RNIC to enable conditional branching. This ability allows the NIC to independently handle more complex tasks. These features make RedN a promising framework that can offload complex operations in key-value stores to the network card without the need for additional hardware. The RedN framework is not currently widely used in LSM Tree-based key-value databases, but the ability of the RNIC to perform conditional judgment and loop traversal provides a direction for optimizing the system performance of key-value databases. Summary of the Invention

[0014] To solve the above problems, the present invention proposes a method and system for accessing an LSM Tree key-value database based on RDMA Chain, which utilizes the RedN framework to optimize the performance of LSM Tree-based key-value storage. By integrating self-modifying RDMA Chain into the MemTable access of LSM-Tree, complex access operations are transferred to the network card, thereby reducing the server CPU overhead and improving the overall system performance.

[0015] The specific solutions are as follows:

[0016] A method for accessing an LSM Tree key-value database based on RDMA Chain includes:

[0017] S1, a database client sends a database operation request to the main node of the database server; the database operation request information includes a version number, a key, and an operation type;

[0018] S2, the main node of the database server parses the request and determines the operation type; if the operation type is a write operation, go to S3; if the operation type is a read operation and the version number is less than 0, go to S4; if the operation type is a read operation and the version number is greater than or equal to 0, go to S7;

[0019] S3, the main node of the database server generates a new sequence number, packs the new sequence number, the key, and the operation type in the operation request information into a skip list node, and inserts the skip list node into the skip list of the in-memory table MemTable to complete the write operation;

[0020] S4, the database server checks the Bloom filter to determine whether the key in the database operation request exists in the skip list of the in-memory table MemTable; if it exists, proceed to S5; if not, proceed to S6;

[0021] S5, the secondary node of the database server publishes a Remote Direct Memory Access Chain (RDMA Chain) corresponding to the request, unloads the query operation of the primary node of the database server from the Central Processing Unit (CPU) to the Remote Direct Memory Access Network Interface Card (RNIC) of the secondary node of the database server. The RNIC queries the skip list of the MemTable of the primary node of the database server according to the RDMA Chain and returns the query result to the primary node of the database server. Then, the primary node of the server returns the query result to the client to complete the query;

[0022] S6, the primary node of the database server notifies the CPU to directly access the Sorted String Table (SSTable) file on the disk to obtain the query result and returns the query result to the client to complete the query;

[0023] S7, the primary node of the database server first queries the skip list node corresponding to the key to be queried under the corresponding version number in the skip list of the MemTable. If the query is successful, the query result is returned to the client; if the query fails, it queries in the SSTable file and returns the query result to the client.

[0024] Furthermore, in S1, the database operation request information further includes the value corresponding to the key and the length of the value corresponding to the key. When the operation type is query or delete, the length of the value corresponding to the key is 0.

[0025] Furthermore, in S1, the version number only takes effect during the query operation. When the version number is greater than or equal to 0, it means that the client requests to query the value corresponding to a certain key under this version number; when the version number is less than 0, it means that the client requests the value corresponding to a certain key under the latest version.

[0026] Furthermore, in S3, the skip list nodes are first sorted in ascending order of keys and then sorted in descending order of serial numbers to ensure that among the different skip list nodes corresponding to the same key, the new skip list nodes are arranged before the old skip list nodes.

[0027] Furthermore, in S4, when the database server checks the Bloom filter, specifically: multiple hash functions are used by the Bloom filter to calculate the hash value corresponding to the key, and the positions of the bit arrays corresponding to these hash values are checked. If the values of all corresponding positions of the bit arrays are 1, it is determined that the key exists in the skip list; otherwise, it is determined that the key does not exist in the skip list.

[0028] The present invention adopts the above technical solutions and has the following beneficial effects:

[0029] (1) The present invention combines a key-value database based on the LSM Tree and the RDMA technology, and unloads the SkipList query operation of the RDMA Chain of the RedN framework to the RNIC, so as to reduce the CPU occupancy rate, release CPU resources and reduce network request latency;

[0030] (2) The task scheduler of the LSM Tree key-value database of the present invention schedules different access modes according to the type of specific tasks, realizing high-performance access. Description of the Drawings

[0031] Figure 1 is a flowchart of the method for accessing an LSM Tree key-value database based on the RDMA Chain according to an embodiment of the present invention;

[0032] Figure 2 is a structural diagram of the CamelDB system according to an embodiment of the present invention;

[0033] Figure 3 is a flowchart of remote access according to an embodiment of the present invention;

[0034] Figure 4 is a latency graph of read requests of different numbers of clients according to an embodiment of the present invention;

[0035] Figure 5 is a latency graph of different numbers of clients under different write ratios according to an embodiment of the present invention;

[0036] Figure 6 is a CPU occupancy rate graph of read requests of different numbers of clients according to an embodiment of the present invention;

[0037] Figure 7 is a CPU occupancy rate graph of different numbers of clients under different write ratios according to an embodiment of the present invention. Detailed Embodiments

[0038] The present invention will be further described in detail below in conjunction with the embodiments and the drawings, but the embodiments of the present invention are not limited thereto.

[0039] As Figure 1 shown, the method for accessing an LSM Tree key-value database based on the RDMA Chain of the present invention includes:

[0040] S1, a database client sends a database operation request to the main node of the database server; the database operation request includes a version number, a key, and an operation type.

[0041] Specifically, the database operation request further includes the value corresponding to the key and the length of the value corresponding to the key. When the operation type is query or delete, the length of the value corresponding to the key is 0.

[0042] Specifically, the version number takes effect only during query operations. When the version number is greater than or equal to 0, it means that the client requests to query the value corresponding to a certain key under this version number; when the version number is less than 0, it means that the client requests to query the value corresponding to a certain key under the latest version.

[0043] Specifically, as Figure 2 shown, the architecture of CamelDB (an RDMA Chain-based LSM Tree key-value database) is presented, mainly consisting of two layers: the application layer and the storage layer. The nodes in the application layer are called database clients, and the nodes in the storage layer are called database servers and support local access and remote access. Local access means that the client sends a request to the server through RPC, and the server processes the request through the CPU and returns the result to the client. Local access is mainly applicable to operations that do not require remote access to data, such as inserting, updating, deleting data in the MemTable or querying data stored in the SSTable. Remote access means that the secondary node of the server publishes the RDMA Chain of the RedN framework, and the RDMA network card directly accesses the skip list in the MemTable of the server's primary node instead of the server's primary node CPU to complete the query operation through cyclic traversal, and then the server returns the queried data to the client.

[0044] S2, the database server primary node parses the request and determines the operation type; if the operation type is a write operation, go to S3; if the operation type is a read operation and the version number is less than 0, go to S4; if the operation type is a read operation and the version number is greater than or equal to 0, go to S7.

[0045] S3, the database server primary node generates a new sequence number, packs the new sequence number, the key, the value, and the operation type in the operation request into a skip list node, and inserts the skip list node into the skip list of the in-memory table MemTable to complete the write operation.

[0046] Specifically, the skip list nodes are first sorted in ascending order of the key, and then sorted in descending order of the sequence number to ensure that among the different skip list nodes corresponding to the same key, the new skip list node is ranked before the old skip list node.

[0047] S4, the database server checks the Bloom filter to determine whether the key in the database operation request exists in the skip list of the in-memory table MemTable; if it exists, go to S5; if it does not exist, go to S6.

[0048] Specifically, when the database server checks the Bloom filter, it specifically calculates the hash values corresponding to the key using multiple hash functions through the Bloom filter, and checks the positions of the bit array corresponding to these hash values. If the values of all corresponding positions in the bit array are 1, it can be determined that the key exists in the skip list; otherwise, it is determined that the key does not exist in the skip list.

[0049] Specifically, in this embodiment, the role of the Bloom Filter is for the read (get) operation. The main difference between the remote access mode and the local access mode lies in whether the key is in the skip list of the MemTable. To facilitate remote access, a Bloom filter is added on the basis of the traditional MemTable to quickly determine whether a certain key is in the Skiplist.

[0050] S5, the secondary node of the database server publishes a Remote Direct Memory Access Chain (RDMA Chain) corresponding to the request, unloads the query operation of the primary node of the database server from the Central Processing Unit (CPU) to the Remote Direct Memory Access Network Interface Card (RNIC) of the secondary node of the database server. The RNIC queries the skip list of the MemTable of the primary node of the database server according to the RDMA Chain, returns the query result to the primary node of the database server, and then the primary node of the server returns the query result to the client to complete the query.

[0051] Specifically, when the secondary node of the server unloads the query task to the RNIC, the RNIC uses a self-modifying RDMA chain to access the skip list of the MemTable of the primary node. In this embodiment, taking the query key x as an example, as Figure 3 shown, the specific steps are as follows:

[0052] (1) The primary node of the server provides the secondary node of the server with the key x to be queried and the address N0 of the first node in the skip list, sets the old value of the Compare-And-Swap (CAS) operation (an atomic operation) of the secondary node of the server to the key x to be queried, and sets the src of the READ operation to N0;

[0053] (2) The RDMA network card executes the READ operation, reads the skip list node according to src, and sets the id and data of the NOOP operation to N i .key and N i .value, where N i is the current node;

[0054] (3) The RDMA network card performs a CAS operation to compare the old value with the id value of the NOOP operation. If the comparison is successful, the opcode of the NOOP operation is set to WRITE and processed according to step (4); if the comparison fails, a READ operation is performed according to step (2) to read the next node. This process continues until the key is found or the end node is reached.

[0055] (4) The RDMA network card performs a WRITE operation to send the query result to the server main node, and the server main node then processes the data and sends it back to the client.

[0056] S6. The database server main node notifies the central processing unit CPU to directly access the sorted string table SSTable file in the disk to obtain the query result and returns the query result to the client to complete the query.

[0057] S7. The database server main node first queries the skiplist node corresponding to the key to be queried under the corresponding version number in the skiplist of the in-memory table MemTable. If the query is successful, the query result is returned to the client; if the query fails, it queries in the sorted string table SSTable file and returns the query result to the client.

[0058] Specifically, in order to extend the applicability of RedN to the key-value storage database based on the LSM tree in this embodiment, CamelDB (LSM Tree key-value database based on RDMA Chain) is proposed. This is a key-value storage database based on the LSM tree that supports RDMA. It introduces a cooperative remote access model between the client and the server node, namely local access and remote access. CamelDB effectively determines whether the key to be queried exists in the MemTable by integrating the Bloom Filter, and adopts different access strategies according to the query result. For the key-value pairs existing in the MemTable, remote access is used, that is, the query task is offloaded to the RNIC through the RDMA Chain of the RedN framework, and it replaces the CPU to complete the query task. For the key-value pairs that do not exist in the MemTable, local access is adopted, and the CPU reads the key-value pairs of the SSTable to complete the query task. At the same time, it also provides efficient support for the write operations (including insertion, update, and deletion) and version control of the MemTable file.

[0059] In a performance study conducted using a cluster supported by CamelDB, it was found that when four clients simultaneously sent requests to the server, the latency decreased significantly. Specifically, when there were no write operations, the latency decreased by 40%; when write operations accounted for 5% of the total, the latency decreased by 37%; when write operations accounted for 20% of the total, the latency decreased by 11%. In addition to these latency improvements, we also observed a significant reduction in the CPU usage of the server. Compared with the leading available RDMA-based key-value storage solutions, the CPU load of the server decreased by 90%. These findings indicate that CamelDB is very effective and efficient in parallel managing multiple client requests, even when mixed operations including write operations are involved. The substantial reduction in latency and CPU usage indicates that our system has a competitive advantage in the field of high-speed data management and storage.

[0060] Specifically, as Figure 4 shown, the latency of read requests under different numbers of clients was evaluated when the keys of all queries existed in the SkipList of the MemTable. In this scenario, CamelDB would adopt the remote access mode. The latency of CamelDB is lower than that of traditional RDMA remote access, especially when the number of clients is large. This is because traditional RDMA requires multiple network round-trips to traverse the Skiplist, while CamelDB can offload the query task to the RDMA network card by using RDMA Chain, and the RNIC reads the working task to loop through the SkipList to query the target node. Therefore, compared with the remote access using traditional RDMA, CamelDB can reduce the latency of read requests by up to 95%. In addition, as the number of clients increases, the latency of CamelDB also increases compared with the CPU local access mode, but the increment is relatively small. This is because the high-speed RDMA network card with RDMAChain can effectively handle the increasing number of concurrent requests, while the CPU local access cannot process requests in a timely manner due to CPU resource constraints and CPU scheduling and other factors, resulting in higher latency.

[0061] As Figure 5As shown, the latency of mixed read and write requests under different write ratios and clients was further evaluated. Since write operations directly insert nodes into the appropriate positions in the SkipList and do not involve I / O operations of SSTables, if the SkipList capacity exceeds the MemTable limit, the data will be flushed to the disk at once in the background, creating a new MemTable to accept new write requests. Compared with the possible SSTable reads during read operations, the latency is lower. Therefore, as the write ratio increases, the latency of the CPU local access mode decreases slightly. In contrast, for CamelDB, as the write ratio increases, the possibility of reading through remote access decreases, and the latency increases instead. Even so, when the write ratio is 20%, the latency of CamelDB is still lower than that of local access.

[0062] Specifically, as Figure 6 shown, when all the keys of the current queries exist in the SkipList of the MemTable, the server CPU occupancy of read requests under different numbers of clients was evaluated. The server CPU utilization of CamelDB is lower than that of the traditional RDMA remote access mode and the CPU local access mode, especially when the number of clients is large. When using traditional RDMA for remote access, although the latency is high due to multiple network round-trips, the CPU occupancy of the server is relatively low. This is because users using traditional RDMA for remote access also use the RDMA network card to process query tasks and do not require the CPU to execute tasks, while the CPU local mode executes tasks through the CPU, resulting in a high CPU occupancy.

[0063] As Figure 7 shown, the server CPU occupancy of mixed read and write requests under different write ratios and different numbers of clients was further evaluated, showing the results of CamelDB and the CPU local access mode. The server CPU occupancy of CamelDB is significantly lower than that of the CPU local access mode, approximately reducing the CPU occupancy by 90%. As the write ratio increases, the server CPU occupancy of CamelDB also increases. This is because write tasks and background compression tasks are still processed by the server, increasing the server's CPU occupancy.

[0064] Although the present invention has been specifically shown and described in conjunction with the preferred embodiments, those skilled in the art should understand that various changes can be made to the present invention in form and detail without departing from the spirit and scope of the present invention defined by the appended claims, and all of them fall within the protection scope of the present invention.

Claims

1. A key-value database access method for LSM Tree based on RDMA Chain, characterized in that, Including: S1. The database client sends a database operation request to the primary node of the database server. The database operation request information includes a version number, a key, and an operation type. S2. The primary node of the database server parses the request and determines the operation type. If the operation type is a write operation, go to S3; if the operation type is a read operation and the version number is less than 0, go to S4; if the operation type is a read operation and the version number is greater than or equal to 0, go to S7. S3. The primary node of the database server generates a new sequence number, packs the new sequence number, the key, and the operation type in the operation request information into a skip list node, and inserts the skip list node into the skip list of the in-memory table MemTable to complete the write operation. S4. The database server checks the Bloom filter to determine whether the key in the database operation request exists in the skip list of the in-memory table MemTable. If it exists, go to S5; if it does not exist, go to S6. S5. The secondary node of the database server publishes a Remote Direct Memory Access Chain (RDMA Chain) corresponding to the request, unloads the query operation of the primary node of the database server from the Central Processing Unit (CPU) to the Remote Direct Memory Access Network Interface Card (RNIC) of the secondary node of the database server. The RNIC queries the skip list of the MemTable of the primary node of the database server according to the RDMA Chain, returns the query result to the primary node of the database server, and then the primary node of the server returns the query result to the client to complete the query. S6. The primary node of the database server notifies the CPU to directly access the Sorted String Table (SSTable) file on the disk to obtain the query result, and returns the query result to the client to complete the query. S7. The primary node of the database server first queries the skip list node corresponding to the key to be queried under the corresponding version number in the skip list of the in-memory table MemTable. If the query is successful, return the query result to the client. If the query fails, query in the SSTable file and return the query result to the client.

2. The method for accessing a key-value database of an LSM Tree based on RDMA Chain according to claim 1, wherein In S1, the database operation request information further includes the value corresponding to the key and the length of the value corresponding to the key. When the operation type is query or delete, the length of the value corresponding to the key is 0.

3. The method for accessing a key-value database of an LSM Tree based on RDMA Chain according to claim 1, characterized in that, In S1, the version number takes effect only during the query operation. When the version number is greater than or equal to 0, it means that the client requests to query the value corresponding to a certain key under this version number. When the version number is less than 0, it means that the client requests to query the value corresponding to a certain key under the latest version.

4. The method for accessing a key-value database of an LSM Tree based on RDMA Chain according to claim 1, wherein In S3, the skip list nodes are sorted first in ascending order of keys and then in descending order of sequence numbers to ensure that among different skip list nodes corresponding to the same key, the new skip list node is arranged before the old skip list node.

5. The method for accessing a key-value database of an LSM Tree based on RDMAChain according to claim 1, characterized in that, In S4, the database server checks the Bloom filter specifically as follows: Use multiple hash functions to calculate the hash value corresponding to the key through the Bloom filter, check the bit array positions corresponding to these hash values. If the values of all corresponding bit array positions are 1, it is determined that the key exists in the skip list; otherwise, it is determined that the key does not exist in the skip list.

Images