A Key-Value Separation Storage Method and System
By adopting the ART prefix tree structure and node management module in the RocksDB database, key-value separation storage is achieved, which solves the performance bottleneck of the database when processing data concurrently and improves read and write performance and CPU cache utilization.
Patent Information
- Application Number
- CN202111461856.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-02
- Publication Date
- 2025-10-31
- Estimated Expiration
- 2041-12-02
AI Technical Summary
The existing RocksDB database has bottlenecks in data write and read performance, especially when processing large amounts of data concurrently, frequent encoding and decoding operations affect IO performance.
We use the ART prefix tree as the data index structure, allocate space for key and value respectively in the node, directly store and read key-value data, reduce encoding operations, and modify the iterator and query tool to directly obtain node information, avoiding memory address decoding.
It improved database read and write performance, reduced IO throughput, increased CPU cache hit rate, and improved data read and write efficiency.
Smart Images
Figure CN114138792B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data storage management technology, and in particular to a key-value separation storage method and system. Background Technology
[0002] In today's network environment, information flows rapidly, involving the input and output of a large number of documents, images, audio, and video. As a result, a large number of users have placed higher demands on the performance of information writing and querying.
[0003] RocksDB is an LSM-tree architecture engine based on LevelDB, providing key-value storage and read / write capabilities. It has the following features:
[0004] 1) High efficiency
[0005] RocksDB uses a log-structured database engine, written entirely in C++ for maximum performance. Keys and values are simply streams of bytes of arbitrary size.
[0006] 2) Optimized for fast storage
[0007] RocksDB is optimized for fast, low-latency storage, such as flash drives and high-speed disk drives. RocksDB fully leverages the high read / write speeds offered by flash or RAM.
[0008] 3) Highly adaptable
[0009] RocksDB can adapt to different workloads. From database storage engines like MyRocks to application data caching and embedded workloads, RocksDB can meet a variety of data needs.
[0010] 4) Basic and advanced database operations
[0011] RocksDB provides basic operations such as opening and closing the database, reading and writing, and more advanced operations such as merging and compacting filters.
[0012] The existing RocksDB database uses a data storage method that first encodes the key, SequenceNumber, ValueType, and value, then stores the encoded data in a single allocated block of memory. When reading key-value information, the entire memory block is retrieved, and then decoding is performed sequentially based on the starting address to obtain the corresponding data. This method reduces data write and read performance to some extent.
[0013] During data migration, it is often necessary to compare key values, which requires parsing the encoded data. However, frequent encoding and decoding during the concurrent processing of large amounts of data will inevitably affect IO performance.
[0014] To ensure the read and write performance of the RocksDB database, this invention proposes a key-value separation storage method and system. Summary of the Invention
[0015] To overcome the shortcomings of existing technologies, this invention provides a simple and efficient key-value separation storage method and system.
[0016] This invention is achieved through the following technical solution:
[0017] A key-value separation storage method is characterized by using an ART prefix tree as the basic data structure for data indexing. Within each data node (Node), space is allocated for storing the key and value values respectively, and the corresponding SequenceNumber and ValueType are stored within the node, thus achieving key-value separation. Furthermore, no encoding is performed during data storage; instead, the key, SequenceNumber, ValueType, and value are directly stored and retrieved, thereby reducing IO throughput, increasing CPU cache hit rate, and improving database read / write performance.
[0018] The implementation process of this key-value separation storage method is as follows:
[0019] S1. When writing data, first create a new node as a version of the key-value data, allocate memory space for the key and value in the current version node, and copy the key and value into the current version node;
[0020] S2. Assign values to the Node members SequenceNumber and ValueType respectively to store the current version information;
[0021] S3. After all the information is stored in the Node, mount the Node address to the ART index tree;
[0022] S4. When reading data, obtain the current node Node and call the corresponding value retrieval function GetXX() to get the information without performing any additional decoding operations.
[0023] The iterator is modified so that data information is no longer obtained through the memory starting address, but directly obtains the relevant information of the current key-value data through the Get operation of the node, reducing the data decoding operation.
[0024] When comparing different versions, the memory starting address (which is not visible to the outside world) is no longer used for decoding and comparison. Instead, the key information obtained from the Node is used directly for comparison.
[0025] When using the Seek tool for query operations, it's unnecessary to pass in a memory address to resolve the key; instead, you can directly pass in a Node pointer to retrieve the corresponding key information. Each Seek query reduces one decoding operation, thus improving read performance to some extent.
[0026] The system based on the Key-value separation storage method of the present invention is characterized by including an ART prefix tree module and a node management module. The ART prefix tree module is used to generate the basic data structure of the data index, and the node management module is used to allocate memory space for storage of key and value, and store the corresponding SequenceNumber and ValueType to achieve key-value separation.
[0027] A computer device, characterized in that it includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described key-value separation storage method.
[0028] A computer-readable storage medium, characterized in that: a computer program is stored thereon, wherein when the computer program is executed by a processor, the steps of the key-value separation storage method are implemented.
[0029] The beneficial effects of this invention are: the key-value separation storage method and system do not require decoding of the corresponding information during data read and write operations, and can directly obtain information using the node, which reduces disk I / O consumption, lowers IO throughput, increases CPU cache hit rate, and improves database read and write performance. Attached Figure Description
[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0031] AppendixFigure 1 This is a schematic diagram illustrating the structure between the ART prefix tree and multi-version key-value data in this invention.
[0032] Appendix Figure 2 This is a schematic diagram of the key-value data in the Node of this invention. Detailed Implementation
[0033] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions in the embodiments of this invention will be clearly and completely described below in conjunction with the embodiments of this invention. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.
[0034] This key-value separation storage method uses the ART prefix tree as the basic data structure for data indexing. Within each data node, space is allocated for storing the key and value values separately, and the corresponding SequenceNumber and ValueType are stored within the node, thus achieving key-value separation. At the same time, no encoding is performed during data storage; instead, the key, SequenceNumber, ValueType, and value are directly stored and retrieved, thereby reducing IO throughput, increasing CPU cache hit rate, and improving database read and write performance.
[0035] Appendix Figure 1 The figure illustrates a three-level ART prefix tree structure. Those skilled in the art will understand that in practical applications, the structure of the ART prefix tree is not limited to the three-level structure shown in the figure.
[0036] The implementation process of this key-value separation storage method is as follows:
[0037] S1. When writing data, first create a new node as a version of the key-value data, allocate memory space for the key and value in the current version node, and copy the key and value into the current version node;
[0038] S2. Assign values to the Node members SequenceNumber and ValueType respectively to store the current version information;
[0039] S3. After all the information is stored in the Node, mount the Node address to the ART index tree;
[0040] S4. When reading data, obtain the current node Node and call the corresponding value retrieval function GetXX() to get the information without performing any additional decoding operations.
[0041] The iterator is modified so that data information is no longer obtained through the memory starting address, but directly obtains the relevant information of the current key-value data through the Get operation of the node, reducing the data decoding operation.
[0042] When comparing different versions, the memory starting address (which is not visible to the outside world) is no longer used for decoding and comparison. Instead, the key information obtained from the Node is used directly for comparison.
[0043] When using the Seek tool for query operations, it's unnecessary to pass in a memory address to resolve the key; instead, you can directly pass in a Node pointer to retrieve the corresponding key information. Each Seek query reduces one decoding operation, thus improving read performance to some extent.
[0044] The system based on this key-value separation storage method includes an ART prefix tree module and a node management module. The ART prefix tree module is used to generate the basic data structure of the data index, and the node management module is used to allocate memory space for key and value storage, and store the corresponding SequenceNumber and ValueType to achieve key-value separation.
[0045] The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the key-value separation storage method.
[0046] The computer-readable storage medium has a computer program stored thereon, which, when executed by a processor, implements the steps of the key-value separation storage method.
[0047] Compared with existing technologies, this key-value separation storage method and system eliminates the need for decoding of corresponding information during data read and write operations. Information can be directly obtained using nodes, reducing disk I / O consumption, lowering IO throughput, increasing CPU cache hit rate, and improving database read and write performance.
[0048] The embodiments described above are merely one specific implementation of the present invention. Ordinary changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included within the protection scope of the present invention.
Claims
1. A key-value separation storage method, characterized in that: Using the ART prefix tree as the basic data structure for data indexing, space is allocated separately for storing the key and value values within each data node, and the corresponding SequenceNumber and ValueType are stored within the node, thus achieving key-value separation. At the same time, no encoding is performed during data storage; instead, the key, SequenceNumber, ValueType, and value are directly stored and retrieved, thereby reducing IO throughput, increasing CPU cache hit rate, and improving database read and write performance. Includes the following steps: S1. When writing data, first create a new node as a version of the key-value data, allocate memory space for the key and value in the current version node, and copy the key and value into the current version node; S2. Assign values to the Node members SequenceNumber and ValueType respectively to store the current version information; S3. After all the information is stored in the Node, mount the Node address to the ART index tree; S4. When reading data, obtain the current node (Node) and call the corresponding value retrieval function GetXX() to get the information; no additional decoding operation is required.
2. The key-value separation storage method according to claim 1, characterized in that: The iterator is modified so that data information is no longer obtained through the memory starting address, but directly obtains the relevant information of the current key-value data through the Get operation of the node, reducing the data decoding operation.
3. The key-value separation storage method according to claim 2, characterized in that: When comparing different versions, you can directly use the Node to obtain the key information for comparison.
4. The key-value separation storage method according to claim 2, characterized in that: When performing a query, you can directly pass in the Node pointer to obtain the corresponding key information.
5. The system of the key-value separation storage method according to any one of claims 1 to 4, characterized in that: It includes an ART prefix tree module and a node management module. The ART prefix tree module is used to generate the basic data structure of the data index, and the node management module is used to allocate memory space for storing the key and value, and at the same time store the corresponding SequenceNumber and ValueType to achieve the separation of key-value.
6. A computer device, characterized in that: It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the key-value separation storage method according to any one of claims 1 to 4.
7. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the steps of the key-value separation storage method according to any one of claims 1 to 4.