Index storage method, system and storage medium

Abstract

The present invention discloses an index storage method, system, and storage medium. The method comprises: determining a subtree in an original index whose number of key-value pairs is less than a set number, storing the data of each subtree in a corresponding page, and obtaining a corresponding SPRT index. The method can optimize range queries, which radix trees are not good at, and effectively reduce the depth of the tree, thereby comprehensively improving the performance of index operations. A runtime recovery setting is also proposed, so that during runtime, SPRT only needs to persist data and a portion of metadata that is rarely modified, effectively reducing persistence overhead.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to an index storage method, system and storage medium. Background Art

[0002] The most significant difference between PM (Persistent Memory) and DRAM (Dynamic Random Access Memory) is data persistence. Persistence means that data written to PM is persisted in the storage medium and survives power outages. Currently, in storage architectures, PM sits behind multiple levels of CPU cache, allowing data to reach PM directly from the CPU cache. However, when writing data to PM, there's no guarantee that the data is already in the PM; it may still be in the CPU cache, necessitating a cache line flush instruction or waiting for the cache line to be implicitly flushed. Due to PM's persistence, byte-granular addressing, nanosecond latency, and storage capacity between DRAM and block devices, it is well-suited for storing index structures, significantly improving overall system performance. Consequently, it is beginning to be adopted by systems across the industry.

[0003] Commonly used indexes include tree indexes and hash indexes. Tree indexes naturally support range queries. Commonly used tree indexes include B+ trees and radix trees. Currently, many industrial systems use B+ trees as indexes. B+ trees support efficient range queries because multiple key-value pairs are stored in a leaf node, and scanning a leaf node does not cause pointer jumps. Range queries are a key feature of relational databases and key-value stores. Furthermore, leaf nodes in a B+ tree can be connected by pointers, further enhancing the advantages of range queries. In other tree structures, such as radix trees and binary search trees, each node can only store a single key-value pair, and data can be stored in leaf nodes or non-leaf nodes. Range queries on these trees must jump between different levels of the tree, resulting in a large number of pointer jumps. Because the performance gap between sequential and random reads in PM is larger than that of DRAM, more random accesses degrade PM range query performance.

[0004] Indexes that support variable-length data storage can be applied to a wider range of applications, because the vast majority of data in the industry is of variable length. However, this support comes at a cost. Many existing persistent tree indexes are deeply optimized for only 8-byte key-value pairs. If you want to make these indexes support variable-length keys, a direct approach is to allocate additional data areas and store the key addresses in the index. However, this modification will incur a large amount of pointer jump overhead. In order to ensure crash consistency, persistent memory indexes usually require a large amount of persistence instruction overhead to ensure that the persisted content is consistent. Summary of the Invention

[0005] The inventors found that in the existing persistent B+ tree index FAST&FAIR, after using an index scheme that supports variable-length data storage to support variable-length data, the five operations of index query / insert / update / delete / range query showed a performance degradation of 3.9 / 1.8 / 2.79 / 2.15 / 1.94 times respectively. The main performance degradation comes from pointer jumps and string comparisons during the traversal process. At the same time, in order to store additional data areas, operations such as insert / update introduce more persistence overhead. When variable-size keys and range queries are required at the same time, B+ trees may not be the best choice for persistent memory indexes. Persistent memory tree indexes cannot achieve high performance at the same time for the two common requirements of variable-length data access and range queries.

[0006] In order to solve the above problems, the inventors have made the present invention, and provide an index storage method, system and storage medium through specific implementation methods.

[0007] In a first aspect, an embodiment of the present invention provides an index storage method, comprising:

[0008] Determine the subtrees in the original index whose number of key-value pairs is less than the set number, store the data of each subtree in a corresponding page, and obtain the corresponding SPRT index.

[0009] In some specific embodiments, storing the data of each subtree in a corresponding page includes the following steps:

[0010] The metadata corresponding to the data in the corresponding subtree is stored starting from the head of a corresponding page in a front-to-back order; the data in the corresponding subtree is stored starting from the tail of the page in a back-to-front order.

[0011] In some specific embodiments, the storage format of each of the metadata includes the SPRT node type, SPRT node depth, SPRT node version, SPRT node prefix, SPRT node lock, the position of the next data insertion, the number of data in the SPRT node, the position of each data in the SPRT node, the length of each data key in the SPRT node, the hash value of each data key in the SPRT node and the sorted position of each data in the SPRT node.

[0012] In some specific embodiments, the storage format of the data in each corresponding subtree includes value, key, valid bit, key length and value length.

[0013] In some specific embodiments, when the SPRT index is used for querying, after finding the corresponding SPRT node, the hash value of each data key in the SPRT node is compared from the SPRT node head to the position of the next data insertion to determine the data key with a matching hash value.

[0014] In some specific embodiments, when the SPRT index is used for querying, after determining the data key that matches the hash value, the valid bit corresponding to the data key is read; based on the valid bit, it is determined whether the corresponding data exists; when the corresponding data exists, the position of the data value is found through the value length of the data, and the value of the data is read out.

[0015] In some specific embodiments, when data is inserted using the SPRT index, the corresponding metadata is updated after the data is stored to the location for the next data insertion.

[0016] In some specific embodiments, when the SPRT index is used to update data, the original data location is not updated, and the updated data is stored in the location of the next data insertion.

[0017] In some specific embodiments, only the SPRT node type, SPRT node depth, SPRT node version and SPRT node prefix are persisted, and are persisted only once when the SPRT node is created.

[0018] In some specific embodiments, after the system is restarted, each time a SPRT node is accessed during operation, the remaining metadata is rebuilt by scanning the key-value pairs from back to front.

[0019] In some specific embodiments, each of the pages is 4KB in size and aligned to 256 bytes.

[0020] In a second aspect, an embodiment of the present invention provides an index storage system, including:

[0021] The SPRT index generation module is used to determine the subtrees in the original index whose number of key-value pairs is less than a set number, store the data of each subtree in a corresponding page, and obtain the corresponding SPRT index.

[0022] Based on the same inventive concept, an embodiment of the present invention further provides a computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions, when executed, implement the aforementioned index storage method.

[0023] The beneficial effects of the above technical solutions provided by the embodiments of the present invention include at least:

[0024] This paper discloses a new slot page radix tree (SPRT) that optimizes range queries, a problem inherent in radix trees. The SPRT structure proposed in this paper can effectively reduce the tree depth, thereby improving the performance of index operations (query / insert / update / delete) across the board.

[0025] In addition to the SPRT structure, the present invention also proposes a runtime recovery setting in some specific embodiments, so that when SPRT is running, it only needs to persist data and a part of metadata that is rarely modified, while all other frequently modified metadata can be restored based on the data when it is accessed for the first time. The amortized recovery overhead is a constant, which effectively reduces the persistence overhead.

[0026] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purposes and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0027] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0029] Figure 1 This is a schematic diagram of generating an SPRT index from an original index in an embodiment of the present invention;

[0030] Figure 2 Schematic diagram of page storage of SPRT index in an embodiment of the present invention;

[0031] Figure 3Schematic diagram of the key-value pair format of the SPRT index in an embodiment of the present invention;

[0032] Figure 4 This is a performance comparison chart of SPRT in an embodiment of the present invention and the latest persistent memory B+ tree and radix tree in the prior art in basic index operations;

[0033] Figure 5 The following is a performance comparison chart of SPRT in an embodiment of the present invention and the latest index in the prior art under four different workloads. DETAILED DESCRIPTION

[0034] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0035] In order to solve the problems existing in the prior art, embodiments of the present invention provide an index storage method, system and storage medium.

[0036] Example 1

[0037] The first embodiment of the present invention provides an index storage method, the process of which is as follows: Figure 1 As shown, the following steps are included:

[0038] Determine the subtrees in the original index whose number of key-value pairs is less than the set number, store the data of each subtree in a corresponding page, and obtain the corresponding SPRT index. Figure 1 As shown, the original index nodes D, E, L3, L4 and L5 are encapsulated in the SPRT node SP3, the original index node L2 is encapsulated in the SPRT node SP2, and the original index nodes C, L0 and L1 are encapsulated in the SPRT node SP1 to obtain the SPRT index corresponding to the original index.

[0039] In some specific embodiments, storing the data of each subtree in a corresponding page includes the following steps:

[0040] like Figure 1 As shown, the metadata corresponding to the data in the corresponding subtree is stored starting from the head of a corresponding page in a front-to-back order; the data in the corresponding subtree is stored starting from the tail of the page in a back-to-front order.

[0041] In some specific embodiments, Figure 2As shown, the storage format of each metadata includes the SPRT node type, SPRT node depth, SPRT node version, SPRT node prefix, SPRT node lock, the position of the next data insertion, the number of data in the SPRT node, the position of each data in the SPRT node, the length of each data key in the SPRT node, the hash value of each data key in the SPRT node and the sorted position of each data in the SPRT node.

[0042] In some specific embodiments, Figure 3 As shown, the storage format of the data in each corresponding subtree includes a value Val, a key Key, a valid bit Valid, a key length KeyLen and a value length ValLen.

[0043] In some specific embodiments, when the SPRT index is used for querying, after finding the corresponding SPRT node, the hash value of each data key in the SPRT node is compared from the SPRT node head to the position of the next data insertion to determine the data key with a matching hash value.

[0044] In some specific embodiments, when the SPRT index is used for querying, after determining the data key that matches the hash value, the valid bit corresponding to the data key is read; based on the valid bit, it is determined whether the corresponding data exists; when the corresponding data exists, the position of the data value is found through the value length of the data, and the value of the data is read out.

[0045] In some specific embodiments, when data is inserted using the SPRT index, the corresponding metadata is updated after the data is stored to the location for the next data insertion.

[0046] In some specific embodiments, when the SPRT index is used to update data, the original data location is not updated, and the updated data is stored in the location of the next data insertion.

[0047] In some specific embodiments, only the SPRT node type, SPRT node depth, SPRT node version and SPRT node prefix are persisted, and are persisted only once when the SPRT node is created.

[0048] In some specific embodiments, after a system restart, each SPRT node accessed during runtime is rebuilt by scanning key-value pairs from the back to the front. This runtime recovery setting for the persistent memory index can effectively reduce runtime persistence overhead, amortizing the persistence overhead introduced to ensure crash consistency to the overhead of restoring metadata the first time a node is accessed during runtime, resulting in a 41% performance improvement for runtime write operations.

[0049] In some specific embodiments, each of the pages is 4KB in size and aligned to 256 bytes.

[0050] In some specific embodiments, for each subtree in the index, if the number of key-value pairs (the number of leaf nodes) in the subtree is less than m (user-defined), then the data in the entire subtree can be stored in a single page. A key-value pair is a pair where each key corresponds to a value. This page has a fixed size, set by default to 4KB (aligned to 256 bytes, which is beneficial for storage and access in persistent memory). Within a page, metadata is maintained starting from the beginning, while data is stored from the end. The metadata primarily includes node type (type), node depth (depth), node version (generation), node prefix (prefix), node lock (lock), next insertion position (next pos), number of data in the node (count), position of each data item in the node (offset array), length of each data key in the node (keylen array), hash of each data key in the node (fingerprint array), and position of each data item in the node after sorting (slot array). Data is stored from the end of the page forward in a five-part format: value, key, valid bits, key length, and value length. The valid bit indicates whether the data has been deleted, and the length of the key value can help calculate the starting position of the key value.

[0051] All five basic index operations (query, insert, update, delete, and range query) first require traversing the index's internal nodes. This traversal is similar to a traditional radix tree, with optimistic locking used for concurrency control. For queries, after finding a leaf node, the index is traversed from 0 to nextpos, comparing the fingerprint array. A match indicates that the key hashes are identical, indicating a possible duplicate key. The offset array is then used to locate the key value. The valid bits are then read to determine if the data has been deleted. If the data exists, the value length is used to locate the value and read it. For inserts, after inserting to nextpos, the corresponding metadata is updated. Updates are similar to queries, except that a different lock is used for the final leaf update. Modifications do not modify in place; similar to inserts, a new key is inserted at nextpos. This operation prevents duplicate data because updates proceed from back to front, while queries proceed from front to back, returning the first matching key value.

[0052] Of the above metadata, only the node type, depth, version, and prefix need to be persisted. These persists only once, when the node is created, and does not change thereafter. All other metadata does not need to be modified. After the system is restarted, each time a node is accessed during operation, the remaining metadata is reconstructed by scanning the key-value pairs from the back to the front.

[0053] The operation method of the novel persistent memory radix tree index SPRT proposed in the above embodiment includes:

[0054] S1. Data query. Start from the root node of the index and traverse downward until the leaf node, and return after finding the corresponding data. The specific process is as follows: (1) When traversing, after reaching an internal node, first obtain the current version number of the node. (2) Then query within the node to find out whether there is a matching character. (3) If not found, it means that the data does not exist, and an empty result is returned, and the operation ends. (4) If found, check the version number of the node at this time. (5) If the version number changes, it means that a concurrent write operation has modified this node, and inconsistent data may be read, so abandon the current operation and return to step (1) to try again. (6) If the version number has not changed, it means that the read result is consistent. According to the read node pointer, find the next node until the leaf node is found.

[0055] Leaf nodes such as Figure 2 As shown in the figure, it is a 4KB page. The page header contains metadata, including the node type (type), node depth (depth), node version (generation), node prefix (prefix), node lock (lock), next insertion position (next pos), number of data in the node (count), position of each data in the node (offsetarray), length of each data key in the node (keylen array), hash of each data key in the node (fingerprint array), and position of each data in the node after sorting (slot array).

[0056] In the data query operation, after reaching the leaf node, (1) first obtain a current version number through lock. (2) Then calculate the hash value h of the key currently being queried. (3) Traverse from 0 to next pos to query the fingerprint array. If the i-th fingerprint is the same as h, it means that the key at position i may be equal to the key to be queried. (4) Therefore, read the i-th in offsetarray and obtain the position of the i-th key-value pair. (5) Then read the i-th key-value pair. Then according to Figure 3In the structure, (6) first determines whether the key-value pair has been deleted based on the valid bit. If it has been deleted, continue to compare in (3). If it has not been deleted, then find the corresponding key and value based on the key length and value length. (7) Then compare whether the keys are the same. If not, return to (3) to continue searching. If they are the same, it means that the data has been found, and return directly, and the operation is completed.

[0057] S2, data insertion. Similar to S1, first traverse the index until the corresponding leaf node is found. The operation on the leaf node is different from S1. The specific steps are: (1) First, add a write lock to the leaf node. (2) Then, according to next pos, find the next insertion position in offsetarray. (3) Write the key-value pair to the data area. (4) Update the metadata (offsetarray, keylen array, fingerprint array, slot array) but do not explicitly persist it. The operations of data update and deletion are similar to query and insertion. The insertion operation is equivalent to writing a new data, while the deletion operation only needs to change the valid bit of the data from 1 to 0.

[0058] S3. Range query. A range query provides a range [start, end]. (1) First, find the edge node location based on start and end. (2) For nodes in the middle of the range, all data is returned directly. (3) For two edge nodes, compare the keys in order using the slot array in the leaf node, find the key value within the range, and then return it.

[0059] Figure 4 The basic operational performance of this embodiment was compared with the latest persistent memory tree indexes in the prior art, P-ART and Fast&Fair. In the figure, Mops represents a unit of performance, representing millions of operations per second. It was found that SPRT comprehensively outperformed current P-ART and Fast&Fair storage systems in five operations (lookup, insert, update, remove, and scan). In particular, range query performance surpassed that of B+ trees, demonstrating the effectiveness of the proposed structure.

[0060] Figure 5 The performance tests of this embodiment, P-ART, and Fast&Fair under four different workloads were demonstrated, and it was found that SPRT has very good scalability under different workloads, far higher than the current P-ART and Fast&Fair systems. Figure 5 The (a) coordinate system in the figure shows the performance of each storage method when 50% of the queries and 50% of the insertions are performed. Figure 5The (b) coordinate system shows the performance of each storage method when 95% of the queries and 5% of the insertions are performed. Figure 5 The (c) coordinate system shows the performance of each storage method when only reading. Figure 5 The (d) coordinate system shows the performance of each storage method when only inserting.

[0061] In the above embodiment, compared with the previous persistent memory radix tree, the single point query can be improved by 20.3%, and the range query can be improved by 65%.

[0062] In the above method of this embodiment, the present invention discloses a new slot page-based radix tree SPRT, which optimizes the range query that the radix tree is not good at, and can effectively reduce the depth of the tree, thereby comprehensively improving the performance of index operations (query / insert / update / delete).

[0063] In addition to the SPRT structure, the present invention also proposes a runtime recovery setting in some specific embodiments, so that when SPRT is running, it only needs to persist data and a part of metadata that is rarely modified, while all other frequently modified metadata can be restored based on the data when it is accessed for the first time. The amortized recovery overhead is a constant, which effectively reduces the persistence overhead.

[0064] Those skilled in the art can change the above sequence without departing from the scope of protection of the present disclosure.

[0065] Example 2

[0066] A second embodiment of the present invention provides an index storage system, including:

[0067] The SPRT index generation module is used to determine the subtrees in the original index whose number of key-value pairs is less than a set number, store the data of each subtree in a corresponding page, and obtain the corresponding SPRT index.

[0068] In the above system, range queries, which the radix tree is not good at, are optimized, and the depth of the tree can be effectively reduced, thereby improving the performance of index operations (query / insert / update / delete) in all aspects.

[0069] Regarding the index storage system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0070] Based on the same inventive concept, an embodiment of the present invention further provides a computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions, when executed, implement the aforementioned index storage method.

[0071] Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention shall still fall within the scope of the patent coverage of the present invention.

Claims

1. An index storage method, characterized in that: The following steps are involved: Determine the subtrees in the original index whose number of key-value pairs is less than the set number, store the data of each subtree in a corresponding page, and obtain the corresponding SPRT index, where SPRT represents a radix tree based on a slot page; Storing the data of each subtree into a corresponding page includes: The metadata corresponding to the data in the corresponding subtree is stored starting from the head of the corresponding page in a forward-to-backward order; the data in the corresponding subtree is stored starting from the end of the page in a backward-to-frontward order; Obtaining the corresponding SPRT index includes: Encapsulate one or more original index nodes in the SPRT node to obtain the SPRT index corresponding to the original index.

2. The method according to claim 1, wherein The storage format of each metadata includes the SPRT node type, SPRT node depth, SPRT node version, SPRT node prefix, SPRT node lock, the position of the next data insertion, the number of data in the SPRT node, the position of each data in the SPRT node, the length of each data key in the SPRT node, the hash value of each data key in the SPRT node and the sorted position of each data in the SPRT node.

3. The method according to claim 1, wherein The storage format of the data in each corresponding subtree includes value, key, valid bit, key length and value length.

4. The method according to claim 2, wherein When the SPRT index is used for querying, after finding the corresponding SPRT node, the hash value of each data key in the SPRT node is compared from the SPRT node head to the position of the next data insertion to determine the data key with a matching hash value.

5. The method according to claim 3, wherein When the SPRT index is used for querying, after determining the data key that matches the hash value, the valid bit corresponding to the data key is read; based on the valid bit, it is determined whether the corresponding data exists; when the corresponding data exists, the position of the data value is found through the value length of the data, and the value of the data is read out.

6. The method according to claim 2, wherein When data is inserted using the SPRT index, after the data is stored in the location for the next data insertion, the corresponding metadata is updated.

7. The method according to claim 2, wherein When the SPRT index is used to update data, the original data location is not updated, and the updated data is stored in the location where the next data is inserted.

8. The method according to claim 2, wherein Only the SPRT node type, SPRT node depth, SPRT node version and SPRT node prefix are persisted, and are persisted only once when the SPRT node is created.

9. The method according to claim 1, wherein After restarting the system, each time a SPRT node is accessed during operation, the remaining metadata is rebuilt by scanning the key-value pairs from back to front.

10. The method according to any one of claims 1 to 9, characterized in that: Each of the pages is 4KB in size and aligned to 256 bytes.

11. An index storage system, characterized in that: include: The SPRT index generation module is used to determine the subtrees in the original index whose number of key-value pairs is less than the set number, store the data of each subtree in a corresponding page, and obtain the corresponding SPRT index. SPRT represents a radix tree based on a slot page; Storing the data of each subtree into a corresponding page includes: The metadata corresponding to the data in the corresponding subtree is stored starting from the head of the corresponding page in a forward-to-backward order; the data in the corresponding subtree is stored starting from the end of the page in a backward-to-frontward order; Obtaining the corresponding SPRT index includes: Encapsulate one or more original index nodes in the SPRT node to obtain the SPRT index corresponding to the original index.

12. A computer storage medium, characterized in that The computer storage medium stores computer-executable instructions, which, when executed, implement the index storage method according to any one of claims 1 to 10.

Images