Log-merge tree key-value store systems and related methods and related devices

Abstract

Embodiments of the present application provide a log-merge tree key-value storage system and related methods and related devices, and relate to the technical field of storage. The system is based on a hybrid storage architecture, which includes a host and a persistent storage medium and a non-volatile storage medium connected to the host, wherein the persistent storage medium stores a plurality of ordered string table layers, each ordered string table layer includes a plurality of ordered string table groups, and each ordered string table group includes a plurality of ordered string tables; the log-merge tree key-value storage system adopts a grouping compression mechanism and includes two-level filters corresponding to the ordered string table groups, which include group filters corresponding to the ordered string table groups and table filters corresponding to the ordered string tables. The embodiments optimize the read amplification problem and the write amplification problem of the storage system, and improve the read performance and the write performance of the storage system.

Description

Technical Field

[0001] The embodiments of this application relate to the field of storage technology, and in particular to a log merge tree key-value storage system based on a hybrid storage architecture, as well as related methods and devices. Background Technology

[0002] In an era of surging information processing volumes, key-value storage systems can meet the demands of data-intensive applications. Typically, disk read / write operations are the performance bottleneck for key-value storage systems, which also suffer from significant read amplification and write amplification issues. Summary of the Invention

[0003] This application discloses an optimization method and related equipment for a log merge tree key-value storage system based on a hybrid storage architecture, which can optimize the read amplification problem of the storage system and improve the read performance of the storage system.

[0004] The first aspect of this application discloses a log merge tree key-value storage system based on a hybrid storage architecture, applied in electronic devices. The hybrid storage architecture includes a host, persistent storage media, and non-volatile storage media. The host is connected to both the non-volatile and persistent storage media. The persistent storage media stores multiple ordered string tables, each ordered string table including multiple ordered string table groups, and each ordered string table group including multiple ordered string tables. The log merge tree key-value storage system employs a group compression mechanism and includes two levels of filters corresponding to the ordered string table groups. The two levels of filters include a group filter corresponding to the ordered string table groups and a table filter corresponding to the ordered string tables.

[0005] The embodiments of this application can solve the problem that read operations may need to perform invalid accesses to multiple ordered string tables within a group due to the grouping and compression strategy of the log merge tree, resulting in additional read overhead and read amplification. This optimizes the read amplification and write amplification problems of the storage system and avoids invalid read operations through a two-level filter. The embodiments of this application can improve the read performance of the storage system.

[0006] In some alternative implementations, the two-stage filters are stored in a non-volatile storage medium.

[0007] Non-volatile storage media combine the excellent storage characteristics of both memory and persistent storage media, improving the read and write performance of storage systems. Storing metadata such as filters on non-volatile storage media reduces write operations and lowers write overhead through in-place update operations. Utilizing high-performance, persistent non-volatile storage media to store metadata such as filters optimizes storage system read performance and reduces long-tail latency by minimizing access to invalid persistent storage media.

[0008] In some alternative implementations, the group filter includes a Bloom filter or a Buzz filter, and the surface filter includes a Bloom filter, a Buzz filter, or a Buzz filter.

[0009] The embodiments of this application can implement group filters.

[0010] In some alternative implementations, the hybrid storage architecture also includes volatile storage media or memory.

[0011] The second aspect of this application discloses a data retrieval method based on the aforementioned log merge tree key-value storage system. The data retrieval method includes: obtaining the target key of the data to be searched; and searching for the target key from the persistent storage medium according to a two-level filter.

[0012] The embodiments of this application can avoid invalid lookups and reduce read latency through a two-stage filter.

[0013] In some alternative implementations, searching for the target key from the persistent storage medium according to the two-level filter includes: searching for the target key from the memory of the electronic device; if the target key is not found in the memory of the electronic device, searching for the target key from the persistent storage medium according to the two-level filter.

[0014] Embodiments of this application are capable of locating target keys from the memory and persistent storage media of electronic devices.

[0015] In some alternative implementations, after obtaining the target key of the data to be searched, the data search method further includes: searching for the target key in the memory of the electronic device; if the target key is not found in the memory of the electronic device, searching for the target key in the persistent storage medium according to a two-level filter.

[0016] Embodiments of this application are capable of locating target keys from the memory and persistent storage media of electronic devices.

[0017] In some optional implementations, searching for the target key from the persistent storage medium based on the two-level filter includes: according to the layer order, for any current ordered string table among multiple ordered string tables, determining whether there exists a target ordered string table group storing the target key in the current ordered string table based on the group key range of the multiple ordered string table groups in the current ordered string table; if there exists a target ordered string table group storing the target key in the current ordered string table, searching for the target key from the target ordered string table group based on the two-level filter.

[0018] The embodiments of this application can determine whether a target ordered string group storing the target key exists in the current ordered string table before searching for the target key in the target ordered string table group in the current ordered string table according to the two-level filter, thereby improving search efficiency.

[0019] In some optional implementations, if the target ordered string table group storing the target key does not exist in the current ordered string table layer, the target key is searched in the next ordered string table layer of the current ordered string table layer.

[0020] Embodiments of this application can search for the target key from the next ordered string table in the current ordered string table in either the order of the layer number or a random order.

[0021] In some optional implementations, searching for the target key from the target ordered string table group based on the two-level filter includes: determining whether the target ordered string table group has the possibility of including the target key based on the group filter in the two-level filter; if the group filter in the two-level filter determines that the target ordered string table group has the possibility of including the target key, determining whether there exists a target ordered string table group storing the target key based on the table key range of multiple ordered string tables in the current ordered string table group; if the table key range of multiple ordered string tables in the current ordered string table group determines that there exists a target ordered string table group storing the target key, determining whether the target ordered string table has the possibility of including the target key based on the table filter in the two-level filter; if the table filter in the two-level filter determines that the target ordered string table has the possibility of including the target key, searching for the data to be searched in the target ordered string table based on the target key.

[0022] The embodiments of this application enable the search for a target key from a target ordered string table using a two-level filter. First, the probability that the target ordered string table contains the target key is assessed. If the target ordered string table has the possibility of containing the target key, it indicates a high probability that the target ordered string table contains the target key. Then, the ordered strings within the target ordered string table are further evaluated to achieve an accurate final determination of whether the target ordered string table contains the target key. Using a two-level filter avoids invalid searches, thereby improving the efficiency of target key search.

[0023] In some alternative implementations, if the target ordered string table group is determined by the group filter in the two-level filter to be unlikely to include the target key, or if the target ordered string table is determined by the table filter in the two-level filter to be unlikely to include the target key, the target key is searched from the next ordered string table in the current ordered string table layer.

[0024] The embodiments of this application can search for the target key in the next ordered string table when the target key does not exist in the current ordered string table. This completes the search of all ordered string tables.

[0025] In some alternative implementations, finding a target key from a persistent storage medium based on a two-level filter includes: finding the target key from multiple ordered string tables in the persistent storage medium in layer order using a two-level filter.

[0026] The embodiments of this application enable the search for a target key from multiple ordered string layers according to the layer number, thereby finding the most recently stored target key.

[0027] In some optional implementations, for any current ordered string table among multiple ordered string tables, searching for the target key in the current ordered string table based on a two-level filter includes: determining whether the target key exists in the current ordered string table based on the two-level filter; if the target key exists in the current ordered string table based on the two-level filter, searching for the data to be searched based on the target key; if the target key does not exist in the current ordered string table based on the two-level filter, searching for the target key in the next ordered string table of the current ordered string table.

[0028] The embodiments of this application enable the search for a target key from the current ordered string surface.

[0029] In some optional implementations, determining whether a target key exists in the current ordered string table based on the two-level filter includes: determining whether a target ordered string table group exists in the current ordered string table based on the group filter in the two-level filter, where the target ordered string table group has the possibility of including the target key; if the target ordered string table group does not exist in the current ordered string table, it is determined that the target key does not exist in the current ordered string table; if the target ordered string table group exists in the current ordered string table, determining whether a target ordered string table exists in the target ordered string table group based on the table filter in the two-level filter, where the target ordered string table has the possibility of including the target key; if the target ordered string table group does not exist, it is determined that the target key does not exist in the current ordered string table; if the target ordered string table group exists, it is determined whether the target key exists in the target ordered string table; if the target ordered string table exists, it is determined that the target key exists in the current ordered string table.

[0030] The embodiments of this application can determine whether a target key exists in the current ordered string table based on the ordered string table group in the current ordered string table using a two-level filter.

[0031] In some optional implementations, determining whether a target ordered string group exists in the current ordered string table layer based on the group filter in the two-level filter, wherein the target ordered string group has the possibility of including the target key, includes: obtaining the group key range of multiple ordered string groups in the current ordered string table layer; determining one or more ordered string groups from the multiple ordered string groups in the current ordered string table layer whose group key range includes the target key, thus obtaining a determined ordered string group; determining whether the determined ordered string group has the possibility of including the target key through the group filter of the determined ordered string group; if the determined ordered string group has the possibility of including the target key, determining that there exists an ordered string group in the current ordered string table layer that has the possibility of including the target key, and using the determined ordered string group as the target ordered string group; if the determined ordered string group does not have the possibility of including the target key, determining that there is no target ordered string group in the current ordered string table layer that has the possibility of including the target key.

[0032] The embodiments of this application can determine whether there is a target ordered string table group with the possibility of the target key in the current ordered string table layer based on the group filter in the two-level filter according to the group key range.

[0033] In some optional implementations, determining whether a target ordered string group exists in the current ordered string table layer based on the group filter in the two-level filter, wherein the target ordered string group has the possibility of including the target key, includes: obtaining ordered string groups in the current ordered string table layer one by one, and determining whether the ordered string group has the possibility of including the target key through the group filter of the ordered string group; if any ordered string group in the current ordered string table layer has the possibility of including the target key, it is determined that a target ordered string group with the possibility of including the target key exists in the current ordered string table layer; if all ordered string groups in the current ordered string table layer do not have the possibility of including the target key, it is determined that no target ordered string group with the possibility of including the target key exists in the current ordered string table layer.

[0034] The embodiments of this application can sequentially obtain each ordered string group in the current ordered string table layer, and determine whether there is a target ordered string group in the current ordered string table layer that has the possibility of containing the target key based on the group filter in the two-level filter. Optionally, the ordered string groups in the current ordered string table layer can be obtained sequentially, or the ordered string groups in the current ordered string table layer can be traversed randomly.

[0035] In some optional implementations, determining whether a target ordered string table exists in the target ordered string table group based on the table filter in the two-level filter, wherein the target ordered string table has the possibility of including the target key, includes: obtaining the table key range of multiple ordered string tables in the target ordered string table group; determining one or more ordered string tables whose table key range includes the target key from the multiple ordered string tables in the target ordered string table group, thereby obtaining a determined ordered string table; determining whether the determined ordered string table has the possibility of including the target key through the table filter of the determined ordered string table; if the determined ordered string table has the possibility of including the target key, determining that there exists an ordered string table in the target ordered string table group that has the possibility of including the target key, and using the determined ordered string table as the target ordered string table; if the determined ordered string table does not have the possibility of including the target key, determining that there is no target ordered string table in the target ordered string table group that has the possibility of including the target key.

[0036] The embodiments of this application can determine whether there exists a target ordered string table in the target ordered string table group that has the possibility of including the target key, based on the table filter in the two-level filter according to the table key range.

[0037] In some optional implementations, determining whether a target ordered string table exists in the target ordered string table group based on the table filter in the two-level filter, wherein the target ordered string table group has the possibility of including the target key, includes: obtaining the ordered string tables in the target ordered string table group one by one, and determining whether the ordered string table has the possibility of including the target key through the table filter of the ordered string table; if any ordered string table in the target ordered string table group has the possibility of including the target key, it is determined that there is a target ordered string table in the target ordered string table group that has the possibility of including the target key; if all ordered string tables in the target ordered string table group do not have the possibility of including the target key, it is determined that there is no target ordered string table in the target ordered string table group that has the possibility of including the target key.

[0038] The embodiments of this application can retrieve the ordered string tables in the target ordered string table group one by one, and determine whether there is a target ordered string table in the target ordered string table group that has the possibility of including the target key based on the table filter in the two-level filter. Optionally, the newly generated ordered string table can be retrieved from the target ordered string table group, or the ordered string tables in the target ordered string table group can be randomly traversed.

[0039] The third aspect of this application discloses a filter update method based on the aforementioned log merge tree key-value storage system. The filter update method includes: determining any one of multiple ordered string tables as the current ordered string table, wherein the multiple ordered string tables are stored in a preset order; obtaining the number of ordered string tables in the current ordered string table and a preset upper limit for the number of tables; if the number of ordered string tables in the current ordered string table is greater than or equal to the preset upper limit for the number of tables, selecting an ordered string table group from the current ordered string table; generating an ordered string table to be merged based on the selected ordered string table group; merging the ordered string table to be merged into the next ordered string table of the current ordered string table to obtain a merged ordered string table; and updating the two-level filter of the next ordered string table of the current ordered string table based on the merged ordered string table.

[0040] During the compression process, the log merge tree hierarchical structure causes the storage system to need to update the metadata blocks and filter blocks of the persistent storage medium, resulting in additional write overhead and write amplification problems. The embodiments of this application can store metadata such as filters on non-volatile storage media, reducing write operations and writing overhead through in-situ updates.

[0041] In some alternative implementations, selecting an ordered string table group from the current ordered string table layer includes: selecting an ordered string table group from the current ordered string table layer based on the number of ordered string tables and the generation time.

[0042] The embodiments of this application enable the selection of an ordered string table.

[0043] In some optional implementations, generating an ordered string table to be merged based on the selected ordered string table group includes: merging and sorting the ordered string tables in the selected ordered string table group to obtain a first ordered string table; and dividing the first ordered string table into at least one second ordered string table based on the key range of the next ordered string table layer of the current ordered string table layer to obtain an ordered string table to be merged that includes at least one second ordered string table.

[0044] The embodiments of this application can generate an ordered string table to be merged based on a selected ordered string table group.

[0045] In some optional implementations, dividing the first ordered string table into at least one second ordered string table based on the key range of the next ordered string table of the current ordered string table includes: if the key range of the first ordered string table is included in any group key range of the next ordered string table of the current ordered string table, the first ordered string table is not divided; if the key range of the first ordered string table is included in any multiple group key ranges of the next ordered string table of the current ordered string table, the first ordered string table is divided according to the multiple group key ranges; if the key range of the first ordered string table is not included in any group key range of the next ordered string table of the current ordered string table, the first ordered string table is divided evenly.

[0046] The embodiments of this application can divide a first ordered string table into at least one second ordered string table based on the key range of the next ordered string table of the current ordered string table.

[0047] In some optional implementations, merging the ordered string tables to be merged into the next ordered string table layer of the current ordered string table layer includes: in the next ordered string table layer of the current ordered string table layer, merging each second ordered string table in the ordered string tables to be merged into an ordered string table group whose group key range includes the table key range of the second ordered string table through multi-threading.

[0048] The embodiments of this application can merge an ordered string table to be merged into the next ordered string table of the current ordered string table.

[0049] In some optional implementations, updating the two-level filter of the next ordered string table based on the merged ordered string table includes: generating corresponding group metadata based on the ordered string table groups generated in the merged ordered string table, and creating corresponding group filters on non-volatile storage media; updating the corresponding group filters on non-volatile storage media based on the modified ordered string table groups in the merged ordered string table.

[0050] The embodiments of this application can update the two-level filter of the next ordered string table of the current ordered string table based on the merged ordered string table. In-place updates reduce write operations and lower write overhead.

[0051] The fourth aspect of this application discloses a non-volatile storage medium management method based on the aforementioned log merge tree key-value storage system. The non-volatile storage medium management method includes: receiving a creation request for a table filter or group filter; converting the creation request into a space allocation request for the non-volatile storage medium; acquiring a lock on a free list of the non-volatile storage medium, the free list including page numbers of free pages in the non-volatile storage medium; extracting multiple page numbers from the free list; creating a storage management object on the non-volatile storage medium based on the multiple page numbers; returning the storage management object; and releasing the lock on the free list.

[0052] The embodiments of this application can solve the space allocation problem of non-volatile storage media. Paging management is implemented for the storage space of non-volatile storage media, encapsulating the underlying paging mechanism for efficient memory allocation. Memory allocation of non-volatile storage media can be implemented based on multi-threading.

[0053] The sixth aspect of this application discloses a non-volatile storage medium management method. Based on the aforementioned log merge tree key-value storage system, the non-volatile storage medium management method includes: in response to data failure of a table filter or group filter, triggering a space reclamation request for the non-volatile storage medium; adding the reclamation request to a reclamation queue and notifying a scheduling thread of the reclamation request; creating multiple reclamation threads through the scheduling thread based on the reclamation requests in the reclamation queue, so as to reclaim failed pages through multiple reclamation threads; updating the free list according to the failed pages, the free list including the page number of the free pages of the non-volatile storage medium; and waiting for the reclamation threads to finish.

[0054] The embodiments of this application can perform page reclamation based on multi-threading, thereby achieving lightweight non-volatile storage media space management.

[0055] In some optional implementations, updating the free list based on the invalid page includes: obtaining the header page number of the free list; assigning the header page number to the next page number of the invalid page; and reassigning the page number of the invalid page as the header page number.

[0056] The embodiments of this application enable updating the free list based on the failed page.

[0057] The seventh aspect of this application discloses a computer device, including a processor and a memory; the memory is used to store instructions; the processor is used to invoke the instructions in the memory, causing the computer device to execute a method related to a log merge tree key-value storage system.

[0058] The eighth aspect of this application discloses a computer-readable storage medium storing at least one instruction, wherein when the at least one instruction is executed by a processor, a method for implementing a log merge tree key-value storage system is disclosed.

[0059] The technical effects brought about by aspects seven to eight can be found in the descriptions of the methods mentioned above, and will not be repeated here. Attached Figure Description

[0060] Figure 1 It is an LSM-Tree organizational structure.

[0061] Figure 2 It is a RocksDb structure.

[0062] Figure 3 It is the compression mechanism of the key-value storage system.

[0063] Figure 4 It is SLM-DB.

[0064] Figure 5 This is the architecture of the log merge tree key-value storage system based on a hybrid storage architecture provided in the embodiments of this application.

[0065] Figure 6 This is a schematic diagram of the structure of a two-stage filter provided in an embodiment of this application.

[0066] Figure 7 This is a schematic diagram of the access to the two-level filter provided in the embodiments of this application.

[0067] Figure 8 This is a flowchart of a data retrieval method for a log merge tree key-value storage system based on a hybrid storage architecture, provided in an embodiment of this application.

[0068] Figure 9 This is a schematic diagram of data retrieval in a log merge tree key-value storage system based on a hybrid storage architecture, provided in an embodiment of this application.

[0069] Figure 10 This is a flowchart of the filter update process provided in an embodiment of this application.

[0070] Figure 11 This is a schematic diagram of filter updates provided in an embodiment of this application.

[0071] Figure 12 This is a flowchart of a mutually exclusive application method provided in an embodiment of this application.

[0072] Figure 13 This is a flowchart of the concurrent recycling method provided in the embodiments of this application.

[0073] Figure 14 This is a data structure logic diagram of the non-volatile storage media management method provided in the embodiments of this application.

[0074] Figure 15 This is a schematic diagram of filter creation provided in an embodiment of this application.

[0075] Figure 16 This is a schematic diagram of the computer device provided in the embodiments of this application. Detailed Implementation

[0076] For ease of understanding, some concepts related to the embodiments of this application are illustrated and explained by way of example for reference.

[0077] It should be noted that in the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects, not to describe a specific order or sequence.

[0078] Key-value (KV) storage systems can be implemented based on Log Structured Merge Trees (LSM-Trees). Key-value storage systems based on Dynamic Random Access Memory (DRAM) typically suffer from disk read / write performance bottlenecks. Furthermore, LSM-Trees exhibit significant read amplification and write amplification issues. Read amplification refers to the ratio of the actual amount of data read or written by the storage system to the amount of data the user expects to write. Read amplification affects the read performance of the key-value storage system, while write amplification reduces the system's lifespan due to wear caused by repeated writes. Non-volatile memory (NVM) possesses the excellent storage characteristics of both main memory and secondary storage. The key-value storage system in the embodiments of this application, based on a hybrid storage architecture of persistent storage media and NVM, along with a log merging tree, can improve data read / write performance. The persistent storage media can be a Solid State Drive (SSD), also known as a solid-state drive, which features large capacity and high stability.

[0079] For ease of understanding, the following sections provide a brief explanation of LSM-Tree, LSM-Tree key-value storage system architecture, compression mechanism of key-value storage system, and non-volatile storage media.

[0080] like Figure 1The diagram shows the LSM-Tree organizational structure. LSM-Tree transforms random writes to persistent storage media into sequential writes, thereby improving write speed. The persistent storage media can be a disk. The index tree structure in an LSM-Tree consists of a small tree (C0) and a large tree (C1, C2...Ck). The small tree is stored in memory, and the large tree can be stored on persistent storage media. The two trees jointly maintain an ordered key space. A received write operation first operates on the small tree C0 in memory. As the small tree in memory grows, when its size reaches a preset condition, a merge operation (MergeSort) is triggered with the large tree (C1, C2...Ck) on persistent storage media. The write operation resulting from the merge operation is a sequential write.

[0081] LSM-Tree key-value store system architectures can include RocksDB or LevelDB structures, etc. Figure 2 The image shows a RocksDb structure. An LSM-Tree can include in-memory structures (memtable, immutable memtable) and on-disk structures (e.g., a sorted string table (SSTable or SST)). The role of the in-memory structure is related to... Figure 1 The function of C0 in an LSM-Tree is similar to that of the disk structure. Figure 1 The function of the large tree in an LSM-Tree is similar. The memtable is a structure for organizing and maintaining data in memory; the immutable memtable is an immutable memtable, also simply called immutable. Before writing data to the memory buffer, a write-ahead logging (WAL) process is performed. Flush refers to importing the data from the memtable into the SSTable, making it persistent storage.

[0082] The data flow path for write operations in the RocksDb structure is: "memtable->immutable memtable->L0SSTables->L1 SSTables->...Ln SSTables". Therefore, SSTables with fewer levels contain newer data, while memory contains the most up-to-date data.

[0083] The read operation process of the RocksDb structure is as follows: After accessing the memory's memtable and immutable memtable sequentially, the target SSTable is located through the SSTable Info Cache maintained in memory. Simultaneously, the SSTable filter corresponding to the target SSTable is used to determine if the key of the target data exists in the target SSTable. If the key exists, the target data is retrieved from the target SSTable. The target data can be retrieved sequentially from the L0, L1, and up to Ln layers on disk. If the search reaches the highest Ln layer and still does not find the key of the target data, it means that the target data does not exist in the RocksDb structure. The lookup mechanism of LSM-Tree and key-value storage systems can achieve this. If multiple target data are stored in the RocksDb structure, the key of the target data read first during the read process is the latest key in time sequence. The SSTable filter is a hash structure used to determine whether an object exists in an SSTable, such as a Bloom filter or a quotient filter.

[0084] like Figure 3 The diagram illustrates the compression mechanisms of key-value stores. LSM-Tree-based key-value stores (such as LevelDB and RocksDB) need to periodically merge lower-level SSTables into higher-level SSTables. The organization and merging methods of SSTables at different levels are called compression mechanisms. Compression mechanisms in key-value stores can include level compaction, tier compaction, and group compaction. Level compaction is the default mechanism in LSM-Tree-based key-value stores like LevelDB and RocksDB, and it divides files at each level according to their key values. Tiered compaction manages and merges SSTables based on file size. Group compaction builds upon level compaction by organizing each level of SSTables into groups.

[0085] In a LSM-Tree-based key-value storage system, if the data volume of the SSTable in the Li layer is equal to or greater than the upper limit, one or more SSTables in the Li layer are merged with the SSTables in the Li+1 layer, the original SSTables in the Li+1 layer are deleted, and the merged SSTable is inserted into the Li+1 layer.

[0086] like Figure 3 Figure A illustrates the hierarchical compression mechanism of a key-value storage system. In this mechanism, SSTables in each level are arranged sequentially, and the key ranges of any two SSTables do not overlap. If the data volume of an SSTable in level Li is equal to or greater than the upper limit, one or more SSTables in level Li are selected and merged with SSTables in level Li+1 whose key ranges overlap. The original SSTables in level Li+1 are deleted, and the merged SSTable is inserted into level Li+1. If the data volume of an SSTable in level Li+1 is also equal to or greater than the upper limit, the merging of levels Li+1 and Li+2 is iteratively performed. Figure 3 As shown in Figure A, the area above the horizontal line represents the memory space, and the area below the horizontal line represents the persistent storage medium. When writing 13 into memory, 7 and 13 in memory are written to layer L0. After writing 7 and 13 to layer L0, the data volume of the SSTable in layer L0 equals the upper limit. 1, 3, 7, 10, 13, and 19 in layer L0 are merged with 2, 4, 8, 12, 15, and 18 in layer L1. The original SSTable in layer L1 is deleted, and the merged SSTable is inserted into layer L1.

[0087] The hierarchical compression mechanism's maintenance of order can cause severe tail latency issues, blocking foreground request responses and causing significant write amplification.

[0088] like Figure 3 Figure B illustrates the hierarchical compression mechanism of a key-value storage system. The hierarchical merging mechanism distributes SSTables across different levels according to file size. SSTables within each level can be arranged out of order, and overlapping key ranges are allowed between any two SSTables within the same level. If the data volume of an SSTable in level Li is equal to or greater than the upper limit, one or more SSTables in level Li that overlap with the key range of level Li+1 are selected and merged in memory. The original SSTables in level Li+1 are deleted, and the merged SSTable is inserted into level Li+1. If the data volume of an SSTable in level Li+1 is also equal to or greater than the upper limit, the merging process for level Li+1 is iterative.

[0089] In a key-value store with a tiered compression mechanism, SSTables are in an unordered state, and the range of table keys between SSTables is also unordered. This results in a slower data read speed for the tiered compression mechanism compared to a key-value store with a hierarchical compression mechanism. Although the keys and values ​​within a single SSTable are ordered, the data read speed of a log merge tree-based storage system is slower than that of a B+ tree-based storage system. In particular, in big data scenarios, where data volume surges, the orderliness between SSTables cannot be completely abandoned.

[0090] like Figure 3 Figure C illustrates the grouping compression mechanism of a key-value storage system. Building upon hierarchical compression, this mechanism divides each SSTable into groups based on its key range. SSTables within the same group can have overlapping key ranges. By maintaining order between groups, read performance can be improved. Figure 3 As shown in C, when writing 13 into memory, 7 and 13 in memory are written to L0 level. After writing 7 and 13 to L0 level, the data volume of the SSTable in L0 level is equal to the upper limit (SSTables need to be written to SSTable groups whose group keys include the table keys of the SSTable, that is, 7 and 13 are written to SSTable groups with group keys ranging from 1 to 100, but cannot be written to SSTable groups with group keys ranging from 101 to 200). Then, 1, 3, 55, 69, 7, and 13 in L0 level are merged with the SSTables in L1 level. Figure 3 If there is no SSTable in L1 layer of C, delete the original SSTable in L1 layer and insert the merged SSTable into L1 layer.

[0091] Non-volatile storage media can include phase-change memory (PCM), resistive RAM (ReRAM), and memristors. Non-volatile storage media possesses the excellent characteristics of both main memory and persistent storage media, exhibiting superior storage performance. Non-volatile storage media has storage latency close to DRAM while providing the same memory access unit (i.e., cache line) as DRAM, allowing addressing on a byte-by-byte basis. This enables the host to access non-volatile storage media directly through the memory bus, just like accessing DRAM, avoiding access through the hardware block device interface and thus avoiding the access wall associated with the hardware block device interface. Furthermore, non-volatile storage media retains the non-volatility of persistent storage media; data stored on non-volatile storage media is not lost when power is cut off. Non-volatile storage media has a higher storage density than DRAM, and its capacity can reach terabyte levels, making it a supplement to DRAM and a medium used to build large-capacity storage systems.

[0092] The following is one way to improve the performance of a log merge tree key-value storage system.

[0093] Typically, SLM-DB (Single-Level Merge DB) leverages NVM to improve the performance of log merge tree key-value stores. NVM is used to store MemTable and Immutable data to enhance consistency while reducing consistency overhead. SLM-DB transforms multi-level data structures on disk into single-level data structures and introduces B+-Trees in NVM as indexes for the single-level data structure. It creates B+-Tree indexes for all key and value locations on disk, reducing disk accesses during data queries by utilizing these indexes. Figure 4 The image shows SLM-DB. Here, Persistent memory refers to persistent memory, and Compaction Log refers to the compressed log.

[0094] Storing B+-Tree indexes for all keys on the NVM requires a large amount of NVM space, and the NVM indexes store variable-length key fields, necessitating a complex NVM management mechanism. The B+-Tree indexes maintained by SLM-DB on the NVM introduce excessively high NVM costs, and due to the complexity of the NVM management mechanism, NVM space utilization is low.

[0095] To better understand the log merge tree key-value storage system, optimization method, and related devices based on a hybrid storage architecture disclosed in the embodiments of this application, the technical problems solved by the embodiments of this application will be described below.

[0096] To address the read / write amplification issue in log merge tree key-value stores, LSM-Tree-based systems use in-memory structures to transform write requests into sequential writes to persistent storage. Data is distributed across different levels of the persistent storage based on its accessibility (hotness or coldness). However, this introduces maintenance overhead. Each compression operation updates the data indexes and filters on the persistent storage multiple times, increasing write amplification. Furthermore, reading data requires accessing different levels of data, including the data block containing the key, as well as the corresponding index and filter blocks, adding further read overhead.

[0097] How to solve the space management problem of NVM? NVM is used to accelerate read and write operations in key-value storage systems, but existing NVM space management methods have performance limitations and do not match the data persistence requirements of LSM-Tree-based key-value storage systems.

[0098] To better understand the log merge tree key-value storage system, optimization method, and related devices based on a hybrid storage architecture disclosed in the embodiments of this application, the architecture of the log merge tree key-value storage system based on a hybrid storage architecture of this application will first be described below. For ease of description, the "log merge tree key-value storage system based on a hybrid storage architecture" will be abbreviated as "log merge tree key-value storage system" below.

[0099] like Figure 5 The diagram shows the architecture of a log merge tree key-value storage system based on a hybrid storage architecture provided in this application embodiment.

[0100] like Figure 5 The hardware architecture shown is that of a log merge tree key-value store system based on a hybrid storage architecture. This architecture includes a host, persistent storage media, and non-volatile storage media. The host is connected to both the non-volatile and persistent storage media. Specifically, the host can connect to the non-volatile storage media via a memory bus, and it can connect to the persistent storage media via PCIe (Peripheral Component Interconnect Express). The non-volatile storage media can be an Optane disk based on 3D Xpoint. The host's Central Processing Unit (CPU) accesses and persists data via the memory bus and x86 instruction set's clfush quality and mefence instructions.

[0101] The hybrid storage architecture also includes volatile storage media (memory), with the host connected to the volatile storage media (memory).

[0102] like Figure 5 The software architecture of the log merge tree key-value store system, shown below, is based on a persistent storage system (such as a single-machine file system, a distributed file system, or a storage system that provides data persistence methods). The log merge tree key-value store accesses and retrieves key-value data (SSTables) from the persistent storage system via key-value (KV) methods. Applications access the log merge tree key-value store system through the KV interface and invoke the necessary storage services. The log merge tree key-value store system utilizes NVM through the NVM lightweight space management mechanism. The persistent storage system runs on persistent storage media.

[0103] In one embodiment of this application, a log merge tree key-value storage system based on a hybrid storage architecture is applied in an electronic device. The hybrid storage architecture includes a host, a persistent storage medium, and a non-volatile storage medium. The host is connected to the non-volatile storage medium, and the host is connected to the persistent storage medium. The compression mechanism of the log merge tree key-value storage system may include a group compression mechanism. The log merge tree key-value storage system includes a two-level filter. The persistent storage medium stores multiple ordered string tables, each ordered string table includes multiple ordered string table groups, each ordered string table group includes multiple ordered string tables, and the two-level filter corresponding to each ordered string table group includes a group filter corresponding to the ordered string table group and a table filter corresponding to the ordered string tables in the ordered string table group.

[0104] In one embodiment of this application, the two-stage filter is stored in the electronic device's memory, non-volatile storage medium, and / or persistent storage medium. The two-stage filter may be stored in a non-volatile storage medium, or optionally in the electronic device's persistent storage medium.

[0105] In one embodiment of this application, a group filter may include a Bloom filter or a Cowboy filter, and a table filter may include a Bloom filter, a quotient filter, or a Cowboy filter. Each ordered string table in the persistent storage medium corresponds to a group filter, which is a hash structure used to determine whether an object exists in the corresponding ordered string table.

[0106] like Figure 6 The diagram shown is a schematic representation of the structure of a two-stage filter provided in an embodiment of this application. Figure 6As shown, NVM stores two two-level filters: one corresponding to ordered string table group 1 (Group1) and another corresponding to ordered string table group 2 (Group2). Dashed lines represent selective storage. The two filters stored in Storage can be selectively stored; that is, Storage can store both two two-level filters represented by the dashed lines, or it can choose not to store them. A two-level filter corresponding to an ordered string table group includes group metadata corresponding to that ordered string table group and table metadata (SSTable metadata) corresponding to the ordered string tables within that ordered string table group. Optionally, the metadata can include a key range, or the metadata can include a key range and filters. For example, the metadata of an ordered string table group is group metadata, and the metadata of an ordered string table is table metadata. The group metadata of an ordered string table group can include the group key range of that ordered string table group, and the table metadata of an ordered string table can include the table key range of that ordered string table. Similarly, the group metadata of an ordered string table group can include the group key range and group filters of that ordered string table group, and the table metadata of an ordered string table can include the table key range and table filters of that ordered string table.

[0107] While the group compression mechanism preserves some of the ordering of SSTables, it requires accessing all SSTables in the ordered string table group to determine if the target key of the data to be searched exists in the ordered string table group. To reduce read amplification in the group compression mechanism, embodiments of this application construct a two-level filter structure including group filters and table filters by establishing group filters for ordered string table groups on NVM. The group filters reduce invalid reads of ordered string table groups, and by storing the two-level filters in NVM, the access of read operations to persistent storage media is reduced.

[0108] In one embodiment of this application, the log merge tree key-value storage system based on a hybrid storage architecture creates a filter (i.e., a table filter) corresponding to all keys in each ordered string table, and creates a filter (i.e., a group filter) corresponding to all keys in each ordered string table group. A two-level filter is formed by the group filter of an ordered string table group and the table filters corresponding to the ordered string tables in the ordered string table group, wherein one of the two-level filters may include one or more table filters.

[0109] Understandably, the group filter of the ordered string table group and the table filter corresponding to the ordered string table in the ordered string table group constitute a two-level filter structure.

[0110] like Figure 7The diagram shown is an access schematic of a two-level filter provided in an embodiment of this application. Figure 7 As shown, the group filter in the two-level filter is a 4-slot Cuckoo filter, and the table filter is a Bloom filter located using 3 hash functions. The Cuckoo and Bloom filters are merely examples to illustrate access to the two-level filter; practical applications are not limited to these examples.

[0111] This embodiment only describes the access process of the two-level filter and does not consider the selection process of SSTable in the ordered string table group. It is known that the target key 45 of the data to be searched will hit the second slot of index 2 in the group filter and access the positions of indexes 1, 2, and 16 in the table filter. The specific process of finding the target key 45 may include:

[0112] A hash value 2 is calculated using the first hash function provided by the group filter, and another hash value 3 is calculated using the second hash function provided by the group filter. The four slots at index 2 of the group filter are traversed, and the hash value within each slot is used to determine if the second slot is the corresponding entry. Therefore, it is determined that the target key 45 may be in this ordered string table group (i.e., this ordered string table group has the possibility of containing the target key 45). If no corresponding slot is found after the traversal, it indicates that the target key 45 is not in this ordered string table group, thus saving the search process for the SSTable file in the ordered string table group and reducing invalid read operations.

[0113] The three hash functions provided by SSTable calculate three hash values: 1, 2, and 16. Accessing the corresponding index positions, index 2 corresponds to a value of 0. Therefore, it is determined that the target key 45 is not in the ordered string table group. If any one of the multiple indexes corresponds to a value of 0, then it can be confirmed that the target key does not exist in the ordered string table group.

[0114] This application provides a data retrieval method for a log merge tree key-value storage system based on a hybrid storage architecture, applicable to electronic devices. Specifically, the data retrieval method includes:

[0115] 101, retrieve the target key of the data to be searched.

[0116] The data to be searched can be key-value data, including the target key and the value corresponding to the target key.

[0117] 102. Locate the target key in the memory of the electronic device.

[0118] Log merge tree key-value storage systems can employ caching mechanisms such as BlockCache and TableCache.

[0119] You can first look up the target key in memory caches such as MemTable, Immutable MemTable, etc.

[0120] 103. If the target key is not found in the memory of the electronic device, search for the target key in the persistent storage medium according to the two-level filter.

[0121] Persistent storage media can include SSDs. If the target key is not found in memory, it is searched for in the SSD using a two-level filter. Furthermore, in response to a query request, the found target key or the search result can be returned.

[0122] (i) In one embodiment of this application, searching for a target key from a persistent storage medium using a two-level filter includes: according to the layer order, for any current ordered string table among multiple ordered string tables, determining whether there exists a target ordered string table group storing the target key in the current ordered string table based on the group key range of the multiple ordered string table groups in the current ordered string table; if there exists a target ordered string table group storing the target key in the current ordered string table, searching for the target key from the target ordered string table group using a two-level filter.

[0123] The layer order can include the order of layer numbers, etc. For example, the target key can be searched in order from layer 0 to layer n of the log merge tree. Before searching for the target key in the current ordered string table layer, a search method can be used to determine whether a target ordered string table group storing the target key exists in the current ordered string table layer based on the group key range of multiple ordered string table groups in the current ordered string table layer. The search method can include binary search or sequential search, etc.

[0124] In one embodiment of this application, if the target ordered string table group storing the target key does not exist in the current ordered string table layer, the target key is searched from the next ordered string table layer of the current ordered string table layer.

[0125] In one embodiment of this application, searching for a target key from a target ordered string table group based on a two-level filter includes: determining whether the target ordered string table group has the possibility of including the target key (i.e., determining whether the target ordered string table group may store the target key) based on the group filter in the two-level filter; if the group filter in the two-level filter determines that the target ordered string table group has the possibility of including the target key (i.e., determining that the target ordered string table group may store the target key), determining whether there exists a target ordered string table group storing the target key based on the table key range of multiple ordered string tables in the current ordered string table group (based on the search method); if the table key range of multiple ordered string tables in the current ordered string table group determines that there exists a target ordered string table group storing the target key, determining whether the target ordered string table has the possibility of including the target key (i.e., determining whether the target ordered string table may store the target key) based on the table filter in the two-level filter; if the table filter in the two-level filter determines that the target ordered string table has the possibility of including the target key (i.e., determining that the target ordered string table may store the target key), searching for the data to be searched in the target ordered string table based on the target key.

[0126] Specifically, if the group filter determines that the target ordered string table group may store the target key, then the target ordered string table group actually stores the target key or does not store the target key; if the group filter determines that the target ordered string table group cannot store the target key, then the target ordered string table group actually does not store the target key.

[0127] The embodiments of this application enable the search for a target key from a target ordered string table using a two-level filter. First, the probability that the target ordered string table contains the target key is assessed. If the target ordered string table has the possibility of containing the target key, it indicates a high probability that the target ordered string table contains the target key. Then, the ordered strings within the target ordered string table are further evaluated to achieve an accurate final determination of whether the target ordered string table contains the target key. Using a two-level filter avoids invalid searches, thereby improving the efficiency of target key search.

[0128] In one embodiment of this application, if the target ordered string table group is determined to be unlikely to include the target key based on the group filter in the two-level filter, or if the target ordered string table is determined to be unlikely to include the target key based on the table filter in the two-level filter, the target key is searched from the next ordered string table layer of the current ordered string table layer.

[0129] (ii) In one embodiment of this application, the target key is searched from multiple ordered string tables in the persistent storage medium according to the layer number order based on a two-level filter. For example, the target key search can be performed in the order from layer 0 to layer n of the log merge tree.

[0130] In one embodiment of this application, for any one of the multiple ordered string tables, the target key is searched from the current ordered string table according to a two-level filter.

[0131] Specifically, the system uses a two-level filter to determine if the target key exists in the current ordered string table. If the two-level filter determines that the target key exists in the current ordered string table, the system searches for the data to be searched based on the target key. If the two-level filter determines that the target key does not exist in the current ordered string table, the system searches for the target key in the next ordered string table.

[0132] In one embodiment of this application, determining whether a target key exists in the current ordered string table based on a two-level filter may include: determining whether a target ordered string table group exists in the current ordered string table based on the group filter in the two-level filter, where the target ordered string table group has the possibility of including the target key (i.e., the target ordered string table group may store the target key). If the target ordered string table group does not exist in the current ordered string table, it is determined that the target key does not exist in the current ordered string table. If the target ordered string table group exists in the current ordered string table, determining whether a target ordered string table exists in the target ordered string table group based on the table filter in the two-level filter, where the target ordered string table has the possibility of including the target key; if the target ordered string table group does not exist, it is determined that the target key does not exist in the current ordered string table. If the target ordered string table group exists, determining whether the target key exists in the target ordered string table; if the target key exists in the target ordered string table, it is determined that the target key exists in the current ordered string table.

[0133] Optionally, determining whether a target ordered string group exists in the current ordered string table layer based on the group filter in the two-level filter, where the target ordered string group has the possibility of including the target key, includes: obtaining the group key range of multiple ordered string groups in the current ordered string table layer; determining one or more ordered string groups from the multiple ordered string groups in the current ordered string table layer whose group key range includes the target key, thus obtaining a determined ordered string group; determining whether the determined ordered string group has the possibility of including the target key through the group filter of the determined ordered string group; if the determined ordered string group has the possibility of including the target key (i.e., the ordered string group may store the target key), determining that a target ordered string group exists in the current ordered string table layer, and using the determined ordered string group as the target ordered string group; if the determined ordered string group does not have the possibility of including the target key (i.e., the ordered string group cannot store the target key), determining that there is no target ordered string group in the current ordered string table layer that has the possibility of including the target key.

[0134] Specifically, a binary search method can be used to determine one or more ordered string groups whose group key range includes the target key from multiple ordered string groups in the current ordered string table layer. The group key range of an ordered string group includes the minimum and maximum keys within that group, and the group key ranges do not overlap between ordered string groups. Therefore, a binary search method can be used to find one or more ordered string groups that may include the target key.

[0135] Understandably, during the search process, for the current ordered string table group in the current ordered string table layer, the filter of the current ordered string table group can be used to determine whether the target key may exist in the current ordered string table group; if it is determined that it is impossible to exist, the search of the current ordered string table group can be skipped and subsequent searches can be performed; if it is determined that it may exist, in order to ensure that the target key actually exists in the current ordered string table group, it is necessary to make further judgments through the table filter of the ordered string table in the current ordered string table group, thereby reducing the false positive rate, reducing the number of accesses to the persistent storage medium, and reducing the read performance degradation caused by the compression method.

[0136] Optionally, the existence of a target ordered string group in the current ordered string table is determined based on the group filter in the two-level filter. The possibility that the target ordered string group may include the target key is determined by: obtaining each ordered string group in the current ordered string table one by one, and determining whether the ordered string group may include the target key through the group filter of the ordered string group; if any ordered string group in the current ordered string table may include the target key, it is determined that there is an ordered string group in the current ordered string table that may include the target key, and the ordered string group that may include the target key is taken as the target ordered string group; if none of the ordered string groups in the current ordered string table may include the target key, it is determined that there is no target ordered string group in the current ordered string table that may include the target key.

[0137] Optionally, determining whether a target ordered string table exists in the target ordered string table group based on the table filter in the two-level filter, where the target ordered string table has the possibility of including the target key, includes: obtaining the table key range of multiple ordered string tables in the target ordered string table group; determining one or more ordered string tables whose table key range includes the target key from the multiple ordered string tables in the target ordered string table group, thus obtaining the determined ordered string table; determining whether the determined ordered string table has the possibility of including the target key through the table filter of the determined ordered string table (i.e., determining whether the determined ordered string table may contain the target key through the table filter of the determined ordered string table); if the determined ordered string table has the possibility of including the target key (i.e., the determined ordered string table may contain the target key), determining that there exists an ordered string table in the target ordered string table group that has the possibility of including the target key, and using the determined ordered string table as the target ordered string table; if the determined ordered string table does not have the possibility of including the target key (i.e., the determined ordered string table cannot contain the target key), determining that there is no target ordered string table in the target ordered string table group that has the possibility of including the target key.

[0138] Specifically, based on the chronological order of the ordered string table, a table filter can be used to determine whether the ordered string table might contain the target key. For example, first determine whether table 1 might contain the target key, then determine whether table 2 might contain the target key, where table 1 was generated later than table 2.

[0139] Optionally, determining whether a target ordered string table exists in the target ordered string table group based on the table filter in the two-level filter, where the target ordered string table has the possibility of including the target key, includes: obtaining the ordered string tables in the target ordered string table group one by one, and determining whether the ordered string table has the possibility of including the target key through the table filter of the ordered string table; if any ordered string table in the target ordered string table group has the possibility of including the target key, it is determined that a target ordered string table with the possibility of including the target key exists in the target ordered string table group; if all ordered string tables in the target ordered string table group do not have the possibility of including the target key, it is determined that no target ordered string table with the possibility of including the target key exists in the target ordered string table group.

[0140] This application provides another data lookup method for a log merge tree key-value storage system based on a hybrid storage architecture, which is applied in electronic devices. Specifically, the data lookup method includes: obtaining the target key of the data to be looked up; and looking up the target key from the persistent storage medium according to a two-level filter.

[0141] like Figure 8 The diagram shows a flowchart of a data search method for a log merge tree key-value storage system based on a hybrid storage architecture provided in this application embodiment. The electronic device determines whether the target key is found in the memory cache; if the target key is found in the memory cache, it returns the target key-value pair corresponding to the found target key; if the target key is not found in the memory cache, it initializes the current ordered string table level to 0; within the current ordered string table level, it uses a binary search to search for an ordered string table group containing the target key; it uses a group filter in the NVM to determine whether the found ordered string table group contains the target key; if the group filter in the NVM determines that the found ordered string table group does not contain the target key, it increments the current ordered string table level by 1; it determines whether the current ordered string table level is greater than or equal to a preset maximum level; if the current ordered string table level is greater than or equal to the preset maximum level, it returns a search result indicating that the target key does not exist; if the current ordered string table level is less than the preset maximum level, the process returns to the next level of the current ordered string table level, where a binary search is used to search for an ordered string table group containing the target key. If the group filter in NVM determines that the found ordered string table group contains the target key, the target key is found through the metadata and table filter of the ordered string table within the ordered string table group in NVM. If the ordered string table within the ordered string table group contains the target key, the target key-value pair corresponding to the target key value is returned. If the ordered string table within the ordered string table group does not contain the target key, the process returns to incrementing the level of the current ordered string table by 1.

[0142] like Figure 9The diagram shown illustrates data lookup in a log merge tree key-value storage system based on a hybrid storage architecture, as provided in an embodiment of this application. The dashed lines represent selective storage.

[0143] Given that the target key is 45, the key value range of MemTable is 3-15, and the key value range of Immutable is 1-10, the files on the SSD have two levels, each containing several groups. The first two groups of level zero each contain three SSTables, with key value ranges of 1-30 and 31-60 respectively. The second group does not contain the target key 45. The first two groups of level one have key value ranges of 1-40 and 41-80 respectively. The first SSTable in the second group does not contain the target key, while the second and third SSTables in the second group both contain the target key 45. The three SSTables in the second group are arranged chronologically from top to bottom, with the first SSTable in the second group being created later than the third SSTable. NVM stores metadata corresponding to four ordered string tables. The search for the target key 45 specifically includes:

[0144] (1) Search for the target key 45 in the MemTable and Immutable MemTable in memory. Based on the key value range, it can be determined that 45 does not exist in the memory cache.

[0145] (2) Search the SSTable in the SSD. First, search in L0. Perform a binary search based on the range of group key values ​​stored in the group metadata of L0 layer to determine that the data exists in the second group of L0 layer.

[0146] (3) The second group of L0 contains three SSTables with key value ranges of 31-41, 33-43, and 47-56, respectively. Before accessing a specific SSTable, the group filter determines that the target key 45 does not exist in the second group of L0, and then continues to access the next level file.

[0147] (4) In layer L0, the binary search method is used to determine that the target key 45 exists in the key value range of the second group in layer L0. Using the Group Indexer strategy, based on the information stored in the second group of layer L0, it can be determined that the target key 45 exists in the first group and the second group of layer L1. That is, the key value range of the second group in layer L0 overlaps with the key value ranges of the first and second groups of layer L1, indicating that the target key 45 may exist in the first and second groups of layer L1.

[0148] (5) Perform a binary search in the first group and the second group of L1 layer based on the range of group key values ​​stored in the group metadata to determine that the data exists in the second group of L1 layer.

[0149] (6) The second group of L1 contains three SSTables with key value ranges of 41-60, 44-55, and 45-75, respectively. Before accessing a specific SSTable, the group filter determines whether the target key 45 exists in the second group of L1.

[0150] (7) Select files whose key range contains the target key 45 based on the metadata information of the L1 layer. The key value range of the three SSTables in the second group of the L1 layer all contain the target key.

[0151] (8) Perform secondary filtering on the three SSTables in the second group of L1 layer in the order of generation time. The secondary filtering can be performed in the order of generation time from late to early.

[0152] (8-1) Use the filter of the first SSTable in the second group of L1 layer to determine if the target key 45 exists. If it exists, use the RocksDB SSTable read interface to access the SSTable, search for the target key in the SSTable, and perform subsequent searches if it is not found; if it does not exist, perform subsequent searches directly.

[0153] (8-2) Use the filter of the second SSTable in the second group of L1 layer to determine that the target key 45 exists, and then find the key-value pair (data to be searched) corresponding to the target key in the second SSTable and return it.

[0154] This application provides a filter update method for a log merge tree key-value storage system based on a hybrid storage architecture, applicable to electronic devices. Specifically, the filter update method includes:

[0155] 201. Select any one of the multiple ordered string layers as the current ordered string layer, wherein the multiple ordered string layers are stored in a preset order.

[0156] 202, Get the number of ordered strings in the current ordered string table and the preset upper limit of the number of tables.

[0157] 203. If the number of ordered strings in the current ordered string table layer is greater than or equal to the preset upper limit of the number of tables, select an ordered string table group from the current ordered string table layer.

[0158] In one embodiment of this application, selecting an ordered string table group from the current ordered string table layer includes: selecting an ordered string table group from the current ordered string table layer based on the number of ordered string tables and the generation time.

[0159] Specifically, you can select the group of ordered strings that has the largest number of ordered strings and the earliest average generation time from the current ordered string table layer.

[0160] 204. Generate an ordered string table to be merged based on the selected ordered string table group.

[0161] In one embodiment of this application, generating an ordered string table to be merged based on a selected ordered string table group includes: merging and sorting the ordered string tables in the selected ordered string table group to obtain a first ordered string table; and dividing the first ordered string table into at least one second ordered string table based on the key range of the next ordered string table layer of the current ordered string table layer to obtain an ordered string table to be merged that includes at least one second ordered string table.

[0162] In one embodiment of this application, dividing a first ordered string table into at least one second ordered string table based on the key range of the next ordered string table of the current ordered string table includes: if the key range of the first ordered string table is included in any group key range of the next ordered string table of the current ordered string table, the first ordered string table is not divided; if the key range of the first ordered string table is included in any multiple group key ranges of the next ordered string table of the current ordered string table, the first ordered string table is divided according to the multiple group key ranges; if the key range of the first ordered string table is not included in any multiple group key range of the next ordered string table of the current ordered string table, the first ordered string table is divided evenly.

[0163] In one embodiment of this application, the size of the first ordered string table is the same as the size of the ordered string table of the next ordered string table layer. Table filters and table metadata corresponding to the first ordered string table are created on NVM.

[0164] 205. Merge the ordered string table to be merged into the next ordered string table of the current ordered string table to obtain the merged ordered string table.

[0165] In one embodiment of this application, merging the ordered string tables to be merged into the next ordered string table layer of the current ordered string table layer includes: in the next ordered string table layer of the current ordered string table layer, merging each second ordered string table in the ordered string tables to be merged into an ordered string table group whose group key range includes the table key range of the second ordered string table through multi-threading.

[0166] 206. Update the two-level filter of the next ordered string table of the current ordered string table based on the merged ordered string table.

[0167] In one embodiment of this application, updating the two-level filter of the next ordered string table of the current ordered string table based on the merged ordered string table includes: generating corresponding group metadata based on the ordered string table groups generated in the merged ordered string table, and creating a group filter corresponding to the generated ordered string table groups on a non-volatile storage medium; and updating the group filter corresponding to the modified ordered string table groups on a non-volatile storage medium based on the modified ordered string table groups in the merged ordered string table.

[0168] The filter update method in this embodiment updates metadata and filters in-situ on NVM, reducing the overhead of write amplification and reclaiming storage space.

[0169] like Figure 10 The diagram shown is a flowchart of the filter update process provided in this application embodiment. The current ordered string table layer is determined to be layer Li; the ordered string table group (merged group) of layer Li is selected and stored in memory; the ordered string table groups in memory are merged and divided (segmented) according to the ordered string table groups of layer Li; a new ordered string table and table filter are generated and written to the corresponding ordered string table group of layer Li+1; the metadata and two-level filters in the NVM are updated in-situ (i.e., the metadata and two-level filters corresponding to the ordered string table group are updated in the NVM according to the new ordered string table group).

[0170] Specifically, the SSTables from the Li layer that are compressed to the Li+1 layer can be read into memory for merging and splitting. The group with the most files in the Li layer of the SSD and the earliest overall file time (i.e., the earliest average file generation time) can be selected. All SSTables in this group (i.e., the ordered string table group) are read into memory, merged, and sorted. The merged SSTables are then split according to the group division of the next layer. After splitting, the key-value ranges to be written to the target layer will not overlap. If there is no overlap with multiple groups, no splitting is performed. After generating a new SSTable and its corresponding filter, the corresponding group in the next layer is selected for writing.

[0171] Specifically, metadata and filter information can be updated. Since the metadata and filters of SSTables and SSTable groups are all stored on NVM, after the merge operation is completed, only the metadata and two-level filters need to be updated in place synchronously, eliminating the need to use copy-on-write technology due to multiple versions.

[0172] like Figure 11 The diagram shown illustrates a filter update provided in an embodiment of this application. The dashed lines represent selective storage.

[0173] First, the conversion between the in-memory Memtable and Immutable Memtable is performed, followed by a compression process from L0 to L1. Assume the files on the SSD are organized into two levels. Level 0 contains two groups, each containing three SSTables. The key value ranges for the two groups are 1–30 and 31–60, respectively. The first two SSTables of the first group in L0 are generated earlier than the first two SSTables of the second group in L0, and the last SSTable of the first group in L0 is generated later than the last SSTable of the second group in L1. Level 1 contains one group with a key value range of 1–60, and contains one SSTable with a key value range of 10–55. The filter update method specifically includes:

[0174] (1) Since the number of SSTable files in the two groups of L0 layer is the same, and the two SSTables in group 1 are earlier than those in group 2, group 1 with the earliest overall file time sequence is selected for compression, and all SSTables in group 1 are read into memory for merging.

[0175] (2) Merge and sort all SSTables read into memory to generate new SSTable files, filters and metadata.

[0176] (3) Perform a split operation on the new SSTable based on the group key value range of the L1 layer. Since there is a group in the L1 layer with a key value range of 1 to 60, and the key value range of the new SSTable file is 1 to 28, which is within the group key value range of the ordered string table group in the L1 layer, the new SSTable does not need to be split and can be directly written to the ordered string table group in the L1 layer.

[0177] (4) Synchronously update the two-level filters and metadata information in NVM.

[0178] This application provides a non-volatile storage media management method for a log merge tree key-value storage system based on a hybrid storage architecture, applicable to electronic devices. Specifically, the non-volatile storage media management method includes:

[0179] 301, Receives a request to create a table filter or group filter.

[0180] 302, converts the creation request into a space allocation request for non-volatile storage media.

[0181] 303, acquire the lock on the free list of non-volatile storage media, which includes the page numbers of the free pages of the non-volatile storage media.

[0182] 304, retrieves multiple page numbers from the free list.

[0183] 305. Creates a storage management object on a non-volatile storage medium based on multiple page numbers.

[0184] 306, Returns the storage management object.

[0185] 307, releases the lock on the free list.

[0186] In one embodiment of this application, the NVM storage space is organized by page. The header page stores overall information about the NVM page; the metadata page stores information about the NVM page; and the data page stores specific data. Data pages can be managed using a bitmap and a free list to maintain information about currently used and free pages. For the lightweight file access method encapsulated by NVM, a file system-like calling interface is provided to encapsulate the underlying NVM paging process, making using NVM as convenient as using memory, while eliminating the need to handle NVM allocation and deallocation.

[0187] In this embodiment, the allocation request uses a "short-circuit" mechanism to complete the allocation directly through the underlying linked list, ensuring that the NVM has completed the response before the requesting thread uses the requested space.

[0188] The non-volatile storage media management method in this embodiment specifically includes a mutually exclusive application method.

[0189] like Figure 12The diagram shows a flowchart of a mutually exclusive allocation method provided in this application embodiment. Specifically, the mutually exclusive allocation method may include: receiving an allocation request (storage request); performing mutex locking; obtaining first_free_page_num from NVMPager and storing it in tmp, where NVMPager manages and describes page information on NVM, tmp is an intermediate variable, and first_free_page_num is the page number of the first free page in the free list. Then, initialize the number of free pages scanned to cnt = 0, and determine whether tmp ! = 0 && cnt < the expected number of pages to be allocated is true. If tmp ! = 0 && cnt < the expected number of pages to be allocated is true, tmp is set to next_page_num (the page number of the next page) of the page corresponding to tmp, cnt is incremented by 1, and the process returns to the step of determining whether tmp ! = 0 && cnt < the expected number of pages to be allocated is true. If tmp ! = 0 && cnt < the expected number of pages to be allocated is false, determine whether cnt is equal to the expected number of pages to be allocated. If cnt equals the expected number of pages to be allocated, modify first_free_page_num in NVMPager to the next_page_num of the page corresponding to tmp, and set the next_page_num of the tmp page to 0. Then, the mutex is unlocked, and the allocation succeeds. If cnt does not equal the expected number of pages to be allocated, the mutex is unlocked, and the allocation fails.

[0190] The application process will be explained further below.

[0191] a. Create new filters in NVM at the end of the merging (compression) process.

[0192] The merge operation creates a new SSTable in the next ordered string table after the current ordered string table, and correspondingly creates a new SSTable filter, which requires access to NVM.

[0193] b. The NVM lightweight space management layer transforms requests to create group filters into storage requests for NVM spaces.

[0194] The NVM Lightweight Space Management System includes an NVM Lightweight Space Management Layer. This layer converts requests for creating group filters into requests for NVM pages. It abstracts the set of pages containing filters that require access as files (a file is a logically contiguous block of space, analogous to a "file" in a general file system; in this embodiment, files involving non-volatile storage media management methods or file services related to non-volatile storage media are logically contiguous blocks of space). When a file is opened or created, an NVMFile handle is constructed in memory. The NVM space can be accessed and used in memory through the NVMFile structure. The core members of this handle are the page number of the page occupied by the group filter on the NVM and an offset representing the logical offset value for file handle read / write operations, analogous to setting the file read / write position in the lseek method of file operations. NVMFile calculates the actual address to be read or written on NVM (the specific location of the specific page) based on the logical offset value, and then uses the starting address of the NVM mapped to the user's logical space via mmap as the base address to directly and accurately access the corresponding content on NVM in the user's logical space.

[0195] Filter construction and retrieval can be performed on NVM using NVMFile, shielding the underlying NVM paging process. The creation of group filters is transformed into the creation of NVMFiles, and then further transformed into scanning of NVM free pages and requesting new pages.

[0196] c. Acquire the lock of the NVM free list, extract a certain number of pages from the free page list, and use these page numbers to create an NVMemHandler object on the NVM.

[0197] The NVM free list is itself a lock-free linked list, but the use of mutex locks prevents parallel allocation-allocation and allocation-deallocation from occurring, while allowing parallel reclamation-deallocation. This ensures the correctness of concurrent allocation and reclamation. Parallel allocation-allocation is disallowed because each allocation operation needs to retrieve the required number of pages from the free list at once, but implementing a lock-free concurrent linked list that deletes multiple elements at once is difficult.

[0198] NVMemHandler is used to describe logically contiguous NVM pages and to manage and access NVM pages referenced by an NVM file.

[0199] d. Release the lock on the free list and return the created NVMMemHandler.

[0200] The entire NVM space allocation operation is in a locking phase. After acquiring the NVM space and creating the NVMMemhandler, the lock on the NVM free page list is released.

[0201] This application provides another method for managing non-volatile storage media in a log merge tree key-value storage system based on a hybrid storage architecture, applicable to electronic devices. Specifically, the non-volatile storage media management method includes:

[0202] 401, in response to data failure in a table filter or group filter, triggers a request to reclaim space stored on non-volatile storage media.

[0203] 402, add the recycling request to the recycling queue and notify the scheduling thread of the recycling request.

[0204] 403, Multiple recycling threads are created by the scheduling thread based on recycling requests in the recycling queue, so that invalid pages can be recycled through multiple recycling threads.

[0205] 404. Update the free list based on the failed pages. The free list includes the page numbers of free pages on non-volatile storage media.

[0206] 405, waiting for the recycling thread to finish.

[0207] In one embodiment of this application, updating the free list based on expired pages includes: obtaining the header page number of the free list; assigning the header page number to the next page number of the expired page; and resetting the page number of the expired page to the header page number. Specifically, the first_free_page_num (header page number, i.e., the page number of the first NVMPageDescriptor (node) in the NVMPager (free list)) can be atomically loaded; the next_page_num (next page number) of the NVMPageDescriptor (expired page) to be reclaimed can be assigned the first_free_page_num; if the first_free_page_num is determined by the atomic operation CAS, the page number of the NVMPageDescriptor is reset to the first_free_page_num.

[0208] The non-volatile storage media management method in this embodiment specifically includes a concurrent recycling method.

[0209] For recycling requests, after the requesting thread submits the sequence of page numbers to be released, it does not need to ensure that the NVM space management returns the specified pages. The requester can continue to execute other processing logic without accessing the address space of the released pages.

[0210] like Figure 13 The diagram shows a flowchart of a concurrent garbage collection method provided in this application embodiment. Specifically, the concurrent garbage collection method may include: receiving a garbage collection request; obtaining a request queue `lock_guard`; inserting the garbage collection request into the request queue; notifying the NVM internal scheduling thread using `notify_one`; the NVMPager internal scheduling thread obtaining the request queue `lock_guard`; and the NVMPager internal scheduling thread determining whether the request queue is empty. If the request queue is empty, it waits to be woken up using the `cv.wait` method. The NVMPager internal scheduling thread wakes up in response to the `notify_one` notification. If the request queue is not empty, the NVMPager internal scheduling thread pops some garbage collection requests from the current queue. The NVMPager internal scheduling thread automatically unlocks `lock_guard` or unlocks `lock_guard` in response to an unlock request. Then, the mutex is locked; the NVMPager internal scheduling thread creates multiple threads to execute garbage collection thread tasks; the NVMPager internal scheduling thread joins multiple threads to complete the garbage collection thread tasks; the mutex is unlocked, and the process of determining whether the request queue is empty is executed. The garbage collection execution thread atomically loads and obtains `first_free_page_num` from NVMPager. The garbage collection execution thread sets the `next_page_num` in the `NVMPageDescriptor` to `first_free_page_num`. The thread then performs an atomic CAS operation to check if the current free list head is still `first_free_page_num`. If it is, the free list head is set to the page number to be collected. The thread then checks if the CAS operation updated the free list head. If yes, the process ends; otherwise, it performs an atomic load operation to retrieve `first_free_page_num` from the `NVMPager`.

[0211] The recycling method will be explained further below.

[0212] e. The merging (compression) process causes metadata such as filters to become invalid, triggering a request to recycle NVM files.

[0213] When an SSTable is compressed and becomes invalid, the page number where the NVMemHandler is located is obtained. The offset value is calculated using the page number. The NVMemHandler on the NVM can be accessed by using the NVM base address based on the offset value, which facilitates the safe reclamation of NVM pages.

[0214] f. Add the recycling request to the recycling queue and notify the internal scheduling thread.

[0215] All operations on the reclamation queue require acquiring a lock. Among multiple reclamation requests, the one releasing the most space is prioritized to reduce the likelihood of allocation requests being unsatisfactory when the number of NVM-allocable pages is limited.

[0216] g. The internal scheduling thread pops up multiple recycling requests, creates multiple worker thread recycling pages, and updates the idle list.

[0217] When a recycling request arrives, NVMPager pops the requests from the waiting queue. If the number of recycling requests is less than the preset frequency (generally), NVMPager's internal scheduling thread will directly create several threads to concurrently execute the recycling requests in the current waiting queue, thus performing the recycling operation concurrently. If the number of recycling requests exceeds the preset number of threads, NVMPager will merge the popped recycling requests into a single recycling operation, directly returning these recycled pages in one go within NVMPager's internal scheduling thread.

[0218] g1. Get the first_free_page_num of the current entire NVM space.

[0219] g2. Set the next_page_num field of the page descriptor of the NVM page to be reclaimed to first_free_page_num.

[0220] g3. Set the current free list head to first_free_page_num using a CAS atomic operation.

[0221] CAS (Compare And Set) is a special atomic operation, typically called as CAS(V, E, N), where V is the address to be updated, and the operation checks if the data in V is the same as that in E. If they are the same, V is set to N. Using CAS ensures that while one thread is successfully inserting data into the linked list, no other thread has successfully performed the same operation.

[0222] h. Wait for all recycling threads to finish.

[0223] like Figure 14 The diagram shown is a data structure logic diagram of the non-volatile storage medium management method provided in the embodiments of this application. Figure 14This describes the logical data structure relationships of the NVM lightweight space management layer. The NVM lightweight space management layer is a component of the NVM lightweight space management system. On NVM, it manages and describes page information through NVMPager and NVMPagerDescriptor, and provides specific file services through NVMmemHandler. In memory, it accesses and uses NVM space through the NVMFile structure.

[0224] like Figure 15 The diagram illustrates the filter creation process provided in this embodiment. The call flow of the NVM lightweight space management system is described by illustrating the process of creating a group filter at the end of a compression operation. At the end of a compression process, while writing a new SSTable file to the SSD, a filter is created in NVM through the NVM lightweight space management system.

[0225] The NVM lightweight space management layer transforms the creation of group filters into the allocation of NVM space. NVMPager schedules space allocation requests concurrently, retrieving a certain number of pages from the list of free pages and creating an NVMemHandler object on the NVM using their page numbers. The created NVMemHandler object is then passed back to the KV system (RocksDB). An NVMFile object is created for later use, and the NVMemHandler is persisted to an SSD to prevent data loss in case of power failure.

[0226] like Figure 16 The diagram shows a computer device provided in an embodiment of this application. The computer device 160 includes a memory 1601, a processor 1602, and computer-readable instructions stored in the memory 1601 and executable on the processor 1602, such as a log merge tree key-value storage system optimization program based on a hybrid storage architecture. When the processor 1602 executes the computer-readable instructions, it implements the steps described in the embodiment of the log merge tree key-value storage system optimization method based on a hybrid storage architecture.

[0227] Those skilled in the art will understand that the illustration Figure 16 This is merely an example of computer device 160 and does not constitute a limitation on computer device 160. It may include more or fewer components than shown, or combine certain components, or different components. For example, computer device 160 may also include input / output devices, network access devices, buses, etc.

[0228] The processor 1602 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or processor 1602 may be any conventional processor. Processor 1602 is the control center of computer device 160, connecting all parts of computer device 160 through various interfaces and lines.

[0229] The memory 1601 can be used to store computer-readable instructions. The processor 1602 implements various functions of the computer device 160 by running or executing the computer-readable instructions or modules stored in the memory 1601 and by calling the data stored in the memory 1601. The memory 1601 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device 160, etc. In addition, the memory 1601 may include a hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, read-only memory (ROM), random access memory (RAM), or other non-volatile / volatile storage devices.

[0230] If the modules integrated into computer device 160 are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium, and when executed by a processor, they can implement the steps of the various method embodiments described above. The computer-readable instructions include computer-readable instruction code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer-readable instruction code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), etc.

[0231] This embodiment also provides a computer storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the log merge tree key-value storage system optimization method based on a hybrid storage architecture as described in the above embodiment.

[0232] This embodiment also provides a computer program product that, when run on an electronic device, causes the electronic device to perform the aforementioned related steps to implement the log merge tree key-value storage system optimization method based on a hybrid storage architecture as described in the above embodiment.

[0233] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component, or module. The apparatus may include a connected processor and a memory. The memory is used to store computer execution instructions. When the apparatus is running, the processor can execute the computer execution instructions stored in the memory to cause the chip to execute the log merge tree key-value storage system optimization method based on hybrid storage architecture in the above-described method embodiments.

[0234] In this embodiment, the electronic device, computer storage medium, computer program product or chip are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding method provided above, and will not be repeated here.

[0235] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0236] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0237] The unit described as a separate component may or may not be physically separate. The component shown as a unit can be one physical unit or multiple physical units, that is, it can be located in one place or distributed in multiple different places. Some or all of the units can be selected to achieve the purpose of the solution in this embodiment according to actual needs.

[0238] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0239] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0240] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A log merge tree key-value storage system, based on a hybrid storage architecture, applied in electronic devices, characterized in that: The hybrid storage architecture includes a host, a persistent storage medium, and a non-volatile storage medium. The host is connected to the non-volatile storage medium and the persistent storage medium. The persistent storage medium stores multiple ordered string tables, each ordered string table includes multiple ordered string table groups, and each ordered string table group includes multiple ordered string tables. The log merge tree key-value storage system adopts a group compression mechanism and includes two levels of filters corresponding to ordered string table groups. The two levels of filters include a group filter corresponding to the ordered string table group and a table filter corresponding to the ordered string table.

2. The log merge tree key-value storage system as described in claim 1, characterized in that, The two-stage filters are stored in the non-volatile storage medium.

3. The log merge tree key-value storage system as described in claim 1, characterized in that, The group filters include Bloom filters or Buzz filters, and the surface filters include Bloom filters, Buzz filters, or Buzz filters.

4. The log merge tree key-value storage system as described in claim 1, characterized in that, The hybrid storage architecture also includes volatile storage media or memory.

5. A data retrieval method, based on the log merge tree key-value storage system as described in any one of claims 1 to 4, the data retrieval method comprising: Get the target key of the data to be searched; The target key is located in the persistent storage medium based on the two-level filter.

6. The data retrieval method as described in claim 5, characterized in that, The step of searching for the target key from the persistent storage medium based on the two-level filter includes: Locate the target key in the memory of the electronic device; If the target key is not found in the memory of the electronic device, the target key is searched for in the persistent storage medium according to the two-level filter.

7. The data retrieval method as described in claim 5, characterized in that, After obtaining the target key of the data to be searched, the data search method further includes: Locate the target key in the memory of the electronic device; If the target key is not found in the memory of the electronic device, the target key is searched for in the persistent storage medium according to the two-level filter.

8. The data retrieval method as described in claim 5, characterized in that, The step of searching for the target key from the persistent storage medium based on the two-level filter includes: According to the layer order, for any current ordered string table among the multiple ordered string table layers, based on the group key range of the multiple ordered string table groups in the current ordered string table layer, it is determined whether there is a target ordered string table group storing the target key in the current ordered string table layer; If a target ordered string table group storing the target key exists in the current ordered string table group, the target key is searched from the target ordered string table group according to the two-level filter.

9. The data retrieval method as described in claim 8, characterized in that, If the target ordered string table group storing the target key does not exist in the current ordered string table layer, the target key is searched for in the next ordered string table layer of the current ordered string table layer.

10. The data retrieval method as described in claim 8, characterized in that, The step of searching for the target key from the target ordered string table group according to the two-level filter includes: Based on the group filter in the two-level filter, determine whether the target ordered string table group has the possibility of including the target key; If the target ordered string table group is determined to have the possibility of including the target key according to the group filter in the two-level filter, it is determined whether there is a target ordered string table storing the target key in the target ordered string table group according to the table key range of multiple ordered string tables in the current ordered string table group; If it is determined that there exists a target ordered string table storing the target key in the target ordered string table group based on the table key range of multiple ordered string tables in the current ordered string table group, then the table filter in the two-level filter determines whether the target ordered string table has the possibility of including the target key; If the target ordered string table is determined to have the possibility of including the target key according to the table filter in the two-level filter, the data to be searched is found in the target ordered string table according to the target key.

11. The data retrieval method as described in claim 10, characterized in that, If the target ordered string table group is determined to have no possibility of including the target key according to the group filter in the two-level filter, or if the target ordered string table is determined to have no possibility of including the target key according to the table filter in the two-level filter, then the target key is searched in the next ordered string table layer of the current ordered string table layer.

12. The data retrieval method as described in claim 5, characterized in that, The step of searching for the target key from the persistent storage medium based on the two-level filter includes: The target key is searched from the plurality of ordered string tables in the persistent storage medium according to the layer order, using the two-level filter.

13. The data retrieval method as described in claim 12, characterized in that, For any one of the plurality of ordered string layers, searching for the target key in the current ordered string layer according to the two-level filter includes: determining whether the target key exists in the current ordered string layer according to the two-level filter; If the target key is determined to exist in the current ordered string surface based on the two-level filter, the data to be searched is found based on the target key; If the two-level filter determines that the target key does not exist in the current ordered string table, the target key is searched for in the next ordered string table of the current ordered string table.

14. The data retrieval method as described in claim 13, characterized in that, The step of determining whether the target key exists in the current ordered string surface layer based on the two-level filter includes: Based on the group filter in the two-level filter, it is determined whether there is a target ordered string table group in the current ordered string table layer, and the target ordered string table group has the possibility of including the target key; If the target ordered string group does not exist in the current ordered string table layer, it is determined that the target key does not exist in the current ordered string table layer; If the target ordered string table group exists in the current ordered string table layer, the table filter in the two-level filter determines whether the target ordered string table exists in the target ordered string table group, and the target ordered string table has the possibility of including the target key; if the target ordered string table group does not contain the target ordered string table, it is determined that the target key does not exist in the current ordered string table layer. If the target ordered string table exists in the target ordered string table group, determine whether the target key exists in the target ordered string table; If the target key exists in the target ordered string table, it is determined that the target key exists in the current ordered string table.

15. The data retrieval method as described in claim 14, characterized in that, The step of determining whether a target ordered string group exists in the current ordered string table layer based on the group filter in the two-level filter, wherein the target ordered string group has the possibility of including the target key, includes: Obtain the group key range of multiple ordered string groups in the current ordered string table layer; From the multiple ordered string groups in the current ordered string table layer, determine one or more ordered string groups whose group key range includes the target key, and obtain the determined ordered string group; The group filter of the determined ordered string group determines whether the determined ordered string group has the possibility of including the target key; If the determined ordered string table group has the possibility of including the target key, it is determined that there exists an ordered string table group in the current ordered string table layer that has the possibility of including the target key, and the determined ordered string table group is taken as the target ordered string table group; If the determined ordered string table group does not have the possibility of including the target key, it is determined that there is no target ordered string table group in the current ordered string table layer that has the possibility of including the target key.

16. The data retrieval method as described in claim 14, characterized in that, The step of determining whether a target ordered string group exists in the current ordered string table layer based on the group filter in the two-level filter, wherein the target ordered string group has the possibility of including the target key, includes: One by one, obtain the ordered string groups in the current ordered string table layer, and use the group filter of the ordered string group to determine whether the ordered string group has the possibility of including the target key; If any ordered string group in the current ordered string table layer has the possibility of including the target key, it is determined that there exists a target ordered string group in the current ordered string table layer that has the possibility of including the target key; If none of the ordered string groups in the current ordered string table layer have the possibility of including the target key, it is determined that there is no target ordered string group in the current ordered string table layer that has the possibility of including the target key.

17. The data retrieval method as described in claim 14, characterized in that, The step of determining whether a target ordered string table exists in the target ordered string table group based on the table filter in the two-level filter, wherein the target ordered string table has the possibility of including the target key, includes: Obtain the table key range of multiple ordered string tables in the target ordered string table group; From the multiple ordered string tables in the target ordered string table group, determine one or more ordered string tables whose table key range includes the target key, to obtain the determined ordered string table; The table filter of the determined ordered string table is used to determine whether the determined ordered string table has the possibility of including the target key; If the determined ordered string table has the possibility of including the target key, it is determined that there exists an ordered string table in the target ordered string table group that has the possibility of the target key, and the determined ordered string table is taken as the target ordered string table; If the determined ordered string table does not have the possibility of including the target key, it is determined that there is no target ordered string table in the target ordered string table group that has the possibility of including the target key.

18. The data retrieval method as described in claim 14, characterized in that, The step of determining whether a target ordered string table exists in the target ordered string table group based on the table filter in the two-level filter, wherein the target ordered string table group has the possibility of including the target key, includes: One by one, obtain the ordered string tables in the target ordered string table group, and use the table filter of the ordered string table to determine whether the ordered string table has the possibility of including the target key; If any ordered string table in the target ordered string table group has the possibility of including the target key, it is determined that there exists a target ordered string table in the target ordered string table group that has the possibility of including the target key. If none of the ordered string tables in the target ordered string table group have the possibility of including the target key, it is determined that there is no target ordered string table in the target ordered string table group that has the possibility of including the target key.

19. A filter update method, based on a log merge tree key-value storage system as described in any one of claims 1 to 4, characterized in that, The filter update method includes: Any one of the plurality of ordered string layers is determined as the current ordered string layer, wherein the plurality of ordered string layers are stored in a preset order; Obtain the number of ordered strings in the current ordered string table and the preset upper limit of the number of tables; If the number of ordered string tables in the current ordered string table layer is greater than or equal to the preset table upper limit, select an ordered string table group from the current ordered string table layer. Generate a list of ordered strings to be merged based on the selected group of ordered strings. The ordered string table to be merged is merged into the next ordered string table of the current ordered string table to obtain the merged ordered string table. The two-level filter of the next ordered string table of the current ordered string table is updated based on the merged ordered string table.

20. The filter update method as described in claim 19, characterized in that, The step of selecting an ordered string group from the current ordered string table includes: Select an ordered string table group from the current ordered string table layer based on the number of ordered string tables and the generation time.

21. The filter update method as described in claim 19, characterized in that, The step of generating the ordered string table to be merged based on the selected ordered string table group includes: The ordered string tables in the selected ordered string table group are merged and sorted to obtain the first ordered string table; The first ordered string table is divided into at least one second ordered string table based on the key range of the next ordered string table of the current ordered string table, resulting in an ordered string table to be merged that includes the at least one second ordered string table.

22. The filter update method as described in claim 21, characterized in that, The step of dividing the first ordered string table into at least one second ordered string table based on the key range of the next ordered string table of the current ordered string table includes: If the table key range of the first ordered string table is included in any group key range of the next ordered string table of the current ordered string table, the first ordered string table will not be divided. If the table key range of the first ordered string table is included in any number of group key ranges of the next ordered string table of the current ordered string table, the first ordered string table is divided according to the number of group key ranges. If the key range of the first ordered string table is not included in any key range of the next ordered string table layer of the current ordered string table layer, the first ordered string table is divided evenly.

23. The filter update method as described in claim 19, characterized in that, The step of merging the ordered string table to be merged into the next ordered string table layer of the current ordered string table layer includes: In the next ordered string table layer of the current ordered string table layer, each second ordered string table in the ordered string table to be merged is merged into an ordered string table group whose group key range includes the table key range of the second ordered string table through multi-threading.

24. The filter update method as described in claim 19, characterized in that, The two-level filter for updating the next ordered string table of the current ordered string table based on the merged ordered string table includes: Based on the ordered string table group generated in the merged ordered string table layer, generate corresponding group metadata and create corresponding group filters on the non-volatile storage medium. Based on the modified ordered string table group in the merged ordered string table layer, the corresponding group filter is updated on the non-volatile storage medium.

25. A method for managing non-volatile storage media, based on the log merge tree key-value storage system as described in any one of claims 1 to 4, characterized in that, The non-volatile storage medium management method includes: Receive a request to create the table filter or the group filter; The creation request is converted into a space allocation request for the non-volatile storage medium; Acquire the lock of the free list of the non-volatile storage medium, the free list including the page numbers of the free pages of the non-volatile storage medium; Extract multiple page numbers from the free list; A storage management object is created on the non-volatile storage medium based on the multiple page numbers; Return the storage management object; Release the lock on the free list.

26. A method for managing non-volatile storage media, based on a log merge tree key-value storage system as described in any one of claims 1 to 4, characterized in that, The non-volatile storage medium management method includes: In response to the data failure of the table filter or the group filter, a request to reclaim the space stored in the non-volatile storage medium is triggered; Add the recycling request to the recycling queue and notify the scheduling thread of the recycling request; The scheduling thread creates multiple recycling threads based on the recycling requests in the recycling queue, so as to recycle invalid pages through the multiple recycling threads; The free list is updated based on the invalid pages, and the free list includes the page numbers of the free pages of the non-volatile storage medium. Waiting for the recycling thread to finish.

27. The non-volatile storage medium management method as described in claim 26, characterized in that, The step of updating the free list based on invalid pages includes: Obtain the header page number of the free linked list; Assign the header page number to the next page number of the invalid page; The page number of the invalid page is reassigned as the header page number.

28. A computer device, characterized in that, The system includes a processor and a memory; the memory is used to store instructions; the processor is used to invoke the instructions in the memory to cause the computer device to run the log merge tree key-value storage system based on a hybrid storage architecture as described in any one of claims 1 to 4, or to cause the computer device to execute the log merge tree key-value storage system optimization method based on a hybrid storage architecture as described in any one of claims 5 to 27.

29. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, which, when executed by a processor, implements the log merge tree key-value storage system based on a hybrid storage architecture as described in any one of claims 1 to 4, or, when executed by a processor, implements the log merge tree key-value storage system optimization method based on a hybrid storage architecture as described in any one of claims 5 to 27.

Images