LSM-Tree optimization method, related device and storage medium
By inserting duplicate data into the B+ tree when the L0 layer of LSM-Tree is merged with the L1 layer and merging it, the problems of read amplification, write amplification and space amplification of LSM-Tree are solved, and query efficiency and storage density are improved.
Patent Information
- Application Number
- CN202411644455.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-18
- Publication Date
- 2025-01-14
AI Technical Summary
LSM-Tree has problems with read, write and space amplification when querying and writing, resulting in increased latency and additional disk IO operations.
When the L0 layer data volume of LSM-Tree reaches the threshold, determine whether it is merged with L1 layer for the first time, and insert the duplicate key value data into the B+ tree. Query and eliminate data through the Bloom filter of the B+ tree to reduce the duplicate data volume , merge data to optimize query and storage.
It effectively reduces the problems of read, write and space amplification of LSM-Tree, improves query efficiency and storage density, and reduces the number of disk IO operations.
Smart Images
Figure CN119311227A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to an LSM-Tree optimization method, related devices and storage media. Background Art
[0002] LSM-Tree (Log-Structured Merge Tree) is a data structure for storing and retrieving large-scale data sets. It is mainly used to solve the problem of slow disk writing speed and has good performance in reading and writing.
[0003] However, LSM trees also have some disadvantages. LSM trees mainly have three problems: read amplification, write amplification, and space amplification. Read amplification means that when querying data, you need to retrieve data structures at multiple levels, which results in multiple disk IO operations. Since data is stored in multiple levels, you need to start the query from the memory table. If it is not found, you need to search down layer by layer until it is found or all levels are traversed. This will increase the query latency and increase the number of disk IO operations. Write amplification means that when writing data, due to the merge operation of the LSM tree, the amount of data written will be greater than the actual amount of data written. When writing data, the data is first written to the memory table. When the memory table reaches a certain size, it is written to the disk table and a merge operation is performed. The merge operation merges duplicate data and sometimes triggers more merge processes. This will increase the amount of data written and increase the number of disk IO operations. Space amplification, the actual disk space occupied by the data is more than the actual size of the data. Summary of the invention
[0004] In view of this, the present application provides an LSM-Tree optimization method, related devices and storage medium to effectively reduce the problems of LSM-Tree read amplification, write amplification and space amplification.
[0005] The first aspect of the present application provides an LSM-Tree optimization method, comprising:
[0006] When the amount of data in the L0 layer of the LSM-Tree reaches the data amount threshold corresponding to the L0 layer, it is determined whether the L0 layer is merged with the L1 layer for the first time;
[0007] If it is determined that the L0 layer is merged with the L1 layer for the first time, the target key value data is inserted into the B+ tree; wherein the target key value data is the data of repeated key values in the L0 layer and the L1 layer, and the B+ tree is a tree data structure for storing the target key value data;
[0008] Delete the target key value data in the L0 layer and the L1 layer to obtain the target L0 layer data and the target L1 layer data;
[0009] Merge the target L0 layer data into the target L1 layer data.
[0010] Optionally, the LSM-Tree optimization method further includes:
[0011] If it is determined that the L0 layer is not merged with the L1 layer for the first time, query each key value in the L0 layer through the Bloom filter of the B+ tree to obtain a query result;
[0012] If the query result indicates that the Bloom filter of the B+ tree determines that the key value exists, the key value is updated to the B+ tree, and the time stamp of the key value is updated;
[0013] If the query result indicates that the Bloom filter of the B+ tree determines that the key value does not exist, then select the next key value, execute the step of querying the key value through the Bloom filter of the B+ tree to obtain the query result;
[0014] When all key values in the L0 layer have been processed, the target key value data in the L0 layer is inserted into the B+ tree and a time stamp is generated;
[0015] Delete the target key value data in the L0 layer and the L1 layer to obtain the target L0 layer data and the target L1 layer data;
[0016] Merge the target L0 layer data into the target L1 layer data.
[0017] Optionally, the LSM-Tree optimization method further includes:
[0018] Receiving a data query request; wherein the data query request includes the target data of this query;
[0019] Query the target data at the L0 layer;
[0020] If the target data is obtained by querying at the L0 layer, the first query result is returned;
[0021] If the target data cannot be found in the L0 layer, the target data is searched in the B+ tree;
[0022] If the target data is found in the B+ tree, the second query result is returned;
[0023] If the target data cannot be found in the B+ tree, query the target data in the L1 layer;
[0024] If the target data is obtained by querying in the L1 layer, a third query result is returned;
[0025] If the target data cannot be obtained in the L1 layer, the next layer of the next L1 layer is selected to query the target data, until the target data is obtained or all layers are queried to generate a final query result.
[0026] Optionally, the LSM-Tree optimization method further includes:
[0027] When the data stored in the B+ tree reaches a storage capacity threshold, the data is eliminated using the least recently used method according to the time stamp of the target key value data in the B+ tree to obtain the data marked for elimination;
[0028] The marked eliminated data is merged with the key-value storage structure of the L1 layer to obtain the merged key-value storage structure of the L1 layer.
[0029] The second aspect of the present application provides an LSM-Tree optimization device, including:
[0030] A judgment unit, configured to judge whether the L0 layer is merged with the L1 layer for the first time after monitoring that the amount of data in the L0 layer of the LSM-Tree reaches a data amount threshold corresponding to the L0 layer;
[0031] a first inserting unit, configured to insert the target key value data into the B+ tree if the determining unit determines that the L0 layer is merged with the L1 layer for the first time; wherein the target key value data is data of repeated key values in the L0 layer and the L1 layer, and the B+ tree is a tree data structure for storing the target key value data;
[0032] A first deleting unit is used to delete the target key value data in the L0 layer and the L1 layer to obtain the target L0 layer data and the target L1 layer data;
[0033] The first merging unit is used to merge the data of the target L0 layer into the data of the target L1 layer.
[0034] Optionally, the LSM-Tree optimization device further includes:
[0035] A first query unit is configured to query each key value in the L0 layer through a Bloom filter of the B+ tree to obtain a query result if the judgment unit judges that the L0 layer is not merged with the L1 layer for the first time;
[0036] an updating unit, configured to update the key value to the B+ tree and update a time stamp of the key value if the query result indicates that the Bloom filter of the B+ tree determines that the key value exists;
[0037] an activation unit, configured to select a next key value if the query result indicates that the Bloom filter of the B+ tree determines that the key value does not exist, and activate the first query unit to execute the step of querying the key value through the Bloom filter of the B+ tree to obtain a query result;
[0038] A second insertion unit is used to insert the target key value data of the L0 layer into the B+ tree and generate a time tag after all key values in the L0 layer have been processed;
[0039] A second deleting unit is used to delete the target key value data in the L0 layer and the L1 layer to obtain the target L0 layer data and the target L1 layer data;
[0040] The second merging unit is used to merge the data of the target L0 layer into the data of the target L1 layer.
[0041] Optionally, the LSM-Tree optimization device further includes:
[0042] A receiving unit, configured to receive a data query request; wherein the data query request includes the target data of this query;
[0043] A second query unit, configured to query the target data at the L0 layer;
[0044] A returning unit, configured to return a first query result if the target data is obtained by querying at the L0 layer;
[0045] The second query unit is further configured to search for the target data in the B+ tree if the target data cannot be obtained by querying in the L0 layer;
[0046] The returning unit is further configured to return a second query result if the target data is found in the B+ tree;
[0047] The second query unit is further configured to query the target data in the L1 layer if the target data cannot be found in the B+ tree;
[0048] The returning unit is further configured to return a third query result if the target data is obtained by querying in the L1 layer;
[0049] The second query unit is further configured to select a layer below the next L1 layer to query the target data if the target data cannot be queried in the L1 layer, until the target data is queried or all layers complete the query to generate a final query result.
[0050] Optionally, the LSM-Tree optimization device further includes:
[0051] An elimination unit is used to eliminate the data stored in the B+ tree by using the least recently used method according to the time mark of the target key value data in the B+ tree when the data reaches the storage capacity threshold, so as to obtain the data marked for elimination;
[0052] The third merging unit is used to merge the marked eliminated data with the key-value storage structure of the L1 layer to obtain a merged key-value storage structure of the L1 layer.
[0053] A third aspect of the present application provides an electronic device, including:
[0054] one or more processors;
[0055] a storage device having one or more programs stored thereon;
[0056] When the one or more programs are executed by the one or more processors, the one or more processors implement the LSM-Tree optimization method as described in any one of the first aspects.
[0057] A fourth aspect of the present application provides a storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the LSM-Tree optimization method as described in any one of the first aspects is implemented.
[0058] As can be seen from the above scheme, the present application provides an optimization method, related device and storage medium of LSM-Tree. When the amount of data in the L0 layer of LSM-Tree reaches the data amount threshold corresponding to the L0 layer, if it is determined that the L0 layer is merged with the L1 layer for the first time, the target key value data is inserted into the B+ tree; wherein the target key value data is the data of repeated key values in the L0 layer and the L1 layer, and the B+ tree is a tree data structure for storing the target key value data; the target key value data in the L0 layer and the L1 layer are deleted to obtain the data of the target L0 layer and the data of the target L1 layer; the data of the target L0 layer is merged into the data of the target L1 layer. The amount of data with repeated key values is effectively reduced, thereby effectively reducing the problems of read amplification, write amplification and space amplification of LSM-Tree. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0060] Figure 1 A specific flow chart of an LSM-Tree optimization method provided in an embodiment of the present application;
[0061] Figure 2 A flowchart of an LSM-Tree optimization method provided in another embodiment of the present application;
[0062] Figure 3 A flowchart of an LSM-Tree optimization method provided in another embodiment of the present application;
[0063] Figure 4 A flowchart of an LSM-Tree optimization method provided in another embodiment of the present application;
[0064] Figure 5 A schematic diagram of an LSM-Tree optimization method provided in another embodiment of the present application;
[0065] Figure 6 A schematic diagram of an LSM-Tree optimization device provided in another embodiment of the present application;
[0066] Figure 7 A schematic diagram of an electronic device for implementing an LSM-Tree optimization method provided in another embodiment of the present application. DETAILED DESCRIPTION
[0067] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0068] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0069] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.
[0070] It should be noted that the concepts such as "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0071] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0072] The present application embodiment provides an optimization method for LSM-Tree, such as Figure 1 As shown, the specific steps include:
[0073] S101. When it is monitored that the data volume in the L0 layer of the LSM-Tree reaches the data volume threshold corresponding to the L0 layer, it is determined whether the L0 layer is merged with the L1 layer for the first time.
[0074] First of all, it should be noted that LSM-Tree consists of multiple levels. The data size of each level is maintained at T times that of the previous level. Each layer consists of multiple SSTables. The SSTables also maintain an orderly relationship with each other. When the data volume of each layer reaches the threshold, it is merged with the next layer. If Ln is the L0 layer, the SSTable of the L0 layer is not merged. When the MemTable of the L0 layer reaches a certain size, it will be converted into an Immutable MemTable. If Ln is not the L0 layer, it is an SSTable.
[0075] Among them, SSTable is a persistent, ordered and immutable key-value storage structure. Its key and value are arbitrary byte arrays, and it provides the functions of searching by specified key and iterating through the key interval of specified range. MemTable is a data structure in memory, which is used to store the most recently updated data and organizes the data in order according to the key. When MemTable reaches a certain size, it will be converted into Immutable MemTable. ImmutableMemTable is an intermediate state of converting MemTable into SSTable. Write operations are handled by the new MemTable, and data update operations are not blocked during the transfer process.
[0076] Specifically, if it is determined that the L0 layer is merged with the L1 layer for the first time, step S102 is executed.
[0077] S102: Insert the target key value data into the B+ tree.
[0078] The target key value data is the data of repeated key values in the L0 layer and the L1 layer. The B+ tree is a tree data structure, which is usually used in the file system of databases and operating systems. The characteristic of the B+ tree is that it can keep the data stable and orderly, and its insertion and modification have a relatively stable logarithmic time complexity. In this application, it is used to store the target key value data and establish a Bloom filter on the B+ tree.
[0079] It is understandable that the data with duplicate key values has a higher probability of being read and written in the future. Therefore, in the specific implementation process of this application, when reading or updating the key value, in addition to the L0 layer, the key value data (target key value data) in the B+ tree is preferentially searched.
[0080] S103 , deleting the target key value data in the L0 layer and the L1 layer, and obtaining the target L0 layer data and the target L1 layer data.
[0081] S104 , merging the data of the target L0 layer into the data of the target L1 layer.
[0082] Optionally, in another embodiment of the present application, an implementation of the LSM-Tree optimization method is as follows: Figure 2 As shown, it also includes:
[0083] S201. If it is determined that the L0 layer is not merged with the L1 layer for the first time, query the key value through the Bloom filter of the B+ tree for each key value in the L0 layer to obtain a query result.
[0084] In the specific implementation process of the present application, since the MemTable of the L0 layer will be converted into an Immutable MemTable after reaching a certain size, it is possible to query the key value for each key value in the Immutable MemTable of the L0 layer through the Bloom filter of the B+ tree to obtain the query result, which is not limited here.
[0085] S202: If the query result indicates that the Bloom filter of the B+ tree determines that the key value exists, the key value is updated to the B+ tree, and the time stamp of the key value is updated.
[0086] Specifically, the key value is updated to the B+ tree, and the time stamp of the key value is updated to mark the update time of the key value data.
[0087] S203: If the query result indicates that the Bloom filter of the B+ tree determines that there is no key value, then select the next key value.
[0088] Considering that there is a certain probability of misjudgment in the Bloom filter, when the key value is not found during the key-value pair B+ tree update, the key value is directly skipped and the next key value is processed. If the Bloom filter determines that the key value does not exist, it is directly skipped and the next key value is processed.
[0089] S204: When all key values in the L0 layer have been processed, the target key value data in the L0 layer is inserted into the B+ tree and a time stamp is generated.
[0090] Specifically, the target key-value data of the L0 layer is inserted into the B+ tree and a time tag is generated to mark the insertion time of the key-value data.
[0091] S205 , deleting the target key value data in the L0 layer and the L1 layer, and obtaining the target L0 layer data and the target L1 layer data.
[0092] S206 , merging the data of the target L0 layer into the data of the target L1 layer.
[0093] Optionally, in another embodiment of the present application, an implementation of the LSM-Tree optimization method is as follows: Figure 3 As shown, it also includes:
[0094] S301: Receive a data query request.
[0095] The data query request includes the target data of this query.
[0096] S302, query target data at the L0 layer.
[0097] S303: If the target data is obtained by querying at the L0 layer, the first query result is returned.
[0098] S304: If the target data cannot be found in the L0 layer, the target data is searched in the B+ tree.
[0099] Specifically, the Bloom filter of the B+ tree determines whether the target data exists in the B+ tree. Considering that the Bloom filter has a certain probability of misjudgment, when the target data cannot be found in the B+ tree or the Bloom filter of the B+ tree determines that it does not exist, a search is performed at the L1 layer.
[0100] S305: If the target data is found in the B+ tree, the second query result is returned.
[0101] Specifically, if the key-value data is found in the B+ tree, the time stamp of the key-value data needs to be updated to mark the query time of the key-value data.
[0102] S306: If the target data cannot be found in the B+ tree, query the target data in the L1 layer.
[0103] S307: If the target data is obtained by querying in the L1 layer, the third query result is returned.
[0104] S308: If the target data cannot be queried in the L1 layer, the next layer of the next L1 layer is selected to query the target data, until the target data is queried, or all layers are queried to generate a final query result.
[0105] Optionally, in another embodiment of the present application, an implementation of the LSM-Tree optimization method is as follows: Figure 4 As shown, it also includes:
[0106] S401. When the data stored in the B+ tree reaches a storage capacity threshold, the data is eliminated using the least recently used method according to the time stamp of the target key value data in the B+ tree to obtain data marked for elimination.
[0107] Among them, the Least Recently Used (LRU) method is a commonly used page replacement algorithm that selects the page that has not been used for the longest time to be eliminated.
[0108] S402 , merging the marked eliminated data with the key-value storage structure of the L1 layer to obtain a merged key-value storage structure of the L1 layer.
[0109] Specifically, the marked obsolete data is merged with the key-value storage structure (SSTable) of the L1 layer to generate a new SSTable, and then the expired data and old SSTable in the B+ tree are cleared.
[0110] like Figure 5 As shown, the numbers in the circles corresponding to the solid arrows are the writing order, and the numbers in the circles corresponding to the dotted arrows are the reading order. Among them, WAL Log (Write Ahead Log) is a common means in database systems to ensure the atomicity and durability of data operations.
[0111] First, the writing order is explained: ① After the written data passes through the WAL Log, ② it reaches the Random Access Memory (RAM), ③ when the MemTable of the L0 layer reaches a certain size, it will be converted into ImmutableMemTable, ④ it will be merged with the B+ tree of L1.
[0112] ⑤Merge with L1.
[0113] In addition, since the B+ tree also has a certain space size, after the B+ tree reaches the threshold, the marking time of the key value data is used to eliminate it by the least recently used (LRU). The marked eliminated data is merged with the SSTable of the L1 layer to generate a new SSTable. Then the expired data and old SSTable in the B+ tree are cleared.
[0114] Reading order: ① Read the MemTable of the L0 layer first, ② Read the Immutable MemTable of the L0 layer, ③ Read the B+ tree; ④ Read the SSTable of the L1 layer. If the target data cannot be queried in the L1 layer, select the next layer of the next L1 layer to query the target data until the target data is queried or all layers have completed the query to generate the final query result.
[0115] It can be seen from the above scheme that the present application provides an optimization method for LSM-Tree. When the amount of data in the L0 layer of the LSM-Tree reaches the data amount threshold corresponding to the L0 layer, if it is determined that the L0 layer is merged with the L1 layer for the first time, the target key value data is inserted into the B+ tree; wherein the target key value data is the data of repeated key values in the L0 layer and the L1 layer, and the B+ tree is a tree data structure for storing the target key value data; the target key value data in the L0 layer and the L1 layer are deleted to obtain the data of the target L0 layer and the data of the target L1 layer; the data of the target L0 layer is merged into the data of the target L1 layer. The amount of data with repeated key values is effectively reduced, thereby effectively reducing the problems of read amplification, write amplification and space amplification of the LSM-Tree.
[0116] The present application embodiment provides an optimization device for LSM-Tree, such as Figure 6 As shown, specifically including:
[0117] The judgment unit 601 is used to judge whether the L0 layer is merged with the L1 layer for the first time after monitoring that the data volume in the L0 layer of the LSM-Tree reaches the data volume threshold corresponding to the L0 layer.
[0118] The first inserting unit 602 is configured to insert the target key value data into the B+ tree if the determining unit 601 determines that the L0 layer is merged with the L1 layer for the first time.
[0119] The target key value data is the data of repeated key values in the L0 layer and the L1 layer, and the B+ tree is a tree data structure used to store the target key value data.
[0120] The first deleting unit 603 is used to delete the target key-value data in the L0 layer and the L1 layer to obtain the target L0 layer data and the target L1 layer data.
[0121] The first merging unit 604 is configured to merge the data of the target L0 layer into the data of the target L1 layer.
[0122] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 1 As shown, no further description is given here.
[0123] Optionally, in another embodiment of the present application, an implementation of the LSM-Tree optimization device further includes:
[0124] The first query unit is used to query each key value in the L0 layer through the Bloom filter of the B+ tree to obtain a query result if the judgment unit judges that the L0 layer is not merged with the L1 layer for the first time.
[0125] The updating unit is used for updating the key value to the B+ tree and updating the time stamp of the key value if the query result shows that the Bloom filter of the B+ tree determines that the key value exists.
[0126] The activation unit is used to select the next key value if the query result shows that the Bloom filter of the B+ tree determines that there is no key value, activate the first query unit to execute the query on the key value through the Bloom filter of the B+ tree, and obtain the query result step.
[0127] The second insertion unit is used to insert the target key value data of the L0 layer into the B+ tree and generate a time tag after all key values in the L0 layer are processed.
[0128] The second deleting unit is used to delete the target key-value data in the L0 layer and the L1 layer to obtain the target L0 layer data and the target L1 layer data.
[0129] The second merging unit is used to merge the data of the target L0 layer into the data of the target L1 layer.
[0130] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 2 As shown, no further description is given here.
[0131] Optionally, in another embodiment of the present application, an implementation of the LSM-Tree optimization device further includes:
[0132] The receiving unit is used to receive a data query request.
[0133] The data query request includes the target data of this query.
[0134] The second query unit is used to query the target data at the L0 layer.
[0135] The returning unit is used to return the first query result if the target data is obtained by querying in the L0 layer.
[0136] The second query unit is further used to search for the target data in the B+ tree if the target data cannot be queried in the L0 layer.
[0137] The return unit is also used to return the second query result if the target data is found in the B+ tree.
[0138] The second query unit is further used to query the target data in the L1 layer if the target data cannot be found in the B+ tree.
[0139] The returning unit is further used to return the third query result if the target data is obtained by querying in the L1 layer.
[0140] The second query unit is also used to select the next layer of the next L1 layer to query the target data if the target data cannot be queried in the L1 layer, until the target data is queried or all layers are queried to generate a final query result.
[0141] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 3 As shown, no further description is given here.
[0142] Optionally, in another embodiment of the present application, an implementation of the LSM-Tree optimization device further includes:
[0143] The elimination unit is used to eliminate the data stored in the B+ tree by using the least recently used method according to the time mark of the target key value data in the B+ tree when the data reaches the storage capacity threshold, so as to obtain the data marked for elimination.
[0144] The third merging unit is used to merge the marked eliminated data with the key-value storage structure of the L1 layer to obtain a merged key-value storage structure of the L1 layer.
[0145] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 4 As shown, no further description is given here.
[0146] As can be seen from the above scheme, the present application provides an optimization device for LSM-Tree. When the amount of data in the L0 layer of the LSM-Tree reaches the data amount threshold corresponding to the L0 layer, if the judgment unit 601 judges that the L0 layer is merged with the L1 layer for the first time, the first insertion unit 602 inserts the target key value data into the B+ tree; wherein the target key value data is the data of repeated key values in the L0 layer and the L1 layer, and the B+ tree is a tree data structure for storing the target key value data; the first deletion unit 603 deletes the target key value data in the L0 layer and the L1 layer to obtain the data of the target L0 layer and the data of the target L1 layer; the first merging unit 604 merges the data of the target L0 layer into the data of the target L1 layer. The amount of data with repeated key values is effectively reduced, thereby effectively reducing the problems of read amplification, write amplification and space amplification of the LSM-Tree.
[0147] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0148] Another embodiment of the present application provides an electronic device, such as Figure 7 As shown, including:
[0149] One or more processors 701.
[0150] The storage device 702 stores one or more programs.
[0151] When the one or more programs are executed by the one or more processors 701, the one or more processors 701 implement the LSM-Tree optimization method as described in any one of the above embodiments.
[0152] Another embodiment of the present application provides a storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the LSM-Tree optimization method as described in any one of the above embodiments is implemented.
[0153] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0154] It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0155] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0156] Another embodiment of the present application provides a computer program product, which, when executed, is used to execute the above-mentioned LSM-Tree optimization method.
[0157] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, the above-mentioned functions defined in the method of the embodiment of the present application are executed.
[0158] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in this application is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing this application.
[0159] Although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present application. Certain features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination.
[0160] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of application involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above application concept. For example, the above features are replaced with (but not limited to) technical features with similar functions applied in the present application.
Claims
1. An optimization method for LSM-Tree, characterized in that: include: When the amount of data in the L0 layer of the LSM-Tree reaches the data amount threshold corresponding to the L0 layer, it is determined whether the L0 layer is merged with the L1 layer for the first time; If it is determined that the L0 layer is merged with the L1 layer for the first time, the target key value data is inserted into the B+ tree; wherein the target key value data is the data of repeated key values in the L0 layer and the L1 layer, and the B+ tree is a tree data structure for storing the target key value data; Delete the target key value data in the L0 layer and the L1 layer to obtain the target L0 layer data and the target L1 layer data; Merge the target L0 layer data into the target L1 layer data.
2. The LSM-Tree optimization method according to claim 1, characterized in that: Also includes: If it is determined that the L0 layer is not merged with the L1 layer for the first time, query each key value in the L0 layer through the Bloom filter of the B+ tree to obtain a query result; If the query result indicates that the Bloom filter of the B+ tree determines that the key value exists, the key value is updated to the B+ tree, and the time stamp of the key value is updated; If the query result indicates that the Bloom filter of the B+ tree determines that the key value does not exist, then select the next key value, execute the step of querying the key value through the Bloom filter of the B+ tree to obtain the query result; When all key values in the L0 layer have been processed, the target key value data in the L0 layer is inserted into the B+ tree and a time stamp is generated; Delete the target key value data in the L0 layer and the L1 layer to obtain the target L0 layer data and the target L1 layer data; Merge the target L0 layer data into the target L1 layer data.
3. The LSM-Tree optimization method according to claim 2, characterized in that: Also includes: Receiving a data query request; wherein the data query request includes the target data of this query; Query the target data at the L0 layer; If the target data is obtained by querying at the L0 layer, the first query result is returned; If the target data cannot be found in the L0 layer, the target data is searched in the B+ tree; If the target data is found in the B+ tree, the second query result is returned; If the target data cannot be found in the B+ tree, query the target data in the L1 layer; If the target data is obtained by querying in the L1 layer, a third query result is returned; If the target data cannot be obtained in the L1 layer, the next layer of the next L1 layer is selected to query the target data, until the target data is obtained or all layers are queried to generate a final query result.
4. The LSM-Tree optimization method according to claim 3, characterized in that: Also includes: When the data stored in the B+ tree reaches a storage capacity threshold, the data is eliminated using the least recently used method according to the time stamp of the target key value data in the B+ tree to obtain the data marked for elimination; The marked eliminated data is merged with the key-value storage structure of the L1 layer to obtain the merged key-value storage structure of the L1 layer.
5. An LSM-Tree optimization device, characterized in that: include: A judgment unit, configured to judge whether the L0 layer is merged with the L1 layer for the first time after monitoring that the amount of data in the L0 layer of the LSM-Tree reaches a data amount threshold corresponding to the L0 layer; a first inserting unit, configured to insert the target key value data into the B+ tree if the determining unit determines that the L0 layer is merged with the L1 layer for the first time; wherein the target key value data is data of repeated key values in the L0 layer and the L1 layer, and the B+ tree is a tree data structure for storing the target key value data; A first deleting unit is used to delete the target key value data in the L0 layer and the L1 layer to obtain the target L0 layer data and the target L1 layer data; The first merging unit is used to merge the data of the target L0 layer into the data of the target L1 layer.
6. The LSM-Tree optimization device according to claim 5, characterized in that: Also includes: A first query unit is configured to query each key value in the L0 layer through a Bloom filter of the B+ tree to obtain a query result if the judgment unit judges that the L0 layer is not merged with the L1 layer for the first time; an updating unit, configured to update the key value to the B+ tree and update a time stamp of the key value if the query result indicates that the Bloom filter of the B+ tree determines that the key value exists; an activation unit, configured to select a next key value if the query result indicates that the Bloom filter of the B+ tree determines that the key value does not exist, and activate the first query unit to execute the step of querying the key value through the Bloom filter of the B+ tree to obtain a query result; A second insertion unit is used to insert the target key value data of the L0 layer into the B+ tree and generate a time tag after all key values in the L0 layer have been processed; A second deleting unit is used to delete the target key value data in the L0 layer and the L1 layer to obtain the target L0 layer data and the target L1 layer data; The second merging unit is used to merge the data of the target L0 layer into the data of the target L1 layer.
7. The LSM-Tree optimization device according to claim 6, characterized in that: Also includes: A receiving unit, configured to receive a data query request; wherein the data query request includes the target data of this query; A second query unit, configured to query the target data at the L0 layer; A returning unit, configured to return a first query result if the target data is obtained by querying at the L0 layer; The second query unit is further configured to search for the target data in the B+ tree if the target data cannot be obtained by querying in the L0 layer; The returning unit is further configured to return a second query result if the target data is found in the B+ tree; The second query unit is further configured to query the target data in the L1 layer if the target data cannot be found in the B+ tree; The returning unit is further configured to return a third query result if the target data is obtained by querying in the L1 layer; The second query unit is further configured to select a layer below the next L1 layer to query the target data if the target data cannot be queried in the L1 layer, until the target data is queried or all layers complete the query to generate a final query result.
8. The LSM-Tree optimization device according to claim 7, characterized in that: Also includes: An elimination unit is used to eliminate the data stored in the B+ tree by using the least recently used method according to the time mark of the target key value data in the B+ tree when the data reaches the storage capacity threshold, so as to obtain the data marked for elimination; The third merging unit is used to merge the marked eliminated data with the key-value storage structure of the L1 layer to obtain a merged key-value storage structure of the L1 layer.
9. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the LSM-Tree optimization method as described in any one of claims 1 to 4.
10. A storage medium, characterized in that: A computer program is stored thereon, wherein when the computer program is executed by a processor, the LSM-Tree optimization method as described in any one of claims 1 to 4 is implemented.