A hybrid storage method and device of B+ tree engine and hierarchical static files

Abstract

The application discloses a kind of B+ tree engine and the mixed storage method and device of hierarchical static file, the method includes: B+ tree file layer is constructed on disk;When memory data is written in disk, whether there is corresponding B+ tree file that can be written according to the key value of memory data in B+ tree file layer is inquired, if the judgment result is yes, then memory data is written in corresponding B+ tree file according to key value, if the judgment result is no, then B+ tree file is newly built in B+ tree file layer, and memory data is written in newly built B+ tree file according to key value;When the data reached in B+ tree file layer reaches preset threshold, data in B+ tree file layer is merged and written in the next level data hierarchical file in disk.The application proposes the method that B+ tree replaces L0 layer, so that data merging is controllable and less time-consuming impact on request;Meanwhile, data to B+ tree can be updated, for writing, the data amount of B+ is less than the data amount of L0 layer, and the query efficiency is also higher than L0 layer.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, specifically to a hybrid storage method and apparatus for B+ tree engine and hierarchical static files. Background Technology

[0002] In the field of disk storage, LSM trees are a classic storage solution. LSM trees improve write performance by using sequential writes and achieve massive data storage by distinguishing between new and old data through hierarchical (memory and file) writing. Specifically, new data is written to memory, and when a certain amount is written, it is written from memory to disk to become old data.

[0003] Data storage schemes based on LSM trees have the following drawbacks:

[0004] 1. The memory capacity is fixed. Once it is full, it needs to be flushed to the disk immediately. Such flushing may trigger multi-level data merging. During multi-level merging, it will consume high hardware resources (CPU, memory). At this time, when LSM tree requests are busy, resource contention will lead to increased request time.

[0005] See details Figure 1 As shown, disk file writing in a classic LSM tree structure typically involves 7 levels (L0 has the smallest capacity, and L1-L6 have progressively increasing capacities). The memory data in an LSM is limited. Once it is full, it will be written to L0. At this point, L0 may not be full, so it can continue writing to the next L0 level. Alternatively, L0 may be full, triggering data merging to L1 (or even further down) levels. This process is uncontrollable, and the merging process is frequently triggered, especially when writing a lot of data. This leads to increased request latency due to resource contention.

[0006] 2. See Figure 1 As shown, a classic LSM tree structure typically contains seven levels for disk file writing (Level L0 has the smallest capacity, with levels L1-L6 progressively increasing in capacity). Each level corresponds to multiple files, and files cannot be modified after being written. If merging between levels is involved, the original files remain unchanged; a new file is generated after the merge, and the original files are deleted. For Level L0, memory is written directly to Level L0. This write generates one or more L0 files. Each write is based on the keys being sorted in memory before writing, meaning that the keys within a single L0 file are ordered. However, multiple writes from memory to L0 will generate multiple L0 files, and the keys stored in these multiple L0 files are not ordered; a single key may exist in multiple L0 files. Therefore, each key lookup will sequentially query multiple L0 files, resulting in a loss of query efficiency.

[0007] In view of this, this invention patent is hereby proposed. Summary of the Invention

[0008] To address the uncontrollable data merging issues in LSM trees and to further optimize query efficiency, this invention provides a hybrid storage method and apparatus combining a B+ tree engine and hierarchical static files. Specifically, the following technical solution is adopted:

[0009] A hybrid storage method combining a B+ tree engine and hierarchical static files includes:

[0010] Build a B+ tree file layer on disk;

[0011] When memory data is written to disk, the B+ tree file layer is queried to see if there is a corresponding B+ tree file that can be written based on the key value of the memory data. If the result is yes, the memory data is written to the corresponding B+ tree file according to the key value. If the result is no, a new B+ tree file is created in the B+ tree file layer, and the memory data is written to the newly created B+ tree file according to the key value.

[0012] When the amount of data written to the B+ tree file layer reaches a preset threshold, the data in the B+ tree file layer is merged and written to the next level data layer file on the disk.

[0013] As an optional embodiment of the present invention, a hybrid storage method for B+ tree engine and hierarchical static files includes a data writing process:

[0014] The B+ tree file layer includes a data storage B+ tree file for storing written memory data;

[0015] The memory data written to the disk is the storage data. Based on the key value of the storage data, a query is performed to see if there is a corresponding data storage B+ tree file in the B+ tree file layer that can be written. If the result is yes, the memory data is written to the corresponding data storage B+ tree file according to the key value. If the result is no, a new data storage B+ tree file is created, and the storage data is written to the newly created data storage B+ tree file according to the key value.

[0016] As an optional embodiment of the present invention, a hybrid storage method for B+ tree engine and hierarchical static files includes a data deletion process:

[0017] The B+ tree file layer includes a data deletion B+ tree file, which is used to store data deletion information in the data hierarchy of the hard disk.

[0018] The memory data written to the disk is the deleted data. The system checks whether the deleted data exists in the current data storage B+ tree file. If it exists, the corresponding stored data in the data storage B+ tree file is deleted. If it does not exist, the deleted data is written to the data deletion B+ tree file according to the key value of the deleted data.

[0019] As an optional embodiment of the present invention, a hybrid storage method of B+ tree engine and hierarchical static files includes a data query process:

[0020] According to the data query request, perform a data query in the data storage B+ tree file;

[0021] If the target data is found, the query result is returned; if the target data is not found, the data is then queried again in the data deletion B+ tree file.

[0022] If the target data is found, the query result is returned. If the target data is not found, the query is performed on the hard drive at each level of the data hierarchy. The query process terminates when the target data is found and the query result is returned. This process continues until all data hierarchy files on the hard drive have been queried, and the query result is returned.

[0023] As an optional embodiment of the present invention, in a hybrid storage method of B+ tree engine and hierarchical static file of the present invention, the data query is performed in the data deletion B+ tree file, and if the target data is found, the query result is returned as the target query data does not exist;

[0024] The process continues until all data levels on the hard drive have been queried. If the target data is found, the query result is returned as the target data; otherwise, the query result is returned as the target data does not exist.

[0025] As an optional embodiment of the present invention, a hybrid storage method for B+ tree engine and hierarchical static files includes a data merging process:

[0026] During the process of merging data in the B+ tree file layer into the next and next level data layer files on the disk;

[0027] Both the data storage B+ tree and the data deletion B+ tree are locked simultaneously, disallowing writing or updating. Data storage cache and data deletion cache are used instead of data storage B+ tree and data deletion B+ tree for data writing and updating.

[0028] After merging the data in the B+ tree file layer into the next and next level data layer files on the disk;

[0029] Clear the data in the data storage B+ tree and data deletion B+ tree in the B+ tree file layer, reinitialize, and write the data stored in the data storage cache and data deletion cache into the data storage B+ tree and data deletion B+ tree.

[0030] As an optional embodiment of the present invention, in a hybrid storage method of B+ tree engine and hierarchical static files, the disk has L1 data level files, L2 data level files, ..., L6 data level files with progressively increasing write capacity. The data of the B+ tree file layer is merged and written into the L1 data level file. After the data stored in the L1 data level file, L2 data level file, ..., L6 data level file reaches a preset threshold, it is merged and written into the next level data level file.

[0031] This invention also provides a hybrid storage device for B+ tree engine and hierarchical static files, comprising:

[0032] B+ tree file layers are built on disk;

[0033] The data writing module, when writing memory data to disk, queries the B+ tree file layer to see if there is a corresponding B+ tree file that can be written, based on the key value of the memory data. If the result is yes, the memory data is written to the corresponding B+ tree file according to the key value. If the result is no, a new B+ tree file is created in the B+ tree file layer, and the memory data is written to the newly created B+ tree file according to the key value.

[0034] The data merging module merges the data in the B+ tree file layer and writes it to the next level data layer file on the disk when the data written in the B+ tree file layer reaches a preset threshold.

[0035] A computer-readable storage medium is characterized in that it stores a computer-executable program, which, when executed, implements the hybrid storage method of a B+ tree engine and hierarchical static files.

[0036] An electronic device includes a processor and a memory, the memory being used to store a computer-executable program, wherein when the computer program is executed by the processor, the processor executes the hybrid storage method of a B+ tree engine and hierarchical static files.

[0037] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0038] The improved LSM tree structure generated by the hybrid storage method of B+ tree engine and hierarchical static files of the present invention uses B+ tree files to replace the L0 level files in the classic LSM tree structure. The improvement scheme mainly includes:

[0039] 1. In the classic LSM tree structure, the files in the L0 layer are immutable. However, in the improved LSM tree structure generated by the hybrid storage method of B+ tree engine and hierarchical static files of this invention, the B+ tree files support addition, update and deletion operations.

[0040] 2. In the classic LSM tree structure, the data of multiple files in the L0 layer is written independently. The data storage in a single L0 file is ordered, and there will be no multiple identical keys. However, L0 files may store the same key. In the improved LSM tree structure generated by the hybrid storage method of B+ tree engine and hierarchical static files of the present invention, each key in the B+ tree file is hashed. The same key will only be written to the same B+ tree, and the keys of multiple B+ trees are not repeated.

[0041] Therefore, in the improved LSM tree structure generated by the hybrid storage method of B+ tree engine and hierarchical static files of the present invention, multiple B+ tree files replace the L0 level file in the classic LSM tree structure, which has the following technical effects:

[0042] 1. When writing memory data to disk, the hybrid storage method of B+ tree engine and hierarchical static file in this embodiment stores memory data on B+ tree file. In this way, each key in the written memory data corresponds to only 1 B+ tree. The query time complexity of key is O(1*log2n), while the query time complexity of the original L0 layer is O(m*log2n), where m is the number of L0 layer files and n is the number of keys stored in each file (B+ tree file or L0 file).

[0043] Therefore, the hybrid storage method of B+ tree engine and hierarchical static files of the present invention reduces the query time complexity when performing data queries, avoids repeated queries of the same key in L0 file, and improves query efficiency.

[0044] 2. For the classic LSM tree structure, since memory data is limited, when it is full, it will be written to the L0 level. At this time, the L0 level may not be full, and it can continue to write to the next L0 level file. It is also possible that the L0 level is full, which will trigger the merging of data to the L1 level (or even more levels down). This process is uncontrollable, and the merging process will be frequently triggered, especially when a lot of data is written.

[0045] This invention discloses a hybrid storage method combining a B+ tree engine and hierarchical static files. The B+ tree approach allows write operations to overcome the limitations of LSM memory data size, increasing the amount of data written from 1-10GB to tens to hundreds of GB. This prevents merging operations from being performed during periods of high write requests (which are also likely to be periods of high read requests), thus avoiding increased request latency caused by merging during periods of high request volume. The timing of the merging operation can be configured according to the user's off-peak periods to minimize the impact of merging on request latency.

[0046] In summary, the hybrid storage method of B+ tree engine and hierarchical static files proposed in this invention addresses the problems of high and uncontrollable query and merge counts in LSM trees at the L0 level. It proposes a method to replace the L0 level with a B+ tree. This improvement makes data merging controllable and has less impact on request time. On the one hand, the merge time can be configured to a period with lower request frequency; on the other hand, the B+ tree can store a large amount of data, reducing the number of merges to 1 / 10 (or lower) of the original L0 level. Simultaneously, data can be updated in the B+ tree. For writing, the amount of data in the B+ tree is less than that in the L0 level (multiple B+ trees do not have duplicate keys), and the query efficiency is also higher than that of the L0 level. This solves the defects of slow query and frequent merging in the L0 level, demonstrating high innovation and practical value. Attached image description:

[0047] Figure 1 A schematic diagram of a classic LSM tree hierarchy;

[0048] Figure 2 A schematic diagram of an improved LSM tree hierarchy generated by a hybrid storage method combining a B+ tree engine and hierarchical static files according to an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0050] Therefore, the following detailed description of embodiments of the present invention is not intended to limit the scope of the claimed invention, but merely illustrates some embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0051] It should be noted that, unless otherwise specified, the embodiments and features and technical solutions in the present invention can be combined with each other.

[0052] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0053] In the description of this invention, it should be noted that the terms "upper," "lower," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use, or the orientation or positional relationship commonly understood by those skilled in the art. These terms are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. In addition, the terms "first," "second," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0054] This embodiment of a hybrid storage method combining a B+ tree engine and hierarchical static files includes:

[0055] Build a B+ tree file layer on disk;

[0056] When memory data is written to disk, the B+ tree file layer is queried to see if there is a corresponding B+ tree file that can be written based on the key value of the memory data. If the result is yes, the memory data is written to the corresponding B+ tree file according to the key value. If the result is no, a new B+ tree file is created in the B+ tree file layer, and the memory data is written to the newly created B+ tree file according to the key value.

[0057] When the amount of data written to the B+ tree file layer reaches a preset threshold, the data in the B+ tree file layer is merged and written to the next level data level file on the disk.

[0058] See Figure 2 As shown, this embodiment illustrates an improved LSM tree structure generated by a hybrid storage method combining a B+ tree engine and hierarchical static files. The improvements mainly include:

[0059] 1. Figure 1 In the classic LSM tree structure, the files in level L0 are immutable, while Figure 2 In the improved LSM tree structure generated by the hybrid storage method of B+ tree engine and hierarchical static files in this embodiment, B+ tree files support addition, update, and deletion operations.

[0060] 2. Figure 1In the classic LSM tree structure, the data of multiple files in the L0 layer is written independently. The data storage in a single L0 file is ordered, and there will be no multiple identical keys. However, L0 files may store the same key. In the improved LSM tree structure generated by the hybrid storage method of B+ tree engine and hierarchical static files in this embodiment, the B+ tree file hashes each key, and the same key will only be written to the same B+ tree. The keys of multiple B+ trees are not repeated.

[0061] Therefore, in the improved LSM tree structure generated by the hybrid storage method of B+ tree engine and hierarchical static files in this embodiment, multiple B+ tree files replace the L0 level file in the classic LSM tree structure, which has the following technical effects:

[0062] 1. When writing memory data to disk, the hybrid storage method of B+ tree engine and hierarchical static file in this embodiment stores memory data on B+ tree file. In this way, each key in the written memory data corresponds to only 1 B+ tree. The query time complexity of key is O(1*log2n), while the query time complexity of the original L0 layer is O(m*log2n), where m is the number of L0 layer files and n is the number of keys stored in each file (B+ tree file or L0 file).

[0063] Therefore, the hybrid storage method of B+ tree engine and hierarchical static files in this embodiment reduces the query time complexity when performing data queries, avoids repeated queries of the same key in L0 file, and improves query efficiency.

[0064] 2. For the classic LSM tree structure, since memory data is limited, when it is full, it will be written to the L0 level. At this time, the L0 level may not be full, and it can continue to write to the next L0 level file. It is also possible that the L0 level is full, which will trigger the merging of data to the L1 level (or even more levels down). This process is uncontrollable, and the merging process will be frequently triggered, especially when a lot of data is written.

[0065] This embodiment presents a hybrid storage method combining a B+ tree engine and hierarchical static files. The B+ tree approach allows write operations to overcome the limitations of LSM memory data size, increasing the amount of data written from 1-10GB to tens to hundreds of GB. This prevents merging operations from being performed during periods of high write requests (which are also likely to be periods of high read requests), thus avoiding increased request latency caused by merging during periods of high request volume. The timing of the merging operation can be configured based on the user's off-peak periods to minimize the impact of merging on request latency.

[0066] In summary, this embodiment presents a hybrid storage method combining a B+ tree engine and hierarchical static files. Addressing the issues of high and uncontrollable query and merging times at the L0 level of LSM trees, it proposes a B+ tree replacement for the L0 level. This improvement makes data merging controllable and has less impact on request latency. Firstly, the merging time can be configured to occur during periods of lower request frequency. Secondly, the B+ tree can store a large amount of data, reducing the number of merges to 1 / 10 (or lower) of the original L0 level. Furthermore, data can be updated in the B+ tree. For writing, the amount of data in the B+ tree is less than that in the L0 level (multiple B+ trees do not have duplicate keys), and the query efficiency is also higher than that of the L0 level. This solves the problems of slow queries and frequent merging in the L0 level, demonstrating high innovation and practical value.

[0067] Furthermore, this embodiment provides a hybrid storage method combining a B+ tree engine and hierarchical static files, including a data writing process:

[0068] The B+ tree file layer includes a data storage B+ tree file for storing written memory data;

[0069] The memory data written to the disk is the storage data. Based on the key value of the storage data, a query is performed to see if there is a corresponding data storage B+ tree file in the B+ tree file layer that can be written. If the result is yes, the memory data is written to the corresponding data storage B+ tree file according to the key value. If the result is no, a new data storage B+ tree file is created, and the storage data is written to the newly created data storage B+ tree file according to the key value.

[0070] In this embodiment, data writing includes both new data writing and data updating. New data writing refers to writing data whose key does not exist in the current B+ tree, while data updating refers to writing data whose key exists in the current B+ tree. When it is new data writing, the data is directly written to the corresponding B+ tree based on the key. When it is data updating, the corresponding key in the B+ tree is queried based on the key of the updated data, and the updated data is replaced with the updated data.

[0071] Furthermore, this embodiment of a hybrid storage method combining a B+ tree engine and hierarchical static files includes a data deletion process:

[0072] The B+ tree file layer includes a data deletion B+ tree file, which is used to store data deletion information in the data hierarchy of the hard disk.

[0073] The memory data written to the disk is the deleted data. The system checks whether the deleted data exists in the current data storage B+ tree file. If it exists, the corresponding stored data in the data storage B+ tree file is deleted. If it does not exist, the deleted data is written to the data deletion B+ tree file according to the key value of the deleted data.

[0074] This embodiment presents a hybrid storage method combining a B+ tree engine and hierarchical static files. When deleting data, if the data to be deleted is found in the data storage B+ tree file, and not found there, the data to be deleted is stored in the data deletion B+ tree file. This is because although the data storage B+ tree file may not contain the data, it may exist in the L1-L6 level files on the disk. By introducing a data deletion B+ tree file to record deleted data, this embodiment ensures that when data is found in the data deletion B+ tree file, the query result can be directly returned as "data has been deleted" or "data does not exist," eliminating the need to perform a layer-by-layer query of the L1-L6 level files on the disk if the data is not found in the data storage B+ tree file. This ensures the accuracy of the data query and improves the efficiency of data retrieval.

[0075] Specifically, this embodiment of a hybrid storage method combining a B+ tree engine and hierarchical static files includes a data query process:

[0076] According to the data query request, perform a data query in the data storage B+ tree file;

[0077] If the target data is found, the query result is returned; if the target data is not found, the data is then queried again in the data deletion B+ tree file.

[0078] If the target data is found, the query result is returned. If the target data is not found, the query is performed on the hard drive at each level of the data hierarchy. The query process terminates when the target data is found and the query result is returned. This process continues until all data hierarchy files on the hard drive have been queried, and the query result is returned.

[0079] Furthermore, in a hybrid storage method of B+ tree engine and hierarchical static files in this embodiment, when data is queried in the data deletion B+ tree file, if the target data is found, the query result is returned as the target data does not exist.

[0080] The process continues until all data levels on the hard drive have been queried. If the target data is found, the query result is returned as the target data; otherwise, the query result is returned as the target data does not exist.

[0081] Furthermore, this embodiment of a hybrid storage method combining a B+ tree engine and hierarchical static files includes a data merging process:

[0082] During the process of merging data in the B+ tree file layer into the next and next level data layer files on the disk;

[0083] Both the data storage B+ tree and the data deletion B+ tree are locked simultaneously, disallowing writing or updating. Data storage cache and data deletion cache are used instead of data storage B+ tree and data deletion B+ tree for data writing and updating.

[0084] After merging the data in the B+ tree file layer into the next and next level data layer files on the disk;

[0085] Clear the data in the data storage B+ tree and data deletion B+ tree in the B+ tree file layer, reinitialize, and write the data stored in the data storage cache and data deletion cache into the data storage B+ tree and data deletion B+ tree.

[0086] Specifically, this embodiment provides a hybrid storage method for B+ tree engine and hierarchical static files. The disk has L1 data level files, L2 data level files, ..., L6 data level files with progressively increasing write capacity. Data from the B+ tree file layers is merged and written into the L1 data level file. After the data stored in the L1 data level file, L2 data level file, ..., L6 data level file reaches a preset threshold, it is merged and written into the next level data level file.

[0087] The L1 to L6 data level files in this embodiment can achieve data storage on a larger scale. In addition, the hybrid storage method of B+ tree engine and hierarchical static files in this embodiment can also work without including L1-L6 data level files, even if only in-memory data and B+ tree are included. The only difference is that the storage scale is reduced.

[0088] This embodiment also provides a hybrid storage device for B+ tree engine and hierarchical static files, including:

[0089] B+ tree file layers are built on disk;

[0090] The data writing module, when writing memory data to disk, queries the B+ tree file layer to see if there is a corresponding B+ tree file that can be written, based on the key value of the memory data. If the result is yes, the memory data is written to the corresponding B+ tree file according to the key value. If the result is no, a new B+ tree file is created in the B+ tree file layer, and the memory data is written to the newly created B+ tree file according to the key value.

[0091] The data merging module merges the data in the B+ tree file layer and writes it to the next level data layer file on the disk when the data written in the B+ tree file layer reaches a preset threshold.

[0092] See Figure 2 As shown in the illustration, this embodiment generates an improved LSM tree structure using a hybrid storage device combining a B+ tree engine and hierarchical static files. The improvement scheme mainly includes:

[0093] 1. Figure 1 In the classic LSM tree structure, the files in level L0 are immutable, while Figure 2 In the improved LSM tree structure generated by a hybrid storage device of a B+ tree engine and hierarchical static files in this embodiment, B+ tree files support addition, update, and deletion operations.

[0094] 2. Figure 1 In the classic LSM tree structure, the data of multiple files in the L0 layer is written independently, and the data storage in a single L0 file is ordered, with no multiple identical keys. However, L0 files may store the same key. In the improved LSM tree structure generated by the hybrid storage device of B+ tree engine and hierarchical static files in this embodiment, the B+ tree file hashes each key, and the same key will only be written to the same B+ tree. The keys of multiple B+ trees are not repeated.

[0095] Therefore, in the improved LSM tree structure generated by the hybrid storage device of B+ tree engine and hierarchical static files in this embodiment, multiple B+ tree files are used to replace the L0 level files in the classic LSM tree structure, which has the following technical effects:

[0096] 1. When writing memory data to disk, the hybrid storage device of B+ tree engine and hierarchical static file in this embodiment stores memory data on B+ tree files. In this way, each key in the written memory data corresponds to only one B+ tree. The query time complexity of the key is O(1*log2n), while the query time complexity of the original L0 layer is O(m*log2n), where m is the number of L0 layer files and n is the number of keys stored in each file (B+ tree file or L0 file).

[0097] Therefore, the hybrid storage device of B+ tree engine and hierarchical static files in this embodiment reduces the query time complexity when performing data queries, avoids repeated queries of the same key in L0 file, and improves query efficiency.

[0098] 2. For the classic LSM tree structure, since memory data is limited, when it is full, it will be written to the L0 level. At this time, the L0 level may not be full, and it can continue to write to the next L0 level file. It is also possible that the L0 level is full, which will trigger the merging of data to the L1 level (or even more levels down). This process is uncontrollable, and the merging process will be frequently triggered, especially when a lot of data is written.

[0099] This embodiment presents a hybrid storage device combining a B+ tree engine and hierarchical static files. The B+ tree approach allows write operations to overcome the limitations of LSM memory data size, increasing the amount of data written from 1-10GB to tens to hundreds of GB. This prevents merging operations from being performed during periods of high write requests (which are also likely to be periods of high read requests), thus avoiding increased request latency caused by merging during periods of high request volume. The timing of the merging operation can be configured based on the user's off-peak periods to minimize the impact of merging operations on request latency.

[0100] In summary, this embodiment of a hybrid storage device combining a B+ tree engine and hierarchical static files addresses the issues of high and uncontrollable query and merging times at the L0 level of LSM trees. It proposes a method to replace the L0 level with a B+ tree. This improvement makes data merging controllable and has less impact on request time. Firstly, the merging time can be configured to occur during periods of lower request frequency. Secondly, the B+ tree can store a large amount of data, reducing the number of merges to 1 / 10 (or lower) of the original L0 level. Simultaneously, data can be updated in the B+ tree. For writing, the amount of data in the B+ tree is less than that in the L0 level (multiple B+ trees do not have duplicate keys), and the query efficiency is also higher than that of the L0 level. This solves the problems of slow queries and frequent merging in the L0 level, demonstrating high innovation and practical value.

[0101] Furthermore, in this embodiment, a hybrid storage device combining a B+ tree engine and hierarchical static files includes a data writing process for the data writing module, which includes:

[0102] The B+ tree file layer includes a data storage B+ tree file for storing written memory data;

[0103] The memory data written to the disk is the storage data. Based on the key value of the storage data, a query is performed to see if there is a corresponding data storage B+ tree file in the B+ tree file layer that can be written. If the result is yes, the memory data is written to the corresponding data storage B+ tree file according to the key value. If the result is no, a new data storage B+ tree file is created, and the storage data is written to the newly created data storage B+ tree file according to the key value.

[0104] The data writing module in this embodiment includes both new data writing and data updating. New data writing refers to writing data whose key does not exist in the current B+ tree, while data updating refers to writing data whose key exists in the current B+ tree. When writing new data, the data is directly written to the corresponding B+ tree based on its key. When updating data, the corresponding key in the B+ tree is queried based on the key of the updated data, and the updated data is replaced with the updated data.

[0105] Furthermore, a hybrid storage device for a B+ tree engine and hierarchical static files in this embodiment includes a data deletion module. The data deletion module performs a data deletion process including:

[0106] The B+ tree file layer includes a data deletion B+ tree file, which is used to store data deletion information in the data hierarchy of the hard disk.

[0107] The memory data written to the disk is the deleted data. The system checks whether the deleted data exists in the current data storage B+ tree file. If it exists, the corresponding stored data in the data storage B+ tree file is deleted. If it does not exist, the deleted data is written to the data deletion B+ tree file according to the key value of the deleted data.

[0108] This embodiment presents a hybrid storage device combining a B+ tree engine and hierarchical static files. When the data deletion module performs data deletion, if the data to be deleted is found in the data storage B+ tree file, and not found there, the data to be deleted is stored in the data deletion B+ tree file. This is because although the data storage B+ tree file may not contain the data, it may exist in the L1-L6 level files on the disk. By introducing a data deletion B+ tree file to record deleted data, this embodiment allows for direct return of "data deleted" or "data does not exist" results when data is found in the data deletion B+ tree file. This eliminates the need to perform a layer-by-layer search of the L1-L6 level files on the disk if the data is not found in the data storage B+ tree file, ensuring the accuracy and efficiency of data retrieval.

[0109] Specifically, this embodiment of a hybrid storage device for a B+ tree engine and hierarchical static files includes a data query module. The data query module executes a data query process including:

[0110] According to the data query request, perform a data query in the data storage B+ tree file;

[0111] If the target data is found, the query result is returned; if the target data is not found, the data is then queried again in the data deletion B+ tree file.

[0112] If the target data is found, the query result is returned. If the target data is not found, the query is performed on the hard drive at each level of the data hierarchy. The query process terminates when the target data is found and the query result is returned. This process continues until all data hierarchy files on the hard drive have been queried, and the query result is returned.

[0113] Furthermore, in a hybrid storage device of B+ tree engine and hierarchical static files in this embodiment, the data query module performs data query in the data deletion B+ tree file. If the target data is found, the query result is returned as the target data does not exist.

[0114] The data query module continues until all data levels on the hard drive have been queried. If the target data is found, the query result is returned as the target data; otherwise, the query result is returned as the target data does not exist.

[0115] Furthermore, a hybrid storage device for B+ tree engine and hierarchical static files in this embodiment includes a data merging module. The data merging module performs a data merging process including:

[0116] During the process of merging data in the B+ tree file layer into the next and next level data layer files on the disk;

[0117] Both the data storage B+ tree and the data deletion B+ tree are locked simultaneously, disallowing writing or updating. Data storage cache and data deletion cache are used instead of data storage B+ tree and data deletion B+ tree for data writing and updating.

[0118] After merging the data in the B+ tree file layer into the next and next level data layer files on the disk;

[0119] Clear the data in the data storage B+ tree and data deletion B+ tree in the B+ tree file layer, reinitialize, and write the data stored in the data storage cache and data deletion cache into the data storage B+ tree and data deletion B+ tree.

[0120] Specifically, this embodiment provides a hybrid storage device for a B+ tree engine and hierarchical static files. The disk has L1 data level files, L2 data level files, ..., L6 data level files with progressively increasing write capacity. Data from the B+ tree file layers is merged and written into the L1 data level file. After the data stored in the L1, L2, ..., L6 data level files reaches a preset threshold, it is merged and written into the next level data level file.

[0121] The L1 to L6 data level files of this embodiment can achieve data storage of a larger scale. In addition, the hybrid storage device of B+ tree engine and hierarchical static files of this embodiment can also work without including L1-L6 data level files, even if it only contains memory data and B+ tree. The only difference is that the storage scale is reduced.

[0122] This embodiment also provides a computer-readable storage medium storing a computer-executable program. When the computer-executable program is executed, it implements the hybrid storage method of B+ tree engine and hierarchical static files.

[0123] The computer-readable storage medium described in this embodiment may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0124] This embodiment also provides an electronic device, including a processor and a memory, wherein the memory is used to store a computer-executable program, and when the computer program is executed by the processor, the processor executes the hybrid storage method of B+ tree engine and hierarchical static files.

[0125] The electronic device is manifested in the form of a general-purpose computing device. It may contain one or more processors that work collaboratively. This invention also does not preclude distributed processing, meaning that processors may be distributed across different physical devices. The electronic device of this invention is not limited to a single entity, but may also be the sum of multiple physical devices.

[0126] The memory stores a computer-executable program, typically machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least some steps of the method.

[0127] The memory includes volatile memory, such as random access memory (RAM) and / or cache memory, and may also be non-volatile memory, such as read-only memory (ROM).

[0128] It should be understood that the electronic device of the present invention may also include elements or components not shown in the examples above. For example, some electronic devices also include display units such as a display screen, and some electronic devices also include human-computer interaction elements such as buttons and keyboards. Any electronic device capable of executing a computer-readable program in its memory to implement the method of the present invention or at least some steps of the method can be considered as an electronic device covered by the present invention.

[0129] From the above description of the embodiments, those skilled in the art will readily understand that the present invention can be implemented by hardware capable of executing specific computer programs, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. included in the system. The present invention can also be implemented by computer software that executes the methods of the present invention, for example, by control software executed by a microprocessor, electronic control unit, client, server, etc. However, it should be noted that the computer software executing the methods of the present invention is not limited to execution in one or a specific set of hardware entities; it can also be implemented in a distributed manner by unspecified hardware. For computer software, the software product can be stored in a computer-readable storage medium (such as a CD-ROM, USB flash drive, portable hard drive, etc.) or distributed across a network, as long as it enables electronic devices to execute the methods according to the present invention.

[0130] The above embodiments are only used to illustrate the present invention and are not intended to limit the technical solutions described herein. Although the present invention has been described in detail with reference to the above embodiments, the present invention is not limited to the specific embodiments described above. Therefore, any modifications or equivalent substitutions to the present invention, as well as all technical solutions and improvements that do not depart from the spirit and scope of the invention, are covered within the scope of the claims of the present invention.

Claims

1. A hybrid storage method combining a B+ tree engine and hierarchical static files, characterized in that, include: Build a B+ tree file layer on disk; When memory data is written to disk, the B+ tree file layer is queried to see if a corresponding B+ tree file exists based on the key value of the memory data. If the result is yes, the memory data is written to the corresponding B+ tree file according to the key value. If the result is no, a new B+ tree file is created in the B+ tree file layer, and the memory data is written to the newly created B+ tree file according to the key value. When the data written in the B+ tree file layer reaches a preset threshold, the data in the B+ tree file layer is merged and written to the next level data level file on the disk. The hybrid storage method also includes a data deletion process: The B+ tree file layer includes a data deletion B+ tree file, which is used to store data deletion information in the data hierarchy of the hard disk. The memory data written to the disk is the deleted data. The system checks whether the deleted data exists in the current data storage B+ tree file. If it exists, the corresponding storage data in the data storage B+ tree file is deleted. If it does not exist, the data deletion B+ tree file is written according to the key value of the deleted data. The hybrid storage method also includes a data query process: According to the data query request, perform a data query in the data storage B+ tree file; If the target data is found, the query result is returned; if the target data is not found, the data is then queried again in the data deletion B+ tree file. If the target data is found, the query result is returned. If the target data is not found, the query is performed on the hard drive at each level of the data hierarchy. When the target data is found, the query process is terminated and the query result is returned. This process continues until all data hierarchy files on the hard drive have been queried and the query result is returned. The hybrid storage method also includes a data merging process: During the process of merging data in the B+ tree file layer into the next and next level data layer files on the disk; Both the data storage B+ tree and the data deletion B+ tree are locked simultaneously, disallowing writing or updating. Data storage cache and data deletion cache are used instead of data storage B+ tree and data deletion B+ tree for data writing and updating. After merging the data in the B+ tree file layer into the next and next level data layer files on the disk; Clear the data in the data storage B+ tree and data deletion B+ tree in the B+ tree file layer, reinitialize, and write the data stored in the data storage cache and data deletion cache into the data storage B+ tree and data deletion B+ tree.

2. The hybrid storage method of B+ tree engine and hierarchical static files according to claim 1, characterized in that, Including the data writing process: The B+ tree file layer includes a data storage B+ tree file for storing written memory data; The memory data written to the disk is the storage data. The B+ tree file layer is queried according to the key value of the storage data to see if there is a corresponding data storage B+ tree file. If the result is yes, the memory data is written to the corresponding data storage B+ tree file according to the key value. If the result is no, a new data storage B+ tree file is created, and the storage data is written to the newly created data storage B+ tree file according to the key value.

3. The hybrid storage method of B+ tree engine and hierarchical static files according to claim 1, characterized in that, The process involves querying the data in the B+ tree file where data is deleted. If the target data is found, the query result is returned as "the target data does not exist". The process continues until all data levels on the hard drive have been queried. If the target data is found, the query result is returned as the target data; otherwise, the query result is returned as the target data does not exist.

4. The hybrid storage method of B+ tree engine and hierarchical static files according to claim 1, characterized in that, The disk has L1 data level files, L2 data level files, ..., L6 data level files with progressively increasing write capacity. The data of the B+ tree file layer is merged and written into the L1 data level file. After the data stored in the L1 data level file, L2 data level file, ..., L6 data level file reaches a preset threshold, it is merged and written into the next level data level file.

5. A hybrid storage device for a B+ tree engine and hierarchical static files, characterized in that, include: B+ tree file layers are built on disk; The data writing module, when writing memory data to disk, queries the B+ tree file layer to see if a corresponding B+ tree file exists for writing based on the key value of the memory data. If the result is yes, the memory data is written to the corresponding B+ tree file according to the key value. If the result is no, a new B+ tree file is created in the B+ tree file layer, and the memory data is written to the newly created B+ tree file according to the key value. The data merging module merges the data in the B+ tree file layer and writes it to the next level data layer file on the disk when the data written in the B+ tree file layer reaches a preset threshold. Including the data deletion process: The B+ tree file layer includes a data deletion B+ tree file, which is used to store data deletion information in the data hierarchy of the hard disk. The memory data written to the disk is the deleted data. The system checks whether the deleted data exists in the current data storage B+ tree file. If it exists, the corresponding storage data in the data storage B+ tree file is deleted. If it does not exist, the data deletion B+ tree file is written according to the key value of the deleted data. Including the data query process: According to the data query request, perform a data query in the data storage B+ tree file; If the target data is found, the query result is returned; if the target data is not found, the data is then queried again in the data deletion B+ tree file. If the target data is found, the query result is returned. If the target data is not found, the query is performed on the hard drive at each level of the data hierarchy. When the target data is found, the query process is terminated and the query result is returned. This process continues until all data hierarchy files on the hard drive have been queried and the query result is returned. Including the data merging process: During the process of merging data in the B+ tree file layer into the next and next level data layer files on the disk; Both the data storage B+ tree and the data deletion B+ tree are locked simultaneously, disallowing writing or updating. Data storage cache and data deletion cache are used instead of data storage B+ tree and data deletion B+ tree for data writing and updating. After merging the data in the B+ tree file layer into the next and next level data layer files on the disk; Clear the data in the data storage B+ tree and data deletion B+ tree in the B+ tree file layer, reinitialize, and write the data stored in the data storage cache and data deletion cache into the data storage B+ tree and data deletion B+ tree.

6. A computer-readable storage medium, characterized in that, The system stores a computer-executable program, which, when executed, implements a hybrid storage method for a B+ tree engine and hierarchical static files as described in any one of claims 1-4.

7. An electronic device comprising a processor and a memory, the memory being used to store a computer-executable program, wherein when the computer program is executed by the processor, the processor performs a hybrid storage method of a B+ tree engine and hierarchical static files as described in any one of claims 1-4.

Images