A key-value storage method based on multi-log-structured merge tree

Abstract

The application discloses a key value storage method based on a multi-log structure merge tree, and comprises the following steps: 1, when a read-write request is received by a multi-log structure merge tree (MLSMT), the hash value of a key in the request is used to determine the log structure merge tree (LSMT) to which the key belongs i , and read-write operation is performed on the LSMT i ; 2, when a scanning request is received by the MLSMT structure, a grouping iterator is established for a file in the log structure merge tree, parallel threads are used to perform a key positioning operation (Seek), and a heap sorting is used to find a next key, so that data is read out; 3, a compaction operation is processed by using parallel threads in the background, and data belonging to a current LSMT i is read out from an LSMT i split from the LSMT s and written into an SST file of the LSMT i . The application can realize accelerated writing and merging of data, reduce the number of layers of a single log structure merge tree, and thus improve the read-write performance of a system.

Description

Technical Field

[0001] This invention belongs to the field of database technology, specifically a key-value storage method based on a multi-log structure merging tree. Background Technology

[0002] The LSM-Tree (LogStructuredMergeTrees) data organization method is used in various databases, such as LevelDB and Cassandra. Compared to B+ trees or other indexed storage implementations, LSM-Trees offer better write performance by implementing random read / write of key-value pairs as in-memory random read / write with sequential disk flushing. This leverages the characteristic that sequential disk writes are superior to random writes, thus improving data write performance. Furthermore, it utilizes the hierarchical, ordered structure of disk data and the merge sorting (compaction) between layers to improve read performance.

[0003] LSM-Tree, with its hierarchical and ordered structure on disk, can indeed improve data read performance when files are randomly stored at a single level. However, this hierarchical structure also introduces read / write amplification issues. When writing data, it is first written to a memory buffer and then appended to a lower level on the disk. Older data is compacted and rewritten to higher-level files. When reading key-value data, it searches the memory buffer. If the data is not found, it searches the disk layer by layer from lowest to highest. If the data is found at a certain level, it is returned directly; otherwise, it indicates that the data does not exist.

[0004] As can be seen from the above read and write process, the written data will undergo a large number of repeated writes, and the read data will also cause a lot of invalid disk I / O because the data is at a high level on the disk, and the system hardware resources are not fully utilized. When the data layer is very deep, the above read and write amplification will be more serious. Summary of the Invention

[0005] This invention aims to address the shortcomings of existing technologies by proposing a key-value storage method based on a multi-log structure merging tree. This method aims to overcome the deficiencies of existing technologies, fully utilize existing hardware resources, accelerate data writing and merging, reduce the number of layers in a single log structure merging tree, and reduce read / write amplification, thereby improving the system's read / write performance and enhancing its parallelism and scalability.

[0006] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0007] This invention discloses a key-value storage method based on a multi-log structure merging tree. This method is applied to a single storage node configured with an MLSMT structure, wherein the MLSMT structure includes: a compaction scheduler, an LSMT information monitor, and N log structure merging trees {LSMT1, LSMT2, ..., LSMT...}. i ,…,LSMT N}, where LSMT i This represents the ith log structure merge tree; 1 ≤ i ≤ N;

[0008] The i-th log structure merge tree LSMT i It includes a disk component and a memory component;

[0009] LSMT i The disk component contains P layers {L i1 ,L i2 ,…,L ij ,…,L ip}, where L ij Let P represent the j-th level structure of the i-th log structure merge tree, where 1 ≤ j ≤ P; L represents the j-th level structure. ij Including Q ij SST files

[0010] {SST ij1 SST ij2 ,…,SST ijk ,…,SST ijQij}, where SST ijk Represents the LSMT of the i-th log structure merge tree. i The j-th layer structure L ij The k-th SST file, 1≤k≤Q ij ;

[0011] The k-th SST file SST ijk Contains several key-value data<key,value> According to the k-th SST file SST ijk The values ​​of all key data in the SST file are used to obtain the k-th SST file. ijk key range (keymin) ijk keymax ijk ), where keymin ijk This represents the k-th SST file. ijk The minimum value of the key, keymax ijk This represents the k-th SST file. ijk The maximum value of the bonds, excluding the first layer structure L i1Apart from the overlapping key ranges between files in the same layer, the key ranges between files in other layers do not overlap and are arranged in an ordered manner.

[0012] Let the k-th SST file be SST ijk The disk space occupied by all key-value data in the SST file is denoted as the k-th SST file. ijk Size ijk The j-th layer structure L ij Total size of all SST files (Size) ij The threshold is denoted as LimitSize ij If j>1, the number of SST files in the first-level structure has a threshold LimitNum. i ;

[0013] Except for the first layer structure L i1 In addition, the (j+1)th layer structure L i(j+1) LimitSize for all SST files i(j+1) It is the j-th layer structure L ij LimitSize for all SST files ij q times, i.e., LimitSize i(j+1) =q×LimitSize ij ;

[0014] The i-th log structure merge tree LSMT i The memory components include the memory buffer MT. i Immutable buffers (IMT) i Two-dimensional metadata management array FM i , and FM i Includes P×Q ij Each element is a structure consisting of the SST file number, the size of the SST file, the reference count of the SST file, the minimum value of the key, the maximum value of the key, and a GC flag field indicating whether garbage collection is required.

[0015] The i-th log structure merge tree LSMT i The included operation is: FH i Operation, Compaction i Operations; among which, Compaction i Operations are divided into MinC i Operations, MaxC i operate;

[0016] FH i Operation refers to memory buffer MT i The key-value data is written to the first-level structure L of the disk component in the form of an SST file. i1 middle;

[0017] MinC i It refers to the first layer structure L i1 and the second layer structure L i2 The selected SST files are sorted by key, merged, and then divided into new SST files, which are then written to L. i2 ;

[0018] MaxC i It refers to the j-th layer structure L ij and the (j+1)th layer structure L ij+1 The selected SST files are sorted by key, merged, and then divided into new SST files, which are then written to L. ij+1 j>1;

[0019] The LSMT monitor is used to obtain monitoring information of N log structure merge trees and calculate the split index, let F i For the i-th log structure merge tree LSMT i The splitting index, F it The threshold for the i-th split metric; the LSMT of the i-th log structure merge tree. i The monitoring information includes: LSMT i The total disk space occupied by all SST files S i Disk space size threshold S it Read / write request count RW i , and the total number of read and write requests RW for all log structure merged trees;

[0020] The key-value storage method is performed according to the following steps:

[0021] Step 1: The current thread uses the MLSMT structure to receive external read / write data requests (req) and determines the type of the request (req):

[0022] If the request `req` is a write request (`Write`), then proceed to step 2, wherein the parameters in the write request include several key-value pairs of data to be written.<key,value> ;

[0023] If the request req is a read request (Get), then proceed to step 2; wherein, the parameters in the read request (Get) include several key data items (key);

[0024] If the request req is a scan request Scan, then proceed to step 5; wherein the parameters in the scan request Scan include the key data range (key1, key2);

[0025] Step 2: The background thread calculates and splits the split index of the log structure merge tree;

[0026] Step 3: The current thread performs a hash operation on the key in the request req to obtain the corresponding hash value. The hash value is then moduloed by the threshold mod to obtain the intermediate value hv, where mod represents the threshold for the number of merged trees in the log structure.

[0027] Step 4: The current thread calculates the minimum number of binary bits n required to represent N, calculates the n-bit value of the tail of hv based on n, and then calculates the sequence number i of the log structure merge tree based on the n-bit value, thus ensuring that the key in the request req belongs to LSMT. i ;

[0028] If the request req is a write request, then execute steps A1 and B1:

[0029] A1. Write the key-value data corresponding to the parameters in the Write request to MT. i And proceed according to steps A1.1-A1.5;

[0030] A1.1 Determine the LSMT of the i-th log structure merge tree. i memory buffer MT i Size M i Is it greater than the threshold Mmax? If it is, proceed to step A1.2; otherwise, change the parameter.<key,value> The corresponding key-value data is written to LSMT i Memory cache MT i And execute B1;

[0031] A1.2 Determine the current log structure merge tree (LSMT) i IMT i If the value is not empty, wait until the Compaction scheduler executes FH. i The scheduling request for the operation, thereby placing the buffer IMT i All data is written to the disk, enabling IMT. i If empty, execute A1.3;

[0032] A1.3, IMT i Pointer to the current memory buffer MT i And generate a new empty memory buffer, making MT i Points to a new memory buffer;

[0033] A1.4, Current thread generates FH i The operation schedules the request, inserts the request into the Compaction task queue, and returns to step 1 to receive the next request req;

[0034] A1.5, Background threads use the Compaction scheduler to execute FH i The operation schedules the request, and then generates a compaction. i The operation scheduler requests the operation and inserts the request into the Compaction task queue, waiting for the Compaction scheduler to execute the Compaction. i The scheduling request for the operation;

[0035] B1. Update LSMT i Parameter S in the monitoring information i RW i RW, which is S i Increase the size of the key-value data in the Write request (RW) i Increase the number of key-value data in the Write request, recalculate RW, and then return to step 1 to receive the next request req.

[0036] If the request req is a read request (GET), then execute steps A2-D2:

[0037] A2. Update LSMT i The parameter RW in the monitoring information i RW, or RW i Increase the number of keys in the read request GET and recalculate the RW value;

[0038] B2, at LSMT i memory buffer MT i Search for the key in the read request GET request. If found, return the corresponding value; otherwise, execute C2.

[0039] C2, at LSMT i Immutable memory buffer (IMT) i Search for the key in the read request GET request. If found, return the corresponding value; otherwise, execute D2.

[0040] D2, from LSMT i The search begins at the first level of the disk component's structure, looking for the key in the read request Get. If the key is found in any level j structure, the corresponding value is returned. Otherwise, the search continues at the next level until the P-th level structure is searched. If the key is not found, null is returned.

[0041] Step 5: Process Scan requests in the MLSMT structure, and execute steps A3-C3:

[0042] A3. The current thread creates several iterators for each log structure merge tree in MLSMT, divides all iterators into G iterator groups, and combines each iterator group into a MergeIter iterator.

[0043] B3. Use multi-threading to process the seek operation of each MergeIter iterator in parallel, that is, each MergeIter iterator points to the position of the first key1 in the scan request Scan that is greater than or equal to the position of the key1 in the scan request Scan.

[0044] C3. Use heap sort to sort the keys at all positions pointed to by the MergeIter iterators in ascending order, and determine the smallest key. min Check if the value is greater than key2. If it is, return to step 1 to receive the next request req; otherwise, output the smallest key key. min and key min The corresponding MergeIter iterator is shifted one position in ascending order of key size, updating the key. min The key size of the corresponding Mergeiter iterator is then used to repeat C3.

[0045] The key-value storage method for the multi-log structure merging tree described in this invention is also characterized in that step 2 includes:

[0046] Step 2.1, according to LSMT i Monitoring information and calculation of split indicators Where P1 and P2 represent the weight ratio of each part, respectively;

[0047] Step 2.2, Determine F i <F it Check if the condition is met. If it is, proceed to step 2.3; otherwise, proceed from LSMT. i A new log structure, LSMT, is split from the middle. i’ and LSMT i FM, a two-dimensional metadata array i Make a copy as the split log structure merge tree LSMT i’ Metadata array FM i’ And update N to N+1;

[0048] Step 2.3: After assigning i+1 to i, return to step 2.1 until i>N, then assign 1 to i and return to step 2.1 to execute.

[0049] In step D2, the key in the read request Get is retrieved according to the following steps:

[0050] D2.1, Let the currently searched LSMTi The number of layers is L ij ;

[0051] If L ij For the first layer L i1 Then, FM files are traversed in descending order of their SST file numbers. i The first layer L in the array i1 In the SST files, find all SST file numbers num that satisfy the conditions in the read request Get where key ≥ keymin and key ≤ keymax, and execute D2.2;

[0052] If L ij For the second layer L i2 To the P-th layer L iP One of the layers, then according to FM i Lth in the array ij The maximum key value (key_max) and minimum key value (key_min) of the layer elements are obtained using a binary search algorithm in FM. i The Lth array ij In the layer, find the SST file number num where key_min ≤ key size ≤ key_max, and execute D2.2;

[0053] D2.2 Determine if the SST file corresponding to the SST file number num is in LSMT i If it is in the current LSMT, then it is in the current LSMT. i Find the SST file corresponding to num in the middle; otherwise, split the LSMT file from the middle. i Log structure merge tree LSMT s Find the SST file corresponding to num in the search;

[0054] D2.3. In the SST file corresponding to num, use the binary search algorithm to find the key in the parameters of the read request GET.

[0055] Step A3 includes:

[0056] A3.1, respectively for LSMT i MT i IMT i Create a memory buffer iterator MTIter i ,IMTIter i ;

[0057] A3.2, any i-th LSMT in all log structure merge trees i In the middle, execute A3.2.1-A3.2.2;

[0058] A3.2.1, from the Lth i1Find the SST files in the layer structure that overlap with the range (key1, key2) in the Scan request parameters, and create an iterator SSTIter for each matching SST file;

[0059] A3.2.2, Establish each L layer in the P-layer structure ij iterator LIter ij j>1;

[0060] A3.3. Using the number of SST files traversed by the iterator as the weight, divide all the created iterators into G iterator groups according to their weights, so that the sum of the weights of each iterator group is uniform; thereby merging all iterators in each group into a MergeIter iterator; where G is determined by the number of background threads.

[0061] The Compaction scheduler in A1.2 executes FH according to the following steps. i The scheduling request for the operation;

[0062] A1.2.1 Generating LSMT i The new SST file for the first layer structure i1new ;

[0063] A1.2.2, IMT i The key-value data in the file is written to the SST file according to the SST file format. i1new ;

[0064] A1.2.3 Record SST i1new Metadata information: SST i1new Document number, SST i1new The file size, minimum key value, and maximum key value will be used to set SST. i1new The file's reference count is set to 0, and the garbage collection flag is set to false;

[0065] A1.2.4, SST i1new File metadata information is inserted into LSMT i FM i In the array.

[0066] 10. The key-value storage method for a multi-log structure merging tree according to claim 5, characterized in that the Compaction scheduler in A1.5 executes Compaction according to the following steps. i The scheduling request for the operation;

[0067] A1.5.1, Let L ij For Compaction i The operation requires processing of LSMTi The layer;

[0068] If Compaction i The operation is MinC i The operation involves processing LSMT. i The first layer L i1 L ij =L i1 ;

[0069] If Compaction i The operation is MaxC i For the operation, select LSMT. i Size of all SSTs in each layer ij With respect to the size threshold LimitSize of the corresponding layer ij The layer with the largest ratio is assigned to L. ij ;

[0070] A1.5.2, According to LSMT i FM in i array, Lth ij The compaction position pointer of the layer structure is used to find the Lth layer. ij The layer structure needs to merge the SST file numbers, and then in the Lth... i(j+1) Layer structure lookup and Lth ij The layered SST file has the number of all SST files with a range of overlapping keys;

[0071] A1.5.3. Based on the searched number, determine the Lth... ij Layer, Lth i(j+1) Is each SST file of the layer in LSMT? i If it is in the current LSMT, then it is in the current LSMT. i Find the corresponding SST file in the middle; otherwise, split the LSMT file. i Log structure merge tree LSMT s Find the SST file corresponding to num in the search;

[0072] A1.5.4, in LSMT i The Lth i(j+1) Create a new SST file in the layer, and put the Lth layer... ij Layer, Lth i(j+1) Key-value data found in all SST files in the layer<key,value> Read the data and sort it by key. Loop through the sorted keys and determine if each key belongs to a Log Merge Tree (LSMT) based on its hash value. i If it belongs to the category, write the key-value data corresponding to the key to the LSMT.i Lth i(j+1) In the new SST file within the layer, otherwise, skip the corresponding key;

[0073] A1.5.5, Record the Lth i(j+1) Metadata information of new SST files in the layer to FM i In the array, and reduce FM. i In the array in LSMT i The reference count of the corresponding file metadata element is found in the LSMT. s The garbage collection flag of the metadata element of the corresponding SST file was found to be true.

[0074] The present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a program that supports the processor in executing the key-value storage method, and the processor is configured to execute the program stored in the memory.

[0075] The present invention discloses a computer-readable storage medium on which a computer program is stored, wherein the computer program is executed by a processor to perform the steps of the key-value storage method.

[0076] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0077] 1. This invention calculates the hash value of the key, takes the modulo of the hash value according to the set maximum number of log structure merging trees, and selects the corresponding log structure merging tree according to the binary value of the hash value after taking the modulo. This realizes data tree storage, reduces the number of layers of a single log structure merging tree, thereby reducing data read / write amplification and improving the system's read / write performance.

[0078] 2. The present invention designs a compaction scheduler, which schedules and processes the flush and compaction operations of multiple log structure merging trees in parallel according to priority. This can make full use of the system's hardware resources, thereby accelerating data writing and reading and enhancing the parallelism of the system.

[0079] 3. This invention retains the hierarchical ordered structure of the log structure merging tree, groups iterators evenly according to their weights, processes the seek operations of the iterators in parallel, and uses the heap sort algorithm to process the data traversal operations. This effectively avoids the problem that hash structure systems do not support scanning operations or have poor scanning performance, thus ensuring the integrity of the system functions.

[0080] 4. This invention monitors the status of the log structure merging tree through an LSMT information monitor, saves the hash binary value of the key after calculation in the SST file, and uses a method of copying metadata information and performing compaction operations to smoothly transition data. This achieves the splitting of the log structure merging tree, overcomes the problem of large performance fluctuations when migrating data in the hash structure, and thus ensures that the overall read and write performance of the system is not significantly affected, thereby enhancing the scalability of the system. Attached Figure Description

[0081] Figure 1 This is a diagram illustrating the overall structure of the key-value storage method based on a multi-log structure merging tree according to the present invention.

[0082] Figure 2 This is a structural diagram of a single log structure merging tree according to the present invention;

[0083] Figure 3 This is a diagram illustrating the process of reading file data using a multi-log structure merging tree, as described in this invention.

[0084] Figure 4 This is a diagram illustrating the Compapction operation process of the present invention.

[0085] Figure 5 This is a diagram illustrating the key-value data entry format of the SST file in this invention. Detailed Implementation

[0086] To describe the technical solution of the present invention more clearly, the specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. In this specific embodiment, the number of log structure merging trees can be dynamically changed with the state of the log structure merging trees.

[0087] In this embodiment, a key-value storage method based on multiple log structure merging trees is applied to a single storage node configured with an MLSMT structure. The MLSMT structure includes: a compaction scheduler, an LSMT information monitor, and N log structure merging trees {LSMT1, LSMT2, ..., LSMT...}. i ,…,LSMT N}, where LSMT i This represents the merging tree of the i-th log structure; 1≤i≤N; for example... Figure 1 As shown, Figure 1The document presents a structure diagram with three LSMTs. The specific descriptions of each part are as follows: When the storage engine receives a read / write request, it first hashes and calculates the log structure merge tree LSMT to which the key belongs. Then, it inserts the key-value data in the request into the corresponding LSMT or reads the key-value data from the corresponding LSMT and updates the status information of the corresponding LSMT. The LSMT information monitor then monitors the status information of all LSMTs to determine whether tree splitting is necessary. The compaction scheduler completes the flush and compaction scheduling requests for each LSMT and writes data from one LSMT to the LSMT it splits out.

[0088] The i-th log structure merge tree LSMT i It includes a disk component and a memory component; the following content is implemented or modified from common storage engines such as LevelDB and RocksDB for log structure merge trees. This example is based on the LevelDB storage engine. Figure 2 As shown, you can see the components and information contained in a single LSMT.

[0089] LSMT i The disk component contains P layers {L i1 ,L i2 ,…,L ij ,…,L ip}, where L ij Let P represent the j-th level structure of the i-th log structure merge tree, where 1 ≤ j ≤ P; L represents the j-th level structure. ij Including Q ij SST files

[0090] {SST ij1 SST ij2 ,…,SST ijk ,…,SST ijQij}, where SST ijk Represents the LSMT of the i-th log structure merge tree. i The j-th layer structure L ij The k-th SST file, 1≤k≤Q ij ;

[0091] The k-th SST file SST ijk Contains several key-value data<key,value> According to the k-th SST file SST ijk The values ​​of all key data in the SST file are used to obtain the k-th SST file. ijk key range (keymin) ijk keymax ijk ), where keymin ijkThis represents the k-th SST file. ijk The minimum value of the key, keymax ijk This represents the k-th SST file. ijk The maximum value of the bonds, excluding the first layer structure L i1 Apart from the overlapping key ranges between files in the same layer, the key ranges between files in other layers do not overlap and are arranged in an ordered manner.

[0092] Let the k-th SST file be SST ijk The disk space occupied by all key-value data in the SST file is denoted as the k-th SST file. ijk Size ijk The j-th layer structure L ij Total size of all SST files (Size) ij The threshold is denoted as LimitSize ij If j>1, the number of SST files in the first-level structure has a threshold LimitNum. i The threshold value here is LimitNum. i This is the default size of 4 used by LevelDB;

[0093] Except for the first layer structure L i1 In addition, the (j+1)th layer structure L i(j+1) LimitSize for all SST files i(j+1) It is the j-th layer structure L ij LimitSize for all SST files ij q times, i.e., LimitSize i(j+1) =q×LimitSize ij Here, q is the default size of 10 in LevelDB.

[0094] The i-th log structure merge tree LSMT i The memory components include the memory buffer MT. i Immutable buffers (IMT) i MTi and IMTi correspond to MemTable and ImmutableMemTable in LevelDB, respectively, and FM is a two-dimensional metadata management array. i , and FM i Includes P×Q ijEach element is a structure consisting of the SST file number, the size of the SST file, the reference count of the SST file, the minimum value of the key, the maximum value of the key, and a GC flag field indicating whether it is garbage collected. Here, the FileMetaData array stored in memory in LevelDB has been modified to add a GC flag field indicating whether it is garbage collected. This is used to indicate that the data that needs to be split to another LSMT has been read and can be deleted during the next garbage collection of the current LSMT.

[0095] The i-th log structure merge tree LSMT i The included operation is: FH i The FHi operation, implemented in LevelDB, refers to the disk flushing and compaction operations performed when writing to MemTable. i Operations; among which, Compaction i Operations are divided into MinC i Operations, MaxC i Operation; Compaction i The operation refers to the compaction operation implemented in LevelDB, which means merging from the j-th level structure L. ij Select several SST files, starting from the (j+1)th level structure L i(j+1) Selecting from L ij After having several SST files with overlapping key ranges, sort the key-value data of all SST files according to the key, then merge and divide them into new SST files and write them to L. i(j+1) In the middle, it is divided into MinC according to different numbers of layers. i Operations, MaxC i operate;

[0096] FH i Operation refers to memory buffer MT i The key-value data is written to the first-level structure L of the disk component in the form of an SST file. i1 middle;

[0097] MinC i It refers to the first layer structure L i1 and the second layer structure L i2 The selected SST files are sorted by key, merged, and then divided into new SST files, which are then written to L. i2 ;

[0098] MaxC i It refers to the j-th layer structure L ij and the (j+1)th layer structure L ij+1 The selected SST files are sorted by key, merged, and then divided into new SST files, which are then written to L.ij+1 j>1;

[0099] The LSMT monitor is used to obtain monitoring information from N log structure merge trees and calculate the split metric, let F i For the i-th log structure merge tree LSMT i The splitting index, F it F is the threshold for the i-th splitting index; it The default threshold is set to 70%, and technical personnel can adjust the threshold by setting different thresholds according to actual conditions. The i-th log structure merge tree LSMT i The monitoring information includes: LSMT i The total disk space occupied by all SST files S i Disk space size threshold S it Read / write request count RW i , and the total number of read and write requests RW for all log structure merged trees;

[0100] S i That is, LSMT i The total size of all SST files in the disk space threshold S it The default setting is 50G. Technicians can set different sizes according to the actual situation. It is recommended to determine the threshold of a single LSMT based on the total data volume of the system and the number of LSMTs. The size threshold of each LSMT can also be set to different values.

[0101] The key-value storage method is performed according to the following steps:

[0102] Step 1: The current thread uses the MLSMT structure to receive read / write data requests (req) from external sources and determines the type of request (req):

[0103] If the request `req` is a write request, then proceed to step 2, where the parameters in the write request include several key-value pairs of data to be written.<key,value> ;

[0104] If the request req is a read request (GET), then proceed to step 2; wherein, the parameters in the read request (GET) include several key data items.

[0105] The amount of data included in the parameters of the Write and Get requests may vary depending on the storage engine; for example, LevelDB only supports single reads.

[0106] If the request req is a scan request Scan, then proceed to step 5; wherein, the parameters in the scan request Scan include the key data range (key1, key2);

[0107] Step 2: The background thread calculates and splits the split index of the log structure merge tree;

[0108] Step 2.1, according to LSMT i Monitoring information and calculation of split indicators Where P1 and P2 represent the weight ratio of each part; the first part of the above formula describes whether the current LSMT space size exceeds the set threshold, and the second part describes whether the current LSMT processes a large number of read and write requests. i When the ratio of read / write requests reaches twice that of read / write (RW / N), a single LSMT handles a larger number of read / write requests. P1 and P2 are set to 0.5 and 0.25 by default, respectively, while the splitting metric threshold F... it The default setting is 80%. When the threshold is exceeded, a split operation is considered to be performed. Technicians can adjust the above weight ratio and the size of the split index threshold according to the actual situation.

[0109] Step 2.2, Determine F i <F it Check if the condition is met. If it is, proceed to step 2.3; otherwise, proceed from LSMT. i A new log structure, LSMT, is split from the middle. i’ and LSMT i FM, a two-dimensional metadata array i Make a copy as the split log structure merge tree LSMT i’ Metadata array FM i’ And update N to N+1; during the split, the current LSMT should be paused. i LSMT i’ Write requests to prevent those belonging to the split LSMT i’ Data inserted into LSMT i First, the metadata information FM is stored in memory. i Copy as FM i’ Then MT i IMT i China belongs to LSMT i’ Data written to MT i’ Then resume writing requests.

[0110] Step 2.3: After assigning i+1 to i, return to step 2.1 until i>N, then assign 1 to i and return to step 2.1 to execute.

[0111] Step 3: The current thread performs a hash operation on the key in the request req to obtain the corresponding hash value. The hash value is then moduloed by the threshold mod to obtain the intermediate value hv, where mod represents the threshold for the number of merged trees in the log structure. The initial value of mod is set to 128, which means that the intermediate value hv can be represented by one byte. Technicians can adjust it according to actual needs.

[0112] Step 4: The current thread calculates the minimum number of binary bits n required to represent N, calculates the n-bit value of the tail of hv based on n, and then calculates the sequence number i of the log structure merge tree based on the n-bit value, thus ensuring that the key in the request req belongs to LSMT. i Assuming the current system is as follows: Figure 1 As shown, with 3 LSMTs, the minimum number of binary bits required for N is 2, that is, 00, 01, and 10 represent the current three LSMTs. If the last n bits of hv are one of 00, 01, and 10, then the key belongs to the LSMT with the corresponding sequence number. If the last n bits of hv are 11, and 11 is found to be greater than the maximum sequence number of the current LSMT, 10, then the LSMTs that are the same except for the highest bit are searched, that is, the key belongs to the LSMT with sequence number 01.

[0113] If the request req is a write request, then execute steps A1 and B1:

[0114] A1. Write the key-value data corresponding to the parameters in the Write request to MT. i And proceed according to steps A1.1-A1.5;

[0115] A1.1 Determine the LSMT of the i-th log structure merge tree. i memory buffer MT i Size M i Is it greater than the threshold Mmax? If it is, proceed to step A1.2; otherwise, change the parameter.<key,value> The corresponding key-value data is written to LSMT i Memory cache MT i And execute B1;

[0116] A1.2 Determine the current log structure merge tree (LSMT) i IMT i If the value is not empty, wait until the Compaction scheduler executes FH. i The scheduling request for the operation, thereby placing the buffer IMT i All data is written to the disk, enabling IMT. i If empty, execute A1.3;

[0117] In this embodiment, the Compaction scheduler in A1.2 executes FH according to the following steps. i The scheduling request for the operation;

[0118] A1.2.1 Generating LSMT i The new SST file for the first layer structure i1new ;

[0119] A1.2.2, IMT i The key-value data in the file is written to the SST file according to the SST file format. i1new ;

[0120] A1.2.3 Record SST i1new Metadata information: SST i1new Document number, SST i1new The file size, minimum key value, and maximum key value will be used to set SST. i1new The file's reference count is set to 0, and the garbage collection flag is set to false;

[0121] A1.2.4, SST i1new File metadata information is inserted into LSMT i FM i In the array.

[0122] A1.3, IMT i Pointer to the current memory buffer MT i And generate a new empty memory buffer, making MT i Points to a new memory buffer;

[0123] The process of executing a Write request and writing data to the memory buffer described above is the same as in LevelDB. When MT i If full, switch to IMT. i Then, a flush operation scheduling request is triggered, which is scheduled and executed by the compaction scheduler.

[0124] A1.4, Current thread generates FH i The operation schedules the request, inserts the request into the Compaction task queue, and returns to step 1 to receive the next request req;

[0125] A1.5, Background threads use the Compaction scheduler to execute FH i The operation schedules the request, and then generates a compaction. i The operation scheduler requests the operation and inserts the request into the Compaction task queue, waiting for the Compaction scheduler to execute the Compaction. iThe scheduling request for the operation;

[0126] The background thread continuously searches for tasks in the Compaction task queue and schedules them in descending order of priority, i.e., FH. i MinC i MaxC i LSMT currently executing read / write requests i Compaction i The LSMT request has already been executed. i’ Compaction i Compression tasks of different LSMTs can be executed in parallel without considering data overlap, making full use of hardware resources such as disk bandwidth and thus improving write performance many times over.

[0127] The compaction scheduler in A1.5 executes compaction as follows: i The scheduling request for the operation;

[0128] Calculate LSMT i Is the number of SST files in the first layer greater than LimitNum? i If it is greater than, then let Compaction i Operation is MinC i If the operation fails, execute A1.5.1; otherwise, calculate the size of all SST files for each layer from layer 2 to layer P. ij With respect to the size threshold LimitSize of the corresponding layer ij If the ratio of the two layers is greater than 1, then execute A1.5.1; otherwise, return to execute the next scheduling request.

[0129] A1.5.1, Let L ij For Compaction i The operation requires processing of LSMT i The layer;

[0130] If Compaction i The operation is MinC i The operation involves processing LSMT. i The first layer L i1 L ij =L i1 ;

[0131] If Compaction i The operation is MaxC i For the operation, select LSMT. i Size of all SSTs in each layerij With respect to the size threshold LimitSize of the corresponding layer ij The layer with the largest ratio is assigned to L. ij ;

[0132] MinC i Operations and MaxC i The only difference between the operations is the number of layers processed. Since the key ranges of the SST files in the first layer overlap, more files are usually read, and the operation needs to have a higher priority than the compaction operation in the lower layer.

[0133] A1.5.2, According to LSMT i FM in i array, Lth ij The compaction position pointer of the layer structure is used to find the Lth layer. ij The layer structure needs to merge the SST file numbers, and then in the Lth... i(j+1) Layer structure lookup and Lth ij The layered SST files have the numbers of all SST files with overlapping keys; the process of selecting merged SST files through compaction is the same as in LevelDB.

[0134] A1.5.3. Based on the searched number, determine the Lth... ij Layer, Lth i(j+1) Is each SST file of the layer in LSMT? i If it is in the current LSMT, then it is in the current LSMT. i Find the corresponding SST file in the middle; otherwise, split the LSMT file. i Log structure merge tree LSMT s Find the SST file corresponding to num in the search;

[0135] Similar to searching for a file in a GET request, the file may exist in a branched tree LSMT. s Or split into LSMT s In the tree, it is necessary to search according to the SST position.

[0136] A1.5.4, in LSMT i The Lth i(j+1) Create a new SST file in the layer, and put the Lth layer... ij Layer, Lth i(j+1) Key-value data found in all SST files in the layer<key,value> Read the data and sort it by key. Loop through the sorted keys and determine if each key belongs to a Log Merge Tree (LSMT) based on its hash value. i If it belongs to the category, write the key-value data corresponding to the key to the LSMT.i Lth i(j+1) In the new SST file within the layer, otherwise, skip the key;

[0137] Similar to the compaction operation in LevelDB, it is necessary to... ij Layer, L i(j+1) Read the layer files, sort them by key size, merge them, and then divide them into new SST files to write to L. i(j+1) layer.

[0138] After using the MLSMT structure, the SST files that require Compression may come from different tree LSMTs. i LSMT s Read from the current LSMT, but only need to read the data belonging to the current LSMT. i Key-value data written to LSMT i Medium is sufficient, the rest is LSMT s The data remains unchanged in the original files. This approach is less efficient, but it reads fewer SST files each time, thus having a smaller impact on read / write requests. For example... Figure 4 As shown, during compaction, it is necessary to place the data located at LSMT. i SST file i13 Located in LSMT s SST file i12 SST i22 Read out the data, sort it, and divide it into new SST files. i23 SST i24 SST i25 Write L i(j+1) layer.

[0139] Besides the methods mentioned above, there is another approach: after reading all key-value data, it belongs to LSMT. i Data written to LSMT i L i(j+1) Layer, belonging to LSMT s Data written back to LSMT s L in s(j+1) Layers, however, when reading files that require Compression, LSMT needs to be... s New files with overlapping ranges are also read. This approach is more efficient, but it reads more SST files each time, thus having a greater impact on read / write requests.

[0140] Furthermore, the log merge tree (LSMT) method for determining the ownership of key-value data requires calculating the binary value 'n' after taking the modulo of the hash value of the key, which wastes a significant amount of time each time. For example... Figure 5As shown, the computational overhead mentioned above can be resolved by modifying the format of the key-value data entries stored in the SST file and adding some space to record n-bit binary values.

[0141] A1.5.5, Record the Lth i(j+1) Metadata information of new SST files in the layer to FM i In the array, reduce FM i In the array in LSMT i The reference count of the corresponding file metadata element is found in the LSMT and will be displayed therein. s The garbage collection flag of the metadata element of the corresponding SST file was found to be true.

[0142] like Figure 4 As shown, after writing back the new SST file and recording the metadata information, the old SST file needs to be... i13 SST i12 SST i22 The reference count is decremented by 1 when LSMT i During garbage collection, the reference count can be used to determine whether to SST old files. i13 Delete. Remove the old file SST. i12 SST i22 On FM s In LSMT, the garbage collection flag GC is set to true. s During garbage collection, the GC uses reference counting and flags to determine whether to save old files to static files (SST). i12 SST i22 delete.

[0143] From the above process, LSMT can be... s Smooth transition of data from LSMT i In this way, it will not have a drastic impact on the overall read and write performance of the system, unlike copying and migrating data.

[0144] B1. Update LSMT i Parameter S in the monitoring information i RW i RW, which is S i Increase the size of the key-value data in the Write request (RW) i Increase the number of key-value data in the Write request, recalculate RW, and then return to step 1 to receive the next request req.

[0145] If the request req is a read request (GET), then execute steps A2-D2:

[0146] A2. Update LSMT iThe parameter RW in the monitoring information i RW, or RW i Increase the number of keys in the read request GET and recalculate the RW value;

[0147] B2, at LSMT i memory buffer MT i Search for the key in the read request GET request. If found, return the corresponding value; otherwise, execute C2.

[0148] C2, at LSMT i Immutable memory buffer (IMT) i Search for the key in the read request GET request. If found, return the corresponding value; otherwise, execute D2.

[0149] D2, from LSMT i The search begins at the first level of the disk component's structure, looking for the key in the read request Get. If the key is found in any level j structure, the corresponding value is returned. Otherwise, the search continues at the next level until the P-th level structure is searched. If the key is not found, null is returned.

[0150] In step D2, the key in the read request GET is retrieved as follows:

[0151] D2.1, Let the currently searched LSMT i The number of layers is L ij ;

[0152] If L ij For the first layer L i1 Then, FM files are traversed in descending order of their SST file numbers. i The first layer L in the array i1 In the SST files, find all SST file numbers num that satisfy the key in the read request Get where key≥keymin and key≤keymax, and execute D2.2; similar to LevelDB, the key ranges of the first-level files overlap, so it is necessary to search for all SST files that satisfy the condition in the order of newest to oldest.

[0153] If L ij For the second layer L i2 To the P-th layer L iP One of the layers, then according to FM i Lth in the array ij The maximum key value (key_max) and minimum key value (key_min) of the layer elements are obtained using a binary search algorithm in FM. i The Lth array ijIn the layer, find the SST file number num where key_min ≤ key size ≤ key_max, and execute D2.2;

[0154] D2.2 Determine if the SST file corresponding to the SST file number num is in LSMT i If it is in the current LSMT, then it is in the current LSMT. i Find the SST file corresponding to num in the middle; otherwise, split the LSMT file from the middle. i Log structure merge tree LSMT s Find the SST file corresponding to num in the search;

[0155] In the current LSMT i FM i The array can find the SST metadata information corresponding to num, but LSMT i Possibly by other tree LSMT s Since it originated from a split, the SST file may exist in the LSMT. s Or split into LSMT s In the log structure merge tree, it is necessary to search in different LSMTs based on the SST position. For example... Figure 3 As shown, LSMT i By LSMT s It was obtained by splitting, and then a new file SST was added. i13 When searching, directly from LSMT i You can read it from the file, and search for the file SST. i22 Then you need to go to LSMT. s Read the file from the middle.

[0156] D2.3. In the SST file corresponding to num, use the binary search algorithm to find the key in the parameters of the read request GET.

[0157] Step 5: Process Scan requests in the MLSMT structure, and execute steps A3-C3:

[0158] A3. The current thread creates several iterators for each log structure merge tree in MLSMT, divides all iterators into G iterator groups, and combines each iterator group into a MergeIter iterator.

[0159] In LevelDB, Scan requests are implemented by creating various iterators from components such as SST files in the system, merging the iterators, and then traversing the data using the iterators. However, when using the MLSMT structure, the number of iterators in each LSMT decreases, but the overall number increases, leading to a decrease in Scan request performance. In this case, it is possible to group the iterators in the system and process them in parallel to speed up Scan request processing.

[0160] A3.1, respectively for LSMT i MT i IMT i Create a memory buffer iterator MTIter i ,IMTIter i ;

[0161] A3.2, any i-th LSMT in all log structure merge trees i In the middle, execute A3.2.1-A3.2.2;

[0162] A3.2.1, from the Lth i1 Find the SST files in the layer structure that overlap with the range (key1, key2) in the Scan request parameters, and create an iterator SSTIter for each matching SST file;

[0163] A3.2.2, Establish each L layer in the P-layer structure ij iterator LIter ij j>1;

[0164] The various iterators established above are the same as those in LevelDB: the iterator for the memory buffer, the iterator for the relevant SST files in Level 1, and the iterator for each level from Level 2 onwards.

[0165] A3.3. Using the number of SST files traversed by the iterator as the weight, divide all the created iterators into G iterator groups according to their weights, so that the sum of the weights of each iterator group is uniform; thereby merging all iterators in each group into a MergeIter iterator; where G is determined by the number of background threads.

[0166] The weight of an iterator is determined by the time spent processing its Seek operation. Therefore, the more files that need to be queried, the more time is spent. Hence, generally, the deeper the iterator, the higher its weight. i(j+1) >Liter ij >SSTIter,IMTIter i ,MTIter i However, since binary search can greatly reduce the query time, the iterator can be directly divided equally according to the number of items.

[0167] B3. Use multi-threading to process the seek operation of each MergeIter iterator in parallel, that is, each MergeIter iterator points to the position of the first key1 in the scan request Scan that is greater than or equal to the position of the key1 in the scan request Scan.

[0168] C3. Use heap sort to sort the keys at all positions pointed to by the MergeIter iterators in ascending order, and determine the smallest key. min Check if the value is greater than key2. If it is, return to step 1 to receive the next request req; otherwise, output the smallest key key. min and key min The corresponding MergeIter iterator is shifted one position in ascending order of key size, updating the key. min The key size of the corresponding Mergeiter iterator is then used to repeat C3.

[0169] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0170] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

Claims

1. A key-value storage method based on a multi-log structure merging tree, characterized by: This is applied to a single storage node configured with an MLSMT structure, which includes: a compaction scheduler, an LSMT information monitor, and N log structure merging trees {LSMT1, LSMT2, ..., LSMT...}. i ,…,LSMT N }, where LSMT i This represents the ith log structure merge tree; 1 ≤ i ≤ N; The i-th log structure merge tree LSMT i It includes a disk component and a memory component; LSMT i The disk component contains P layers {L i1 ,L i2 ,…,L ij ,…,L ip }, where L ij Let P represent the j-th level structure of the i-th log structure merge tree, where 1 ≤ j ≤ P; L represents the j-th level structure. ij Including Q ij A SST file {SST ij1 SST ij2 ,…,SST ijk ,…,SST ijQij }, where SST ijk Represents the LSMT of the i-th log structure merge tree. i The j-th layer structure L ij The k-th SST file, 1≤k≤Q ij ; The k-th SST file SST ijk Contains several key-value data<key,value> According to the k-th SST file SST ijk The values ​​of all key data in the SST file are used to obtain the k-th SST file. ijk key range (keymin) ijk keymax ijk ), where keymin ijk This represents the k-th SST file. ijk The minimum value of the key, keymax ijk This represents the k-th SST file. ijk The maximum value of the bonds, excluding the first layer structure L i1 Apart from the overlapping key ranges between files in the same layer, the key ranges between files in other layers do not overlap and are arranged in an ordered manner. Let the k-th SST file be SST ijk The disk space occupied by all key-value data in the SST file is denoted as the k-th SST file. ijk Size ijk The j-th layer structure L ij Total size of all SST files (Size) ij The threshold is denoted as LimitSize ij If j>1, the number of SST files in the first-level structure has a threshold LimitNum. i ; Except for the first layer structure L i1 In addition, the (j+1)th layer structure L i(j+1) LimitSize for all SST files i(j+1) It is the j-th layer structure L ij LimitSize for all SST files ij q times, i.e., LimitSize i(j+1) = q × LimitSize ij ; The i-th log structure merge tree LSMT i The memory components include the memory buffer MT. i Immutable buffers (IMT) i Two-dimensional metadata management array FM i , and FM i Includes P×Q ij Each element is a structure consisting of the SST file number, the size of the SST file, the reference count of the SST file, the minimum value of the key, the maximum value of the key, and a GC flag field indicating whether garbage collection is required. The i-th log structure merge tree LSMT i The included operation is: FH i Operation, Compaction i Operations; among which, Compaction i Operations are divided into MinC i Operations, MaxC i operate; FH i Operation refers to memory buffer MT i The key-value data is written to the first-level structure L of the disk component in the form of an SST file. i1 middle; MinC i It refers to the first layer structure L i1 and the second layer structure L i2 The selected SST files are sorted by key, merged, and then divided into new SST files, which are then written to L. i2 ; MaxC i It refers to the j-th layer structure L ij and the (j+1)th layer structure L ij+1 The selected SST files are sorted by key, merged, and then divided into new SST files, which are then written to L. ij+1 j>1; The LSMT monitor is used to obtain monitoring information of N log structure merge trees and calculate the split index, let F i For the i-th log structure merge tree LSMT i The splitting index, F it The threshold for the i-th split metric; the LSMT of the i-th log structure merge tree. i The monitoring information includes: LSMT i The total disk space occupied by all SST files S i Disk space size threshold S it Read / write request count RW i , and the total number of read and write requests RW for all log structure merged trees; The key-value storage method is performed according to the following steps: Step 1: The current thread uses the MLSMT structure to receive external read / write data requests (req) and determines the type of the request (req): If the request `req` is a write request (`Write`), then proceed to step 2, wherein the parameters in the write request include several key-value pairs of data to be written.<key,value> ; If the request req is a read request (Get), then proceed to step 2; wherein, the parameters in the read request (Get) include several key data items (key); If the request req is a scan request Scan, then proceed to step 5; wherein the parameters in the scan request Scan include the key data range (key1, key2); Step 2: The background thread calculates and splits the split index of the log structure merge tree; Step 2.1, according to LSMT i Monitoring information and calculation of split indicators Where P1 and P2 represent the weight ratio of each part, respectively; Step 2.2, Determine F i <F it Check if the condition is met. If it is, proceed to step 2.3; otherwise, proceed from LSMT. i A new log structure, LSMT, is split from the middle. i’ and LSMT i FM, a two-dimensional metadata array i Make a copy as the split log structure merge tree LSMT i’ Metadata array FM i’ And update N to N+1; Step 2.3: After assigning i+1 to i, return to step 2.1 until i>N, then assign 1 to i and return to step 2.1 to execute. Step 3: The current thread performs a hash operation on the key in the request req to obtain the corresponding hash value. The hash value is then moduloed by the threshold mod to obtain the intermediate value hv, where mod represents the threshold for the number of merged trees in the log structure. Step 4: The current thread calculates the minimum number of binary bits n required to represent N, calculates the n-bit value of the tail of hv based on n, and then calculates the sequence number i of the log structure merge tree based on the n-bit value, thus ensuring that the key in the request req belongs to LSMT. i ; If the request req is a write request, then execute steps A1 and B1: A1. Write the key-value data corresponding to the parameters in the Write request to MT. i And proceed according to steps A1.1-A1.5; A1.1 Determine the LSMT of the i-th log structure merge tree. i memory buffer MT i Size M i Is it greater than the threshold Mmax? If it is, proceed to step A1.2; otherwise, change the parameter.<key,value> The corresponding key-value data is written to LSMT i Memory cache MT i And execute B1; A1.2 Determine the current log structure merge tree (LSMT) i IMT i If the value is not empty, wait until the Compaction scheduler executes FH. i The scheduling request for the operation, thereby placing the buffer IMT i All data is written to the disk, enabling IMT. i If empty, execute A1.3; A1.3, IMT i Pointer to the current memory buffer MT i And generate a new empty memory buffer, making MT i Points to a new memory buffer; A1.4, Current thread generates FH i The operation schedules the request, inserts the request into the Compaction task queue, and returns to step 1 to receive the next request req; A1.5, Background threads use the Compaction scheduler to execute FH i The operation schedules the request, and then generates a compaction. i The operation scheduler requests the operation and inserts the request into the Compaction task queue, waiting for the Compaction scheduler to execute the Compaction. i The scheduling request for the operation; B1. Update LSMT i Parameter S in the monitoring information i RW i RW, which is S i Increase the size of the key-value data in the Write request (RW) i Increase the number of key-value data in the Write request, recalculate RW, and then return to step 1 to receive the next request req. If the request req is a read request (GET), then execute steps A2-D2: A2. Update LSMT i The parameter RW in the monitoring information i RW, or RW i Increase the number of keys in the read request GET and recalculate the RW value; B2, at LSMT i memory buffer MT i Search for the key in the read request GET request. If found, return the corresponding value; otherwise, execute C2. C2, at LSMT i Immutable memory buffer (IMT) i Search for the key in the read request GET request. If found, return the corresponding value; otherwise, execute D2. D2, from LSMT i The search begins at the first level of the disk component's structure, looking for the key in the read request Get. If the key is found in any level j structure, the corresponding value is returned. Otherwise, the search continues at the next level until the P-th level structure is searched. If the key is not found, null is returned. Step 5: Process Scan requests in the MLSMT structure, and execute steps A3-C3: A3. The current thread creates several iterators for each log structure merge tree in MLSMT, divides all iterators into G iterator groups, and combines each iterator group into a MergeIter iterator. B3. Use multi-threading to process the seek operation of each MergeIter iterator in parallel, that is, each MergeIter iterator points to the position of the first key1 in the scan request Scan that is greater than or equal to the position of the key1 in the scan request Scan. C3. Use heap sort to sort the keys at all positions pointed to by the MergeIter iterators in ascending order, and determine the smallest key. min Check if the value is greater than key2. If it is, return to step 1 to receive the next request req; otherwise, output the smallest key key. min and key min The corresponding MergeIter iterator is shifted one position in ascending order of key size, updating the key. min The key size of the corresponding Mergeiter iterator is then used to repeat C3.

2. The key-value storage method for a multi-log structure merging tree according to claim 1, characterized in that, In step D2, the key in the read request Get is retrieved according to the following steps: D2.1, Let the currently searched LSMT i The number of layers is L ij ; If L ij For the first layer L i1 Then, FM files are traversed in descending order of their SST file numbers. i The first layer L in the array i1 In the SST files, find all SST file numbers num that satisfy the conditions in the read request Get where key ≥ keymin and key ≤ keymax, and execute D2.2; If L ij For the second layer L i2 To the P-th layer L iP One of the layers, then according to FM i Lth in the array ij The maximum key value (key_max) and minimum key value (key_min) of the layer elements are obtained using a binary search algorithm in FM. i The Lth array ij In the layer, find the SST file number num where key_min ≤ key size ≤ key_max, and execute D2.2; D2.2 Determine if the SST file corresponding to the SST file number num is in LSMT i If it is in the current LSMT, then it is in the current LSMT. i Find the SST file corresponding to num in the middle; otherwise, split the LSMT file from the middle. i Log structure merge tree LSMT s Find the SST file corresponding to num in the search; D2.

3. In the SST file corresponding to num, use the binary search algorithm to find the key in the parameters of the read request GET.

3. The key-value storage method for a multi-log structure merging tree according to claim 2, characterized in that, Step A3 includes: A3.1, respectively for LSMT i MT i IMT i Create a memory buffer iterator MTIter i ,IMTIter i ; A3.2, any i-th LSMT in all log structure merge trees i In the middle, execute A3.2.1-A3.2.2; A3.2.1, from the Lth i1 Find the SST files in the layer structure that overlap with the range (key1, key2) in the Scan request parameters, and create an iterator SSTIter for each matching SST file; A3.2.2, Establish each L layer in the P-layer structure ij iterator LIter ij j>1; A3.

3. Using the number of SST files traversed by the iterator as the weight, divide all the created iterators into G iterator groups according to their weights, so that the sum of the weights of each iterator group is uniform; thereby merging all iterators in each group into a MergeIter iterator; where G is determined by the number of background threads.

4. The key-value storage method for a multi-log structure merging tree according to claim 3, characterized in that, The Compaction scheduler in A1.2 executes FH according to the following steps. i The scheduling request for the operation; A1.2.1 Generating LSMT i The new SST file for the first layer structure i1new ; A1.2.2, IMT i The key-value data in the file is written to the SST file according to the SST file format. i1new ; A1.2.3 Record SST i1new Metadata information: SST i1new Document number, SST i1new The file size, minimum key value, and maximum key value will be used to set SST. i1new The file's reference count is set to 0, and the garbage collection flag is set to false; A1.2.4, SST i1new File metadata information is inserted into LSMT i FM i In the array.

5. The key-value storage method for a multi-log structure merging tree according to claim 4, characterized in that, The compaction scheduler in A1.5 executes compaction according to the following steps. i The scheduling request for the operation; A1.5.1, Let L ij For Compaction i The operation requires processing of LSMT i The layer; If Compaction i The operation is MinC i The operation involves processing LSMT. i The first layer L i1 L ij =L i1 ; If Compaction i The operation is MaxC i For the operation, select LSMT. i Size of all SSTs in each layer ij With respect to the size threshold LimitSize of the corresponding layer ij The layer with the largest ratio is assigned to L. ij ; A1.5.2, According to LSMT i FM in i array, Lth ij The compaction position pointer of the layer structure is used to find the Lth layer. ij The layer structure needs to merge the SST file numbers, and then in the Lth... i(j+1) Layer structure lookup and Lth ij The layered SST file has the number of all SST files with a range of overlapping keys; A1.5.

3. Based on the searched number, determine the Lth... ij Layer, Lth i(j+1) Is each SST file of the layer in LSMT? i If it is in the current LSMT, then it is in the current LSMT. i Find the corresponding SST file in the middle; otherwise, split the LSMT file. i Log structure merge tree LSMT s Find the SST file corresponding to num in the search; A1.5.4, in LSMT i The Lth i(j+1) Create a new SST file in the layer, and put the Lth layer... ij Layer, Lth i(j+1) Key-value data found in all SST files in the layer<key,value> Read the data and sort it by key. Loop through the sorted keys and determine if each key belongs to a Log Merge Tree (LSMT) based on its hash value. i If it belongs to the category, write the key-value data corresponding to the key to the LSMT. i Lth i(j+1) In the new SST file within the layer, otherwise, skip the corresponding key; A1.5.5, Record the Lth i(j+1) Metadata information of new SST files in the layer to FM i In the array, and reduce FM. i In the array in LSMT i The reference count of the corresponding file metadata element is found in the LSMT. s The garbage collection flag of the metadata element of the corresponding SST file was found to be true.

6. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing any of the key-value storage methods of claims 1-5, and the processor is configured to execute the program stored in the memory.

7. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program, when executed by a processor, performs the steps of any of the key-value storage methods described in claims 1-5.

Images